Using Delivery Simulation Experiences in Hiring in the Technology Field

The following was a paper submitted for EPY 690 Introduction to the Learning Sciences. Great thanks to Dr. E. Michael Nussbaum, Instructor, and Dr. Vanessa Vongkulluksn, Advisor, for their helpful feedback and comments and to Paula Paul of Greyshore Partners for the inspiration to research this topic.

Introduction

The hiring process in the technology field is broken. The traditional approach to technical interviewing is rife with problems. Software candidates cite concerns around the hiring and interviewing process for technical roles. An alternative to the traditional technical interview is creating delivery simulation experiences (DSEs). A DSE integrates practices that the learning sciences have shown support learning outcomes. While the idea is not without its disadvantages, there are benefits to both the candidate and the company seeking employees.

The hiring process is broken

Behroozi et al. (2019) identify six concerns software candidates have about technical interviews and hiring: relevance, anxiety, affect, affordances, practice, and missing out. The traditional hiring process relies heavily on solving problems on a whiteboard. This practice captures several of the concerns of Behroozi et al. (2019). Solving technical problems at a whiteboard is a high cognitive load process. The high cognitive load is due in part to the public nature of the task as well as the lack of tools which provide affordances like code completion, a tool built into integrated development environments that produces automated suggestions about what code should come next, and syntax highlighting which makes code easier to read and navigate (Behroozi et al., 2018). The lack of everyday software development tools in the interview experience is an example of relevance, problem solving not grounded in real-world, scenarios or constraints. The cognitive overload caused by this scenario produces anxiety and affect. Affect refers to the frustration and possible humiliation experienced during a technical interview (Behroozi et al., 2019). As a result of this subjective process, candidates experiences during the hiring process are not a reflection of the actual day-to-day work and the candidates selected may or may not have the true abilities to do the day-to-day job. This mismatch leads to dissatisfaction on the part of the developer and the company which leads to high turnover.

Using a delivery simulation

A DSE could be used for hiring in multiple ways. First, a DSE could be run in boot camp fashion where participants pay to attend and companies use the DSE as a target for recruitment; this example is described below. The 8-12 week DSE could be abbreviated to a half to multi-day experience with a much smaller project to replace the traditional interviewing process which is typically drawn out over several weeks or months. Finally, a DSE could be used in the full 8-12 week capacity as a contract-to-hire environment.

In a delivery simulation, a participant works alongside a team of other novice developers, designers, and analysts with scaffolding from an experienced leader who facilitates most of the learning and a series of subject matter experts on quality analysis, product design, data, and infrastructure. In this environment which takes from the elements of both project and problem-based learning, the candidate and their cohort engage in agile practices in order to work through several iterations over 8-12 weeks to produce a mock product like a website for a product owner. During that time, the participants would engage in collaboration, knowledge building, and argumentation. Through elements of cognitive apprenticeship like articulation and reflection, they would produce blog posts and building a portfolio of artifacts that reflect their knowledge of what they've learned over the experience. The portfolio would also showcase their knowledge of technical topics and agile processes and include feedback from mentors and team members as well. The use of a DSE would address all of the concerns of software candidates: relevance, anxiety, affect, affordances, practice, and missing out (Behroozi et al., 2019).

Working in a realistic simulation

One of the benefits to a DSE over a traditional technical interview is that it offers a realistic simulation that negates some of the stress-inducing, cognitive overload situations that traditional interviews do. Because a DSE is grounded in real-world scenario and constraints, the simulation candidates experience in a DSE should mirror the actual job for which they're being hired. This assessment reflects actual job performance much more closely than a whiteboard problem which in turn should mean better skills to job matching and less likelihood of turnover.

While the delivery simulation does not check all the boxes for project and problem-based learning, it does encompass many of them. Project-based learning is based on the constructivist perspective that when students actively construct their understanding they gain a deeper understanding of what they are learning (Krajcik & Shin, 2014). While there may not be an explicitly stated driving question, there is problem to be solved. The participants are building a website that meets the requirements of the product owner. This is where there is a bit of overlap with problem-based learning. In problem-based learning, learners are attempting to solve complex, ill-structured problems (Lu et al., 2014). A benefit of using a problem solving approach is the ability to transfer knowledge to new problems (Lu et al., 2014). Problem solving is a requirement of software development in every area from coding to product.

Several other aspects of project-based learning are present in a DSE. The features of project-based learning present in the DSE are: learning performances, collaboration, and the creation of artifacts (Krajcik & Shin, 2014). A showcase is a demonstration to stakeholders including the product owner of what work has been accomplished in the prior sprint. This is a type of learning performance which is a feature of project-based learning. Collaboration will be discussed later. Finally, participants will create a set of artifacts to demonstrate learning. Each participant will produce a publicly-sharable portfolio that includes blog posts about technical topics and lessons learned, feedback from teachers and fellow participants, and code snippets from what they worked on.

Scaffolding is how, with the help of someone more knowledgeable and experienced, a learner can solve problems they would not otherwise be capable of solving (Reiser & Tabak, 2014). Scaffolding is essential to the DSE. The participants are new to the field of technology and software development. They are also likely new to agile processes. Agile is a category of methodologies for approaching project management based on the agile manifesto (Beck et al., 2001). Agile methodologies emphasize aim to be collaborative, fast, flexible, and adaptive. This is in contrast to a more traditional waterfall approach to project management which makes many if not all decisions up front and executes based on a pre-determined plan. As Reiser and Tabak (2014) point out, the challenges in the problem solving process are figuring out which actions to take, when, and how. Engaging in means-ends analysis particularly in a case where the learner does not have the requisite schema from which to draw, is an expensive process for working memory capacity (Sweller et al., 1998). Reducing that cognitive load using scaffolding will improve learning outcomes and the experience for the participants.

Working in a team

During a DSE, participants work in a team. Throughout the experience participants will be working using the techniques of mob and pair programming. Pair programming is where two developers work on the same problem together. Mob pairing is when more than two developers participate in collaboration to solve a problem. This approach addresses the remaining concerns raised by Behroozi et al. (2019): anxiety, affect, and missing out. The experience happens over an extended period, no advanced practice is required, and part of the experience is building rapport within and relying on a team. Thus, the frequent formative assessment instead of one high-stakes white boarding session, would likely reduce anxiety and the negative affect of frustration and humiliation. The evaluation of a candidate’s skills would be thoroughly vetted and the artifacts produced would reduce the likelihood that a candidate would be passed over unfairly.

The DSE would involve all four reasons to study collaboration - collaboration-as-a-window, collaboration-for-distal-outcomes, collaboration-for-proximal-outcomes, and collaboration-as-learning. Examining just one of those, collaboration-for-distal-outcomes looks at the processes happening at the collective, group level and analyzes how they impact the individual’s distal outcomes (Enyedy & Stevens, 2014). By teaching and purposely integrating a technique like revoicing in which a participant in a pair or conversation talks and someone else in the group restates what the first person said and elaborates on it, participants will learn skills beneficial to the workplace in a lower risk environment. Revoicing has been linked to increased engagement (Enyedy & Stevens, 2014).

“Groups can learn – that is, acquire skills and understandings best described at the group level…and people are recognized for contributions they make to the organization’s or community’s knowledge” (Scardamalia & Bereiter, 2014, p. 397). Knowledge building is best done as a group activity. The DSE is centered around the work of the group. The actual job of software development involves working with a team which makes the experience relevant, reduces negative affect and anxiety, does not require additional practice outside the experience, and ensures the task and expectations match the skills of the learner.

The tenets of collaborative argumentation show in the DSE in the use of spikes. Spikes are pieces of work in which research is done to answer questions and present viable solutions for discussion within the team in order to solve some sub-problem of the larger task. Spikes often result in Architectural Decision Records which are records about decisions related to the architecture of the software. For example, if the team was deciding on what Javascript framework to use for the creation of their website, they might look at several options. The team would investigate the options and document their research. This makes knowledge explicit, increases articulation, and learners are co-elaborating on the new knowledge (Andriessen & Baker, 2014). Questions that arise are brought to the team for discussion and the team experiences conceptual change based on the conversation and new information. These skills are also used in less formal ways during pair and mob programming sessions while participants are actively working to solve a problem. Thus, instead of asking a candidate about a time when they had a disagreement with a colleague and how it was resolved or other tangential questions designed to interrogate their ability to work with others, the candidate can demonstrate those abilities and the potential employer can observe them.

Disadvantages

Using DSEs to replace traditional technical interviews is not without its disadvantages. On the surface, this is a more labor intensive process than technical interviews from a company’s perspective, however, it may not actually be more costly depending on how much a company spends per candidate on the hiring process. The organizations running the boot camp-style DSEs might be a good place from which companies could recruit. Companies could also consider running whole or half day experiences in lieu of a lengthy, multiphase interview process that takes place over several months.

Conclusion

The technical hiring process is broken. Software candidates highlighted six concerns: relevance, anxiety, affect, affordances, practice, and missing out (Behroozi et al., 2019). An alternative to traditional hiring would be to use a delivery simulation environment. A delivery simulation environment reduces or eliminates the frustrations outlined by Behroozi et al. (2019). By working in a realistic simulation and in teams, participants would have the pedagogical advantages of features of project and problem-based learning, scaffolding, collaboration, knowledge building, and collaborative argumentation. These advantages would provide a more robust representation of candidate skills and a simulation that reflects the tools and constraints of the day-to-day job would increase the likelihood of candidate-to-job match thus decreasing the chances of dissatisfaction and turnover. While there are disadvantages in the form of cost to the participant in the boot camp-style version of the DSE and the possibly more labor intensive process, the delivery simulation environment is a concept worth further experimentation and research.

References

Andriessen, J., Baker, M., & Sawyer, R. K. (2014). Arguing to Learn. In The Cambridge Handbook of the Learning Sciences (Second, pp. 439–460). essay, Cambridge University Press.

Beck, K., Beedle, M., van Bennekum, A., Cockburn, A., Cunningham, W., Fowler, M., Grenning, J., Highsmith, J., Hunt, A., Jeffries, R., Kern, J., Marick, B., Martin, R. C., Mellor, S., Schwaber, K., Sutherland, J., & Thomas, D. (2001). Manifesto for Agile Software Development. Retrieved October 12, 2021, from https://agilemanifesto.org/.

Behroozi, M., Lui, A., Moore, I., Ford, D., & Parnin, C. (2018). Dazed. Proceedings of the 40th International Conference on Software Engineering, 93–96. https://doi.org/10.1145/3183399.3183415

Behroozi, M., Parnin, C., & Barik, T. (2019). Hiring is Broken: What Do Developers Say About Technical Interviews? 2019 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), 2019, 1–9. https://doi.org/10.1109/VLHCC.2019.8818836

Behroozi, M., Shirolkar, S., Barik, T., & Parnin, C. (2020). Debugging hiring: What went right and what went wrong in the technical interview process. Proceedings - International Conference on Software Engineering, 71–80. https://doi.org/10.1145/3377815.3381372

Enyedy, N., Stevens, R., & Sawyer, R. K. (2014). Analyzing Collaboration. In The Cambridge Handbook of the Learning Sciences (Second, pp. 191–212). essay, Cambridge University Press.

Karray, F., Alemzadeh, M., Abou Saleh, J., & Nours Arab, M. (2008). Human-computer interaction: Overview on state of the art. International Journal on Smart Sensing and Intelligent Systems, 1(1), 137–159. https://doi.org/10.21307/ijssis-2017-283

Krajcik, J. S., Shin, N., & Sawyer, R. K. (2014). Project-Based Learning. In The Cambridge Handbook of the Learning Sciences (Second, pp. 274–297). essay, Cambridge University Press.

Lu, J., Bridges, S., Hmelo-Silver, C.E. & Sawyer, R. K. (2014). Problem-Based Learning. In The Cambridge Handbook of the Learning Sciences (Second, pp. 298-318). essay, Cambridge University Press.

Reiser, B.J., Tabak, I., & Sawyer, R. K. (2014). Scaffolding. In The Cambridge Handbook of the Learning Sciences (Second, pp. 44-62). essay, Cambridge University Press.

Scardamalia, M., Bereiter, C., & Sawyer, R. K. (2014). Knowledge Building and Knowledge Creation: Theory, Pedagogy, and Technology. In The Cambridge Handbook of the Learning Sciences (Second, pp. 397-417). essay, Cambridge University Press.

Sweller, J., van Merrienboer, J. J., & Paas, F. G. W. . (1998). Cognitive Architecture and Instructional Design. Educational Psychology Review, 10(3), 251–296. https://doi.org/10.1023/A:1022193728205