Modernizing the Code Testing practice for AI

Bringing qualified candidates onto the team requires an investment of developer and leadership time.

When it comes to developers, we’re usually a lot more interested in their ability to write, debug, and improve applications than their ability to be unflappable in an interview.

There’s a lot to say about improving the hiring pipeline and providing team leads, HR and management with better tools.

Let’s look at how we bring on new talent and optimize the process.

In the approach above, we don’t have a mechanism for technical filtering ahead of technical interviews. The process will struggle to scale.

Improving the curve through testing

Reducing the area under the curve allows us to scale our hiring practices.

A well-designed test should:

• reduce the number of unsuccessful interviews overall
• increase the breadth of resumes we can examine
• allow greater time to be spent on team and fit and deep dive interviews
• and increase the number of candidates suitable to hire and the end of a cycle

Good quality Code Tests are a top line improvement to any pipeline

They’re a great aid to introduce to any organization that has begun to significantly scale.

However, Copilot/ChatGPT and other ML models have the ability to compose reasonable functions from written descriptions, meaning outdated tests are trivially answerable without the candidate performing any action.

2025 brings us to the point of needing to update our testing process to retain the strengths of administering short tests and to ensure we’re picking up talent that can handle complex problems outside of the scope of a query to copilot.

For teams that haven’t run their code tests through Copilot, they may be surprised to learn how well it performs on many types of questions. Code generation is an active area of research, and specialized models are able to obtain satisfactory answers to written prompt questions as evaluated by a human on upwards of 80% of questions.

We’ll explore what makes some questions trival and what makes them challenging.

A Quick Review of Testing Process

Tests are suitable for all roles that involve technical skills. The content will vary and may not involve “code” for some positions.

Versions of the test can be produced for customer-facing developers, data analyst and data scientist roles, UI and UX positions. Generally, the overarching lessons here apply to all technical testing, not just development.

Good tests for candidates typically have a few qualities in common:

a) a quick test: no more than 1-2 hours in length

b) low burden: quickly understandable by hiring lead, reduce workload

c) valuable: the answers should tell us quite a bit about the candidate

Having a refined design like the Five Questions Template above gives us something we can quickly scan candidates and filter out obvious non-fits on a performance basis rather than assumptive.

During any hiring cycle, we’d like a good return for developer/leadership/HR time spent and  have a broad funnel for potential candidates. Excessive commitments of candidate time narrows this field, so a quick test outperforms a long test in the real world.

Tests that build our confidence in a candidate accelerate the in-person process; we can shorten the phase of the interview that establishes basic skill and re-commit that energy to fit and aptitude and a more finely tuned evaluation of particularly valuable skills.

Questions that are now trivial

Code tests have consistently evolved over the last thirty years in order to meet the reality of tools available.

Widespread internet access, quality google results and well-answered stackoverflow questions has rendered the idea of trivia-style questioning impractical.

We usually now test for resolving more complex scenarios, however, not all ‘complex’ scenarios are actually difficult for code generators.

Many ‘complex’ problems can also be solved trivially

Solvability is not limited to simple problems: if the language model can correctly decompose a question into reasonable steps, it can still rapidly complete even large problems with  a correct response.

Let’s review some question styles where we’ve “increased” the difficulty, but which remain trivially solvable for commercial LLMs.

Question A)

“Write a function in javascript that takes as input a number N, randomly selects 5 animals, one of which is a bird, from a list of 15 animals, and returns up to a maximum of N array of arrays of different possible combinations of animal names in a random order.”

It’s difficult to obtain an answer to this just by using a search engine, because the combination of requirements is sufficiently unique that it does not produce matching questions.

Here’s what Microsoft Copilot, AI chatbot developed by Microsoft, based on the GPT-4, handled that question in 2024.

function getRandomCombinations(N) {
    const animals = ["Lion", "Tiger", "Elephant", "Giraffe", "Zebra", "Kangaroo", "Panda", "Koala", "Penguin", "Eagle", "Parrot", "Owl", "Falcon", "Sparrow", "Peacock"];
    const birdList = ["Penguin", "Eagle", "Parrot", "Owl", "Falcon", "Sparrow", "Peacock"];
    
    function getRandomInt(min, max) {
        return Math.floor(Math.random() * (max - min + 1)) + min;
    }

    function shuffleArray(array) {
        for (let i = array.length - 1; i > 0; i--) {
            const j = getRandomInt(0, i);
            [array[i], array[j]] = [array[j], array[i]];
        }
    }

    function getRandomAnimals() {
        const selectedAnimals = [];
        const bird = birdList[getRandomInt(0, birdList.length - 1)];
        selectedAnimals.push(bird);

        const remainingAnimals = animals.filter(animal => animal !== bird);
        shuffleArray(remainingAnimals);

        for (let i = 0; i < 4; i++) {
            selectedAnimals.push(remainingAnimals[i]);
        }

        shuffleArray(selectedAnimals);
        return selectedAnimals;
    }

    const combinations = [];
    for (let i = 0; i < N; i++) {
        combinations.push(getRandomAnimals());
    }

    return combinations;
}

// Example usage:
console.log(getRandomCombinations(3));

getRandomCombinations(3);

[
  [
    "Falcon",
    "Peacock",
    "Owl",
    "Penguin",
    "Parrot"
  ],
  [
    "Panda",
    "Parrot",
    "Penguin",
    "Peacock",
    "Koala"
  ],
  [
    "Penguin",
    "Eagle",
    "Kangaroo",
    "Falcon",
    "Parrot"
  ]
]

Copilot’s answer is correct and insufficiently distinguishable from work done by humans.

The inclusion of undefined components did not pose issues.

Essentially, models that can handle composition of arbitrary functions can answer many questions, especially where there is a strong similarity to well-published questions.

Questions that resemble textbook questions or homework questions asked on stackoverflow are particularly trivial, even if their combination is unique with distinct elements.

Unfortunately, these questions are quite common on code tests for candidates.

Because answers to this appear human-written and are reasonably high quality, we’re not going to get far unless we change our approach to what we test and why we test it.

Question types that Copilot struggles with

We can produce questions the AI will struggle to answer.

<br>
<hr>
<div id="box">
	<div class="square">
	1
	</div>
	<div class="circle">
	2
	</div>
</div>

<STYLE>

#box .square {
	border-radius: 16px;
	width: 32px; height: 32px; 
	background-color: limegreen;
	position: absolute;
	top: 15px;
	left: 15px;
	cursor: pointer;
}

#box .circle {
	width: 32px; height: 32px;
	background-color: pink;
	position: absolute;
}

#box {
	position: absolute; 
}

</STYLE>

Expansive questions provide higher quality results

In general, access to LLMs exacerbate a weakness present in many testing scenarios, which is questioning of candidates tests only a surface knowledge of a topic.

The characteristic of being answerable through repetition rather than understanding or effort is also what makes a question trivial to copilot. Refining our questions to a higher quality satisfies both the need for better questions and produces challenging results for LLMs.

In the next question, we’ll explore a question that not only confounds Copilot but also produces a greater insight into the experience and competence of the candidate.

NameVirtualHost 192.168.1.200:80

<VirtualHost mytestsite.tld:80>
	ServerAdmin general@mytestsite.tld
	DocumentRoot /var/www/vhosts/mytestsite.tld/htdocs
	ServerName mytestsite.tld
	ServerAlias www.mytestsite.tld
	ErrorLog logs/mytestsite.tld-error_log
	CustomLog logs/mytestsite.tld-access_log common
	DirectoryIndex index.html index.htm welcome.html
</VirtualHost>

<VirtualHost testsite.tld:80>
	ServerAdmin general@testsite.tld
	DocumentRoot /var/www/vhosts/testsite.tld/htdocs
	ServerName testsite.tld
	ServerAlias www.testsite.tld
	ErrorLog logs/testsite.tld-error_log
	CustomLog logs/testsite.tld-access_log common
	DirectoryIndex index.html index.htm welcome.html
</VirtualHost>

Debugging questions are a gold standard for testing questions

Complex tasks, like genuinely debugging a hidden issue, can fall outside of copilot’s scope by a considerable margin.

This is a debugging style question. This kind of bug is invisible to AI-assist applications like copilot.

<!DOCTYPE html>
<html lang="en">
<head>
	<meta charset="UTF-8">
	<meta name="viewport" content="width=device-width, initial-scale=1.0">
	<style>
        body { height: 100vh; background-color: #f0f0f0; padding: 20px; box-sizing: border-box; }
        #calendar { display: block; position: static; }
        .calendar-container { display: flex; flex-direction: column; justify-content: center; align-items: center; width: 100%; max-width: 1200px; margin: 0 auto; }
        .calendar { width: 100%; }
        .month { display: grid; grid-template-columns: repeat(7, 1fr); gap: 1px; border: 2px solid #000; margin-bottom: 20px; }
        .month-header { grid-column: span 7; text-align: center; font-weight: bold; padding: 10px; background-color: #ddd; }
        .day-header { font-weight: bold; text-align: center; padding: 10px; background-color: #eee; }
        .day { border: 1px solid #ddd; background-color: #fff; box-sizing: border-box; padding: 8px; min-height: 100px; display: flex; flex-direction: column; }
        .date-header { font-weight: bold; margin-bottom: 8px; }
        .staff { margin-bottom: 4px; }
	</style>
	<title>Calendar for Karaoke Night - This month and next month</title>
</head>
<body>
<section id="calendar">
	<h3>Karaoke Hosts - this month and next month</h3>
	<div class="calendar-container">
		<div class="calendar" id="calendar"></div>
	</div>
</section>

<script>
	// Generate a UUID (one-liner function)
	const generateUUID = () => 'xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx'.replace(/[xy]/g, c => (c === 'x' ? (Math.random() * 16 | 0) : ((Math.random() * 16 | 0) & 0x3 | 0x8)).toString(16));

	// generates an example of a JSON staffBookings structure
	function loadStaffBookings() {
		// Example structure created by loadStaffBookings:
		// {
		//   "startDate": {
		//     "bookingID": { "staffName": "Alice", "period": "afternoon" },
		//     "bookingID": { "staffName": "Bob", "period": "evening" }
		//   },
		//   "startDate": { ... }
		// }

		const staffNames =
			Array.from({ length: 10 }, () => {
				const uuid = generateUUID();
				const names = ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve', 'Frank', 'Grace', 'Heidi', 'Ivan', 'Judy'];
				return { [uuid]: { staffName: names.shift() } };
			}).reduce((acc, curr) => ({ ...acc, ...curr }), {});

		const bookings = {};
		const today = new Date();

		for (let i = -30; i <= 60; i++) {
			const date = new Date();
			date.setDate(today.getDate() + i);
			const dateString = date.toISOString().split('T')[0];
			bookings[dateString] = {};
			const periods = ['afternoon', 'evening'];
			const filledPeriods = Math.random() < 0.9 ? periods : [periods[Math.floor(Math.random() * periods.length)]];

			filledPeriods.forEach(period => {
				const bookingID = generateUUID();
				const staffID = Object.keys(staffNames)[Math.floor(Math.random() * Object.keys(staffNames).length)];
				bookings[dateString][bookingID] = { staffID, period, staffName: staffNames[staffID].staffName };
			});
		}

		window.application = window.application || {};
		window.application.staffBookings = bookings;
	}

	// Generate calendar with staff bookings
	function generateCalendar(startDate, numMonths, staffBookings) {
		const calendarContainer = document.querySelector('.calendar-container .calendar');
		const daysOfWeek = ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday'];
		const benchmarks = [];
		let currentDate = new Date(startDate);

		for (let i = 0; i < numMonths; i++) {
			const monthContainer = document.createElement('div');
			monthContainer.className = 'month';

			const monthHeader = document.createElement('div');
			monthHeader.className = 'month-header';
			monthHeader.textContent = `${currentDate.toLocaleString('default', { month: 'long' })} ${currentDate.getFullYear()}`;
			monthContainer.appendChild(monthHeader);

			// Add day of the week headers
			daysOfWeek.forEach(day => {
				const dayHeader = document.createElement('div');
				dayHeader.className = 'day-header';
				dayHeader.textContent = day;
				monthContainer.appendChild(dayHeader);
			});

			const monthStart = new Date(currentDate.getFullYear(), currentDate.getMonth(), 1);
			const monthEnd = new Date(currentDate.getFullYear(), currentDate.getMonth() + 1, 0);

			let startDay = monthStart.getDay();
			startDay = startDay === 0 ? 6 : startDay - 1;  // Adjust to have Monday as the first day

			for (let blank = 0; blank < startDay; blank++) {
				const blankDiv = document.createElement('div');
				blankDiv.className = 'day';
				monthContainer.appendChild(blankDiv);
			}

			for (let day = 1; day <= monthEnd.getDate(); day++) {

				const dayDiv = document.createElement('div');
				dayDiv.className = 'day';

				const dateHeader = document.createElement('div');
				dateHeader.className = 'date-header';
				dateHeader.textContent = `${day}`;

				dayDiv.appendChild(dateHeader);

				const dateString = new Date(currentDate.getFullYear(), currentDate.getMonth(), day).toISOString().split('T')[0];


				if (staffBookings[dateString]) {
					Object.values(staffBookings[dateString]).forEach(booking => {
						const staffDiv = document.createElement('div');
						staffDiv.className = 'staff';
						staffDiv.textContent = `${booking.staffName} (${booking.period})`;
						dayDiv.appendChild(staffDiv);
					});
				}

				monthContainer.appendChild(dayDiv);
				currentDate.setDate(currentDate.getDate() + 1);
			}

			calendarContainer.appendChild(monthContainer);
			currentDate.setMonth(currentDate.getMonth() + 1); // Move to the next month
			currentDate.setDate(1); // Reset to the first day of the month

		}



	}

	document.addEventListener('DOMContentLoaded', () => {
		loadStaffBookings();
		setTimeout(() => {
			const today = new Date();
			generateCalendar(today, 2, window.application.staffBookings); // Generate calendar using staff bookings
		}, 200);
	});
</script>

</body>
</html>

Modifying the question for other languages or frameworks

We can explore or reframe this style of questions in many different ways, frameworks and styles.

 We’re still looking for something contrived to keep the user’s setup time down.

A real application might swap DateTime for luxon, the string interpolation may be handled by a library, and of course we wouldn’t be generating a sample JSON structure to work with.

(Some very quick examples in Angular and Svelte in the spoiler tags below)

<!DOCTYPE html>
<html lang="en" ng-app="calendarApp">
<head>
	<meta charset="UTF-8">
	<meta name="viewport" content="width=device-width, initial-scale=1.0">
	<link href="https://cdn.jsdelivr.net/npm/tailwindcss@2.0.1/dist/tailwind.min.css" rel="stylesheet">
	<title>Calendar for Karaoke Night - This month and next month</title>
</head>
<body class="h-screen bg-gray-100 p-5 box-border">
<section id="calendar" ng-controller="CalendarController as ctrl">
	<h3 class="text-xl font-bold mb-5">Karaoke Hosts - this month and next month</h3>
	<div class="calendar-container flex flex-col justify-center items-center w-full max-w-5xl mx-auto">
		<div class="calendar w-full" id="calendar">
			<div ng-repeat="month in ctrl.months" class="mb-5">
				<div class="month border-2 border-black grid grid-cols-7 gap-px mb-5">
					<div class="month-header col-span-7 text-center font-bold p-2 bg-gray-300">
						{{month.name}}
					</div>
					<div class="day-header font-bold text-center p-2 bg-gray-200" ng-repeat="day in ctrl.daysOfWeek">
						{{day}}
					</div>
					<div ng-repeat="blank in month.blanks" class="day p-2 bg-white"></div>
					<div ng-repeat="day in month.days" class="day border border-gray-200 bg-white box-border p-2 min-h-[100px] flex flex-col">
						<div class="date-header font-bold mb-2">{{day.date}}</div>
						<div class="staff mb-1" ng-repeat="staff in day.staffs">
							{{staff.staffName}} ({{staff.period}})
						</div>
					</div>
				</div>
			</div>
		</div>
	</div>
</section>

<script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.8.2/angular.min.js"></script>
<script>
	const app = angular.module('calendarApp', []);

	app.controller('CalendarController', function ($scope) {
		const generateUUID = () => 'xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx'.replace(/[xy]/g, c => (c === 'x' ? (Math.random() * 16 | 0) : ((Math.random() * 16 | 0) & 0x3 | 0x8)).toString(16));

		const loadStaffBookings = () => {
			const staffNames = Array.from({ length: 10 }, () => {
				const uuid = generateUUID();
				const names = ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve', 'Frank', 'Grace', 'Heidi', 'Ivan', 'Judy'];
				return { [uuid]: { staffName: names.shift() } };
			}).reduce((acc, curr) => ({ ...acc, ...curr }), {});

			const bookings = {};
			const today = new Date();

			for (let i = -30; i <= 60; i++) {
				const date = new Date();
				date.setDate(today.getDate() + i);
				const dateString = date.toISOString().split('T')[0];
				bookings[dateString] = {};
				const periods = ['afternoon', 'evening'];
				const filledPeriods = Math.random() < 0.9 ? periods : [periods[Math.floor(Math.random() * periods.length)]];

				filledPeriods.forEach(period => {
					const bookingID = generateUUID();
					const staffID = Object.keys(staffNames)[Math.floor(Math.random() * Object.keys(staffNames).length)];
					bookings[dateString][bookingID] = { staffID, period, staffName: staffNames[staffID].staffName };
				});
			}

			return bookings;
		};

		const generateCalendar = (startDate, numMonths, staffBookings) => {
			const daysOfWeek = ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday'];
			const months = [];
			let currentDate = new Date(startDate);

			for (let i = 0; i < numMonths; i++) {
				const month = {
					name: `${currentDate.toLocaleString('default', { month: 'long' })} ${currentDate.getFullYear()}`,
					blanks: [],
					days: []
				};

				const monthStart = new Date(currentDate.getFullYear(), currentDate.getMonth(), 1);
				const monthEnd = new Date(currentDate.getFullYear(), currentDate.getMonth() + 1, 0);

				let startDay = monthStart.getDay();
				startDay = startDay === 0 ? 6 : startDay - 1;

				for (let blank = 0; blank < startDay; blank++) {
					month.blanks.push({});
				}

				for (let day = 1; day <= monthEnd.getDate(); day++) {
					const dateString = new Date(currentDate.getFullYear(), currentDate.getMonth(), day).toISOString().split('T')[0];
					const staffs = staffBookings[dateString] ? Object.values(staffBookings[dateString]) : [];
					month.days.push({ date: day, staffs });
					currentDate.setDate(currentDate.getDate() + 1);
				}

				months.push(month);
				currentDate.setMonth(currentDate.getMonth() + 1);
				currentDate.setDate(1);
			}

			return { daysOfWeek, months };
		};

		$scope.ctrl = {};

		$scope.ctrl.daysOfWeek = ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday'];
		$scope.ctrl.months = generateCalendar(new Date(), 2, loadStaffBookings()).months;
	});
</script>

</body>
</html>

<script>
  import Calendar from './Calendar.svelte';
</script>

<main>
  <Calendar />
</main>

<style>
  main {
    padding: 1rem;
    font-family: Arial, sans-serif;
  }
</style>

<script>
  import { onMount } from 'svelte';

  let staffBookings = {};

  const generateUUID = () => 'xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx'.replace(/[xy]/g, c => (c === 'x' ? (Math.random() * 16 | 0) : ((Math.random() * 16 | 0) & 0x3 | 0x8)).toString(16));

  function loadStaffBookings() {
    const names = ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve', 'Frank', 'Grace', 'Heidi', 'Ivan', 'Judy'];
    const staffNames = names.reduce((acc, name) => {
      const uuid = generateUUID();
      acc[uuid] = { staffName: name };
      return acc;
    }, {});

    const bookings = {};
    const today = new Date();

    for (let i = -30; i <= 60; i++) {
      const date = new Date();
      date.setDate(today.getDate() + i);
      const dateString = date.toISOString().split('T')[0];
      bookings[dateString] = {};
      const periods = ['afternoon', 'evening'];
      const filledPeriods = Math.random() < 0.9 ? periods : [periods[Math.floor(Math.random() * periods.length)]];

      filledPeriods.forEach(period => {
        const bookingID = generateUUID();
        const staffID = Object.keys(staffNames)[Math.floor(Math.random() * Object.keys(staffNames).length)];
        bookings[dateString][bookingID] = { staffID, period, staffName: staffNames[staffID].staffName };
      });
    }

    return bookings;
  }

  function generateCalendar(startDate, numMonths, staffBookings) {
    const daysOfWeek = ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday'];
    const months = [];
    let currentDate = new Date(startDate);

    for (let i = 0; i < numMonths; i++) {
      const month = {
        name: `${currentDate.toLocaleString('default', { month: 'long' })} ${currentDate.getFullYear()}`,
        blanks: [],
        days: []
      };

      const monthStart = new Date(currentDate.getFullYear(), currentDate.getMonth(), 1);
      const monthEnd = new Date(currentDate.getFullYear(), currentDate.getMonth() + 1, 0);

      let startDay = monthStart.getDay();
      startDay = startDay === 0 ? 6 : startDay - 1;

      for (let blank = 0; blank < startDay; blank++) {
        month.blanks.push({});
      }

      for (let day = 1; day <= monthEnd.getDate(); day++) {
        const dateString = new Date(currentDate.getFullYear(), currentDate.getMonth(), day).toISOString().split('T')[0];
        const staffs = staffBookings[dateString] ? Object.values(staffBookings[dateString]) : [];
        month.days.push({ date: day, staffs });
        currentDate.setDate(currentDate.getDate() + 1);
      }

      months.push(month);
      currentDate.setMonth(currentDate.getMonth() + 1);
      currentDate.setDate(1);
    }

    return { daysOfWeek, months };
  }

  onMount(() => {
    staffBookings = loadStaffBookings();
    const today = new Date();
    calendarData = generateCalendar(today, 2, staffBookings);
  });

  let calendarData = { daysOfWeek: [], months: [] };
</script>

<section id="calendar">
  <h3>Karaoke Hosts - this month and next month</h3>
  <div class="calendar-container d-flex flex-column justify-content-center align-items-center w-100 max-w-5xl mx-auto">
    {#each calendarData.months as month}
      <div class="calendar w-100 mb-5">
        <div class="month border border-dark d-grid gap-1 mb-5">
          <div class="month-header col-span-7 text-center font-weight-bold p-2 bg-light">
            {month.name}
          </div>
          {#each calendarData.daysOfWeek as day}
            <div class="day-header font-weight-bold text-center p-2 bg-light">
              {day}
            </div>
          {/each}
          {#each month.blanks as blank}
            <div class="day p-2 bg-white"></div>
          {/each}
          {#each month.days as day}
            <div class="day border border-light bg-white box-border p-2 d-flex flex-column">
              <div class="date-header font-weight-bold mb-2">{day.date}</div>
              {#each day.staffs as staff}
                <div class="staff mb-1">
                  {staff.staffName} ({staff.period})
                </div>
              {/each}
            </div>
          {/each}
        </div>
      </div>
    {/each}
  </div>
</section>

<style>
  body { height: 100vh; background-color: #f0f0f0; padding: 20px; box-sizing: border-box; }
  .calendar-container { display: flex; flex-direction: column; justify-content: center; align-items: center; width: 100%; max-width: 1200px; margin: 0 auto; }
  .calendar { width: 100%; }
  .month { display: grid; grid-template-columns: repeat(7, 1fr); gap: 1px; border: 2px solid #000; margin-bottom: 20px; }
  .month-header { grid-column: span 7; text-align: center; font-weight: bold; padding: 10px; background-color: #ddd; }
  .day-header { font-weight: bold; text-align: center; padding: 10px; background-color: #eee; }
  .day { border: 1px solid #ddd; background-color: #fff; box-sizing: border-box; padding: 8px; min-height: 100px; display: flex; flex-direction: column; }
  .date-header { font-weight: bold; margin-bottom: 8px; }
  .staff { margin-bottom: 4px; }
</style>

While we can introduce framework specific issues to compound the complexity of the problem, we have a goal to constrain realistic time spent to less than forty minutes on this question.

Good questions are about fundamental debugging skills, and expressible in many ways.

Using naive and un-optimized code within our questions provide excellent launching points for deeper conversations and candidate insight.

Incorporating AI tasks into the testing process

For organizations that include the use of AI tools to generate code for their projects as a core function of how they operate, it can be useful to directly test a candidate’s ability to operate with AI generated code.

Here’s a solvable task that will cause some LLM models to loop through a series of bad results, requiring human intervention.

Question D (1st alternative):

Suppose you have a snippet of HTML that is defined by an application not operated by your organization. You have permission to only adjust the contents of style=”” attribute.

Adjust the given prompt so that Copilot will reliably produce a correct answer, even if unprompted with other data on a fresh session. Copilot must not introduce new tags of any kind and correctly return a properly styled table that meets the defined requirements of the original prompt.

Optionally, ensure the prompt also produces a visually pleasing set of style attributes in screen sizes above and below 800px;

<!--
PROMPT: 

Cause this table with inline styles to hide the center column and place the two cells stacked vertically at screen sizes smaller than 760 pixels wide. No new tags of any kind can be introduced and existing tags cannot be renamed.

-->

<div style="width: 800px;">
<table style="border-collapse: collapse; width: 100%;"> <tbody> <tr> 

<td style="width: 50;">An arbitrarily long series of words about text</td> 
<td style=""> </td> 
<td style="width: 50%;">And other criteria that can exist without complication.</td>

</tr> </tbody> </table>
</div>

Each task should have some complicating factor that frustrates an LLM’s naive approach and requires the user to critically think about what they are designing and trying to do when they write their prompt.

In the above example, that wrinkle is the restriction on new tags.  It’s a quick solution (an experienced front-end developer has many tools to resolve this without javascript or media queries).

This type of exercise can be expanded to other tasks, such as “adjust this prompt so that, given an arbitrary database definition (sample included below), copilot will generate a class with a complete set of filter methods for every field that will only allow permitted values.”

In the database example, the wrinkle we could introduce is a JSON field that is described in a way that the LLM does not understand.

A lack of expertise here will become especially informative when the prompt is, subtly, asking for the wrong thing and the user does not know how to describe what is they are trying to achieve.

By asking a question in this style from a domain outside of a user’s expertise, the question will also give us an assessment of the candidate’s ability not just to use LLM models, but to learn on their own and critically evaluate their output.

Producing a stronger gradient of results

Sharp-eyed readers will have noticed Question D (the calendar question prior to the AI Prompt writing question) has more shortcomings than just the listed bugs.

When developing the final advanced questions (Question E), remember to seed them with other issues that will not appear on a cursory inspection or test run.

Finally, improving the hiring pipeline by investing in test tooling

A quality test with great questions greatly enhances the process. We can also invest time in tooling.

What we’re really hiring for is the ability to do work and solve complex issues, so tools that allow us to perform deeper assessments are worth examining.

Improving our tooling around testing candidates will make

Playpen and jsFiddle: At the most basic level, we can supply candidates with tests using services like jsFiddle and other playpen examples. These take not much more time to set up, and if set to private, are reasonably secure.

Screen Shares: when we have capacity, we can do developer screen shares. Since the end of the pandemic, it seems less necessary. We usually find administering the short test and then spending the time in person talking with a real IDE open to be more productive than adding a screen share session back in.

Containerized Test Environment: At a higher level, we can supply a candidate with a rough approximate of what the workplace development environment looks like in a container, complete with buildscripts, and provide them with tasks within that environment. Credentials and instructions are easy to set up and reset.

Of course, it’s important that the account we test under have Full Relationship Isolation for security purposes, but once that is accomplished, we gain the following things:

• Testing against very large data sets (100 gb+) which are impractical by email
• Testing candidate ability to navigate build environments
• Broad-spectrum questions that touch on multiple systems (from backend to front-end)
• Lower work burden to create “depth” questions from whole cloth (questions 3 and 5)
• A very real sense how quickly the developer could pick up technology and paradigms at the workplace, and, from question 4, their opinions about whether they would like working with them

Additionally, because the developer does not have to perform routine tasks like setting up an environment or downloading components, we can spend more of the test time deep diving into hard questions.

Containerized environments are beneficial for both developers and non-developers

The uncapped data component is useful for any positions that test developer understanding of scalability and performance, and profoundly useful for testing data analyst and engineer skills.

Questions like “this environment performs in aberrant fashion. Identify the issue from the logs and resolve it,” become possible in containerized settings.

A ready-made containerized application is also exceptional for front-end developers.

We’re a bit less interested in “explain border box to me in this constructed example” and a lot more interested in “there are visual defects in this page. Identify and correct them,” or “this page renders poorly and slowly on mobile devices. Explore the animation techniques and optimize them.”

Since the advent of Copilot, we’ve cared less about knowing a specific framework or technology, as it accelerates learning on unfamiliar formats. However, being able to present someone with a simple task in a legacy framework (YUI or, say, ZK) and seeing how they handle it can be a useful data point.

Techniques & Terms

Not getting the results you’re looking for? Troubleshooting the process of code testing

For those who already have a code test, or have tried to implement one and are running into issues in the process, here’s a quick troubleshooting Q&A derived from some experiences with different outcomes.

END