All about coding tests for hiring: Definitive guide

Everything you need to know to implement a coding test in your hiring process

BY DEEPTI CHOPRA

Co-founder of Adaface

Hiring a new software engineer is a complex process that involves reviewing dozens of hundreds of applications and resumes, sending out coding tests or take-home assignments, conducting remove pair programming interviews, and setting up in-person interviews.

While in-person interviews are a good way to evaluate developers, most companies receive far too many applications to be able to interview everyone. That's where coding test come into play- reducing your candidate pool from hundreds of applications to the top 20% qualified candidate.

This guide goes over everything you need to know about coding tests as a recruiter in the tech space. Here's what this guide covers:

What are coding tests
When should you use a coding test
Coding tests vs take home assignments
Candidate sentiment for coding test
Pros and cons of using coding tests
What questions should you use on a coding test
Should we use coding tests for fresh grads or for experienced hires also?
Testing for framework skills
Coding test vs pair programming interview
History of coding tests
How to use a candidate friendly coding test
Best practices for implementing coding tests
Tools to automate coding tests
Elimination vs selection
How to drive up completion rates
Candidate friendly coding tests

What are coding tests?

Coding tests or programming tests evaluate a programmers's ability to solve problems by analyzing and writing code and fixing errors.

Companies use coding challenges to serve as first round filters, and send these to interested candidates to get an initial feel for their technical ability. Most companies tend to use coding tests before onsite technical and cultural fit interviews to make sure they are spending time only on qualified candidates.

Since most programmers are terrible at writing code (60-80% can't write a simple Fizz Buzz) in spite of their keyword filled resumes, the coding test serves as a great filter. While a coding test is a terrible way to decide which candidate to hire, it can be a great tool for companies to filter down to a handful of candidates to do in-person interviews with and hire the best fit candidate.

Automated coding tests

Automated coding tests involve candidate solving simple coding challenges in a browser based IDE. These tasks typically take between 30 minutes and 1 hour to complete.

Online assessment platforms involve candidates writing some code inside a basic browser based IDE that solves a given problem. Candidates typically have between 30 minutes and 1 hour to complete the given task.

Automated coding tests focus on testing for these:

Correctness: Can the candidate write code that works (i.e. returns the expected output)
Optimization/ time complexity: Is this the most efficient way to solve this problem? (time and space complexity)

Typically candidates are emailed a URL to a coding challenge. There are several tools on the market that make it a seamless experience for recruiters to send coding tests to candidates, and few companies prefer to use their internal tools. The format is to solve a given number of questions in a set time frame. Depending on the role, the candidate might be able to pick a programming language of their choice to complete the challenge in.

While the ability to write code to solve simple problems is not an accurate measurement of a developer's ability, it is a fundamental requirement that a lot of developers don't fulfil. This is what making automated coding tests an effective screen.

What questions should you use on a coding test

The biggest problem with coding tests today is that they’re not always aligned with what the role requires. An Android developer does not necessarily need to know how to invert a binary tree. Or maybe they do. But the point is that if you’re using that on a coding test for that role, you should be very deliberate about it.

Questions where you might still hire the person even if they couldn’t solve it/ got it completely wrong are pointless at the screening stage, such as a coding test.

Check out this guide on How to setup an effective coding test (for hiring managers).

Coding test vs take home assignments

The argument for take home assignments:

They reflect real world tasks more accurately
They're offline so programmers can complete them in their element, and there is less pressure.

The arguments against take home assignments:

Most take home assignments are too long (4 hours to a week)
Expectations are often unclear.
Hiring managers need to evaluate each assignment manually, costing the company expensive engineering time.

Conclusion

Despite the pros of a take home assignment, most companies report a completion rate of 30% (since they're too long).

A take home assignment is most effective as a filter when you have ~10-20 candidates for a role. Having only a handful of candidates also means that you can build a relationship with each one before asking them to complete the assignment, driving up completion rates.

For roles with > 50 candidates, coding tests are a much better way to filter out unqualified candidates.

Why the coding test is an effective filter

When hiring a software developer, these are the things you want to filter for:

Are they a competent coder?
Are they a nice person and fun to work with?
Are they committed to doing this and can they handle the pressure?

Most software engineers fail at condition #1, making it an effective filter and removing the maximum number of unfit candidates in one shot. Most people aren't jerks, and it's hard to filter for commitment at scale, without face time whereas a substantial number of candidate fail at very simple coding questions making a coding test the best first filter for tech roles.

Once we've narrowed it to the the candidates who atleast have the bare minimum qualifications, 2 and 3 are easy to figure out with interviews.

Candidate sentiment

What unites software developers of different experience levels, in different geographies, with different areas of expertise is their hatred for traditional coding tests. Traditional assessments use the equivalent of puzzles/ obscure computer science questions or niche algorithms that an engineer would never really use in practice.

The primary reasons software engineers are unhappy with coding tests being used as filters for software engineering roles are:

They test for algorithmic skill rather than the ability to write code: While it's great if someone's good at algorithms (even though this skill can be improved with practice), this is not a strong indicator of how good of an engineer someone is/ how good they're going to be in the role.
They're too big an ask: Asking the candidate to spend more than 60 minutes on a coding test before you've spent any time is unfair.
It's harder to code in an unfamiliar environment: Most developers have a preference of IDE (integrated development environment) that they've customized in a way that helps them write code seamlessly. A test environment is unfamiliar, and it's harder for a software engineer to function optimally.

Read more about why engineers refuse to do coding tests.

Candidate friendly coding tests 🤩

At Adaface, we set up 45 min candidate-friendly assessments that test for on-the-job skills.

Here's what we're doing differently from the status quo:

Shorter assessments (45-60 mins) to make sure engineers can do it ASAP, they are investing as less time as possible, while still enough to showcase their expertise.
Custom assessments tailored to the requirements of the role (NO trick questions) ☺️
Questions at the simpler end of the spectrum (it is a screening interview) with a very generous time allowance (3x what it takes our team to write code for it)
Extremely granular scoring that eliminates false positives and false negatives
Friendly candidate experience (hints for each question, friendly messaging and chatbot; average candidate NPS is 4.4/5). Check out our wall of love ❤️.

What questions should you use on a coding test

Questions where you might still hire the person even if they couldn’t solve it/ got it completely wrong are pointless at the screening stage, such as a coding test.

We’ve found that the most useful way to do it would be to use 2 must-be-able-to-code questions, one easier than the other and give some benefit of doubt to candidates who solve it partially. This eliminates candidates who basically can’t code (quite a lot of them), and then you can take the rest to the final rounds of interview.

Here is a detailed guide on how to approach setting up a coding test and what questions to include in the test.

Prepare Your Prospective Candidates

Given the sentiment around traditional coding tests, it is not unusual that senior engineers will ignore a cold email asking them to attempt a 1-2 hour coding test only to never hear from the company again.

Even in the automated screening process, it is crucial that you make your candidates feel valued.

On the other hand, if you set up a thoughtful coding test that takes 45 minutes or lesser to complete, take time to communicate the expectation and your reason behind using a coding test, they will feel respected and will likely take the time to complete the challenge.

How to drive up completion rates for coding tests:

Make sure your coding test is short and relevant for the role, don't set up a quiz with trick questions or puzzles they would never encounter on the job.
Send personalized emails (can be automated) informing them about the coding test. Clearly communicate expectations, and explain the rationale behind your hiring process.
Candidates don't like the idea of giving a "test". Try friendly conversational assessments instead.
Move fast.

Use Candidate feedback to Calibrate Your Process

Implementing a coding test and inviting candidates to complete is only a start.

Candidates not completing the coding challenge is also feedback you need to take seriously. Before you dismissing candidates who drop out as being lazy, or not being serious enough, you need to take a hard look at your screening process and assess if it is candidate friendly.

One of the best ways to understand candidate sentiment for your screening process is to assign 5-10 minutes to take their feedback on the process when you invite them over for interviewers. This not helps you improve your process for future candidates, but also signals to the candidate that you care and are committed to setting up a candidate friendly hiring process. You might be surprised at the insights you get from speaking with your candidates about your process.

Concerns against coding tests:

The primary concerns recruitment teams have against coding tests are the following:

Candidates can cheat on remotely administered tests: While this was true until a few years ago, technological advances enable features like web proctoring, webcam proctoring, session replay, non-googleable questions and question leak checks to assure you that assessment scores are reflective of a candidate's ability.
Tests can intimidate and drive away candidates: While this a genuine concern, using short, candidate-friendly assessments with relevant questions and moving fast (< 24 hours of the candidate applying) can circumvent the concern and actually improve candidate experience.
Coding tests make the hiring process longer: Since the first step of the hiring process has the maximum number of unqualified candidates, and coding tests reduce your pool to the top 20% qualified candidates, coding tests are seen to reduce time-to-fill.

Check out how Adaface pre-employment assessments address recruiter concerns with the latest updates in technology in depth.

Job hunting is stressful, and hiring processes for software engineering roles can be nerve-wracking. What recruitment teams and hiring managers need is to have empathy for the candidates, set up developer friendly coding tests, communicate expectations and move fast.