A group of subject matter experts from a diverse set of industries & experience levels determines the knowledge and skills that are critical for a role, to decide what topics and concepts the questions should focus on testing.

This group of SMEs develops and ranks a list of topics that they believe the minimally qualified candidate should have expertise on, for different levels of seniority for that role (junior, intermediate and senior). If a particular topic is not as important, or its use is not as frequent, it is not included in the final list. This is done to ensure the topics included in the skills assessment represent those actually needed on the job. This final weighted list becomes the foundation for the skills assessment template and helps determine the number or percentage of questions that should come from each topic.

Once the list of topics is determined, the SMEs design questions to appropriately measure whether the minimally qualified candidate has the ability to perform the role. Several factors must be kept in mind while designing the questions to provide a desirable mix of validity, reliability, fairness, and practicality:

• Question should not focus on test recall, and instead they should test some understanding of the core concepts.
• Scenario based questions do a better job of a hands-on evaluation and also test critical thinking skills, as compared to a theoretical question lifted out of a textbook.
• Problem-solving skills are proven to be closely related to knowledge, so evaluating knowledge can be an indirect measure of skills.
• The subject matter experts also need to identify a reference to support the correct answer.

There are additional criteria that must be met for each question:

• The question must clearly and completely present the problem without using a complex sentence structure.
• The questions should focus on testable and reproducible scenarios and should not be subjective or opinion-based.
• The question must avoid gender, cultural, and vernacular bias.
• As far as possible, negatively worded questions, such as “which of the following are not...” must be avoided.

After a question is proposed by a subject matter experts, it is peer reviewed for the following:

• Is the question at the correct level of difficulty?
• Is this something a minimally qualified candidate for that role should know?

It is also reviewed by a team who review the questions against this questionnaire:

• Does the question have proper grammar, spelling, and punctuation?
• Is the question understandable by a non-native English speaker?
• Does the question contain vocabulary or language that might have different meaning for different subgroups of candidates?
• Is the question culturally sensitive?

If the question fails the questionnaire, it is re-evaluated and may be either rewritten or removed.

Before being used as a scored question on a test, all questions must pass a beta test. Some tests have a small percentage of questions that are not used in the determination of the candidate’s final score. For example, for a test with 20 questions, there might be 1 question that is being beta tested and does not count towards the final score.

The results of the responses to the beta questions are evaluated before allowing it to become a scored question on a future test. Questions that are found to be too easy, too hard, or are misunderstood are re-evaluated and may be either rewritten or removed.

By removing the very easy and very hard questions, we ensure that the final questions are those that can truly differentiate between candidates who possess the requisite knowledge and skills and those who do not.

All questions that pass the beta test are made live.