An introduction to: Cut-off scores

One of the most frequent questions our Science team gets is about setting cut-off scores: What they are, how they are set, and their benefits and risks. 

In this introductory guide, we’ll explain why TestGorilla does not recommend cut-off scores to our customers and take a look into the intricacies of setting cut-off scores for pre-employment tests. We’ll discuss the importance of considering their legal defensibility and fairness in hiring. We’ll also offer insights into common misconceptions about their use and share some best practices to consider if you decide to use them.

What are cut-off scores?

Cut-off or cut scores are predefined thresholds set by employers to determine the minimum acceptable performance level on pre-employment tests for a job role. Candidates scoring above this threshold are considered for further evaluation, while those scoring below typically do not progress in the hiring process. 

This binary (pass/fail, hire/don’t hire, recommend/do not recommend) nature of cut-off scores starkly contrasts with benchmarks, which provide a comparative analysis of an individual's performance against a group (e.g., scoring within the top 25% of candidates). 

Because cut-off scores are used to make important hiring decisions, a lot of work needs to go into setting cut-off scores to ensure these thresholds are legally defensible. Poorly set cut-off scores can negatively affect your candidates and your organization.

TestGorilla’s approach to cut-off scores

At TestGorilla, we understand that setting cut-off scores can be very complex. That's why we don't recommend universal cut-off points. Using generic cut-off scores can raise fairness and discrimination concerns, as it's challenging to prove they're job-related and necessary for all positions in different organizations.

We advocate for creating and using context-specific cut-off scores (rather than a one-size-fits-all approach), as we recognize the profound impact that role- and organization-specific criteria can have on the effectiveness and fairness of the selection process. Defining cut-off scores based on these specific criteria also ensures employers make informed, legally defensible hiring decisions that enhance job performance, reduce turnover, and maintain the integrity of the organizational culture. 

We encourage our customers to analyze the skills and competencies crucial for success in their specific organizational context. This analysis can help create tailored cut-off scores that align with the job requirements and consider the organization's culture, goals, and strategy.

The legal defensibility of cut-off scores

When done correctly, cut-off scores can make the hiring process efficient (by reducing the number of underqualified candidates under consideration) and fair (by giving all candidates a clear standard to meet). However, poorly set cut-off scores can lead to significant issues, such as overlooking potentially successful candidates, discriminating unfairly against certain groups, or advancing those unlikely to perform well in the role. 

For these reasons, cut-off scores cannot be set arbitrarily. To be legally defensible, they must be done using one of several cut-off setting approaches from the psychometric literature (e.g., a modified-Angoff approach, Bookmark, Contrasting Groups, Borderline Groups method). This is particularly crucial when your organization uses these scores to make decisive classifications such as pass/fail, hire/don’t hire, or recommend/don’t recommend. If you are doing this, you most likely need a cut-off score-setting study to ensure its legal defensibility and avoid discrimination claims.

Here are some key considerations that you may want to take into account when setting cut-off scores for tests or assessments:

• Demonstrate job specificity: Cut-off scores must be demonstrably relevant to the specific job they're used for. This means that the score threshold chosen should have a clear, empirically validated connection to job performance or critical job-related competencies.

• Ensure non-discriminatory practices: It's crucial to analyze the impact of cut-off scores on different demographic groups to ensure they do not inadvertently discriminate against any group. If disparate impact is detected, organizations must be prepared to justify their cut-off scores as a business necessity or consider alternative assessments.

• Document the decision-making process: Maintain thorough documentation of how cut-off scores were determined, including any validation studies conducted and the rationale behind chosen thresholds. This documentation can be crucial for defending the use of cut-off scores in legal challenges.

Common misconceptions about cut-off scores

Higher cut-off scores always lead to better hires

Many believe setting higher cut-off scores will invariably result in better hires by filtering for only the top performers. However, excessively high cut-off scores can exclude potentially great candidates who might otherwise excel or bring other valuable skills not captured by the assessment.

Cut-off scores are one-size-fits-all

A prevalent misconception about cut-off scores is the idea of a "universal good score" applicable across various roles or organizations. This notion fails to recognize the unique demands of different job roles and the distinct performance standards of organizations. Cut-off scores must be customized to the specific requirements of the position and aligned with the organization's expectations for performance. This customization ensures that scores are meaningful and relevant, facilitating a more targeted and effective selection process. 

Contrary to the comparative nature of scoring benchmarks, which assess how candidates' scores relate to broader norms, cut-off scores are fundamentally about meeting the bespoke criteria set for job success within the specific context of an organization.

Cut-off scores are static and unchanging

There's a misconception that once a cut-off score is set, it remains effective indefinitely. However, as job roles evolve and organizations grow, these scores need regular review and adjustment to remain relevant and effective.

Cut-off scores can completely remove bias from the hiring process

While cut-off scores can help standardize the evaluation process, they're not a panacea for bias. The involvement of people in the making of hiring decisions can introduce a tremendous amount of bias, underscoring the need for ongoing efforts to identify and mitigate biases throughout  the hiring process.

Best practices for implementing cut-off scores

Adopt role- and organization-specific cut-off scores

Adopting role- and organization-specific cut-off scores is a critical best practice in pre-employment testing, emphasizing the necessity to customize these thresholds according to the unique demands of each job and the context of every organization. This tailored approach is grounded in the understanding that generic cut-off scores cannot accurately reflect the diverse competencies and skills essential for success across varied roles and workplaces. 

Conduct a criterion validation study

Before setting cut-off scores, conduct a criterion-related validation study to ensure that the tests or assessments accurately predict job performance. This involves correlating test scores with performance metrics to identify the score thresholds that best distinguish between high and low performers.

Conduct a cut-off score-setting study

Psychometric literature suggests many methods, such as the Angoff, modified Angoff, Bookmark, Contrasting Groups, Borderline Groups, Hofstee, and Ebel methods. Which approach you choose depends on several factors, including whether you have existing data and whether you can easily recruit a panel of subject matter experts.

The modified-Angoff method is one of the most popular methods. It involves a mix of quantitative estimates and expert judgment. At its core, this method requires a group of subject matter experts (SMEs) to estimate how a minimally competent candidate (MCC) would perform on each test item, blending content analysis with performance expectations. 

The process starts with training SMEs to understand their role and the concept of an MCC, followed by independent item rating rounds to gauge the difficulty for an MCC. Discussions among SMEs to address discrepancies in ratings lead to a consensus-driven adjustment in a second round of ratings. 

This iterative process aims to refine the cut-off score to accurately reflect the requirements needed for competence. Final evaluations of these ratings, alongside statistical measures of inter-rater reliability and expected pass rates, culminate in a recommended cut-off score. Typically, the whole process is led by a psychometrician experienced with setting cut-off scores.

Involve multiple stakeholders

Include input from HR professionals, hiring managers, and possibly even high-performing employees in the process of setting cut-off scores. This ensures a comprehensive understanding of the job's requirements and the qualities of successful employees.

Ensure alignment with organizational goals

Cut-off scores should reflect the minimum qualifications for a role and your organization's broader talent acquisition strategies. Consider how these scores fit into your workforce development and diversity goals.

Regularly review and adjust cut-off scores

The relevance of cut-off scores may change as job roles evolve or as new data becomes available. Regularly review these scores against actual job performance data to ensure they continue to serve their intended purpose effectively.

Committed to setting cut-off scores? Consider the alternatives

In sum, adopting role- and organization-specific cut-off scores is not merely a recommendation but a critical component of a legally compliant and effective selection process. This approach ensures that the hiring process is fair and equitable and closely aligned with organizational needs and goals, ultimately leading to a stronger, more cohesive workforce.

Instead of going through the time-consuming and expensive process to set legally-defensible cut-off scores, it is likely to serve your needs to focus your screening efforts on the higher scoring candidates in your hiring pool. Work with those candidates most likely to perform well on the job for your subsequent stages of hiring.

Looking for more strategies to improve your hiring? Check out these science series blog posts.

• A brief introduction to classical test theory

• What is predictor-criterion congruence and how can it help me hire successfully?

• Legal risk and skills-based hiring: The legal defensibility of TestGorilla’s assessments

Disclaimer: The information contained in this site is provided for informational purposes only and should not be construed as legal advice on any subject matter. If you have any questions on the legal defensibility of tests in your area, please consult with a lawyer. Please also familiarize yourself with the laws and regulations in your specific country and region regarding the use of pre-employment tests in selection.

References

Mueller, L., & Munson, L. (2015). Setting cut scores. In C. Hanvey & K. Sady (Eds.). Practitioner's guide to legal issues in organizations (pp. 127–161). Springer.

Plake, B., & Cizek, G. J. (2012) Variations on a Theme: The Modified Angoff, Extended Angoff, and Yes/No Standard Setting Methods. In G. J. Cizek (Ed.). Setting Performance Standards: Foundations, Methods, and Innovations. Routledge.

Zieky, M. J., Perie, M., & Livingston, S. A. (2008). Cutscores: A manual for setting standards of performance on educational and occupational tests. Educational Testing Service.