Structured reference checks: a scoring guide

An unstructured reference check is a conversation: the caller asks what comes to mind, the reference says what they choose, and the notes reflect both. A structured reference check fixes the parts that vary. The questions come from the job, every reference answers the same ones, and each answer is rated on a scale that is defined in advance.

Structure does not make a reference more honest. It makes references comparable, which is what a committee choosing between finalists needs.

What the research says

Most of the evidence on structure comes from interviews, and it is consistent: structured interviews predict job performance better than unstructured ones. A widely cited 1998 meta-analysis by Schmidt and Hunter put reference checks below structured interviews, with validity estimates of .26 and .51. A 2022 reanalysis by Sackett and colleagues lowered most of the older estimates and still ranked structured interviews first, but left reference checks out for lack of sufficient research.

Research on reference checks themselves is thinner. One 2004 study found that a short structured telephone reference check, in which references rated defined dimensions, predicted later supervisor ratings. It was a single organization hiring for entry-level roles, so it shows what structure can do rather than what every campus will see. The U.S. Office of Personnel Management’s guidance makes the practical point plainly: standardized questions based on an analysis of the job can greatly improve reference checking.

Step 1: pick the competencies from the job

Start with the position description and choose the three to five things that matter most. More than five and references give thin answers to all of them.

• Faculty: teaching and mentoring, research and scholarship, service and collaboration, professional integrity.
• Academic leaders: judgment, building trust, managing people and budgets, communication across the institution.
• Staff: reliability, communication with students and colleagues, judgment, handling confidential information.

Write a one-sentence definition of each competency for the committee. If two committee members would define "collegiality" differently, the scores will mean different things.

Step 2: write two questions per competency

For each competency, ask for a rating and for an example. The rating makes references comparable; the example tells the committee what the rating is based on. Our faculty reference check questions are written in this pattern and grouped by competency.

Teaching and mentoring

1. Compared with others you have worked with at a similar career stage, how would you rate the candidate’s teaching? (1 to 5, or not able to judge)

2. Please describe a course, session or student you saw them teach or mentor, and what came of it.

Step 3: define the scale in words

A bare one-to-five scale invites every reference to use it differently. Describe each point, anchor it to the reference’s own experience of similar people, and always offer a way to say "I did not see this".

Treat "not able to judge" as missing, never as a low score. A reference who did not see someone teach has told you nothing about their teaching.

Step 4: score across references

• Show each reference’s score, not only the average. Two references at 5 and one at 1 average to a respectable number that describes none of them.
• State each reference’s relationship to the candidate next to their scores. A supervisor and a co-author see different parts of the work.
• Keep every rating with its example. The committee should be able to read why a score was given without asking.
• Compare finalists competency by competency, not by one total. A strong researcher and a strong teacher can have the same total for different reasons.

When references disagree

Disagreement is information, not noise. Before the committee discusses it, check three things: whether the references saw the same part of the work, how recently each worked with the candidate, and whether one of them gave an example while the other gave an impression.

If the disagreement still matters to the decision, follow up with a call, ask both references the same follow-up question and add the answers to the packet with the date. A structured process makes the follow-up shorter, because you already know exactly what to ask. Reference check red flags covers how to tell a real concern from a policy or faint praise.

Common mistakes

• Too many competencies, so every answer is short.
• A numeric scale with no description of what the numbers mean.
• Scores reported without the examples behind them.
• Changing the questions partway through a search.
• Averaging away a disagreement instead of looking at it.
• Counting "not able to judge" as a low score.

None of this needs a large program. A short competency list, two questions each and a defined scale, used the same way for every finalist, already puts the committee on firmer ground than a stack of call notes.