Developing the Right Questions

One of the pitfalls for organizations that adopt structured interviewing yet don’t achieve the benefits is question sets that lack validity. So while the mechanics are in place, it is all for naught as a candidate’s appraisal against an irrelevant set of questions doesn’t inform a hiring decision. Usually, this occurs because subject matter experts [SMEs] weren’t responsible for question formulation.

Like many of the recommendations in this resource, upon reading the above, it may seem blatant that questions written by non-SMEs may lack predictive value. But it is more common than you may think! This happens mainly because organizations, especially smaller ones, need to hire the “first-of” for functional roles as they grow without the benefit of an expert already performing that role. So, for example, suppose a CEO in a small startup is performing the company’s HR work out of necessity. Of course, this doesn’t mean that the CEO is doing it exceptionally well or that they are in a position to know the “must-have” competencies or how to differentiate between a compelling interview response and one that seems convincing but is lacking.

An early step in job analysis is convening a group of SMEs—be they from within the company or external. These experts would review the task inventory and competencies to understand what a specific job means to this organization. Many people with the same job title are expected to perform different duties depending on how a particular job fits into the fabric of a unique organization.

Another often overlooked detail in identifying who is an SME and who isn’t is to include a successful hiring track record as part of the scope of required expertise of an SME. For example, not every expert bus driver is also an expert at identifying other expert bus drivers through questioning. That itself is its own competency.

Procedurally speaking, SMEs will collaborate on brainstorming experience-based scenarios related to the job’s competencies. Then, they will convert those scenarios into questions. Ideally, a second set of SMEs would review the questions and provide feedback.

Before getting too deep into this topic, we will warn you to avoid inadvertently telling the candidate how you will rate their response through the words you use when asking your question, aka “telegraphing.” In interviews, some candidates may lie or embellish their answers, and telegraphing provides a strategic advantage to such candidates, undermining your ability to rank their responses accurately. Below is an example of two similar questions, one that telegraphs and one that doesn’t.

In this question, the candidate knows you want a positive review of Trello:

At our company, we love Trello for managing small projects. Have you ever used Trello or a similar tool for task management?

In this question, it is less clear what your position on Trello may be:

In this role, we have seen that sometimes digital tools have been helpful, and other times they have caused challenges. From your experience, what are the benefits and limitations of Trello or similar tools for managing small projects?

In some of the worst cases of telegraphing, a candidate unaware of a topic can provide a compelling response by restating parts of the question and filling in gaps with context clues. So, don’t telegraph!

Due to the limited time available to interview a candidate, sometimes there is a tendency to add additional sub-questions making questions compound. This is problematic for two reasons:

For candidates, it may cause confusion. While answering part one of your question, they must keep part two (and three, and four!) in their memory. This can result in partial responses or situations where interviewers feel they need to coach candidates to provide a complete answer.

For interviewers, this may mean that your question touches on multiple competencies, making it harder to rate a candidate’s response. For example, what if part of their response was good for competency “A” but bad for competency “B”? Ideally, a question targets a single “must-have” competency.

When conducting an interview, asking open-ended questions is a great idea. Focusing on what you want to know rather than how you ask is easy. However, how you ask questions matters in terms of what and how much you can discover; you can learn unexpected and significant things with this easy technique.

Open-ended questions are questions that allow someone to give a free-form answer.

Closed-ended questions can be answered with “Yes” or “No,” or they have limited possible answers. These are good for surveys, not for interviews.

Closed Example:

Have you applied the agile project management methodology in a recent role?

Open-Ended Example:

Based on recent experience, can you describe your opinion of the strengths and weaknesses of the agile project management methodology?

There are essentially two types of questions that get asked in structured interviews

1. Behavioral: based on the candidate’s past actions, literally how they behaved.
2. Situational: based on how a candidate would respond in a hypothetical situation.

Remember, no matter what type of question you ask, questions must be connected to clear “must-have” competencies. Sometimes a question sounds great, but if the needs of the job don’t inform it, it isn’t a question worth asking.

Behavioral questions are the most predictive type when estimating work performance. By discovering more about how a candidate behaved in similar situations, employers may foretell how they will perform in the future.

The other type is situational interview questions which probe the candidate on how they would respond to a hypothetical scenario. Because candidates don’t know how to react to a situation that might not happen, behavioral is considered a more reliable way to predict future behavior. That being said, it could be splitting hairs because when situational questions are connected to “must-have” competencies (and not overly abstract or theoretical), it is often clear that a candidate is drawing on specific past experiences in their answer.

At FirstWho, we find using both question types is effective. Still, situational gets used less, primarily to create more variety and engagement in the interview to improve the canddiate experience.

Situational Example

In this role as a social media manager, how would your structure a marketing campaign for a new initiative to increase bike riding in cities?

Behavioral Example

From your experience managing social media campaigns, can you step us through the rollout of a successful advocacy campaign for which you were directly responsible?

Have we mentioned that questions should pertain to “must-have” competencies?

Additionally, questions should be concise.

They should be free of any jargon that may be organization-specific and exclusionary. For example, including a sports analogy in your question might be engaging to a sports fan but could confuse a person who doesn’t follow the referenced sport.

Use superlative adjectives such as “most,” “last,” “worst,” “least,” and “best.”

Provide a setup or context for each question to further achieve clarity.

What not to do:

How do you prevent a team member from going AWOL when a poorly scoped project starts getting into extra innings?

A more clear approach:

In our industry, it is not uncommon for staff members to work extra hours to get a project completed on time. Can you describe when you needed to support your team when they were under the most intense deadline pressure?

Candidates may or may not have experience with the commonly used STAR (Situation, Task, Action, Result) method of answering. STAR encourages candidates to describe a goal, key tasks, actions they were responsible for, and the outcome—which usually results in complete answers. Sometimes, as you consider a candidate’s response, you might want to ask additional probing questions to get more details.

Probes are follow-up questions to interview questions. This medical-sounding term comes from the practice of “probing” for additional information. Probes often make an interview flow smoother, leading to a pleasant experience for both candidate and interviewer, but they should be used thoughtfully.

A correct use for probes is to get extra clarity about an answer that seems accurate, but some additional detail would give interviewers more rating confidence.

A harmful use for probes is to bail out candidates who answer poorly and then awarding them an enhanced rating if they improve. Now, you might think, why would anyone do that? Well, interviews can be an intense experience for both the interviewer and the candidate! As a result, sometimes interviewers offer too much help to smooth tensions and awkwardness. While that might not sound so bad—it’s nice to be helpful after all—but there are a few issues:

1. A low rating is a valid assessment when a candidate doesn’t supply a satisfactory answer.
2. Due to unconscious bias, candidates are unlikely to get helped equally, so it’s best not to assist candidates in interviews.
3. Uneven support undermines the validity of ratings. It is assumed that candidates arrived at their answers independently. Mixing in some helped ratings cast doubt on all ratings.

Example of a suitable probe:

Thank you for that response. You mentioned your experience on-call. Can you provide more details about that?

Example of a harmful probe:

Thank you for that response. A moment ago, I thought you were about to describe your on-call experience, but you didn’t mention it. That is quite important for this role. Do you have experience being on an on-call rotation?

Sometimes an organization will allow for helpful probes, but even that should be intentional. This is usually because highly-competent candidates are in short supply or over budget, and the organization has training capacity. The training capacity is critical here because if the organization can’t train a poorly assessed or borderline candidate that gets hired, they are not setting this person up for success. Later, this decision may stress other employees and the candidate, especially if the pace of learning is slow, leading to morale issues and turnover.