Developing the Right Questions


Questions are the primary research tool organizations use to inform their assessments and recommendations. As such, there is no overstating how crucial it is to ask the right questions. Between the limited time of getting to know a candidate and the strict competency requirements of a job, there is no time to waste on the wrong questions.

💡At FirstWho, we sometimes use the phrase “illuminating questions” to highlight that “good questions” or the “right questions” help your organization discover candidates’ skills and abilities. Sometimes hiring is considered a somewhat dry business endeavor, but genuinely learning about other humans and considering how they might support your organization’s mission is a spirited endeavor.

The Role of Subject Matter Experts

One of the pitfalls for organizations that adopt structured interviewing yet don’t achieve the benefits is question sets that lack validity. So while the mechanics are in place, it is all for naught as a candidate’s appraisal against an irrelevant set of questions doesn’t inform a hiring decision. Usually, this occurs because subject matter experts [SMEs] weren’t responsible for question formulation.

Like many of the recommendations in this resource, upon reading the above, it may seem blatant that questions written by non-SMEs may lack predictive value. But it is more common than you may think! This happens mainly because organizations, especially smaller ones, need to hire the “first-of” for functional roles as they grow without the benefit of an expert already performing that role. So, for example, suppose a CEO in a small startup is performing the company’s HR work out of necessity. Of course, this doesn’t mean that the CEO is doing it exceptionally well or that they are in a position to know the “must-have” competencies or how to differentiate between a compelling interview response and one that seems convincing but is lacking.

An early step in job analysis is convening a group of SMEs—be they from within the company or external. These experts would review the task inventory and competencies to understand what a specific job means to this organization. Many people with the same job title are expected to perform different duties depending on how a particular job fits into the fabric of a unique organization.

Another often overlooked detail in identifying who is an SME and who isn’t is to include a successful hiring track record as part of the scope of required expertise of an SME. For example, not every expert bus driver is also an expert at identifying other expert bus drivers through questioning. That itself is its own competency.

Procedurally speaking, SMEs will collaborate on brainstorming experience-based scenarios related to the job’s competencies. Then, they will convert those scenarios into questions. Ideally, a second set of SMEs would review the questions and provide feedback.

A Warning About Telegraphing

Before getting too deep into this topic, we will warn you to avoid inadvertently telling the candidate how you will rate their response through the words you use when asking your question, aka “telegraphing.” In interviews, some candidates may lie or embellish their answers, and telegraphing provides a strategic advantage to such candidates, undermining your ability to rank their responses accurately. Below is an example of two similar questions, one that telegraphs and one that doesn’t.

In this question, the candidate knows you want a positive review of Trello:

At our company, we love Trello for managing small projects. Have you ever used Trello or a similar tool for task management?

In this question, it is less clear what your position on Trello may be:

In this role, we have seen that sometimes digital tools have been helpful, and other times they have caused challenges. From your experience, what are the benefits and limitations of Trello or similar tools for managing small projects?

In some of the worst cases of telegraphing, a candidate unaware of a topic can provide a compelling response by restating parts of the question and filling in gaps with context clues. So, don’t telegraph!

Challenging but not “Trick” Questions

Rigorously developing your interview questions shows that you care about who gets hired and what work gets done at your organization, which is a positive signal for candidates. However, the goal during the interview is to listen to candidates and provide the best opportunity for a well-suited candidate to reveal themselves. Generally, this means making your questions thoughtful while avoiding trickery or puzzles, which may result in a good candidate answering poorly and forming a negative view of your organization.

Research from Glassdoor shows that candidates who experience more challenging interviews have greater job satisfaction later.

Hiring reliable people capable of your “must-have” competencies is vital. Unfortunately, this may mean that some questions are hard, leading to awkward moments with less suitable candidates. ☑️ Remember: it’s better to have a few awkward moments in an interview than to have a series of difficult conversations later with a struggling staff member.

Avoiding Compound Questions

Due to the limited time available to interview a candidate, sometimes there is a tendency to add additional sub-questions making questions compound. This is problematic for two reasons:

For candidates, it may cause confusion. While answering part one of your question, they must keep part two (and three, and four!) in their memory. This can result in partial responses or situations where interviewers feel they need to coach candidates to provide a complete answer.

For interviewers, this may mean that your question touches on multiple competencies, making it harder to rate a candidate’s response. For example, what if part of their response was good for competency “A” but bad for competency “B”? Ideally, a question targets a single “must-have” competency.

Be Creative, and Have Fun!

Even though questions in a structured interview are consistent, they can still be thought-provoking and fun! For example, suppose you were hiring a manager, and you were assessing their ability to have difficult conversations with staff. The following example shows a creative approach that often results in interesting responses that show how a candidate developed a competency over time by learning from mistakes.

Creative Example:

What was a time you had to have a difficult conversation with a person you managed that did NOT go well?

Open-Ended vs. Closed Questions

When conducting an interview, asking open-ended questions is a great idea. Focusing on what you want to know rather than how you ask is easy. However, how you ask questions matters in terms of what and how much you can discover; you can learn unexpected and significant things with this easy technique.

Open-ended questions are questions that allow someone to give a free-form answer.

Closed-ended questions can be answered with “Yes” or “No,” or they have limited possible answers. These are good for surveys, not for interviews.

Closed Example:

Have you applied the agile project management methodology in a recent role?

Open-Ended Example:

Based on recent experience, can you describe your opinion of the strengths and weaknesses of the agile project management methodology?

Behavioral and Situational Interview Questions

There are essentially two types of questions that get asked in structured interviews

  1. Behavioral: based on the candidate’s past actions, literally how they behaved.
  2. Situational: based on how a candidate would respond in a hypothetical situation.

Remember, no matter what type of question you ask, questions must be connected to clear “must-have” competencies. Sometimes a question sounds great, but if the needs of the job don’t inform it, it isn’t a question worth asking.

Behavioral questions are the most predictive type when estimating work performance. By discovering more about how a candidate behaved in similar situations, employers may foretell how they will perform in the future.

The other type is situational interview questions which probe the candidate on how they would respond to a hypothetical scenario. Because candidates don’t know how to react to a situation that might not happen, behavioral is considered a more reliable way to predict future behavior. That being said, it could be splitting hairs because when situational questions are connected to “must-have” competencies (and not overly abstract or theoretical), it is often clear that a candidate is drawing on specific past experiences in their answer.

At FirstWho, we find using both question types is effective. Still, situational gets used less, primarily to create more variety and engagement in the interview to improve the canddiate experience.

Situational Example

In this role as a social media manager, how would your structure a marketing campaign for a new initiative to increase bike riding in cities?

Behavioral Example

From your experience managing social media campaigns, can you step us through the rollout of a successful advocacy campaign for which you were directly responsible?

On Clarity

Have we mentioned that questions should pertain to “must-have” competencies?

Additionally, questions should be concise.

They should be free of any jargon that may be organization-specific and exclusionary. For example, including a sports analogy in your question might be engaging to a sports fan but could confuse a person who doesn’t follow the referenced sport.

Use superlative adjectives such as “most,” “last,” “worst,” “least,” and “best.”

Provide a setup or context for each question to further achieve clarity.

What not to do:

How do you prevent a team member from going AWOL when a poorly scoped project starts getting into extra innings?

A more clear approach:

In our industry, it is not uncommon for staff members to work extra hours to get a project completed on time. Can you describe when you needed to support your team when they were under the most intense deadline pressure?

Providing Interviewers with Positive and Negative Indicators

Due to the avoidance of telegraphing, interviewers may need clarification about assessing an answer. For this reason, questions should get paired with negative and positive indicators that guide interviewers to accurate and consistent ratings. This could be a bullet-pointed list for each question. Additionally, showing the specific competency evaluated by the question improves rating quality.

Probing Questions & Policies

Candidates may or may not have experience with the commonly used STAR (Situation, Task, Action, Result) method of answering. STAR encourages candidates to describe a goal, key tasks, actions they were responsible for, and the outcome—which usually results in complete answers. Sometimes, as you consider a candidate’s response, you might want to ask additional probing questions to get more details.

Probes are follow-up questions to interview questions. This medical-sounding term comes from the practice of “probing” for additional information. Probes often make an interview flow smoother, leading to a pleasant experience for both candidate and interviewer, but they should be used thoughtfully.

A correct use for probes is to get extra clarity about an answer that seems accurate, but some additional detail would give interviewers more rating confidence.

A harmful use for probes is to bail out candidates who answer poorly and then awarding them an enhanced rating if they improve. Now, you might think, why would anyone do that? Well, interviews can be an intense experience for both the interviewer and the candidate! As a result, sometimes interviewers offer too much help to smooth tensions and awkwardness. While that might not sound so bad—it’s nice to be helpful after all—but there are a few issues:

  1. A low rating is a valid assessment when a candidate doesn’t supply a satisfactory answer.
  2. Due to unconscious bias, candidates are unlikely to get helped equally, so it’s best not to assist candidates in interviews.
  3. Uneven support undermines the validity of ratings. It is assumed that candidates arrived at their answers independently. Mixing in some helped ratings cast doubt on all ratings.

Example of a suitable probe:

Thank you for that response. You mentioned your experience on-call. Can you provide more details about that?

Example of a harmful probe:

Thank you for that response. A moment ago, I thought you were about to describe your on-call experience, but you didn’t mention it. That is quite important for this role. Do you have experience being on an on-call rotation?

Sometimes an organization will allow for helpful probes, but even that should be intentional. This is usually because highly-competent candidates are in short supply or over budget, and the organization has training capacity. The training capacity is critical here because if the organization can’t train a poorly assessed or borderline candidate that gets hired, they are not setting this person up for success. Later, this decision may stress other employees and the candidate, especially if the pace of learning is slow, leading to morale issues and turnover.