How to Tell if Executive Search Firms are Data-Driven and Evidence-Based
From my work with search committees, one question that often comes up is how to evaluate whether a search firm is evidence-based or data driven. Staffing Advisors' entire approach to hiring is evidence-based. But how can that be proven, or compared to another search firm?
There are several elements to consider. Sadly, a firm’s stated retention rate cannot be one of them. Because it cannot be independently verified or observed, anyone can claim any retention rate on their placements. For example, our rate is 85% retention at 3 years, but there is no realistic way a client could verify that number. It would require knowing every single one of our searches, reaching out to every single client, and confirming the employment dates for every placement we've done.
Then even if someone did all that, it still wouldn't be a perfect comparison. Even if you assumed an honest intent to report retention rates accurately, the math to calculate retention can be fuzzy, with wildly varying definitions between firms. For example, we have an 85% retention rate on our placements at 3 years. But to calculate our 3 year retention rate, we only consider people who we placed at least 3 years ago. If we looked at everyone we placed within the past 3 years, our rate would be much higher, because it would include people who started work last week.
So what questions can you ask to help determine if a search firm is “evidence-based” vs. opinion-based?
1. How will the search firm gather input for the position overview?
In the early search committee meetings, what will be discussed? A rigorous evaluation of the business problem to be solved, or a general discussion of strategic goals and ideal candidate attributes? When speaking with stakeholders, is the information gathered in a similar vein?
Is the search committee discussing problems to be solved and discussing the tangible measurable competencies needed to drive results (knowledge skills and abilities that can be tested)? Or is the search committee primarily discussing attributes? (Attributes are generally opinions that can be open to interpretation depending on who is doing the evaluation, such as: bold, courageous, innovative, mature, etc.) Attributes can be a problem because research has shown that more than half of your evaluation of another person is a reflection of the you, and not the person being evaluated.
2. Ask for examples of position overviews the search firm has developed.
Are they generally similar to each other or meaningfully different? Are they full of vague generalities that might apply to a variety of organizations, or do they clearly signal the search committee’s intentions? Do they identify the business problem to be solved, indicate the nature of the challenges, and outline the amount of running room the new leader will have? Are the messages interesting or even inspiring to read? And most importantly, is it clear to you what performance is expected? Top performers want to know that the board (or search committee) has thought deeply about what is needed, and is clear about performance expectations, otherwise your job could be a massive career risk for them.
3. Ask how the candidate pool will be developed and what selection criteria will be used to identify who will be contacted.
Data-driven recruiting offers the additional benefit of being more inclusive of diversity – a “Moneyball” approach to recruiting. Diversity comes from the right message being delivered to the right audience, right from the start. Diversity comes from thinking expansively about the underlying pool of who will be contacted. It is not about awarding points to certain people, or just posting ads on so-called “diversity job boards.”
Listen for if the search firm is using selection criteria that actually predict success on the job. (More on that here.) Recruiting often looks like a modern integrated marketing campaign. There are metrics, A/B testing, a variety of message delivery mechanisms, and a feedback loop that includes the search committee. Be concerned if the candidate selection criteria veer toward elements that are not predictive of success on the job (such as education, years of experience, the prestige of the candidate’s current employer, confidence, being known by the search consultant, etc.) These elements invariably exclude some very well-qualified candidates.
And be positively alarmed about vague answers suggesting that, “Everyone is in our database” or, “Everyone takes the call when it is from our firm” or any reference to, "Good people know good people" (because those "good people" mostly tend to know demographically similar people, and that can be remarkably limiting to the development of your candidate pool.)
4. How will the candidate pool be vetted by the search consultant?
When it's time to select candidates for interview, what information will be shared with the search committee beyond the resume? Will the search committee receive some kind of pretty “candidate pitch” document (like a PR agent might produce), or receive some type of briefing from the search consultant (also designed to pitch the candidate), or will you see something else?
We prefer to share information specifically related to the competencies the search committee has discussed that are most likely to result in success on the job. The candidate submits this written supplemental information on their own, so it is both a writing sample and an evaluation of how they think, what they value, and what they prioritize. No fluff, no pitch, just data that is likely to predict success on the job.
5. How are search committee interview questions developed, and how do they relate to competencies that drive success on the job? Are those competencies also being evaluated in reference checks?
Interview questions are often suggested by board members or other stakeholders, and may not be specifically related to competencies that drive results. This can result in nearly meaningless questions like, “What is your philosophy about leadership?” It is best to avoid the realm of theoretical questions in the interview process. Talking about work and doing work are quite different skills.
6. Beyond the interview questions, how will competencies be put to the test during the hiring process?
Does the search firm suggest ways to add rigor to the hiring sequence with work-sample testing? Do they help you create an environment where candidates might fail, or do they suggest a less rigorous hiring sequence? (Side note: Generally, I do not recommend introducing pre-employment personality testing to the executive search process unless the organization already has a long history of success with a particular instrument. I explain why in this post.)
7. Ultimately you are trying to find out how the proposed hiring process will reduce your risk of a hiring mistake.
The search firm should be current with the latest research on hiring practices, and well aware of the fact that typical hiring practices are laden with bias and prone to error. Whatever data-driven approach they suggest should be designed to zealously guard against the very human nature to prefer “people like us” and help the committee focus on factors that predict success on the job, and avoid reflexively gravitating toward the person “you’d like to have a beer with.”
If a search consultant is surfacing issues that make you stop and think at regular intervals, you are probably on the right track. (Then you know they are thinking equally hard.) But if they readily accept simple answers to complex questions, you might want to find another consultant.