Executive Hires: The Case for Extreme Referencing

For CEOs, especially first timers, one of your most difficult tasks will be hiring truly great executives. There is no substitute for experience in hiring. I learned this the hard way as a first time entrepreneur with absolutely no hiring expertise. And yet, after hiring more than 100 people – and making a lot of mistakes – by far my best executive hire was my last one: David Yu, now CEO of Betfair.

The key lesson? If you are a young entrepreneur you need to surround yourself with investors, advisers and executives who are experienced and talented, especially when it comes to hiring other top executives.

An engineer will have a body of work that can be evaluated and tests that can be given. Technical talent can be evaluated for the first 90 days and replaced if they disappoint. However, if you are hiring a CFO, CTO, VP of Business Development or top marketing and sales executives, it’s not so easy. These executives are sophisticated at selling their experience in a compelling way, and once hired it takes time to evaluate their effectiveness. The mis-hires often do parts of the job well, making it more complicated to remove them. And, once the decision is made to remove them, there is a much greater risk of damage to the organization.

But the good news is that top executives’ experience is also more suitable for extreme referencing, as they tend to have had extended professional interaction with a much greater number of people.

Base 80 percent of your hiring decision on thorough reference checking

In my experience the leading cause of executive hiring mistakes can be traced to waiting to make (perfunctory) reference checks until the final step in the hiring process, perhaps even after the decision has been made.

It’s a tough pitfall to avoid. You invest great effort to get your executives and investors to interview a number of candidates. You laboriously negotiate to pick your favorite and only then do you ask the candidate for a few references to call. After all this time and effort, which you feel is a distraction from your true job of running the company, you call those references with a confirmatory bias, practically begging them to tell you that the interviewee is a great marketer, so you can get her on board and get back to work.

Your real work as CEO is to infuse top talent into the organization. You need to block time on your schedule to hire. If you don’t have the time, it is doubly important to build the team so you can hand off other work to focus on making the best hires.

Why references are so important

Resumes, CVs and interviews are all sales pitches. The candidates have carefully crafted the attributes they show you in order to seem perfectly suited to the job.

Some typical pitfalls in a cleverly crafted executive resume:

• Omits or minimizes negative experiences and cherry-picks the most compelling successes;
• Blurs a candidate’s role or responsibilities to make it tough to evaluate true contributions to an initiative, campaign or idea;
• Takes credit when they are just part of a rising tide, i.e. everyone who worked at Google from 2000-2005 delivered unbelievable numbers. Not all of them were studs.

By all means build your interviewing skills so you become adept at smoking out pretenders through great questioning, but know that as the candidate’s level of sophistication increases at the executive level, so too will your difficulty in spotting the posers.

Why rely on a two-hour chat with a candidate to try and divine their strengths, personality, ambition and drive when you can get a firsthand account of how they performed in the real world? Why not speak to the people who watched them perform and get their evaluation?

When you think about the most critical attributes you are trying to evaluate for a startup, they can be especially hard to divine from a resume or interview. Skills, experience and intelligence are well suited to interviews. But the things that cause someone to thrive at a startup – drive, teamwork, leadership and cultural fit – are much harder to divine.

Why do many smart founders reference check so poorly?

• They know people given by the candidate as references will have a vested interest in helping the person who referred them;
• They believe many references are reluctant to say anything negative due to policy or liability concerns;
• Checking references is very time consuming.

These concerns are real. The first two can be mitigated with good technique. On the third, I’m sorry but I’m going to make it worse – I need you to spend even more time on references than you do today.

Begin with the right expectations

• Be mentally prepared to eliminate a candidate. Go back to the drawing board if the references are less than stellar. Expect at least a 50 percent chance of eliminating the candidate based on references. If you are not in this mindset, you will only hear what you want to – that you’ve found the perfect person.

• Allocate 10-15 hours of total time to spend checking each candidate’s references. This includes research such as question preparation and tracking down additional references as well as about 30 mins per call for 5-10 calls. You need to block this time on your calendar. To repeat: if you spend less than 10 hours on extreme referencing a top hire, you need to come up with better questions and find more people to call.

• Look for negative information and largely devalue the positive information you hear about the candidate. Because of a natural positive bias in reviews, you have to assume that a lot of the positive feedback may be overstated. Hence, if someone with no clear axe to grind with the candidate gives you relevant negative information, you need to take this very seriously.

• Notice what is NOT being said. Narrowly defined praise can often be used to show you the weaknesses of a candidate. Sometimes when a candidate is “smart”, it may mean he lacks drive or interpersonal skills.

• When you start hearing the same weaknesses and strengths repeated about a candidate, it’s a good sign you’ve completed your extreme referencing.

• You should not outsource reference checking, especially to executive recruiters. The incentives are all wrong. Despite the fact that someone else may be skilled at it, you cannot outsource this critical bit of judgment in shaping your team.

Would you let a recruiter conduct all the interviews, and pick your new CTO by reading the recruiter’s notes? Obviously not, and you need to be doubly reluctant to rely on your recruiter to conduct your reference checks, a task requiring far more nuance and judgment.

There are three basic mistakes people make when checking executive references: they call too few people, they don’t talk to the right people, and they don’t ask the right questions.

Your goal of extreme referencing is to discover new people to talk to until you find someone who has nothing to lose by telling you the truth about the candidate. You may sometimes get an unbiased view from the list of people the candidate gives you to call. This can happen due to skilled questioning, or just be in the nature of the individual. However, one of the main goals of calling the candidate’s references directly is to find out who else the candidate worked with, and in what capacity, so you can track down those people and talk to them.

You should begin by setting the candidate’s expectation that you will be referencing them extensively from the list they give you and also referencing anyone else you find who is relevant. This is the point when you need to clarify with the candidate if there is anyone you should not call. Be sure to take a hard look at the reasons for any exclusions.

Make a list of people you are going to call. Start with the names the candidate gives you. If there are any clear omissions given your knowledge of their work history, be sure to ask for those names. Then, as you call candidates on the list, names of other people will surface. Add them to the list. Prioritize the list from most to least recent and most to least relevant. Work down the list until all your questions about the candidate are answered and you have clarity about the hire.

LinkedIn is a fantastic option for finding extreme referencing sources; this is the primary reason I have built out my full network on the site. When I want to reference a candidate my first step is to punch in their name and see who I know that is connected to them. This is your best chance to get reference gold: someone who knows the candidate well but knows you better.

You should plan to talk to all their direct supervisors, some peers, and at least a few people who reported to them. And remember that it is important to qualify the reference, especially to get a sense of how they rank employees or what “great” means to the reference. Are they used to working with really high caliber people? One tactic is to quickly cross-reference the reference when speaking to other references from the same company. Another is to ask yourself – would you hire the reference person?

If there are people on your reference list you want to call that the candidate did not give you, there are a number of good sources for finding their contact information. Do not hesitate to cold call them. In addition to LinkedIn other top sources: Facebook, Google, PeopleSearchPro.

If you’ve done all this, you now have a long list of people to call.

Ask the right questions

You can and should ask all the standard questions of a reference call. I want to pay special attention to two categories: key questions you might not think to ask, and a list of typical questions that are phrased in a way that’s different and more effective.

Here are some key questions that you might not think to ask explicitly, often because you assume you already know the answer:

• Did Joe report directly to you? For what portion of his tenure at the company? Who else did Joe report to?
• To the best of your knowledge, did Joe resign, or was he asked to leave?
• Ask the recruit, bluntly, “What was the biggest mistake you made in the role” – then reference check against his answer.
• (To build your list of references) Everyone I speak with says such great things about Joe. Is there anyone Joe didn’t see eye-to-eye with? Why do you think?
• Why wasn’t Joe further promoted at your company? An interesting answer sometimes is that there wasn’t a spot (i.e. they would have to replace their boss). If that’s the case, what skills/ability does the boss possess that Joe lacks? How do you think Joe would perform in the boss’s role? (This one might be pretty hard to ask if you are talking to their boss).

It’s really important to ask hard questions. You can elicit useful information by asking questions that have no easy way out, even in cases where the reference has a close connection to the candidate.

Most people find it tough to ask hard questions for fear of seeming rude. If this is the case, it’s better to mitigate the rudeness by a preface to the question, rather than by softening the question itself. I’ll throw some examples into the hard questions below.

Hard questions to ask

• Joe is by far the best candidate I’ve seen. Still, everyone has their short suits. What are Joe’s?
• We’re inclined to make Joe an offer, and I hope he will take the position. If he does, how would you suggest I manage Joe to get the best performance out of him?
• (If the candidate had peers in the role) You say, “Joe was good.” Compared to Tom and Mary was he clearly the best at the job? Or did they each have their strengths and weaknesses? What was Mary better at than Joe?
• Did Joe have setbacks? How did he handle them?
• Would you hire him again? If so, would you rehire him without hesitation? Create a role for him if you didn’t have one? Only hire him in the right role?
• How would you change the role we’re considering him for to best suit his abilities?
• If I were to hire Joe and six months from now you heard he was fired without other context, what would be your best guess as to why he was fired?
• What are Joe’s passions in and out of work?
• How hungry is Joe for results, for example, what is the craziest thing you’ve ever seen him do to achieve a goal?
• Joe mentioned the launch of the new website was one of his big achievements. What was his actual role in the launch? Was he solely responsible, or was it a team effort?

How to evaluate the answers

The rest of this reference guide is brute force: if you work hard at it, I am confident you will be successful. But evaluating what you hear is an exercise in nuance and judgment, which you will develop over time as you do more hiring (and extreme referencing). If you’re still developing your judgment, lean on experienced and talented advisers who can help you piece through the information you collect.

Startups have a lot of setbacks along the way. I think that digging into times of adversity or failure of the candidate can be enlightening. If they haven’t worked in roles with real setbacks, it is something to consider.

One consequence of extreme referencing is that you will hear more bad news than you did previously. You need to have tolerance for some faults in the candidate as there is no perfect person. If he or she didn’t scale sufficiently at a company that grew past 300 employees, but you have just eight employees in your startup, it’s a risk you might be willing to take.

When hiring into a young startup be especially vigilant on cultural issues. If you’re evaluating someone coming from a more established company, or if you are young founders hiring some “adult supervision,” you should work hard to get a good sense of the candidate’s drive and work ethic. Consider the impact of them leaving at 6 p.m. while the rest of the team codes until midnight.

As CEO of a startup, your top job is to infuse as much talent into the organization as possible. Make it a priority and devote energy to extreme referencing, and you will be doing your job to give your business the best chance to succeed.

Thanks to Ben Elowitz, Jack Herrick, Alastair Mitchell, Mark Peters, Antonio Rodriguez and David Skok for their help on this post.  Special thanks to Irv Grousbeck, mentor to a whole generation of great entrepreneurs.

More thanks:  Thomas

Want to share this post?

Click to Tweet: Executive Hires: The Case for Extreme Referencing via @jdh http://www.joshhannah.com/2010/06/executive-hires-the-case-for-extreme-referencing/
Like what you've read? Click here to subscribe to this blog!
Written by Josh Hannah
Josh Hannah joined Matrix Partners after a career as a serial entrepreneur (Betfair, eHow, wikiHow.) Read more about Josh.
  • http://www.thesecomefromtrees.com petekazanjy

    This is pretty awesome Josh. And you make the case for exactly why we think that Unvarnished is going to be helpful in allowing entrepreneurs (and candidates, too) to do more easily exactly what you're talking about.

  • joshhannah

    Great point. I agree!

  • LesleyYoung

    Great advise and well articulated. Bringing in executives who have experience and skills to fill skill gaps in the existing management team are the ones who build strong, effective, high growth execution teams.

  • http://profiles.yahoo.com/u/7ZOOMI6GFLHHCMOVMIAHY65TGY TerryD

    I guess this is good advice, but I'm wondering what qualifies you to speak on the matter. It cracks-me-up all the 22-year-olds giving all kinds of advice.

  • http://twitter.com/bryanrbeal Bryan Beal

    Great concept, but not entirely based in the realities of hiring an executive. This may work when hiring an Exec from one startup into another startup, but this process won't always work when hiring an executive with experience from a major company. The reason – many large organizations have very strict policies on current employees giving references on past employees. They are worried about libel litigation, and as such restrict what employees can and cannot say about former employees. It is common for a large company to limit reference information to very quantitative, verifiable data (salary, employment dates, etc) and ban subjective statements (would you hire them again, what were they weak at, etc). I supposed someone could violate corporate policies and give the reference anyway, but is that really what you want to promote? I am not surprised to see such an opinion being voiced by someone who's career has been focused purely within startups, but you won't be hiring an exec out of many major IT companies with this approach.

  • joshhannah

    Corporate reference policies are a legitimate challenge to doing a good reference check. In my experience, if you work hard at it and are pursuing a big list, you can still get people to talk. I've certainly been successful referencing out of Fortune 500 companies and large tech companies, but it's not 100% cooperation.

    My experience has been that negative references are more likely to hide behind a corporate policy than positive ones. When I'm having a hard time getting cooperation I take that as a red flag and work doubly hard to reference the candidate. Ultimately I don't feel comfortable hiring someone I haven't been able to fully reference, the cost of a mishire is just too great.

    TJ at Cypress Semiconductor, a hiring innovator, would not allow any hires in the entire company unless they had been endorsed by someone age Cypress employee had a strong personal relationship with. Without a first-degree reference, for any employee, they felt it better to pass them to take the risk. Now that is extreme reference checking!

  • http://twitter.com/bryanrbeal Bryan Beal

    I like that policy. It is very smart to leverage internal referrals. Brilliant people often know other brilliant people, and they are typically careful to recommend someone when they know their own reputation is on the line within their own company.

  • http://profiles.yahoo.com/u/6VKTNMCTJEYS6CCW5DJTZBJLBQ Leake

    Nice for in the trenches but be careful of your biases – the diversity of your team may still predicate your successes. There is no substitute for first person due diligence, and I think that is your major point here. I might add that basing the success of any strategy/tactic on the performance of another entity (esp. TJ Rogers @ Cypress) might be spurious. Good luck

  • http://twitter.com/bhalligan Brian Halligan

    One of my favorite questions is to give references a net promoter test: On a scale of 1 to 10 (with 10 being the highest), how strong is your referral. If less than 10 (I've yet to get a 10 on a blind reference), my next question is a simple “why not a 10?” After they get the first reason out, I ask why again and again until they run out of things to say.

    A fellow CEO was checking on a guy he was hiring with me and his first question was “What does this guy suck at?” He said it very forcefully in a way that didn't give me time to think about it and the first thing that came into my mind ended up coming out of my mouth. …I thought this was good too.

  • http://twitter.com/williambharding Bill Harding

    Holy cow those are really great questions to ask. Worth the price of admission in themselves!

  • Jumpshot

    Much of this advice on 'extreme referencing' applies equally from the candidates perspective as well. Just because the board of a start up has some great VC company names on it – it doesnt mean you will be working with/for the A team. Understanding the company you might join and the dynamics of the Board and incumbent team can help candidates in equal measure.

    Many of the questions proposed focus on the negative side of the ledger. Understanding the positive traits of a candidate or employer can be equally insightful – and as you say, when times get tough, these traits will be important.



From the Blog

Box, Uber and 29-cent Tacos

My twitter is alive today with stories of Box, now offering *unlimited* file storage for just $35.  What a deal!  Consumers love...

What’s the Big Idea?

“To me, ideas are worth nothing unless executed. They are just a multiplier. Execution is worth millions.” — Steve Jobs

Conventional wisdom in...

The Power of Preference

Uber is getting a lot of attention for the eye-popping price of their lastest financing round ($17B, in case you missed it).  It’s amazing...