This is first in a multi-part series on designing and building a technical organization. My emphasis is on computational engineering, which is a blend of software development and quantitative analysis for product development. Another way to put it is how to build organizations where data is the heart of the product. This first installment discusses the importance of defining and communicating your culture for hiring. I argue that most companies do not know how to characterize their culture in concrete terms, which is the biggest impediment to ensuring employee productivity and success in the company. The second installment explores the difference between problem solving and critical thinking, specifically the impact this distinction has on culture and employee performance. I will also discuss why I do not advocate STEM-centric educations, which is probably anathema coming from someone who is a poster-child for STEM. In the third installment I will discuss how to create a custom model that accurately selects people for your culture. It should be obvious that culture is derived from the leadership of an organization. This observation leads to two important conclusions: culture is malleable and an excellent culture must be maintained with vigilance; leaders must be chosen based on their values and vision for the culture.

As a point of departure, let’s look at Google’s technical interview process, which has recently been the subject of numerous key strokes. Lazlo Bock, SVP of People Operations at Google, confessed that academic performance, test scores, brain teasers, etc. have little predictive power in forecasting future employee performance. Like many things it turned out that this was largely research of the obvious but took a population of 50,000 people to determine it definitively. The results seemed to please a lot of people, particularly the ones that have a disdain for brain teasers and tests of esoteric knowledge. But there is more to it than that. The conclusion is that experiential questions provide more insight into future performance than GPA, education level, and even mastery of brain teasers. Why this is the case can be answered glibly: humans are  too complex to be distilled into broad measures as those listed above. Throw in the management factor, politics, family and personal issues, etc. and clearly these so-called objective metrics will have little predictive power. In more precise terms, Google has chosen a poor set of explanatory variables (GPA, education level, mastery of brain teasers) to model their response variable (employee performance).

The case of the mistaken culture

The problem with Google’s approach and the reason why people in general have so many poor hires is because people fundamentally don’t know what they are hiring for, in other words the type of person that thrives within their specific culture. I’ll stick my neck out and say that cultural fit is the single most important factor in future employee success. How do people go about describing their culture to outsiders? It depends. It used to be that there were the job descriptions where people say they want superstars or code ninjas or whatever and then describe their culture as fast-paced, innovative, yadda yadda. Nowadays things are more polished. Instead you get descriptions of culture like this: ping-pong tournaments, social groups for cycling, open layout with lots of space for relaxation (?!), cold brew, free metrocards, etc. This is not culture per se, but it does imply that the company believes that social interaction is more  important than personal space (and productivity), and that culture is defined by the treats given to employees. To be fair this is a one-sided assessment, but if you are not explicit about your culture, how can you prevent interpretations such as these?

It is common to see companies conflate perks with culture or fool themselves into thinking that some exciting, forward-thinking adjectives are an appropriate proxy for a culture. Other times the lack of any culture is so prominent that it’s hard to ignore. In the below example, you have to read between the lines of the job description to uncover aspects of the culture:

The role will represent the technical team within the management group including the CEO, COO and VP of Product. … The role will report to the VP of Product. We are looking for the right fit.

 

The fact that the most senior technical person in the company reports to the VP of Product speaks volumes about their subordinate view of technology and engineering. This is in a company that sells software services.

To ensure that talented staff will thrive in an organization, a company needs to communicate clearly both their mission statement (purpose) and their culture statement (method). Companies typically focus on the former while falling down on the latter. It’s only after the hiring failures that people start wondering what it is about them (and by extension their culture) that is problematic. The mission statement alone does not communicate much about the working method. Even Google fails miserably here, since their mission statement is broadcast to the world (in very large font, mind you) while their culture statement is anemic:

We strive to maintain the open culture often associated with startups, in which everyone is a hands-on contributor and feels comfortable sharing ideas and opinions.

 

In a company with 50k+ employees, this is a remarkably vague statement. Note also how this statement stops at “sharing ideas and opinions”. This is actually quite different from a startup where you are required to own your ideas and make them real. What outsiders know about the Google ‘culture’ are the legendary perks: free sushi, free massages, etc. The slideshow below the culture statement highlights more great perks: a workspace with a tatami mat in Japan, a pub-style lounge in Ireland, a climbing wall, a bowling alley. Again, these are things, not a culture.

Hiring as an optimization problem

My operating principle is that knowing your culture will improve hiring success. The Google method is to look at hiring as a random process and then conduct research to determine which factors have the most significance. Hiring is not analogous to a physical phenomenon with universal laws that must be discovered, so this sort of experimental process doesn’t make sense (not to mention the gross selection and survivorship biases). Nor is it some anthropological study where researchers are not allowed to engage/interact with their subjects. Instead it is better to think of it as an engineering problem, with a goal and a set of constraints. In other words define your culture as your constraint and then optimize the interviewing process (and management process) to fit the culture. In essence an interview is a function that maps a candidate to a culture given a set of questions, or

The challenge in hiring is that the interview is a multivariate function where it is difficult to hold both arguments constant. (We could complicate it further by adding the interviewer as another argument, but let’s keep this simple.) The problem can be described using mathematical programming.

The idea is that maximizing the interview function yields the candidates that best fit the culture. How closely a candidate must fit the culture is governed by , while how closely the interview questions must represent the culture is codified by . Most people should have a small whereas will vary depending on circumstances. Characterizing hiring in this fashion is particularly effective because it explicitly acknowledges the dependence that hiring has on the culture.

Codification of culture

While no two company cultures will be the same, the process for describing the culture has a standard template. Over the years I’ve reflected on why some people excel at a company and why others fail. From this exercise I’ve developed some questions whose answers reveal a company’s true nature aka their culture. This is not an exhaustive list as it is meant primarily to illustrate the framing of a company culture.

1. Incentive structure
   1. What motivates the culture – Money, fame, pursuit of excellence, pursuit of truth, intellectual rigor, intellectual freedom, pushing limits
   2. How is success rewarded – Financially, emotionally, socially, no
   3. How is failure handled – Publicly, privately, with fear, constructively
2. Employee independence
   1. How are decisions made – Top down, bottom up, by cabal, by popular vote
   2. How are projects managed – Team selection, reporting, milestones, status, management
   3. How diverse is the work environment – Telecommuting, personal space, privacy, working hours, quiet spaces
3. Organizational model
   1. What is the management structure – Hierarchical with silos, matrix, flat
   2. How is conflict resolved – Directly, via subterfuge, going over someone’s head, via arbitration
4. What is not tolerated in your culture – Dishonesty, cowardice, “fitting in”
5. How consistent is your culture – Within a silo, ad hoc, cult

Note that these questions do not quibble over whether the cafeteria should stock sushi versus sausage or a kegerator over a chemex. They are about two core things: how much control the executives of the company desire and how much respect employees are given.

Once the culture is defined, it is fairly easy to map interview questions and other objective measures to evaluate whether a candidate is a good fit. There is nothing that says you can’t use GPA, education level, etc. as factors in the hiring process. What’s important is knowing how well these measures map to your culture. For example, despite the current open disdain for brain teasers, I am an advocate of asking people Fermi problems (of the “how many golf balls fit into a bus” variety). The reason is that it maps to my culture of intellectual curiosity, being explicit about assumptions, and being able to estimate error. This is because I hire for computational engineers, which requires the ability to think critically. People who avoid vagueness or don’t know how to eliminate ambiguity is a big red flag for me (high management overhead, productivity drag), and Fermi questions identify that quickly. Where possible it is good to use or follow up with a real-world Fermi problem (e.g. how do you detect regime change in an irregular univariate time series?).

Using experiential questions does not save you from needing to determine how well the questions (and expected answers) map to the culture. In some ways it is harder, since answers can be all over the place (as people’s experiences will likely be all over the place). Furthermore if your culture is unique, it is unlikely that candidates will have experienced situations that overlap with your culture. For example someone may have all their work experience in large companies with a lot of managerial hierarchy. They may also be extremely unhappy because they would be better suited to a small company with a flatter structure. Suppose you have a flat structure. The way to determine whether someone can thrive in your environment is not by asking “do you prefer flat organizations or hierarchical organizations?” nor experiential forms like “tell me about the team you enjoyed working in the most”. The first is easy to read, so people may bias their answers. The second is biased by their experience and may not accurately reflect their true values. A better question is conditionally hypothetical: “describe what you do when your boss tells you to do something you know is incorrect”. If they go into detail about how they’ve handled the situation in a hierarchical organization, you have to eliminate that constraint for them: “if you weren’t in a hierarchical organization, how would you handle it?”

A brief case study

The culture I have defined as Chief Scientist/CTO of my current company is based on critical thinking, openness, and ownership. My belief is that work should be intellectually challenging and rewarding, so I look for people who have a demonstrated sophistication in their knowledge, are intellectually curious, and want to learn a lot in a short amount of time. To do this people need to be mature and responsible to know how to commit to a schedule, know when to ask for help, and know when enough research/design/rigor is enough. I don’t tolerate drama nor people who don’t take ownership of their work. Weeding out the poor fits is an important step that enables you to offer more to the good fits. With the right group of people you can offer tremendous amounts of freedom: we don’t have set working hours, locations, nor silos of responsibility. The result? My staff tell me that this is the hardest job they’ve ever had and simultaneously that it’s the most rewarding. I’m also getting tremendous productivity out of a group of people with little programming experience and lacking ivy league degrees.

Achieving this sort of organizational success involves the following types of questions. In addition to a Fermi-like problem, I require two code reviews: one of their code (whatever they want to discuss), and one of ours. Not only does this eliminate the stress of writing code on the spot in an unfamiliar work environment (try writing an algorithm in Google Docs when you normally write code in vi), it allows people to put their best foot forward and discuss something they are proud of. The meta part of the interview is seeing how they react to criticism (although biased because of the interview setting) and whether they recognize the tradeoffs in their design. Understanding how well someone can work with others (and particularly an environment with a lot of freedom) is arguably more difficult to measure. This is done through targeted experiential questions: “tell me a situation where you’ve disagreed with a peer”, “describe a situation where you thought the outcome was unfair to you”, “tell me how you deal with a junior teammate that doesn’t do things how you want them to”.

Hiring is inherently difficult. By focusing on your culture and finding candidates that fit your culture, long term success can be improved. Applying an analytical approach to this process can reduce the variability and seeming randomness often associated with hiring.