Kelly: Pete, what abilities make an analyst successful?
Pete: There are clearly technical skills and communication skills. We often think quite a lot about the technical side, and try to assess whether or not someone has those technical chops; but, equally important, is the ability to communicate your results. Communication might mean translate a question from something ambiguous into a direct mathematical representation. Or it might mean explain the results in a way that are consumable and actionable. So it’s not just that you are able to run some complicated piece of analysis over a giant data set. It is also about being able to understand what your coefficients mean or what your outcomes mean and then be able to explain those.
Kelly: Interesting. I think you’ve touched upon couple of key points. One is a background in statistics. To understand what you are doing is not the same as simply executing a statistical procedure. You have to really understand it to communicate it effectively. Two, is the ability to translate a very complex problem into a set of hypothesis that can be tested. Can you elaborate on this a bit more?
Pete: Sure. So, as an analyst, the keeper of a company’s data, you’re often asked a lot of questions. These are often narrowly defined ad-hoc requests. A very junior approach to that is to go ahead and answer it. A great analyst, however, is not just an answerer. A great analyst will try to figure out what the customer is trying to get at with that data and is that the best data to answer the question?
To do that, it is very important to understand what actions will be taken as result of the answer. Your job is actually to set up the horse race, the competing hypotheses with different actions associated with them, to get to the result. So, I think a more advanced skill as an analyst is to be able to actually understand what the customer is asking and not just be an answerer. That’s very, very important.
Also, there’s a clichéd phrase among statisticians that correlation is not causation. Typically when we write down a model, we’re looking at things that are co-varying together. The right side is the independent variable and the left side is your outcome or dependent variable of interest.
And, what’s assumed is that the relationship always goes from the right side variable causing the left side variable. A great analyst is one who doesn’t get duped into the easy trap of correlation versus causation and will try to understand to what extent is this question like an experiment and to what extent it is not.
Kelly: So, it really comes down to habits of mind. You are looking for certain habits of mind. How do you assess that, in an interview? Do you have a favorite question or type of questions you like to ask?
Pete: We typically pick up people with great training in statistics or the scientific method and have built up years of expertise and experience in research. I don’t care how many techniques you know for testing the difference between two samples. What I care about is that your training has subconsciously forced you to think like a scientist, forced you to think about things, again, like an experiment. What’s valuable is not your explicit knowledge but your mindset your training has built into you.
So, I typically like to ask open ended questions around an imaginary data set and see whether or not the person is able to identify the hidden assumptions within that data set and their implications.
Kelly: Do you have a more concrete example?
Pete: So, one of my favorite questions is taken from the OkCupid blog, I don’t know if you’re familiar with the OkCupid blog? It is a dating website.
Kelly: Not really, but I’m going to check it out after this.
Pete: Fantastic. And what I typically ask somebody to do is critique one of these blog posts. Look at what the author is implying based on the data and what sort of assertions he’s making or advice that he’s giving to the customer, in this case, the users of the site, and what might be wrong about that implication.
For instance, one of the questions I like to ask is, “How long should a message be on OkCupid and imagine that you had a full complete data set of all the messages?” And it would be wrong just to explicitly look at what message length has been the most successful because the people wrote those messages weren’t experimentally put into those groups. Often people confuse data generated in a clean way through experiment with data generated with some other purpose in mind.
Kelly: So, it is a “trick” question.
Kelly: Pete, you mentioned something interesting last time we spoke. You said to develop a top analyst, you have to allow people to fail. Can you elaborate on that a little bit more?
Pete: So, we hire a lot of people with a lot of good adjacent experience, but not necessarily direct experience, for instance, people directly out of advanced degree program that have not had much or any work experience.
To transform this raw talent to someone who is an independent contributor, you have to put them in positions to both succeed and fail. Micromanagement in analytics doesn’t work. You want people to independently solve problems. It might take them twice as long, ten times as long, or even longer initially, but what you want is for that speed to converge on your general analyst population. And for junior analyst, you want to shorten your feedback cycle as well.
Kelly: Fantastic. We covered a lot of grounds today, Pete. Thanks for your time.
Pete: It was fun. Thank you.
About Pete Fishman
Peter Fishman is the Director of Analytics at Yammer, the enterprise social network. He leads a team of analysts and data engineers that seek to optimize Yammer features and present usage statistics to internal and external customers. Prior to joining Yammer, Dr. Fishman worked as a social gaming economist at Playdom/Disney and as a statistician in the front office of the Philadelphia Eagles. He received his B.S. from Duke University and a Ph.D. in economics from UC Berkeley.
About bizi LLC
bizi is a niche resource partner for life sciences manufacturers specializing in commercial planning, market access and HEOR via high end independent consultant support and executive search.