Value Digest by bizi - a monthly newsletter highlighting value, access, reimbursement and regulatory updates impacting biopharma, device and diagnostics firms.

Check out our July issue here.

To receive the newsletter as soon as it is released, sign up here.

One of the first books I’ve read when I started bizi is Jack Shostak’s SAS Programming in the Pharmaceutical Industry.  I had the privilege to get on the phone with Jack on a sunny Saturday morning and got him talking about his book and much more.  

Kelly:     Jack, what are some of the common attributes shared by the top statistical programmers you’ve worked with in the past?

Jack:    The number one attribute is something you don’t necessarily pick up in school, or get from books.  It’s just something you are going to have, and that would be attention to details.  The work we do is varied and dynamic.  You can’t necessarily make people more detailed.  It’s an inherent thing. If somebody has good attention to detail in their work, then all other things can be overcome. They can learn whatever they need to learn.  So, attention to detail is probably the number one thing I would say.  And that’s really hard to pick up on a resume or judge by what classes they’ve taken.  It’s something you find out when you work with them.
 
Kelly:    How about the ability to learn new things then, Jack?  As you alluded to, the nature of the work is dynamic; the industry is constantly changing as well from regulations to new data standards. 
 
Jack:    Well, it’s like with all things, you have to keep learning.  People come to statistical programming historically from all kinds of disciplines.  Maybe they had some SAS in school, took a SAS certification class, or dabbled in analytics.  But things have changed over time.  

For instance, SAS is kind of the Swiss army knife of statistical programming but it’s not the only tool in the arsenal you can use.  R is coming up fast.  So, to stay versatile, statistical programmers may want to learn R.   

Aside from the statistical tools, now really to be successful, statistical programmers need to also be fluent in data standards, they need to understand and develop expertise in CDISC SDTM and CDISC ADaM. 

So things are changing and you always have to keep learning.  You have to watch the horizon, and decide what it is you want to specialize in.       
  
Kelly:    In my working with statistical programming managers, I experienced two schools of thoughts.  One school of thought believes statistical programmers should consciously develop their clinical knowledge and skills so that they can dialogue more effectively with statisticians and clinicians.  The other school of thought deems “clinical knowledge” at best a nice to have for statistical programmers. 

So, how important is clinical knowledge to statistical programmers?
 
Jack:    That’s an excellent question. 

The individual that says statistical programmers do not need to have clinical knowledge probably is coming from a heavily siloed environment.  I am trying to not be negative when I say that.  In that environment, statistical programmers are head down, and generate data sets and TFLs.  That’s what they do and if you spend more than 15 minutes generating that one listing then you’re off task.  The programmers are almost an automaton here.  They’re executing the instructions.  They are executing the specifications.

There is something to be said for that.  However, I don’t really like that because when programmers work in that fashion, they’re disengaged from what’s really going on.  And by that, I mean if statistical programmers don’t have therapeutic area knowledge, then they will not be able to properly comprehend a protocol, read a statistical analysis plan, or understand what the key endpoints are, what the therapeutic intervention is, how does this intervention act on patients, and what are the safety concerns.  In contrast, if programmers do understand those things, then in their work whether they’re deriving variables or generating output, they can look at it and understand better what they’re seeing because they understand clinically what’s going on.  
    
Kelly:    Sort of like understanding the big picture, eh? 

Jack:    Exactly.  And that’s what’s valuable.  I can’t count the number of times that statistical programmers have caught problems in the data, or problems in the analysis.   If a statistical programmer can churn out a listing or table in 10, 15 minutes but he doesn’t really understand what he is churning out, it could be garbage.  

It’s really the lead statistician and the lead statistical programmer that know the study better than anybody else.  And if that lead statistical programmer is just executing, and he doesn’t understand what he is looking at, you’re going to have a problem. 

So, it’s a balance.  Speed is important.  So are attention to details, and understanding clinically what’s going on.  

Kelly:    Another common complaint from statistical programming managers is that statistical programmers’ resumes often look alike.  It is difficult to tell who is a better candidate for the job.  This may have to do with the standard execution part of a statistical programmer’s job.  How do you tackle this issue?
 
Jack:    An excellent question again.  Unfortunately, the solution to that problem is to pick up the phone.  You have to talk to people because, guess what, a lot of the time the resume is not necessarily reflective of what the person knows.  So you can get on the phone fairly easily and assess with a few questions what they actually know and what they don’t.
  
Kelly:    So, what are some of your favorite questions to ask in a phone screen?

Jack:    One blatant thing is whether what the person is telling me they know is reflective of what’s on their resume.  If you seemingly only know two or three of the twelve things on your resume, I already have a trust issue.  It is not going to go very far.  It’s better to be honest on your resume.  Put the things on there that you know. 

Another thing is frankness in conversation, honesty in conversation.  If I feel like somebody is not being honest, if I feel like they are stretching their abilities that is a big turn off.  Eagerness, desire to do the work, positive attitude go a really long way as well.

Kelly:    I see.  How about technical competency?  Do you assess that in a phone screen?  

Jack:    Yes, but I tend to stay higher level.  For example if I’m hiring a SAS Programming position, I’m going to hit the higher level areas.  What have you done?  Do you know specific areas of SAS?  How are you with say ODS Graphics, Macro language, Data Step, SQL?  What have you done with the language?  But it’s not going to get super deep.  Usually a phone screen runs for me about half an hour to an hour. 

Then after that, if I want to bring that person in and have them meet the team, then we will administer a don’t-lie-to-me quiz.  It’s a basic competency quiz.   By bringing somebody in and doing the in-person interview you also get the sense of personal fit with the team.

Kelly:    Talking about fit with the team, statistical programmers can be quiet by nature.  Many prefer to focus on the technical issues at hand and minimize interactions with others.  However, if you do want to be a valuable contributor to the team, the ability to work and communicate effectively with others such as teammates, statisticians, clinicians, data managers, and regulatory folks becomes increasingly critical.  Do you have some advice for the quieter type of programmers on how to communicate better?  
    
Jack:    Yes, that’s an excellent question as well.  Many people go into mathematics and statistics are—how do we put this—not the most outgoing, gregarious, communicative people.  They can be fairly quiet, heads down working. Unfortunately, that can be career limiting.  What we see happening is the software and the tools that we use are getting better and better.  So the ability to sling code isn’t going to be as valuable as the ability to speak to a clinician and a statistician and wield software to produce what you need to do.

So there’s going to be a shift from being technically brilliant to being interpersonally brilliant.  The need to interact, the need to be able to take requirements from somebody who’s not technology savvy and make software do what it needs to be done is going to become more and more important. 

And along with that, the ability to write is critical as well.  We live in a world of twitter.  People are forgetting how to write.  It is dangerous because if you can’t express your thoughts coherently to somebody, it doesn’t matter how technically brilliant you are.  You can’t get your message through.

Kelly:    Last but not least, Jack, I want to talk about your book, SAS Programming in the Pharmaceutical Industry.  This is the first and one of the best books I’ve read on the topic.  I know the updated edition will be out in March 2014.  So what can people expect from this book and the March release? 
 
Jack:    The book is designed for beginner to intermediate SAS programmers in the pharmaceutical industry.  It defines the context for the work and it gives you background on the industry including all the people you will work with in the industry.  Then it takes you through the types of programming that you will need to do to be successful. So, pulling data, massaging it, building your analysis data set, generating tables, figures, listings and all the challenges involved in doing that.  It ties into regulatory issues as well.  Things you need to know for regulatory submissions.

But it doesn’t go super deep into the statistics. If you want to do that, there is another book I co-authored with Glenn Walker that covers the statistics and that is Common Statistical Methods for Clinical Research with SAS Examples, Third Edition.  There’s another book that I did with Chris Holland on implementing CDISC that came out last year, and that is called Implementing CDISC Using SAS: An End-to-End Guide.

I wrote the first edition of SAS Programming in the Pharmaceutical Industry over 10-years ago.  So it needed some updating.  The upcoming March release has undergone a complete overhaul to make it more current with SAS and industry standards.  For instance, every example in the book now is based on CDISC data models, so it uses SDTM and ADaM data slices.  All of the old SAS GRAPH technology has been retired in the book, and it’s all based on ODS graphics now.  Then, some updating on the coding information, but the framework of the book is similar to the way it was. 

Kelly:    That’s a great overview, Jack.  Good luck with the new release and thank you for taking the time today to do the interview. 

Jack:    Thank you very much, Kelly. I really appreciate your time.

About Jack Shostak
Jack Shostak, Associate Director of Statistics, manages a group of statistical programmers at the Duke Clinical Research Institute. A SAS® user since 1985, he is the author of SAS® Programming in the Pharmaceutical Industry, and coauthor of Common Statistical Methods for Clinical Research with SAS® Examples, Third Edition, as well as Implementing CDISC Using SAS®: An End-to-End Guide. Shostak has published papers for the Pharmaceutical SAS Users Group (PharmaSUG) and the NorthEast SAS Users Group (NESUG), and he contributed a chapter, "Reporting and SAS Tool Selection," in the book Reporting from the Field. He is active in the Clinical Data Interchange Standards Consortium (CDISC) community, contributing to the development of Analysis Data Model (ADaM), and he serves as an ADaM trainer for CDISC. Shostak received an MBA from James Madison University and a BS in Statistics from Virginia Polytechnic Institute and State University.

About bizi LLC
bizi is a boutique consultancy offering consultant support for clinical statistics and outcomes research projects for life sciences and healthcare companies.

I met Pete at a big data analytics conference last year.  And he was one of my favorite presenters there.  Aside from leading an analytics team at Yammer, Pete runs a bacon hot sauce business on the side.   I recently got him to open up on his favorite subject: what makes an analyst successful?

Kelly:    Pete, what abilities make an analyst successful?

Pete:    There are clearly technical skills and communication skills.  We often think quite a lot about the technical side, and try to assess whether or not someone has those technical chops; but, equally important, is the ability to communicate your results.  Communication might mean translate a question from something ambiguous into a direct mathematical representation.  Or it might mean explain the results in a way that are consumable and actionable.  So it’s not just that you are able to run some complicated piece of analysis over a giant data set.  It is also about being able to understand what your coefficients mean or what your outcomes mean and then be able to explain those.

Kelly:    Interesting.  I think you’ve touched upon couple of key points.  One is a background in statistics.  To understand what you are doing is not the same as simply executing a statistical procedure.  You have to really understand it to communicate it effectively.  Two, is the ability to translate a very complex problem into a set of hypothesis that can be tested.  Can you elaborate on this a bit more?

Pete:    Sure. So, as an analyst, the keeper of a company’s data, you’re often asked a lot of questions.  These are often narrowly defined ad-hoc requests.  A very junior approach to that is to go ahead and answer it.  A great analyst, however, is not just an answerer.  A great analyst will try to figure out what the customer is trying to get at with that data and is that the best data to answer the question?    

To do that, it is very important to understand what actions will be taken as result of the answer.  Your job is actually to set up the horse race, the competing hypotheses with different actions associated with them, to get to the result.  So, I think a more advanced skill as an analyst is to be able to actually understand what the customer is asking and not just be an answerer.  That’s very, very important.

Also, there’s a clichéd phrase among statisticians that correlation is not causation.  Typically when we write down a model, we’re looking at things that are co-varying together. The right side is the independent variable and the left side is your outcome or dependent variable of interest.

And, what’s assumed is that the relationship always goes from the right side variable causing the left side variable.  A great analyst is one who doesn’t get duped into the easy trap of correlation versus causation and will try to understand to what extent is this question like an experiment and to what extent it is not.      

Kelly:    So, it really comes down to habits of mind.  You are looking for certain habits of mind.  How do you assess that, in an interview?  Do you have a favorite question or type of questions you like to ask?

Pete:    We typically pick up people with great training in statistics or the scientific method and have built up years of expertise and experience in research.  I don’t care how many techniques you know for testing the difference between two samples. What I care about is that your training has subconsciously forced you to think like a scientist, forced you to think about things, again, like an experiment.  What’s valuable is not your explicit knowledge but your mindset your training has built into you.

So, I typically like to ask open ended questions around an imaginary data set and see whether or not the person is able to identify the hidden assumptions within that data set and their implications.

Kelly:    Do you have a more concrete example?  

Pete:    So, one of my favorite questions is taken from the OkCupid blog, I don’t know if you’re familiar with the OkCupid blog?  It is a dating website.

Kelly:    Not really, but I’m going to check it out after this.

Pete:    Fantastic. And what I typically ask somebody to do is critique one of these blog posts.  Look at what the author is implying based on the data and what sort of assertions he’s making or advice that he’s giving to the customer, in this case, the users of the site, and what might be wrong about that implication.

For instance, one of the questions I like to ask is, “How long should a message be on OkCupid and imagine that you had a full complete data set of all the messages?”  And it would be wrong just to explicitly look at what message length has been the most successful because the people wrote those messages weren’t experimentally put into those groups.  Often people confuse data generated in a clean way through experiment with data generated with some other purpose in mind.

Kelly:    So, it is a “trick” question.

Pete:    Yes.

Kelly:    Pete, you mentioned something interesting last time we spoke.  You said to develop a top analyst, you have to allow people to fail. Can you elaborate on that a little bit more?

Pete:    So, we hire a lot of people with a lot of good adjacent experience, but not necessarily direct experience, for instance, people directly out of advanced degree program that have not had much or any work experience.  

To transform this raw talent to someone who is an independent contributor, you have to put them in positions to both succeed and fail. Micromanagement in analytics doesn’t work.  You want people to independently solve problems.  It might take them twice as long, ten times as long, or even longer initially, but what you want is for that speed to converge on your general analyst population. And for junior analyst, you want to shorten your feedback cycle as well.

Kelly:    Fantastic.  We covered a lot of grounds today, Pete.  Thanks for your time.

Pete:    It was fun.  Thank you.

About Pete Fishman
Peter Fishman is the Director of Analytics at Yammer, the enterprise social network. He leads a team of analysts and data engineers that seek to optimize Yammer features and present usage statistics to internal and external customers. Prior to joining Yammer, Dr. Fishman worked as a social gaming economist at Playdom/Disney and as a statistician in the front office of the Philadelphia Eagles. He received his B.S. from Duke University and a Ph.D. in economics from UC Berkeley.

About bizi LLC
bizi is a NJ-based BI consulting firm offering analytics professionals to help companies transform data into information, information into insight.