Data Analyst vs Data Scientist: Industry Perspectives

in

Both "Data Analyst" (DA) and "Data Scientist" (DS) are titles that vary greatly between industries and even amongst individual organizations within industries. As the roles behind titles change over time, it is natural for some teams to ask themselves the following questions: should we have distinct roles or just stick to one? How would we differentiate the roles in a way that fulfills our organization's needs and is generally consistent with similar organizations? Do we want to consider a DS to be equivalent to a Sr. DA, the only difference being the title? Answering these questions not only establishes clear responsibilities and expectations, but enables hiring managers and recruiters to communicate clearly with potential applicants in the future (in job postings, for example).

"data scientist vs data analyst" search results

"data scientist vs data analyst" search results

Search the Internet for "data scientist vs data analyst" and you will find plenty of people who don't know what the difference is (nor if there even is one anymore), and you will find plenty of people who think they know the definitions and differences. You will find an abundance of opinions but very little consistency! When I asked my followers on social media what they personally think the differences are, not everyone shared the same opinion but some interesting camps of thought emerged.

This is my effort to summarize the many replies I received, so here are certain important points, recurring themes, and somewhat overlapping camps of thought:

  • Single/primary distinction: DS is a DA who can code
    • In summary: the kinds of questions that a DA can answer and the kinds of tasks a DA can work on are a subset of DS’s because GUI tools limit what can be done, but a DS – by knowing programming – can answer way more kinds of questions and work on way more kinds of tasks.
    • Leads to reproducibility1, scalability
    • See discussion with Hadley Wickham
  • Single/primary distinction: statistical and machine learning (ML) modeling
  • No DAs, just two types of DSs: “Type A” vs “Type B” (refer to Doing Data Science at Twitter) came up a few times
  • Emily Robinson brought up that “Data Scientist” is now also used as an umbrella term and specialties are specified in the title as needed4
  • Some big tech companies like Facebook, Spotify, and some departments within Apple are moving away from having DAs to just having DSs5
  • Practical considerations for NY/SF/Austin tech scene:
    • “DS title will need a higher salary.”
    • “You will lose talent because of the DA title. It is seen as less prestigious.”
    • “You may have to work harder for diverse pool of applicants w/ DS title.”
      • “That latter comes from one company I know who’s had a harder time getting female applicants for DS positions vs DA (when they’re fairly similar responsibilities)”6
  • Lucas Meyer voiced support for a classic: Drew Conway’s infamous Venn diagram7
  • A coworker of mine shared that at one of his previous employments his organization identified three data scientist personas/profiles:
    • DS, Operations provides data & insights for resourcing decisions through ad-hoc analyses, dashboards, defining KPIs, and A/B testing.
      • This is the role of a Data Scientist in Product who creates reports and dashboards for management and executives. - MP
    • DS, Product delivers data science as product (and not to be confused with Data Scientists in Product). These folks build predictive models, AIs, matchmaking systems.
      • In some orgs this might be an ML Engineer or an AI Engineer or just a Data Scientist? - MP
    • DS, Research experiments and innovates. Not everything they work on ends up in production or utilized, but they are free to be creative and take chances.
      • In some orgs this might be the Research Scientist? - MP
    • Thinking of it this way, you might envision a scenario/pipeline wherein a Research DS prototypes a new recommender system (RS) algorithm, then an Operations DS helps determine (through A/B testing and qualitative user research together with a Design/UX Researcher) whether it's worth the costs to productionize (perhaps with the input of a Business/Financial Analyst), and then a Product DS scales the RS (possibly in collaboration with a Data Engineer) and deploys it to production. - MP

Closing thoughts

I hope for some that this is an eye-opening moment and that they now realize that there's no single distinction everyone agrees on. Everyone is coming into it with their own backgrounds, experiences, thought processes, and ideas. None of these are wrong! So if you're in a hiring position, please remember to be specific when writing a job description. You can't just write "Data Analyst" or "Data Scientist" at the top and expect everyone else to share your assumptions, it's a recipe for misunderstanding and failure.

I would like to thank everyone who responded, and especially Emily Robinson and Renee M. P. Teate. Thank you everybody for taking the time to write and in some cases discuss nuances in spun-off threads! If you want to explore all the replies yourself, here's root.

I would also like to point out that is not even representative of how data professionals perceive these roles globally. All of the responses were from English-literate people, most (if not all) of the responses were from people living and working in U.S., and many of them are specifically people who follow me on Twitter. I know for a fact that there are so many more data professionals (data engineers have opinions on this too!) who aren't in any of those groups. These are professionals who have their own perceptions, who operate in different cultures, under different expectations all across the world, and someone out there is probably writing a similar post within their own community.

Resources for learning to visualize data with R/ggplot2

in ,

"I'm currently learning visualisation with R/ggplot2 and was wondering whether you could share tips/links/videos/books/resources that helped you in your journey"

— Raya via Twitter

Tips

The only tip I'll give is that you should strive to make every chart look exactly how you want it to look and say exactly what you want it to say. You will learn in the process of doing. When it's time to visualize the data and you have an idea for a very specific look and story, don't give it up or compromise on your vision just because you don't know how to do it. Trust me, there is so much documentation out there and so many posts on Stack Overflow that you will be able to figure it out. (But also it's totally fine to get 90-95% of the way there and call it done if that last 5-10% sprint is driving you bonkers.)

Here are some charts I made in the past two and a half years at The Wikimedia Foundation:

In each case I distinctly remember wanting to express an idea in a specific way and had to learn something new or learn how to think about a problem in a new way. If you click on one of those, you will see a link to its source code on GitHub that you are welcome to look at and learn from.

Resources

I would recommend checking out the following:

…and many more! There are a lot of folks sharing their knowledge and techniques out there, and I cannot stress enough how many of them you can find out by going through Mara's exhaustive catalogue of cool things made by people in R.

Also, it just dawned on me that I can totally make this a series. I've previously written blog posts as responses to the following questions:

So I've gone back and added the new "Ask Popov" tag to those. Hopefully there will be more questions because these advice posts feel good to write!

The journey so far…

in

I recently received an email which said, "I'm interested in learning more about you and your journey to where you are today," so I thought I'd describe how I went from studying visual arts to analyzing data at Wikimedia Foundation (WMF).

I entered California State University - Fullerton (CSUF) with the intention to do a double major in visual arts (concentration in illustration) and mathematics (concentration in pure math). I've been doing art for years at that point (here is some of my stuff), so I wanted to become a professional. I also wanted to study math because I really enjoyed AP Calculus BC, but I specifically wanted to study pure mathematics because I wanted to go into cryptanalysis after reading The Code Book: The Science of Secrecy from Ancient Egypt to Quantum Cryptography.

Since I was putting quite a lot of hours into my job at a grocery store to pay for school, I realized after my first semester that I just didn't have time or energy to pursue both of my passions. So I had a decision to make, and I chose math over art. That was a major turning point in my life, and I still second-guess myself about it every now and then (despite a successful career in data science).

I had several amazing math teachers in high school, and one of them introduced us to rings, groups, and fields from abstract algebra in a two-week detour during Honors Algebra 2, and I really liked that stuff! Even though I hated probability problems all throughout high school, a year into my undergraduate education, something in my head flipped. I found a calling in another branch of mathematics — a more applied branch based on probability — and I switched my concentration from Pure Mathematics to Statistics.

Dr. Sam Behseta took me under his wing as my advisor and invited me to get involved in an undergraduate research project dealing with application of statistics to neuroscience, along with two other students. We performed an assessment of several probability-distance measures such as Kullback–Leibler divergence for comparing neuron firing counts from peristimulus time histograms (PSTHs). I went on to do another undergraduate research project with him, this time using model-based clustering to find common patterns in PSTHs.

I wasn't sure about going on to do a PhD in Statistics, but I wanted to learn more. I applied as a Statistician to Valve Software, but was rejected after the phone screen. I needed to learn more before I could get a job in the industry. I applied to Carnegie Mellon University's Master in Statistical Practice (MSP) program run by Drs. Joel Greenhouse and Howard Seltman and got extremely lucky because I was actually admitted. I packed up my things and moved across the country to Pittsburgh, PA, where I learned data mining, time series analysis, survival analysis, and even got to contribute my skills to Carnegie Mellon University Department of Statistics Census Research Node (one of the few NSF-Census Research Network nodes working with U.S. Census Bureau in preparation for the 2020 census).

The focus of the MSP program was statistical consulting, and our final project involved actual consulting for actual clients. In my case, my partner and I performed statistical analysis of fMRI data for Dr. Mina Cikara. At the same time, Dr. James T. Becker at University of Pittsburgh (in partnership with CMU via CNBC) was looking for a jack-of-all-trades for his neuropsychology research program (NRP) at University of Pittsburgh Medical Center (UPMC). I've been doing neuroscience-related stuff for two years at that point and I liked it, so this opportunity was a natural fit.

I worked at NRP/UPMC for the next two years, analyzing MRI scans for studies of Alzheimer's disease and HIV-associated neurocognitive disorders. I also organized the past twenty years of data, performed ad-hoc analyses, and developed a novel method of calculating associations between MRI data and Bayesian posterior parameter estimates. But this was a soft-money position (I was hired on a grant) and I couldn't stay with them longer than those two years, so I started looking for other opportunities.

In the months leading up to that, I had serendipidously become friends with Oliver Keyes — the sole data analyst in the Discovery department at WMF at the time — on Twitter. We connected through our mutual love for R and social justice, and when they found out that I was looking for a job, they suggested I apply to the opening for Discovery's second data analyst because my statistics-focused skillset was a complement to their skillset. Of course I applied to work at Wikipedia! After going through the interview process (which was the basis for the one I would later write about), I was offered the job and Oliver and I became a team. This was two and a half years ago.

I don't know what's next in store for me. Two years in, and I'm still very happy at WMF and I get to perform lots of really cool analyses. Heck, I'm even supported in learning (and encouraged to learn) new things that aren't directly related to my job description.

Cheers~

Advice for graduates applying for data science jobs

in

Getting into a technical field like data science is really difficult when you're fresh out of school. On the off-chance that your potential employer actually gets the hiring process right, most organizations are still going to place a considerable amount of weight on experience over schooling. Like, yeah there are certain schools that make it a lot easier to go from academia to industry, but otherwise you're dealing with the classic catch-22 situation.

Something that can help you – and what I would notice when reviewing applications – is having something original and interesting (even if just to you) to show and talk about. It doesn't have to be published original research. It doesn't have to be a thesis. It just has to show that you can:

  • Work with real data: In most academic programs, methods are taught using clean, ready-to-use data. So it's important to show that you can take some data you found somewhere and process into something that you can glean insights from. It also gives you a chance to work with data about a topic that you personally find interesting. Possible sources of data include:
  • Explore it: Once you have a dataset that actually excites you, you should perform some EDA. Produce at least one (thoroughly labeled) visualization that shows some interesting pattern or relationship. I want to see your curiosity. I want to see an understanding that you can't just jump into model-fitting without developing some familiarity with your data first.
  • Analyze it: You're going to lose a lot of interest if you just show and talk about how you followed the steps of some tutorial verbatim. If you learn from the tutorial and then apply that methodology to a different dataset, that's basically what "experience" means. And don't try to use an overly complicated algorithm/model if the goal doesn't require it. You might get incredible accuracy classifying with deep learning, but you'll probably have a more interesting story to tell from inference with a logistic regression. Heck, at Wikimedia we use that in our anti-harassment research.
  • Present your work: It can be a neat report with an executive summary (abstract) or it can be an interactive visualization or a slide deck. Just something better than zip archive of scripts or Jupyter notebooks.
  • Explain your work (however complex) and results in a way that can be understood: This is where the first point is really important. If you're describing your analysis of data from a topic you're familiar with and are interested in, you're going to have a much easier time explaining it to a stranger. Be prepared to talk about it to a non-technical person. Be prepared to talk about it to a technical person who may not be familiar with your particular methodology. Your interviewer may have done a lot of computational lingustics & NLP but no survival analysis, so get ready to give a brief lesson on K-M curves (and vice versa).
  • Perform an analysis from start to finish: Because that's what we look for when we assign a take-home task to our candidates.

A lot of times the job postings will include a number of years as a requirement, but that's not as need-to-have as you or they might think. Secretely, it's actually a nice-to-have because "experience" is mostly a proxy for "candidate has previously used real data to solve a problem in a way that can be understood and used to inform a decision-making process." If you don't have experience, you can still demonstrate that you've done what a data scientist does.

Good luck~

Acknowledgement: I would like to thank Angela Bassa (Director of Data Science at iRobot) for her input on this post. In particular, the last paragraph is based entirely on her suggestions. She also created the Data Helpers website that lists data professionals who are able to answer questions, promote, or mentor newcomers into the field.