Data Scientist: Disclosing the Profession of Tomorrow

Despite being very profitable, data analysis is a rather ungrateful career choice to be pursued. Sometimes it requires almost an impossible level of execution from a specialist – scavenging through the vast amounts of data and then coming up with the best solution. Not every scientist can combine wise analytic skills with smart thinking and sharp decision making all at once. As a result, data scientists are forced to have a number of challenges and face them every day. 

An interesting survey has been made by Kaggle where they asked over 16,000 respondents the following question: “At work, which barriers or challenges have you faced this past year?”. Results were the following:

  1. Dirty data (36% reported)
  2. Lack of data science talent (30%)
  3. Company politics (27%)
  4. Lack of clear question (22%)
  5. Data inaccessible (22%)
  6. Results not used by decision makers (18%)
  7. Explaining data science to others (16%)
  8. Privacy issues (14%)
  9. Lack of domain expertise (14%)
  10. Organization small and cannot afford data science team (13%)

Results also have revealed, that the average amount of challenges that one specialist can experience is around three, but drastically varies from position to position. Predictive Modelers and Data Scientists can face up to four challenges, where Programmers usually face only one. 

Baring everything in mind, I decided to look at some of the most common challenges, discuss them and maybe even try to come up with a working solution. 

Dirty Data

In order to get a solid analysis, you will obviously need to gain access to the solid kind of data. Getting this access can prove to be both challenging and time-consuming. Common issues range from the insufficient level of data to the less variety in the kind of data. Data is often can be spread unequally between various types of business, meaning that laying hands on all of it can become very difficult. 

Director of data and analytics at Snowflake, Scott Hoover, says that “the overwhelming majority of effort a typical data scientist puts forth has to do with creating a clean data set with useful information, all before any of the compelling machine learning or statistical models can be applied. This is the part of the job that’s almost considered an art or a craft. Just like any artist or craftsperson, there’s untold effort that largely goes unnoticed when viewing the final product.”

Thoughts: Data Scientists need to expert data management system, as well as other forms of information integration tools. Various data integration programs can also help to connect with external data sources to get their internal insight perspective of the overall workflow. 

Lack of data science talent

Data Scientists should not try to achieve everything at once, but rather narrow their skills and professional focus onto one specific area. At the same time, they need to have a domain knowledge and gain a subject matter expertise. Another challenge pops up when they need to later apply this knowledge to a concrete business solution. Since Data Scientists can be viewed as a metaphorical bridge between top management and IT department, domain expertise is crucial for conveying the needs of management to the IT Department.

Jim Polk, CTO at Elinext states the following: “Data Science is a multi-directional phenomenon in its essence that puts together all kinds of knowledge elaborately knitted. A Data Scientist is an expert who self-develops creative thinking and rely on multiple tools and approaches.”

Thoughts: Data scientists should pay close attention to the business requirements, as well as improve their technical and statistical tools. They should be understanding the problems of business at hand, getting valuable insights and analyzing and modeling a suitable solution. And do not stick to one work model behavior – flexibility of approach is often a key to finding an optimal solution. 

Lack of clear question

This is proved to be one of the major issues, that people in the industry face constantly since in order to properly analyze something and on top of that come up with a working solution, you need to be able to define and understand its every aspect. Data scientists not only should think about “how”, but also “why” things should be done this way – it is not a random “sifting through data to find a connection” type of work. Many are sadly opt for the mechanical approach and are ready to work with all the data sets, without first understanding both business and client requirements. 

Chief scientist at SnapLogic, Greg Benson thinks that: “Data scientists often run into the issue of trying to add artificial intelligence or machine learning capabilities without concrete objectives. This is a waste of time. Start by asking how your customer experience will improve at a high level.”

Tal Kedar also gives a good example on the matter: “If you are building a self-driving machine, you need to know what makes a good driver, and be well-versed in the challenges and outcomes that accompany safe or reckless driving, and then have those reflected in the algorithms that drive the car. This is why I encourage data scientists to always make sure they have a coherent, clear narrative linking the business problem at hand to their choice of algorithms.”

Thoughts: Initial workflow should be first well-defined before you should begin to actually analyze the given data. Start with the identifying the problem first, then design a solution, while building a supportive checklist doc, to track your progress every step of the way. 

Closing thoughts

Today we have learned that Data professionals experience huge challenges in their data science and machine learning career – for most of them, it is three per year. We have learned that a theoretical aspect of the work is only the tip of the iceberg, and you should approach your work with a serious level of pragmatism if you want to see formidable results. You should start form first mastering the basics and only focus on tools, platforms, and areas you want to dive deep, instead of trying to accomplish everything at once. Be ready to face the real world challenges, because they are the ones, that will help you to become more innovative and proactive at your work. 

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Scroll to Top