So this is a question for those of you in the comments.

I'm finishing up a Ph.D. in engineering (heavy into climate change research, so tons of programming + mathematical + statistical knowledge in addition to combing through TBs of data with R and other languages).

What kinds of problems are frequently present in the data science industry that differs from academic research?

I realize this isn't a proper answer to your question, but it reminds of a tweet from Monica Rogati:

  "A decade in academia taught me a bunch of sophisticated algorithms; a decade in industry taught me when not to use them."

Source: https://twitter.com/mrogati/status/726115691703619584

Welp that's really fair. Thanks for the quote.

Thing I've noticed the most is that many people don't fully understand why we need to apply a specific test.

The core concept many people seem to miss is that the point of data science is to find meaning in large quantities of data, to recognize patterns, and to present them in a meaningful and easy-to-understand way. Really to allow for educated data-driven decision making. Each approach is a tool for you to make an informed idea, but if you apply them the wrong way then... well, could be worrisome down the road.

Understanding the full problem and then finding the right tools or approaches to solve it is necessary instead of putting everything inside a black-box model.