Spark is not too tricky to dive into, even though you can't really take advantage unless you have a big cluster to use :)
if you want to practice data-manipulation, and a lot of the map reduce type stuff you can do with spark, I find Pandas useful for small datasets (And a lot of overlap in functionality as far as Dataframes are concerned)
For pipeline stuff, definitely take a look at Luigi, but again without a cluster it'll be less fun. Still, if you can try automating tasks with a mini luigi scheduler on your localhost, it would be good practice
This is a really interesting initiative! Thanks for sharing! Was there anything surprising you all found? Have you had a lot of feedback from underrepresented groups?
1) Only two companies in the entire ranking have outperformed the American population in terms of diversity. We all know that tech can do better, but this is pretty surprising given that most companies on the list are based out of SF and NYC, two areas are quite diversity relative to the rest of America.
2) Because this is the first time we can compare this amount of standardized diversity data across companies, it's the first time we're really able to see who is doing better than who. It's interesting to see some companies that are often lauded and branded as diversity thought leaders, doing less well than we'd all expect.
3) Some patterns have emerged: NYC based companies tend to do better than SF based companies. Ecommerce companies tend to outperform security companies (though we saw that one coming). More importantly, this data shows how company structure and sector can make it easier or harder to be inclusive in certain ways. That's why we've included sector averages. We hope those numbers help people make more informed analyses about how well a company is performing given their role make up.
We've had a lot of feedback from underrepresented groups that will inform product improvements moving forward--from accessibility to which companies people want to see on the list next. Most of all, we've had affirming feedback that these numbers really do help people make the case for diversity initiatives and kickstart conversations at their companies. We're excited about the prospect of seeing real initiatives come into existence because of DTP data, and that's what we really hope to support and see more of!
This is a nice project, and very interesting to see how demographics differ in different niches within a profession.
I know that hiring for increased diversity in certain professions is very difficult. In particular this is difficult in professions where the education-to-profession pipeline for qualified individuals is "bottlenecked" from earlier points in peoples' lives. Company initiatives may not be nipping the problem in the bud, but rather are counteracting a problem that has already started earlier.
That is, not only are certain groups underrepresented in some profession, but they are also underrepresented in the population of students and self-learners that are interested in that profession. It is indeed a complex problem which may require "uprooting" of trends that penetrate deep in family traditions, ethnic economic niches, or improving social networks from people that have an educational or an income disadvantage.
they are. back in the day they were charged with terrorism, principally. Among sabotage and other crimes. One of the rare examples of terrorism being used outside of either a wartime threat or leftist organization/labor union.
