Professionalization and the Lemon Problem in Data Science
I have recently been reading – and cannot recommend highly enough – Rakesh Khurana’s From Higher Aims to Hired Hands: The Social Transformation of American Business Schools and the Unfulfilled Promise of Management as a Profession. While this may sound like an incredibly abstruse and dry topic, it has turned out to be an incredibly wide-ranging book that covers virtually the entire history of American business and its practitioners within wider American society. Nearly every page or two has an incredibly thought-provoking fact or thought. A few that stuck out to me, ranging from macro to micro:
“The goal of the professionalization project in American management, carried out by the university-based business school, was to achieve control in a specific area – the large, publicly traded corporation – and protect that control from competing groups, namely shareholders, labor, and the state.”
“Raising the esteem enjoyed by business as an occupation thus promised to shore up the status of marginal members of the traditional elite even while providing access to increased occupational status for upwardly mobile managers”
“In 1800, three quarters of the states maintained educational requirements for the practice of law; by 1860, only a quarter of them had any such requirements…in 1800 almost every state had medical licensing was, while by 1860 none of them did.”
The re-emergence of the professions in the Progressive era was in some measure a solution to the Lemon Problem:
“a new foundation for trust had to be created to facilitate the more impersonal, purely transactional relationships that had come to dominate social life.”
It also exposed the weak professionalization of what I do for a living – so-called “Analytics and Data Science”. Khurana identifies a number of markers of a profession: a formal curricula, professional schools, an ethic of “disinterested expertise”, certification and trade associations. These are both structural and purely social, but both are an important parts of the definition.
Analytics and Data Science are clearly a most immature profession. One perennial gripe for both hiring managers and applicants in the data science world is a weak shared conception of what constitutes a “data scientist”. To some people a data scientist builds systems to manage constrained resource optimization at scale; to others a data scientist performs detailed causal analysis on limited observational or experimental data. To be clear to anyone far from this world – those two tasks have nothing to do with each other in either degree or kind.
While there are now schools offering professional training, that hardly seems to solve the issue. There’s an important Lemon Problem in the Analytics/Data Science profession – quality assessment of applicants is extremely difficult along axes of both quality of skills possessed and relevance of skills for the needs. This is the business case for e.g., Kaggle – it provides a disinterested qualification mechanism for applicants to credibly signal skills. Which at the very least should offer some information along the quality axis, though it communicates less about quality of fit. This is why the use of tests has become commonplace for this role, including on my team at ZipRecruiter.
Developing a career track and path to advancement for budding analysts/data scientists will run headlong into this professionalization problem. It’s difficult for hiring managers to assess their skills, hurting labor market liquidity. It’s difficult for outsiders to assess their skills, hurting intra-firm political clout. It’s difficult to define precisely what a “data scientist” does, hurting their incorporation into existing institutions.
The best prognosis for the discipline here is probably what will happen – slow establishment of uniform curricula, formation of professional associations, and some sort of licensing regime. I’ve never been particularly excited about that path, but Khurana has finally convinced me of the value for it.
After all – it’s certainly worked out well for the management profession!