The Great Stagnation: Problems Are Harder, and/or Talent is Misallocated

On Rationally Speaking, Julia Galef interviews Michael Webb about increasing research inefficiency - for example, Webb cites the statistic that today, to get another Moore's-Law-Doubling, it takes twenty times as many researchers as it did in the 1970s. It's not obvious that research is more and more inefficient because it's still producing improvements at the same rate, but only by consuming more and more resources to maintain the same rate. He uses the analogy of mining, where you have to keep going further and further into the ground to get to the gold, or the coal, or whatever it is. The longer the mine is operating (assuming a single central shaft) the bigger this distance gets:

[The pre-work you have to do in order to make a contribution] is a lot further today than it ever was. The amount of knowledge you have to have as a scientist to be able to get to the frontier, to make these contributions, is just so much larger today. And you can see this from the amount of time of it takes to do a PhD, how old an inventor is the time they first take out a patent, the size of research teams. Ben Jones, he's a fantastic economics professor at Kellogg, has papers that document these things.

That means that for individuals, they could either end up spending more time studying, which is what you see in the PhD length, or you see that they just focus on narrower and narrower fields. For a given amount of time, you only learn something about a much, much narrower field. Which might mean that you just have less good insights if it turns out that for all you progress, the fields...The wider field you have to be combining with some knowledge from quite distributed science.

I had previously argued for exactly this idea as an explanation for technological stagnation (or, prior to that, increasing research inefficiency), and with admitted nerve called this ultimate economic heat death "Caton-Schumpeter stasis."

Another factor is the availability of talent, which operates on the assumption that talent is unevenly distributed in the population and is a constraint on technological progress. Consequently there are also the ideas of talent dilution and talent mis-selection.

Talent dilution is the idea that there are only so many Fermis and Oppenheimers, and there is a negative marginal utility to adding more people to the research endeavor. The otherwise productive people are overwhelmed with meetings and emails and swamped by mediocrity. This is actually optimistic, as it suggests that we could return to research productivity by restricting the size of research teams. That this is not already happening suggests that either this idea is wrong, or that people putting the teams together have perverse incentives (quite possible) but, since these are mostly private sector endeavors, somehow overwhelm the profit incentive without unsustainably driving the enterprise into the ground - which seems hard to believe on its face.

Talent mis-selection is a little more subtle. The track to become a physical scientist or semiconductor engineer in the mid-20th century was not as "artificial" (i.e., externally imposed) and clear as it is now. The cause of your having a career in STEM was likely early achievement in that field, because your primary motivation is to explore things in STEM, not to make money or move up in a hierarchy.* Getting good test scores, being a well-behaved student, and knowing how to game your applications is probably much more important now than it was then, and may not be sorting for the actual most productive talent. On top of this, the world today is just a lot more interesting, with a lot more (easy!) options, for someone who's good at quantitative thinking, and the best may not be going into research - they're going to Wall Street or heading to startups. (There are pretty solid statistics that med school applications drop when the economy is good and vice versa - I'd wager that the correlation is even more true for physical science and engineering graduate programs.) By selecting for the type of person who focuses for their first quarter century of life on collecting prestige coupons, climbing hierarchies and gaming applications, you are very likely selecting against exactly those people who will be most productive in STEM, i.e. the kind of person who is directly motivated and rewarded by work in STEM. (For a great discussion about the gap in social cognition or lack thereof between STEMmy and other types of people, see this Slate Star Codex post.)

To put a finer point on the idea of talent mis-selection, let's look at another domain of achievement. Imagine a national program claiming to identify "the nation's top talent in military conquest", complete with an entrance exam and rigorous interviews. You need a reference from a military historian. Those not wearing a tie to their interview are shown the door for their disrespectful and noncomformist behavior. How likely would it be to find the next Genghis Khan or Hannibal this way? The most interesting thing to do for a real potential conqueror would be to go wherever there is active conflict, and the "successful applicants" would likely be annihilated in a real war by the person who went to Syria and became a warlord.

[Update: you may be aware of the Thiel fellowship where students are paid to drop out of college and pursue a business. Business Insider has been following up on how its Fellows have been doing. The reporting certainly shows survivor bias since I don't see a clear "out of Y fellowships awarded, X are currently successful outside education" - and a lot of these students would have been successful anyway so we don't know the denominator. Still, I suspect the fellowship as an intervention is increasing the rate. Still: what gets measured gets addressed, which is why every metric ends up getting gamed, and looking entrepreneurial is no exception, so people are no doubt trying to game the fellowship, and we're back to the mis-selection problem again: "Entrepreneurship has become a line you put on your resume," Thiel says, apparently non-ironically - and to paraphrase Thiel's complaint, in the businesses his fellows are founding, he's getting lots of Facebooks but few flying cars (one exception in this list here.)

[Added later: here's great article summarizing a paper, which simulated how the funding and promotion incentives of scientists are degrading the average quality of work, and unsurprisingly a reproducibility crisis in multiple fields. This would be true even with good talent allocation - because the entire system is selecting for publishable but not necessarily true findings. You might call this third problem talent distraction.]

[Historical example: in discussing the erosion of China's technological and administrative lead over Europe during the second millennium CE, Brad DeLong offers the following as one among many causes:

Perhaps the root problem was that with triple-cropping rice strains the wet-rice fields were too fertile, the governmental bureaucracy too effective, and the avenues of establishment-oriented upward mobility to the striving and aggressive too open. After making a little money the logical next step was to buy some land. Because the land was rich, because labor was plentiful and cheap, and because the empire was (most of the time) strong internally, one could live well after turning one's wealth into land. One could also easily make the important social contacts to pave the way for one's children to advance further. And one's children could do the most important thing needed for upward mobility: study the Confucian classics and do well on the examinations: first the local shengyan, then the regional juren, and then the national jinshi. Those who had successfully written their eight-legged essays and made proper allusions to and use of the Confucian classics would then join the landlord-scholar-bureaucrat aristocracy that ruled China and profited from the empire. In the process of preparing for the examinations and mastering the material needed to do well on them, they would acquire the habits of thought and values of a Confucian aristocrat landlord-scholar-bureaucrat. Entrepreneurial drive and talent was thus molded into an orthodox Confucian-aristocratic pattern and harnessed to the service of the regime and of the landlord class: good for the rents of the landlords, good for the stability of the government, but possibly very bad indeed for the long-run development of technology and organization.

This is a nightmare, real-world example of talent distraction and would also produce talent mis-selection, and Delong's thesis merits further study.]

*I'm all for scientists getting paid. A statistician once pointed out to me that if statistician jobs were suddenly paying 10x more, you might not get the best statisticians - you would get the people best at obtaining stable large paychecks signed by someone else, and some of them will hopefully be good statisticians.