Interesting article from Derek Thompson at The Atlantic. The title is "The Forgotten Stage of Human Progress." The basic idea is that we should ask the question: why do we have great technology that we don't deploy? He goes down this thread to ask whether technological progress is enough to have actual progress. (Punchline: no.)
Personally, I find Thompson's question intriguing, but think that his diagnosis is directionally wrong.
He emphasizes the importance of implementation, and thinks that this is where we are falling short. I disagree. I would call what he's talking about "scaling" and think that this is part of the technology. He gives the example of the computer and claims we had all the components in the 50s-70s, which is laughable. We did have semiconductors, but there were, to put it mildly, many innovations that needed to happen for computers to work the way they do today. So I simply don't buy this.
There is a more interesting question: why do we seem to have a number of great innovations that we choose not to use? Or choose not to invest in despite great promise?
This is reminiscent of a common criticism of technological theories of progress. The one that sticks with me most is Michael Mann's critique of Bill Gates' book on climate change. Mann argues that we have the solutions to climate change but just don't use them; what's needed is political change, not technological change. (Though I don't think Mann is totally right, it is an important challenge we need to consider because many forms of improvement aren't implemented for unclear reasons).
Nuclear, which Thompson mentions in his article, is a great example. Why don't we invest more in nuclear? It has the potential to be a great solution that makes lots of people happy, so why haven't we done it? Another powerful example for me (that Thompson doesn't really explore) is the mRNA vaccines. They have the potential to be quickly updated for new strains. Yet the vaccines we use today, including all the boosters, are still wild-type vaccines. mRNA absolutely allows for this on a cheaper scale than traditional vaccines. Why? And with other vaccines, why were we choosing to go backward before COVID-19, resulting in outbreaks of long-forgotten diseases like whooping cough and measles--in children, no less?
This seems like a topic worth exploring. I do think that some of this is news-driven, where people see headlines about technological or scientific progress they don't understand and think that a solution or invention is further along than it really is. Paxlovid is an example, where we have proof something works but manufacturing is highly nontrivial. We might ask "why don't we have a better scale-up infrastructure?" but that isn't exactly evidence of a lack of progress. (I don't have any inside information as to whether the issue is a scale-up problem, a supply chain problem, or even a regulatory problem, though my bet is on the former).
This also made me think about why we refuse to learn. This was triggered by news today in my home state of California, where a powerful legislator wants to spend $10 billion of our (temporary bumper) surplus subsidizing housing. Policy is a form of progress too, and we have been spending the past 2 years understanding how this just inflates costs. Why, despite all the evidence, do they just refuse to allow us to build more housing? The past few years have taught me how many people truly refuse to learn from experience, even at great personal cost, and it is the thing that depresses me most about our prospects for progress.
If you haven't read it already, 'Where's my Flying Car' is basically a direct answer to the question you pose here. Hall has scientific, technological, political, and cultural analyses of several possibly impactful technologies like flying cars, nuclear energy, nanotech, etc.
I know it but it’s still pretty specific to the technologies and products the author is interested in. I’m more interested in a general question of: there are a number of technologies where the implementation barriers seem nontechnical, and it seems like it’s getting worse. Curious why.
He uses specific case studies but the book definitely synthesizes general principles which can explain the nontechnical barriers in many technologies. Things like 'failures of nerve' and imagination and 'the Machiavelli effect' are illustrated with cases but are applicable to a wide range of technologies. Jason's summary is great but I really think you would enjoy the book. It's fun to read, not too long, and I think it is the most comprehensive answer to your question out there right now.
My summary/review: https://rootsofprogress.org/where-is-my-flying-car
For some reason, the media really doesn't want to spread the message "we need to build more housing". One theory is that many of the older journalists own property and don't want more construction in their neighborhoods. This doesn't seem like a very good explanation as then we might predict the younger journalists who don't own property would push to build more.
A second theory is that the media is currently pushing the narrative of rich oppressing the poor and this explanation doesn't fit with this narrative. This seems more likely. Many journalists are struggling financially due to the shift to online, so even if the housing market were fixed it likely wouldn't fix their issues. Hence they are incentivised to push for a more extensive restructuring of society.