I recently wrote a blog post Deep Neural Nets: 33 years ago and 33 years from now that was partly a case study on the nature of progress in deep learning, which people here may find interesting. (I would encourage people to read it briefly and return for a few more progress studies - specific comments below).

What strikes me the most is that progress in deep learning has for decades been upper bounded by the computing infrastructure and the software ecosystem. For example, our ability to automate sight famously made a leap in 2012 AlexNet. The neural network architecture and the training algorithm would have been extremely recognizable to LeCun 1989. So who deserves the credit for this leap? I'm inclined to say that it is the thousands of engineers who made computer chips faster and cheaper. Who made the hard drives bigger. Who developed Google Image Search, which was used to seed the labels. Who built Amazon Mechanical Turk that allowed the ImageNet project to clean the labels. Who developed GPUs and wrote CUDA. And of course Alex, Ilya and Geoff for the "final assembly". In particular, there is no single big eureka moment in this story, only the tens of thousands of smaller eureka moments hidden out of sight.

As a result I am fascinated by the "dark matter" of progress in progress studies - the little incremental improvements (usually by large organizations) that improve on the collective infrastructure and unlock a final assembly of exceptional results. I am also looking for equivalents of this in other fields, e.g. I recall reading an article a while ago that suggested that one of the reasons the Romans would find it hard to industrialize is that material science, precision manufacturing and the associated industry had to advance further to enable the building of all the necessary machines and experimental tools. I am not qualified to tell, but perhaps something along those lines is also "dark matter", hidden behind the more standard narratives of the invention of the steam engine and the like.

Finally, if it were the case that progress often takes this form (does it?), what could be done to best accelerate it? For example, what could be done to make the AlexNet happen 10 years earlier? It feels hard to come up with any one single thing, except for actions that support and encourage an ecosystem of large organizations incrementally improving the collective ...

Posts

Wiki Contributions

Comments