Portability

"The Grand Illusion: The Myth of Software Portability and Implications for ML Progress* explores the challenges and implications of specialized hardware in machine learning. Co-authored with Fraser Mince, Dzung Dinh, Jonas Kgomo, and Sara Hooker, this paper investigates how the shift toward specialized chips (GPUs, TPUs) may impact the future of AI innovation and software portability."

The Motivation Behind The Grand Illusion

"The motivation for the paper came out of work that Sara Hooker and I had published independently with the Association for Computing Machinery (ACM). In both cases, we discussed the trend towards specialized computing.

Historically, central processing units (CPUs) were designed to be very general and do lots of different things. Increasingly we have seen a move toward specializing chips by creating graphics processing units (GPUs) or tensor processing units (TPUs), which are tailored to be good at certain tasks. In essence, there is a trend of picking and choosing where to enhance performance and where to give it up.

The consequence of moving to specialized chips means that we have less flexibility. For example, GPUs allow us to do many more calculations in parallel, but at the cost of it being harder to manage very unpredictable workloads. These imposed limitations can impact our ability to innovate and explore uncharted areas downstream. This can be a problem. With emerging technologies, we don’t necessarily know where innovation will come from or how we may end up wanting to use the technology. For example, when personal computers were invented, who would have realized that the spreadsheet was going to be the killer app.

One of my colleagues here at MIT, Joel Emer, likes to call them computational black swans."

The Role of the Team

"The paper then sprung from these concerns on the impact of specialized chips and software programs and whether it hurts our ability to explore different use cases and innovation down the line. We pulled together a team of experts from the community to work on it, including Fraser Mince, Dzung Dinh, and Jonas Kgomo."