A Union Square Ventures post from last fall called out the fact that throughout the course of technology history, apps beget the infrastructure that supports them. The concept of an “app” is loosely defined as something that directly touches the end-user (examples included light bulbs, planes, email, etc.) In the case of web applications, this analogy is relatively simple and accurate. But, in the infrastructure phase of machine learning, the analogy doesn’t exactly hold together -- in between an end-user “app” and the infrastructure are the necessary steps of running a data science model on the infrastructure for training and inference.
The problem with the “apps beget infrastructure” argument for machine learning is that many of the applications of neural networking are inherently limited by the constraints of modern infrastructure. In other words, the apps cannot exist in the real world (outside the theoretical) without the infrastructure first.
The history of deep learning spans about a half century (h/t to Andrey Kurenkov for this excellent series on the topic). Neural networks were considered largely theoretical math (mostly a progression of incremental algorithm improvements) until the Stanford University paper “Large-Scale Deep Unsupervised Learning Using Graphics Processors.” Using graphics processors (GPUs), researchers proved that machine learning algorithms could be processed 70x faster than on general purpose CPUs.
This infrastructure discovery unleashed an era of experimentation, led by large companies with deep learning research divisions, including Microsoft and Google. In this case, the infrastructure very much begat the apps, and proved that complex deep learning computations were possible (and accurate) for breakout applications including speech and image recognition.
Hitting an Infrastructure Wall
Today, many data scientists are hitting another infrastructure wall when it comes to the types of deep learning applications they can develop (and the speed at which they can develop them). Complex use cases such as medical imaging require that extremely large, 3D datasets get processed with an exceptionally high degree of precision to be useful. Using a GPU, TPU or similar hardware isn’t always an available option, due to these machines’ high cost, relative scarcity, and inability to process large datasets efficiently. (If you’ve ever had to wait in line to use a GPU at a research lab, you’d get it.)
In this scenario, much of a data scientist’s time is spent manipulating data to get it to fit inside the GPU memory, since otherwise the performance gain is lost. Down- and up-sampling, reducing kernel size, compressing information, removing layers and reducing the number of convolutions are all tricks to get a model and dataset into a GPU. Many times, if these tricks don’t work -- or the model takes too long to train -- a data scientist won’t even bother trying. In this very specific scenario, the “app” is left on the drawing board because of the infrastructure’s limitations.
The Infrastructure Phase of Machine Learning
Therefore, the next leap in deep learning infrastructure is required to beget a new set of killer apps. To solve new problems and to ask bigger questions, we need that next great infrastructure unlock. Current machine learning is severely limited for applications such as deploying AI on the edge (on devices like self-driving cars or robots built to assist the elderly), or processing deep learning models for vast amounts of high resolution videos and images. People want to create these types of “applications,” but there’s a significant infrastructure gap they need to overcome to get there.
This point in particular from the USV piece was interesting, as viewed through the lens of a machine learning researcher, “if we think about an ‘infrastructure phase’ divorced from the apps that will use it, we run the risk of building too far ahead, in a speculative vacuum.” AI and machine learning was born out of academic research -- a process that was built to create and test assumptions. The assumption comes first, testing second.
Academia and corporate research labs are arguably giant, speculative vacuums that have produced some of the most interesting innovations of our time. We ultimately just need the right infrastructure to help prove that our crazy ideas are possible.