A few years ago, artificial intelligence struggled to write coherent sentences. Now, it has surpassed human performance on a growing number of academic and professional tasks.
Ever since ChatGPT came out in 2022, AI has made huge progress and is improving at an astonishing rate. According to a Sky News Report, leading models have continued to double in performance every seven months.
Despite many researchers’ optimism with the development of AI, one underlying problem remains unresolved: AI is absorbing data at a faster rate than it can be generated. This is currently considered one of the largest obstacles to continued AI development.
The importance of data in AI development
Most of the time, AI training data consists of raw information, such as text, images and audio recordings. Sometimes, raw data is converted into a mathematical format that is easier for models to process. Data is a core part of an AI model’s training process, and without high quality data, AI could have never reached such an advanced stage.
A simple example of this process is demonstrated by a perceptron, one of the basic building blocks of AI. A perceptron takes in data and tries to separate it into two categories, such as “dog” or “not dog,” although it cannot classify multiple categories like “dog,” “cat” or “mouse.” It separates data by drawing a boundary line between groups of data so that the two categories are on different sides of the line, then repeatedly adjusts the line to minimize error.
Large language models (LLMs) like ChatGPT rely on a similar process, except with more complex, nonlinear mathematical models and billions of parameters.
AI does not inherently know anything; its capabilities are highly dependent on the training data given. Its responses aren’t formed through reasoning but are predictions made through mathematical models, which in turn are optimized through data.
Why high quality data is running out
With such rapid technological development, AI has been using up all kinds of data — textual, visual and physical — much faster than it can be produced. Although available data doubles every 3-4 years, AI models’ data demands are increasing by approximately four times per year.
Increasingly, models have started to train on overlapping data, which limits its capabilities. When duplicate data is used to train AI models, they don’t take in any new information, leading to smaller gains in performance. In some cases, it can even reduce the accuracy of models’ predictions.
Despite the public internet being a huge source of data, only a fraction of it is suitable for training AI models. It is estimated that the internet contains around 180 billion terabytes of data but only 100,000 are suitable for training.
Suitable, high-quality data needs to be accurate, complete and relevant, or else it creates inconsistencies in AI performance. Text that contains many misspellings and grammatical mistakes, for instance, serves as noise in an LLM’s training data since it doesn’t help the model learn proper grammar in writing.
Another limiting factor is legal restrictions, as shown in recent years with court cases involving OpenAI and Anthropic.
In 2025, a group of authors sued Anthropic for copyright infringement. The accusation was that the company was training its AI model, Claude, without their permission, leading to the case Bartz v. Anthropic. The Supreme Court ruled that training on legally obtained books is allowed, although Anthropic’s data contained millions of pirated books. In the end, Anthropic agreed to a $1.5 billion settlement.
Books, research papers and newspaper articles are often protected by copyright, and using them as training data raises questions about intellectual property rights. Although this debate is still ongoing and some companies use protected data regardless, it still greatly restricts the amount of high-quality data available to AI developers.
For AI robotics, the data shortage challenge is even more impactful.
Not only is high-quality data limited, the complexity of the physical world poses unusual challenges as well. Unlike LLMs, which only process digital data formats, AI robots must perceive and respond to their surroundings in real time, similar to how humans use their five senses.
This means their training data requires spatial, auditory and tactile aspects, which are significantly more difficult to obtain than just scraping text off the internet. This type of data must be actively gathered through multiple cameras, microphones, depth sensors and force sensors.
Some organizations, like NASA, are trying to train robots using 3D virtual simulations. To do this, a version of a robot operates in a computer-generated environment, where it repeats tasks many times and generates large amounts of training data with great efficiency.
However, robots often have trouble adapting to the real world after training in simulations because simulations are unable to account for physical factors such as friction, air resistance and heat deformation. Essentially, AI robots trained through virtual reality are learning from an oversimplified data set; it doesn’t sufficiently prepare them for real world settings.
Researchers agree that there’s no way around all these issues — AI models need unique, real-world data to train on in order to continue improving.
Experts say that language models will have fully consumed their data between this year and 2032 if they continue training at their current pace. If AI runs out of data, the growth of AI models will be significantly hindered and eventually plateau.
Models might even deteriorate when training on AI-generated data due to a phenomenon called model collapse. This happens because AI generates content based on mathematical predictions, which does not fully reflect a real-world scenario. AI-generated data is likely to contain flaws, and when other AI models are trained on these inaccuracies, they often aggregate and eventually lead to a high level of error.
How companies are responding
Now that data is in huge demand, many businesses are working aggressively to collect huge quantities of it and employ data-efficient training processes.
Yangming Huang, a former Apple engineer (and the uncle of the writer of this story), founded his own data collection company, coScene, in 2022. It focuses on gathering physical data for procedures such as grabbing objects and navigating real-world environments. This kind of input is crucial for both AI robotics and world models — AI systems that simulate the physical world.
“AI does not understand the physical world,” Huang said. “That’s why models sometimes generate videos with weird behavior, like a hand with six fingers or a motorcycle that goes up into the sky.”
While humans are able to understand the physical world through interacting with it, AI doesn’t get the same experience when training on images. This is why world models — AI models trained to understand physical laws and spatial relationships — are crucial for improving the accuracy of image and video generation.
When collecting data, Huang’s team uses teleoperation, a way to remotely control robots to complete tasks like folding clothes. During the process, the robots’ various sensors gather information about their surroundings. Then, the team stores and processes the data so that it’s usable for training AI models.
Although the task seems simple, Huang emphasized collecting physical data comes with many obstacles.
“You can’t collect data repeatedly of the same scenario,” Huang said, “The data must be diversified for different kinds of robots that focus on different skills.”
Since robots vary widely in capabilities and design, the same dataset cannot be reused across all systems. As a result, Huang’s team must gather large volumes of varied, high-quality data tailored to each type of machine. This need for diversity makes physical data collection both time-consuming and resource intensive.
Meanwhile, others are working to improve AI’s training efficiency. According to Huang, there are many ways efficiency can be achieved, such as through online learning.
In online learning, an AI model keeps updating its dataset based on new information. Instead of starting from scratch every time new data arrives, the model makes small adjustments over time. This way, AI systems adapt faster and do not need long, intensive retraining sessions.
Another useful method for optimizing the AI model is zero-shot learning, where an AI model performs a task that it has never been explicitly trained on. For example, zero-shot learning would teach an AI model that a zebra is a horse-like animal with black and white stripes, while traditional machine learning would feed it thousands of zebra images.
By using generalized patterns, concepts and language relationships, AI models are able to apply existing knowledge and perform well in unfamiliar situations.
“Physical data is really limited, and we need maybe 10,000 times more than what’s currently existing,” Huang said. “This is why world models are becoming the new trend in AI development.”
