Big Tech’s Quest for More Robust AI Agents: The Role of Reinforcement Learning Environments

For years, executives from major tech companies have envisioned autonomous AI agents capable of executing tasks using various software applications. However, testing today’s consumer AI agents, like OpenAI’s ChatGPT Agent and Perplexity’s Comet, reveals their limitations. Enhancing AI agents may require innovative techniques currently being explored.

The Importance of Reinforcement Learning Environments

One of the key strategies being developed is the creation of simulated workspaces for training AI agents on complex, multi-step tasks—commonly referred to as reinforcement learning (RL) environments. Much like how labeled datasets propelled earlier AI advancements, RL environments now appear essential for developing capable AI agents.

AI researchers, entrepreneurs, and investors shared insights with TechCrunch regarding the increasing demand for RL environments from leading AI laboratories, and numerous startups are emerging to meet this need.

“Top AI labs are building RL environments in-house,” Jennifer Li, a general partner at Andreessen Horowitz, explained in an interview with TechCrunch. “However, as you can imagine, creating these datasets is highly complex, leading AI labs to seek third-party vendors capable of delivering high-quality environments and assessments. Everyone is exploring this area.”

The drive for RL environments has spawned a wave of well-funded startups, including Mechanize and Prime Intellect, that aspire to dominate this emerging field. Additionally, established data-labeling companies like Mercor and Surge are investing significantly in RL environments to stay competitive as the industry transitions from static datasets to interactive simulations. There’s speculation that major labs, such as Anthropic, could invest over $1 billion in RL environments within the next year.

Investors and founders alike hope one of these startups will become the “Scale AI for environments,” akin to the $29 billion data labeling giant that fueled the chatbot revolution.

The essential question remains: will RL environments truly advance the capabilities of AI?

Understanding RL Environments

At their essence, RL environments simulate the tasks an AI agent might undertake within a real software application. One founder likened constructing them to “creating a very boring video game” in a recent interview.

For instance, an RL environment might mimic a Chrome browser, where an AI agent’s objective is to purchase a pair of socks from Amazon. The agent’s performance is evaluated, receiving a reward signal upon success (for example, making a fine sock purchase).

While this task seems straightforward, there are numerous potential pitfalls. The AI could struggle with navigating dropdown menus or might accidentally order too many pairs of socks. Since developers can’t predict every misstep an agent will take, the environment must be sophisticated enough to account for unpredictable behaviors while still offering meaningful feedback. This complexity makes developing environments far more challenging than crafting a static dataset.

Some environments are highly complex, allowing AI agents to utilize tools and interact with the internet, while others focus narrowly on training agents for specific enterprise software tasks.

The current excitement around RL environments isn’t without precedent. OpenAI’s early efforts in 2016 included creating “RL Gyms,” which were similar to today’s RL environments. The same year, Google DeepMind’s AlphaGo, an AI system, defeated a world champion in Go while leveraging RL techniques in a simulated environment.

Today’s environments have an added twist—researchers aspire to develop computer-using AI agents powered by large transformer models. Unlike AlphaGo, which operated in a closed, specialized environment, contemporary AI agents aim for broader capabilities. While AI researchers start with a stronger foundation, they also face heightened complexity and unpredictability.

A Competitive Landscape

AI data labeling agencies such as Scale AI, Surge, and Mercor are racing to build robust RL environments. These companies possess greater resources than many startups in the field and maintain strong ties with AI labs.

Edwin Chen, CEO of Surge, reported a “significant increase” in demand for RL environments from AI labs. Last year, Surge reportedly generated $1.2 billion in revenue by collaborating with organizations like OpenAI, Google, Anthropic, and Meta. As a response, Surge formed a dedicated internal team focused on developing RL environments.

Close behind is Mercor, a startup valued at $10 billion, which has also partnered with giants like OpenAI, Meta, and Anthropic. Mercor pitches investors on its capability to build RL environments tailored to coding, healthcare, and legal domain tasks, as suggested in promotional materials seen by TechCrunch.

CEO Brendan Foody remarked to TechCrunch that “few comprehend the vast potential of RL environments.”

Scale AI once led the data labeling domain but has seen a decline after Meta invested $14 billion and recruited its CEO. Subsequent to this, Google and OpenAI discontinued working with Scale AI, and the startup encounters competition for data labeling within Meta itself. Nevertheless, Scale is attempting to adapt by investing in RL environments.

“This reflects the fundamental nature of Scale AI’s business,” explained Chetan Rane, Scale AI’s head of product for agents and RL environments. “Scale has shown agility in adapting. We achieved this with our initial focus on autonomous vehicles. Following the ChatGPT breakthrough, Scale AI transitioned once more to frontier spaces like agents and environments.”

Some nascent companies are focusing exclusively on environments from inception. For example, Mechanize, founded only six months ago, ambitiously aims to “automate all jobs.” Co-founder Matthew Barnett told TechCrunch that their initial efforts are directed at developing RL environments for AI coding agents.

Mechanize is striving to provide AI labs with a small number of robust RL environments, contrasting larger data firms that offer a broad array of simpler RL environments. To attract talent, the startup is offering software engineers $500,000 salaries—significantly higher than what contractors at Scale AI or Surge might earn.

Sources indicate that Mechanize is already collaborating with Anthropic on RL environments, although neither party has commented on the partnership.

Additionally, some startups anticipate that RL environments will play a significant role outside AI labs. Prime Intellect, backed by AI expert Andrej Karpathy, Founders Fund, and Menlo Ventures, is targeting smaller developers with its RL environments.

Recently, Prime Intellect unveiled an RL environments hub, aiming to become a “Hugging Face for RL environments,” granting open-source developers access to resources typically reserved for larger AI labs while offering them access to crucial computational resources.

Training versatile agents in RL environments is generally more computationally intensive than prior AI training approaches, according to Prime Intellect researcher Will Brown. Alongside startups creating RL environments, GPU providers that can support this process stand to gain from the increase in demand.

“RL environments will be too expansive for any single entity to dominate,” said Brown in a recent interview. “Part of our aim is to develop robust open-source infrastructure for this domain. Our service revolves around computational resources, providing a convenient entry point for GPU utilization, but we view this with a long-term perspective.”

Can RL Environments Scale Effectively?

A central concern with RL environments is whether this approach can scale as efficiently as previous AI training techniques.

Reinforcement learning has been the backbone of significant advancements in AI over the past year, contributing to innovative models like OpenAI’s o1 and Anthropic’s Claude Opus 4. These breakthroughs are crucial as traditional methods for enhancing AI models have begun to show diminishing returns.

Environments form a pivotal part of AI labs’ strategic investment in RL, a direction many believe will continue to propel progress as they integrate more data and computational power. Researchers at OpenAI involved in developing o1 previously stated that the company’s initial focus on reasoning models emerged from their investments in RL and test-time computation because they believed it would scale effectively.

While the best methods for scaling RL remain uncertain, environments appear to be a promising solution. Rather than simply rewarding chatbots for text output, they enable agents to function in simulations with the tools and computing systems at their disposal. This method demands increased resources but, importantly, could yield more significant outcomes.

However, skepticism persists regarding the long-term viability of RL environments. Ross Taylor, a former AI research lead at Meta and co-founder of General Reasoning, expressed concerns that RL environments can fall prey to reward hacking, where AI models exploit loopholes to obtain rewards without genuinely completing assigned tasks.

“I think there’s a tendency to underestimate the challenges of scaling environments,” Taylor stated. “Even the best RL environments available typically require substantial modifications to function optimally.”

OpenAI’s Head of Engineering for its API division, Sherwin Wu, shared in a recent podcast that he is somewhat skeptical about RL environment startups. While acknowledging the competitive nature of the space, he pointed out the rapid evolution of AI research makes it challenging to effectively serve AI labs.

Karpathy, an investor in Prime Intellect who has labeled RL environments a potential game-changer, has also voiced caution regarding the broader RL landscape. In a post on X, he expressed apprehensions about the extent to which further advancements can be achieved through RL.

“I’m optimistic about environments and agent interactions, but I’m more cautious regarding reinforcement learning in general,” Karpathy noted.

Update: Earlier versions of this article referred to Mechanize as Mechanize Work. This has been amended to reflect the company’s official name.