Real-world workflows are becoming the next generation of training data.

In recent conversations with various AI labs and data companies, we've observed a clear trend: real-world data, the data generated in enterprises' actual workflows, is emerging as a new priority for model training.

Compared to internet data that has been thoroughly mined, this remains an untapped mineral deposit, with prospectors only just starting to arrive.

The interesting part of this market is that data and the ability to leverage it are distributed at opposite ends: model companies are typically MLE (Machine Learning Engineer)-rich but data-poor. Enterprises are the exact opposite — data-rich, but MLE-poor.

Against this backdrop, we've seen two distinct types of companies emerge in the market, representing two sides of the same coin:

• Human data company: Serves model labs by providing specialized datasets.

• RLaaS (RL-as-a-service) company: Serves enterprises by converting their business workflows into trainable model environments. Essentially, they deliver outsourced MLE consulting, agent system implementation, and post-training services.

This article compiles 20 reading notes and takeaways from recent discussions about this data sector, mostly observations and retrospectives from frontline practitioners. Full references are attached at the end for further reading.

01.

The Data Market Is a Surfing Business

⁠1. The data market is a surfing business. The demand for data will persist long-term, but every time a capability bottleneck is broken, the "most valuable data" shifts to a new type and form. As a result, players in the entire data market must continuously adapt alongside evolving model training paradigms.

⁠2. In the past few years, each wave of new data demands has created a new set of winners. Scale AI seized the data labeling opportunities for autonomous driving and early LLMs; Mercor, Surge AI, and Handshake rode the expert data wave. As training shifts further toward real-world workflows, new companies like Protege, Sunset, and SF Data are beginning to appear.

At the same time, established players are also adjusting their offerings to align with the needs of frontier labs. Both Mercor and Handshake are now emphasizing enterprise data use cases to position themselves ahead of the next wave.

⁠3. The new wave: Real-world Data.

⁠Over the past few years, the budget priorities of frontier labs have shifted from expert labeling to RL environments, and now they are turning their attention to real-world data.

⁠The core reason is that Long Horizon is the most critical direction for model advancement. The unit of tasks that models can handle is evolving from a single code modification, to a single file, to an entire codebase, and eventually to a full project.

⁠The longer the task sequence, the harder it becomes to manually construct environments. Real projects are full of historical states, tool dependencies, organizational rules, and unexpected edge cases — details that experts cannot easily fabricate out of thin air in a sandbox. Therefore, frontier labs are beginning to source real-world data.

⁠4. The data and RL market is far from saturated. While the gold rush in this space is heating up rapidly, overall supply has not kept pace with demand. A vast amount of real white-collar work, enterprise processes, and professional knowledge has not yet been converted into model-learnable data or reusable training environments. In the future, we will see more companies focused on "data production" and "environment creation," and potentially new types of intermediary firms (e.g., enterprise data brokers, expert networks), forming a more mature, finely divided supply chain.

⁠02.

How to Build a Successful Data Company

⁠1. Based on their sources, data available in the market today can be divided into two categories:

⁠• Type 1: Data captured from real work

⁠This is similar to "recording work processes," such as session replay, screen operations, and internal company collaboration logs. The best Type 1 data not only records actions, but also restores the intentions behind those actions.

⁠For example, GitHub is a great source of Type 1 data. Commit messages, issues, and ticket resolutions are linked together, almost completely preserving the full lifecycle of a problem from emergence to resolution: what a person was trying to solve, what modifications they attempted, why they made those changes, and whether the final result was accepted.

⁠• Type 2: Artificially constructed data

This is what the vast majority of human data companies are currently doing: recruiting domain experts → manually designing tasks → having experts complete tasks in pre-built environments → collecting outputs → processing them into formats suitable for model training.

⁠2. Type 2 data is well-suited for pre-training or early-stage capability development. But as models begin to handle longer sequences of tasks with higher economic value, Type 1 data will become increasingly important.

⁠However, pure Type 1 data is extremely difficult to obtain. In practice, a more feasible approach is to first achieve an intermediate Type 1.5 state, where Type 2 data is made to resemble Type 1 data as closely as possible. This typically includes:

⁠• Building long-term partnerships with a small number of high-quality experts, and ensuring they understand basic reward shaping and model training logic;

⁠• Redesigning incentive mechanisms. Humans are often better at reward hacking than models, so safeguards must be put in place to prevent annotators from producing low-quality labeled data;

⁠• Implementing multi-layer QA (quality assurance). Have operations teams QA outsourced work, ML engineers QA the operations teams, and ensure all stakeholders perform checks from different perspectives before final delivery. Gradually industrialize mature inspection methods, including anomaly detection, contributor behavior analysis, and using actual training performance to validate data quality in reverse.

⁠3. When designing training data, the truly critical factor is hillclimbability.

⁠Many companies follow this task design logic: as long as we create a task that the current model cannot solve, it proves there is an opportunity. But the problem is, designing a task that state-of-the-art models cannot handle is not difficult. The real challenge is to create tasks that sit exactly at the model's capability boundary — challenging enough, but still allowing the model to make steady progress through training.

Two scenarios can be envisioned:

⁠• pass@1 = 0%, pass@32 = 30%.

This means the model cannot solve the task on its first attempt, but after allowing it to explore independently 32 times, it has a 30% success rate. This proves the model has occasionally stumbled onto a successful path, but has not yet stabilized. This task is perfectly positioned at the model's capability boundary and provides high training value.

⁠• pass@1 = 0%, pass@256 = 0%.

After 256 attempts, there is still no success, indicating the task is far beyond the model's current capability, or the task itself is too niche. This is rarely a good training benchmark.

⁠4. Looking at today's data market, four key problems remain:

• Data distortion: Type 2 data is often misrepresented as Type 1 data captured from real work.

⁠• Eval distortion: Data companies design evaluations that are not sufficiently aligned with real work. When SOTA models can already perform well on certain evaluations out of the box, vendors deliberately increase difficulty to create the illusion that "models still have much room for improvement." This tactic is especially effective when researchers or procurement teams are not domain experts themselves.

⁠• Non-scalable QA: Many companies treat QA as an operational problem, solving it by adding more manpower and layers of manual review. But QA is essentially an engineering problem — quality must be stably embedded into the production pipeline through automated inspection, data traceability, environment generation, and evaluation systems. If engineering teams are constantly busy with one-off custom client requests, it will be difficult in the long run to build truly scalable data capabilities.

⁠• Demand transmission distortion: Researchers themselves sometimes cannot clearly articulate data requirements or properly conduct QA. But if model labs create a dedicated internal data procurement team, it adds another layer of communication between researchers and external vendors, introducing more information loss. This is why many labs still allocate data procurement budgets directly to researchers.

⁠5. A data vendor's ability to gain the trust of model labs typically depends on three core capabilities:

⁠• Data taste: The ability to recognize what high-quality data looks like, meaning the competence to ensure data quality;

⁠• Research taste: Understanding exactly where the model is currently bottlenecked, and knowing what type of data is needed to address the problem;

⁠• Scalability: The ability to maintain consistent quality as operations scale. This is easily overlooked — many companies recruit top experts during the demo phase, with founders personally overseeing projects and heavy manual QA, so small-scale deliveries look impressive. But this does not mean they can replicate that same quality 10x or 100x over.

⁠6. As the gold rush of "selling RL environments / data" heats up in San Francisco's tech circles, more and more entrepreneurs are entering this market. But researchers at model labs have grown tired of the generic pitches from these companies.

⁠To truly earn researchers' trust today, you need to demonstrate the three aforementioned capabilities. This can be done by:

⁠• Publishing benchmarks and technical papers;

⁠• Training models using your own data to demonstrate measurable performance gains;

⁠• Providing auditable data lineage, detailing data sources, cleaning processes, expert qualifications, and QA mechanisms.

⁠7. A company's DNA can be roughly inferred from its hiring structure:

⁠Companies that focus more on Type 2 data typically prioritize project management and operations, frequently recruiting SPLs (Special Project Leads). Essentially, their model revolves around taking orders, expanding teams, and delivering custom work. By contrast, companies aiming to move toward Type 1 data almost exclusively hire "members of technical staff," focusing on data engineering, environment engineering, and QA automation.

⁠The former is "selling manpower," while the latter is "building a factory."

⁠As data value continues to shift toward Type 1 data, the window of opportunity for pure Type 2 companies may close in as little as two years.

⁠Therefore, if you are a VC, you should not invest in a company that has no roadmap for Type 1 data. If you are an SPL, you should not lock yourself into long-term project delivery. Your connections in labs are extremely valuable — consider supplementing your technical skills to launch a more technically focused startup.

⁠03.

How RL Continues to Evolve

⁠1. What exactly is an RL environment?

⁠An RL environment can be understood as a software space where models can operate, invoke tools, complete tasks, get evaluated, and receive rewards.

⁠Take a sales operations agent as an example: its environment might include simulated Salesforce instances, email systems, customer databases, product documentation, and approval workflows.

Suppose we assign the agent a task: identify the 20 customers at highest churn risk over the past three months, and prepare personalized renewal emails for each of them.

To complete this task, the model needs to: query data → understand customer history → assess churn risk → access the CRM system → draft emails → generate quotes → submit for approval.

The environment then scores the agent based on predefined rules: were the correct customers identified? Is the churn risk assessment reasonable? Do the emails meet requirements? Are the quotes appropriate? Was the task completed within an acceptable timeframe?

⁠2. A clear trend is that agent environments and tasks are becoming increasingly complex. Four major evolving directions include:

⁠• Expanded action space: An agent no longer completes all work within a single sandbox, but can seamlessly move across multiple interconnected environments.

⁠• Dynamic tools: In the future, agents may directly use tools with uncertain outputs. For example, the "search tool" in an environment might not be a traditional search API, but another SOTA model. This means agents must also learn to adjust parameters, evaluate outputs, and perform cross-validation.

⁠• Organizational complexity: Complex tasks will be broken down by one agent and distributed to multiple sub-agents, with results aggregated centrally. In this scenario, reward functions must not only measure task completion, but also account for cost and latency. Two identical results — one completed in 40 minutes, and another in 3 minutes — have vastly different commercial value.

⁠• Shifting task boundaries: As models become more powerful, task decomposition patterns will change. Tasks that once required 20 steps to train may now be reliably completed in one step. As a result, the training objectives for RL environments will keep moving upward — from training models to perform single actions, to completing full sub-processes, to planning and orchestrating entire end-to-end workflows.

⁠3. What kind of RL data is most suitable for model learning?

⁠Jason Wei once proposed the Verifier's Law: The ease of training an AI to solve a task is directly proportional to the task's verifiability. Ultimately, anything that can be measured can be optimized. This verifiability has seven key dimensions:

4. An additional critical point to emphasize is that the key to RL is not just "whether the result is verifiable," but also "whether the environment can be reset repeatedly," allowing the model to get sufficient practice.

Coding is particularly well-suited for reinforcement learning, not just because it has clear reward signals, but because the entire environment is easily replicable, parallelizable, and resettable. A single code repository can be cloned into tens of thousands of containers, where models can continuously modify code, run tests, and restart after failures. Even if individual learning iterations are inefficient, capabilities can be accumulated through massive trial and error.

⁠But the real world does not have so many save points. For example, "successfully purchasing an item on Amazon" is certainly verifiable, but it is impossible to spin up 10,000 real Amazon instances simultaneously for the model to place orders, make payments, and restart. Even manually building a replica e-commerce site requires maintaining huge amounts of detail around inventory, payments, accounts, and logistics.

⁠Entering the real world is even more challenging. Activities like founding a startup, management, investment, legal work, and sales all have measurable outcomes, but you cannot spawn 10,000 parallel universes for the model to repeatedly practice in. These activities have slow feedback loops, ambiguous causal relationships, and non-repeatable opportunities.

⁠Therefore, for AI to enter the real world in the future, it cannot forever rely on brute-force trial and error. It must improve its sample efficiency — quickly extracting transferable patterns from a small number of non-repeatable, ambiguously signaled experiences. This remains one of the most distinct advantages humans have over models.

⁠5. Several approaches are being explored to improve sample efficiency, including:

⁠• Idea 1: OPSD (on-policy self-distillation)

⁠This can be thought of as an "experience compression" mechanism.

⁠Assume a model has been working alongside users for a week. By the seventh day, it will gradually build a deep understanding of the organization — knowing which issues users have repeatedly corrected, and which approaches have been tested and proven ineffective.

⁠At this point, the model behaves like a long-tenured, experienced employee familiar with company workflows.

⁠Next, this "veteran model" can act as a teacher, guiding a base model that has not been exposed to the full historical record. The student does not need to remember every detail of the past week, but must learn the judgment framework the teacher has developed: which information is more important, when to ask clarifying questions, and under what circumstances to take direct action.