Demonstration Data for Physical AI: Connecting Human Tasks, Robot Execution, and Training

Physical AI Has a Data Problem

Physical AI systems need data that describes more than what the world looks like. They need records of what happened in that world: the task being performed, the actions taken, the objects involved, the sequence of events, and the outcome. Text, images, and video are abundant, but structured records of actions and outcomes inside a known 3D environment are much harder to find.

That creates a particular challenge for robotics teams collecting demonstration data. A video of someone completing a task can show what happened visually, but it does not necessarily preserve the underlying spatial record: where the person was looking, how their hands moved, which inputs they provided, what they interacted with, when important events occurred, and whether the task ultimately succeeded.

Robot execution produces another important stream of data. Each attempt can contain machine actions, movement, sensor perspectives, interventions, task events, and outcomes. These signals become more useful when they remain connected to the task and environment in which they occurred, rather than being treated as separate sources of telemetry.

For physical AI, the opportunity is to preserve these activities as structured spatial data. Cognitive3D provides a way to record how people demonstrate tasks and how robots perform them, keeping actions, inputs, context, and outcomes connected within a continuous spatial record. That record can then be inspected, compared across episodes, and made available to the systems and models that use the data.

Two Sources of Demonstration Data, One Spatial Record

There are two important data streams in this workflow: human demonstrations and robot execution. Human demonstration data does not have to begin with a robot. A person can perform a task in a virtual environment or an instrumented real-world setup while Cognitive3D captures the first-person, or egocentric, perspective alongside available head and hand movement, inputs, object interactions, task events, and timing.

The sequence matters because the value of demonstration data is not limited to the individual signals being captured. Rather than treating movement, inputs, interactions, and outcomes as independent measurements, Cognitive3D keeps them connected to the task events that occurred during the same episode. Repeating the task across people, setups, and conditions can then broaden the dataset and capture variation in how people perform it.

Teleoperation adds another form of demonstration data. Here, a person directly controls a robot, allowing operator inputs to be connected with the robot’s resulting actions and task outcome. Instead of separating the human side from the machine side, teams can preserve the relationship between the command, the physical response, and what happened next.

Robot execution provides the second data stream. Teams can record machine actions, available sensor perspectives, interventions, task context, and outcomes during each attempt. Either stream can be useful independently, but together they provide a richer record of how a task is demonstrated, how a machine performs it, and what happens as a result.

Keep the Episode Intact

A major challenge with physical AI data is not simply capturing more signals. It is maintaining the context that makes those signals meaningful. Consider a robotic pick-and-place task. Knowing that an attempt failed is useful, but the failure label alone does not explain why. Teams may need to know the starting conditions, what objects were present, which actions preceded the failure, whether a person intervened, where the robot was positioned, and at what point the attempt broke down.

Cognitive3D organizes that information as a continuous episode around the task and outcome. Human-side data can include head pose and gaze where available, hands and joints, controller inputs, movement, interactions, and timing. Robot-side data can include available sensors and views, joints and end-effector activity, base pose and motion, task events, interventions, and outcomes. The result is a spatial and semantic record rather than a collection of disconnected logs.

That context is valuable when an episode succeeds, but it can be especially valuable when something goes wrong. A failed attempt is not necessarily bad data. If the record preserves the actions, interventions, conditions, and outcome surrounding the failure, teams can inspect where the attempt broke down and compare it with other episodes. Those useful failures can become part of the dataset rather than disappearing behind a binary success metric.

From Capture to Training: Inspect the Data Before You Use It

Collecting demonstration data is only the beginning. Teams also need ways to understand what they have captured and determine which records are useful for what comes next. Cognitive3D structures this workflow around four connected capabilities: Track, Explore, Analyze, and Connect.

Track captures the scene, inputs, goals, events, and available sensor data that make up the spatial record. Explore lets teams inspect individual episodes through 3D replay, paths, and a synchronized timeline, making it possible to reconstruct an attempt and examine movement, gaze where recorded, actions, timing, and events together rather than relying only on a success label or raw log.

Analyze helps teams look across records, conditions, and objectives. Teams can compare operators, setups, task conditions, and outcomes to identify recurring issues, coverage gaps, patterns, and useful failures. Connect then makes the resulting record available to the customer’s systems and models.

That final step matters for physical AI because Cognitive3D is not the learning system itself. It provides access to the spatial records so teams can select data and use it within their own training, evaluation, and refinement workflows. The process moves from Capture → Inspect → Assess and Compare → Select → Train, Evaluate, and Refine → Run the Next Attempt.

This is where spatial analytics moves beyond observability. Replay helps a team understand an individual episode, while the broader workflow makes the underlying demonstration and execution data easier to inspect, assess, select, and use in the systems responsible for what happens next.

Human Demonstrations Are More Than Teleoperation

Teleoperation is an important source of robotics data, but it is not the only way to capture human demonstrations. A person performing a task directly can also generate valuable demonstration data. In a VR environment or an instrumented real-world setup, teams can capture an egocentric record of how someone completes a task, including available perspective, movement, interactions, inputs, timing, task progression, and outcome.

This matters because physical tasks contain information that can be lost when they are reduced to video alone. Consider a person assembling a component. The useful record may include how they approach the object, how their hands move, which interaction happens first, how long individual steps take, what they are looking at, and how the sequence changes when something does not go as expected.

Capturing the demonstration as a continuous spatial episode preserves more of that structure. Repeating the task across people and conditions then creates variation, allowing teams to build datasets containing different approaches, environments, corrections, and outcomes rather than collecting a single idealized example.

Teleoperation extends the same principle to human-guided robot operation. Operator inputs can be tied directly to robot actions and task outcomes, providing another form of demonstration data for teams working on physical AI.

Robot Execution Completes the Picture

Once a robot begins performing the task, the same spatial record can be used to capture what happened during the machine’s attempt. The task, objects, and starting conditions provide context, while robot actions and available sensor data describe the execution. Human interventions can show where assistance was required, and the outcome records whether the attempt completed, failed, or was aborted while keeping the surrounding context intact.

This makes it possible to inspect an outcome rather than simply count it. Imagine a set of repeated pick attempts where failures are associated with a blocked sensor view at the point of grasp. A team can replay an affected episode, inspect the surrounding actions, and then compare the same pattern across failed attempts and different stations.

Instead of treating those failures as unusable records, the team can identify useful failures, flag capture faults, adjust its collection protocol, and pass selected data to the learning team. This is the relationship that matters for physical AI: record the attempt, understand the data, select what is useful, and make it available to the systems responsible for the next learning cycle.

Built to Move Spatial Data Into Your Stack

A demonstration-data platform also needs to fit into the systems robotics teams already use. Cognitive3D supports capture across Unity, Unreal, C++, ROS 2, visionOS, Android XR, and WebXR, with more than 40 integrations represented in the current platform. Spatial records can then be accessed through CSV, JSON, REST API, CLI, and an MCP Server. 

That separation between capture and access is important. Cognitive3D records the task, objects, events, outcomes, and available spatial signals as a continuous record. Teams can inspect and analyze that record inside Cognitive3D, but the data does not have to stay there. It can be made available to the systems and models that need it.

For robotics teams, this means the platform can sit alongside the existing stack rather than attempting to replace it. ROS 2, learning systems, simulation environments, internal pipelines, and model infrastructure continue doing their respective jobs, while Cognitive3D provides the spatial record connecting what happened during the episode to the systems that use the resulting data.

Where a team uses demonstration data for imitation learning or other model-development approaches, selected records can move into those workflows. Where the priority is evaluation, debugging, dataset curation, or protocol improvement, the same record can support those activities as well. In either case, the important point is that the data remains portable and machine-queryable.

A Decade of Spatial Data Applied to Physical AI

Cognitive3D is bringing this approach to robotics from a substantial existing record in spatial analytics. The platform has captured more than 350 million minutes of activity, 27 million sessions for replay, and more than one trillion device inputs. More than 1,400 enterprise and research teams have used Cognitive3D, and the platform is SOC 2 Type II and Vanta-monitored. 

This experience matters because physical AI introduces a data problem that is inherently spatial. The specific devices and applications may change, but many of the underlying requirements are familiar: record movement and interactions in three dimensions, preserve task progression and outcomes, reconstruct individual attempts, identify patterns across large numbers of records, and provide structured access to the resulting data.

For XR analytics, understanding behaviour can itself be the objective. For physical AI, the spatial record can also become an input to the systems teams use to train, evaluate, and refine their models. That makes the ability to connect as important as tracking, exploring, and analyzing, because the value of the record is not only that a person can inspect it. The record is able to move into the workflows where teams put that data to work.

Walk Into the Next Program Review With Evidence

Your robotics program may already be producing valuable demonstration data through human task capture, teleoperation runs, simulated task scenarios, and robot execution. The challenge is keeping that data connected to enough context to make it useful. Teams need to understand what the task was, what the person or robot did, which objects were involved, where someone intervened, which attempts succeeded or failed, and which episodes are worth carrying forward into evaluation or training.

Cognitive3D brings those pieces together in a continuous spatial record. Teams can capture the demonstration or robot attempt, inspect it in context, compare it with other episodes, select the data that matters, and connect that data to the systems and models they already use.

The goal is not another robotics dashboard. It is a structured, portable record of actions and outcomes in 3D that can move with the physical AI workflow, giving teams evidence they can inspect, compare, select, and use.