AuRA: From Atoms to Bits through Autonomous Robot Interaction

Abstract

Simulation enables robot-learning experiments at scale, but acquiring physics-informed digital twins often requires human manipulation and manual annotation of recorded interactions. To reduce this reliance on human effort, we present AuRA, a system that automates robot interaction and data preparation for digital-twin reconstruction. AuRA combines execution-aware grasp selection, gripper-motion measurements derived from robot states, and registered pre-interaction geometry and appearance priors. These choices enable autonomous collection, avoid reliance on visual gripper tracking, and recover object information obscured by the robot.

Across 13 deformable objects, AuRA reduces active human labor and wall-clock collection time per successfully reconstructed asset by 88.5% and 55.4%, respectively. Relative to the human-collected PhysTwin baseline, AuRA improves seven of nine metrics for future prediction under known gripper motion, including a 28.1% reduction in one-way L₁ Chamfer distance. It also achieves similar mean silhouette IoU and Chamfer distance under the evaluated unseen interactions. These results highlight the potential of autonomous robot interaction to support more efficient digital-twin acquisition with comparable reconstruction quality.

AuRA compared with human-operated collection: robot interaction with robot states and pre-interaction priors versus human interaction with visual tracking and manual annotation, both producing a spring-mass plus 3D Gaussian digital twin.

From physical interaction to digital twins. AuRA replaces manual manipulation with autonomous robot interaction, derives controller motion from robot states instead of visual tracking, and uses pre-interaction priors to supplement robot-occluded regions, while retaining the same spring–mass and 3D Gaussian representation.

Approach

Replacing the human operator with a robot raises three coupled challenges. AuRA answers each with one design choice, on top of the PhysTwin reconstruction backend.

1 · Interaction

Execution-aware grasp selection

A visually plausible grasp may violate joint limits or collide with the table or cameras.

AnyGrasp candidates from a multi-view RGB-D scan are checked for tabletop clearance and joint-limit margins along the planned motion. The robot executes the best feasible grasp-and-lift while recording RGB-D videos and robot states.

2 · Motion

Controller trajectories from robot states

Visually estimated gripper trajectories suffer from segmentation, tracking, and depth errors.

Recorded joint states and calibrated gripper geometry give the paths of gripper-attached controller points directly, even when the gripper is occluded. Visual tracking is used only for the object's response.

3 · Occlusion

Pre-interaction priors

The gripper and arm hide parts of the object's geometry and appearance.

A static point cloud and a Gaussian appearance model built from the pre-interaction scan are registered to the first interaction frame and fill in the regions the robot obscures.

AuRA pipeline: data acquisition (multi-view scan, grasp generation, execution and recording), data post-processing (pre-interaction prior registration, object and controller trajectory processing), and asset reconstruction (spring-mass and 3D Gaussian digital twin).

AuRA's autonomous physical-to-digital acquisition pipeline. Multi-view RGB-D capture supports grasp generation and execution-feasibility filtering. The robot executes the selected grasp and records the interaction with synchronized robot states. During processing, visual tracking provides object trajectories, while robot poses and calibrated gripper geometry provide controller trajectories. AuRA also constructs a static point cloud and trains a Gaussian appearance model from its pre-interaction captures, then aligns both with the first interaction frame to supplement robot-occluded regions. These inputs support spring–mass optimization and Gaussian rendering through the PhysTwin reconstruction backend.

Results

We evaluate on 13 deformable objects with a Franka robot arm and three RGB-D cameras, comparing against human-operated collection with the PhysTwin pipeline by operators in three experience groups (amateur, trained, expert).

1. Collection Efficiency

Human labor, collection time, and end-to-end reconstruction yield per asset

88.5%

less active human labor per asset

55.4%

less wall-clock collection time per asset

2.8×

higher end-to-end reconstruction yield

AuRA produces 26 reconstructed assets from 35 acquisition attempts (74.3%), whereas human-operated collection produces 21 from 78 attempts overall (26.9%). Per reconstructed asset, AuRA needs 126.7 s of wall-clock time and only 32.7 s of human labor, since a person is needed only to place the object and start the system, compared with 284.4 s for human collection overall.

Collection costs exclude data processing, reconstruction training, and untimed backup trials.

2. Reconstruction Quality

Resimulation and future prediction against the human-collected PhysTwin baseline

Following PhysTwin, the first 70% of each interaction is used for training and the remaining 30% for testing under known controller motion. AuRA improves seven of nine future-prediction metrics, with clear gains in silhouette and geometry: the one-way L₁ Chamfer distance drops by 28.1% and depth coverage rises by 9.9 points. PhysTwin, which retains about 2.4× more tracking points, fits the training frames more closely.

Metric	Resimulation		Future Prediction
Metric	PhysTwin	AuRA	PhysTwin	AuRA
PSNR (dB) ↑	12.289	11.549	10.325	10.590
SSIM ↑	0.9856	0.9852	0.9805	0.9797
LPIPS ↓	0.0126	0.0148	0.0233	0.0230
IoU ↑	0.6675	0.6090	0.4607	0.5001
Precision ↑	0.7521	0.7021	0.5899	0.6132
Recall ↑	0.7972	0.7791	0.6260	0.7085
Depth MAE (mm) ↓	21.797	24.093	27.784	34.923
Depth Coverage ↑	0.8152	0.8050	0.6392	0.7386
One-Way L₁ Chamfer (mm) ↓	11.225	10.104	23.335	16.783

Metrics are averaged within each object and then equally across 13 objects. PhysTwin uses 29 human trajectories; AuRA uses 26 robot trajectories.

3. Unseen Interactions

Reconstructed assets driven by new interactions

For each object, one asset per method is driven by a new, teleoperated drag-and-lift interaction. AuRA with static Gaussians and PhysTwin reach similar mean performance (mask IoU 0.3861 vs. 0.3849, one-way Chamfer 43.554 mm vs. 43.430 mm, PSNR 10.379 dB vs. 10.309 dB). Static Gaussians trained on the pre-interaction scan also beat first-frame Gaussians on every appearance and silhouette metric, because they keep the appearance of regions the robot later covers.

Data collection
frame 0 (GT)

Rendering overlay on
unseen interaction

Unseen interaction
frame 0 (GT)

AuRAStatic GS

AuRA static Gaussian rendering overlaid on the unseen interaction

AuRAFirst-frame GS

AuRA first-frame Gaussian rendering overlaid on the unseen interaction

PhysTwin

PhysTwin unseen interaction ground truth

Object appearance at the first frame of an unseen interaction. The rendered asset is manually aligned to the unseen interaction and overlaid 50/50 on its ground truth.

4. Ablations

Controller trajectory source, static geometry prior, and object tracker

Controller trajectory source. On the same 26 robot-collected trajectories, controllers derived from robot states enable successful reconstruction in 26/26 cases (100.0%), compared with 2/26 (7.7%) for controllers obtained through visual tracking, depth lifting, and PhysTwin-style filtering.
Static geometry prior. Adding the registered pre-interaction point cloud improves most metrics for every object tracker tested (CoTracker3, MVTracker, MV-TAP).
Object tracker. Among point-cloud-initialized trackers, MV-TAP generally performs best in both resimulation and future prediction.

BibTeX

@article{yu2026aura,
  title={{AuRA}: From Atoms to Bits through Autonomous Robot Interaction},
  author={Yu, Xinchen and Jung, Yoonkyo and Chen, Haoran and Cai, Zikui and Talreja, Aayush and Lee, Seungjae and Kim, Hyunjin and Huang, Furong},
  journal={arXiv preprint arXiv:2610.34452},
  year={2026}
}