VLA Deployment

TL;DR

This project started as an industrial-funded VLA deployment problem and ended as a working real-robot pipeline. I owned the full stack: data collection on real robots, teleoperation, camera calibration, model training, evaluation, sim-real alignment, and deployment.

By May 31, 2026, the robot could roll out four blender manipulation subskills in the real world, each clearing a conservative 70%+ internal success-rate bar:

• Open the blender lid
• Place fruit into the blender
• Close the blender lid
• Turn on the blender

The important result is not only that the robot can do these behaviors. The delivery is that I built the loop that makes new behaviors trainable, measurable, and deployable.

Why This Was Hard

A VLA model is only one part of a robot product. The real work is making the observation, action, data, training, and deployment contracts line up under real hardware constraints.

For this project, that meant building a practical target-domain layer around a foundation-model workflow:

• Collect clean demonstrations from the Franka Research 3 with synchronized multi-view RealSense observations.
• Convert robot data into a training format compatible with policy learning workflows.
• Train and evaluate policies with explicit success criteria instead of relying on one-off demos.
• Align real and simulated camera observations so pretraining and synthetic data could become useful.
• Deploy policies back onto the robot through a real-time control stack.

The First Closed Loop

The first milestone was proving the whole stack could run end to end. With no inherited VLA or ROS data pipeline to build on, I bootstrapped robot bring-up, cameras, teleoperation, data logging, training, and deployment from scratch.

The system architecture used a distributed ROS 2 setup with Zenoh, an RTX 5090 machine for inference and data collection, ROS2 Control for the robot interface, and a real-time control path for the Franka Research 3. The goal was simple: get from demonstration data to a real rollout without changing tools halfway through.

Reliable data collection also required comfortable teleoperation. I fixed the Franka teleoperation setup with gravity compensation so demonstrations could be collected through a leader-follower workflow.

Sim-Real Alignment

Pretraining is useful if the model’s observations are close enough to the real deployment domain. Before asking whether a checkpoint could transfer, I treated camera calibration as a model-performance problem, not just a robotics setup detail.

If the real camera is not calibrated against the robot base, the model is learning from a view that the simulator does not actually share. I calibrated the camera transform relative to the base frame, then re-rendered the simulation from that matched camera pose so the observations became more alike.

Base-to-camera transform calibration. The goal was to make real and simulated observations agree before judging model transfer.

Real observation

Simulation before calibration

Simulation after calibration

That calibration work became reusable infrastructure:

• A Visual Guide to Hand-Eye Camera Calibration in Robotics
• franka_moveit_camera_calibration, a no-Docker ROS 2 and MoveIt calibration repo for Franka Research 3 and RealSense cameras

Why Pretraining Mattered

I used RoboCasa365 as context for the training strategy, not as a benchmark to reproduce wholesale. The key lesson from the paper is practical: broad pretraining helps downstream robot tasks, but the target domain still needs a dedicated adaptation phase.

RoboCasa365 studies generalist robot training across 365 household tasks, 2,500 simulated kitchen scenes, and more than 2,000 hours of robot interaction data. Its foundation-model experiments show that pretraining followed by target post-training improves downstream data efficiency. It also reports that joint one-phase training on pretraining and target data can underperform a two-stage pretrain-then-fine-tune setup.

That shaped my project framing: use simulation and pretrained checkpoints as leverage, then invest in real target data, calibration, and deployment discipline.

a RoboCasa-pretrained GR00T checkpoint produced more useful target-domain behavior than the no-pretraining baseline under the same small-data setting.

Real Data, Real Evaluation

For the blender subskills, I collected real-robot demonstrations, trained policies, and evaluated rollouts as repeatable task attempts rather than isolated demo clips. The project moved from “can it work once?” to “can I collect, train, evaluate, diagnose, and redeploy?”

The evaluation view paired simulation and real deployment for each subskill, making it easier to compare behavior across domains instead of judging isolated clips.

The same loop also made it possible to probe generality: after collecting demos on a seen object, I tested whether the trained skill could handle related object variation in real rollouts.

Generalization study panel for the place-into-blender skill. The collected demos used a seen object; the rollout probes below tested related variation.

The evaluation mindset mattered because the system had several possible failure sources: camera pose mismatch, teleoperation quality, object placement variation, action scaling, gripper timing, policy undertraining, and control latency. Treating the robot as a full pipeline made those failures debuggable.

Synthetic Data Path

After the real-world skills worked, I explored how to reduce the amount of expensive real data needed for future tasks. This became a second pipeline: reconstruct or approximate the environment, generate synthetic trajectories, and test whether a model could begin to act before seeing real demonstrations.

Gaussian Splatting reconstruction.

Gaussian Splatting cleanup.

Synthetic data in LeRobot format

Zero-Real-Data Rollout Probe

Start frames

End frames

This was not the final production recipe by itself. It was a probe that made the project more valuable: the pipeline could now test where simulation helps, where real data is still required, and which parts of the environment must be aligned before the model can transfer.

What I Delivered

This project finished with a deployable real-robot VLA workflow and a clearer understanding of where the engineering leverage lives.

The most important deliverables were:

• A real-robot data collection stack with synchronized multi-camera observations.
• A teleoperation workflow suitable for collecting demonstrations on Franka Research 3.
• A training and evaluation loop for target-domain VLA skills.
• Four real-world blender subskills crossing a conservative 70%+ success threshold.
• Camera calibration and sim-real observation alignment infrastructure.
• A synthetic-data exploration path using Gaussian Splatting, simulation, and motion planning.
• Public technical writeups and reusable calibration code.

The project is finished, but the lesson carries forward: deploying VLAs is full-stack robotics work. The model matters, but the moat is the pipeline around it.

Written Output

• How to Enable Zero-Gravity Teleoperating a Franka Robot?
• Understanding PD Controllers in Robotics and Robot Learning
• Multi-Machine ROS2 for Robot Learning: A Guide to RMW
• Understanding Action Chunking with Flow Matching
• Why Naive Behavioral Cloning Doesn’t Work?
• Why the Gymnasium API Looks the Way It Does?