← all reports
compute training & simulation off the robot 2026-08-21

Where To Train

The Jetson can train, but barely — and it is the same machine that has to run the robot. This is how to move training and simulation onto someone else's hardware, what it costs, and what to buy only if renting stops making sense.

Training needs the dataset, not the robot

Nothing about training touches the arms, the serial ports, or the Jetson. It needs one thing: the dataset, which is already on the Hugging Face Hub at Suyang99/xlerobot-cup-grasp-20260820-0230223 MB, nine files, three MP4 video streams plus a parquet. That downloads in under a minute anywhere in the world.

So a teammate with a GPU can train without ever being in the same room, the same network, or the same country as the robot. They need a read token for the dataset (it is private) and nothing else. Trained weights go back to the Hub; the Jetson pulls them down for the hardware trial.

01 — How much compute is actually needed

Measured on the Jetson, not estimated

These come from the two training logs. They set the scale of everything below — and the headline is that this is a small job.

VRAM used
1.6–1.9 GB
at batch 2 — VRAM was never the limit
Speed
1.0–1.7 step/s
Jetson, batch 2
Run length
4–5 h
for 18–20k steps
Dataset
223 MB
transfer is a non-issue

The per-step breakdown says where the time went: data_s was 0.003–0.005 s while updt_s was ~0.6 s. The job is GPU-bound, not data-bound — so a real GPU converts directly into speed, and CPU, RAM speed and disk throughput barely matter.

02 — The three routes

In the order worth trying them

route 1 · free

Google Colab

A free T4 with 16 GB VRAM, and the notebook already exists in the repo at 05-training/train_smolvla_colab.ipynb. Zero cost, zero setup, nothing to buy.

Limits: the free tier disconnects after a few hours and can drop you when GPUs are busy. Mount Drive and checkpoint often, or push checkpoints to the Hub as they are written. Fine for a run of a few hours; painful for the full official recipe.

route 2 · a few dollars

Rent a GPU by the hour

Marketplaces (vast.ai, RunPod) rent consumer cards — 3090, 4090 — by the hour from other people's machines. Managed clouds (Lambda, Paperspace) cost more but are tidier.

This is almost certainly the right answer for a hackathon. You need a handful of runs, not a permanent machine. See the cost arithmetic and the step-by-step below.

route 3 · hundreds+

Buy a machine

Only worth it if someone will keep using it after this project — for coursework, other robots, or a simulation track that runs for weeks. For one hackathon the maths does not work: the price of one mid-range GPU buys hundreds of rental hours.

03 — What it costs

The arithmetic, with the assumptions visible

Rental prices move constantly and vary by provider, region and demand — treat the rates below as a range to check, not a quote. The step counts and the conclusion are robust to the price drifting.

RunSteps × batchEst. hours on one 4090Est. cost
Repeat what was already run20k × 2under 1 hwell under $1
A properly sized run
(batch 32, grad-accum to 64)
50k × 32~3–6 h$1–4
SmolVLA's own documented recipe200k × 64~8–20 h$3–15
Five experiments at the middle size~20–30 h$8–25
$

Typical rates, to sanity-check against

Consumer cards on marketplaces (RTX 3090 / 4090) have generally sat in the region of a few tens of cents to under a dollar per hour; datacenter cards (A100, H100) run several times that. Check the live price on the provider before booking — that is the only number that counts.

The point that survives any price change: a full, correctly sized training run for this project costs single-digit dollars. The Jetson spends 4–5 hours of its own life on a run that a rented card finishes over lunch for the price of a coffee.

04 — Renting, step by step

From an empty account to a running job

Written for a teammate who has never rented a GPU. The gotchas marked ⚠ are the ones that have already cost this project time.

  1. Pick the card by VRAM, not by name. 24 GB (3090 or 4090) lets you run batch 64 without tricks. 16 GB works with gradient accumulation. Anything with 12 GB still beats the Jetson by a wide margin. A used-market 3090 is often the cheapest 24 GB per hour on a marketplace.
  2. ⚠ Set the disk size deliberately — this is the one that has bitten us twice. Rented instances default to a small disk and charge separately for more. Checkpoints here are 869 MB each for SmolVLA and 198 MB for ACT, written every 2,000 steps.
    # budget before you book:
    checkpoint size × (steps ÷ save_freq) + dataset + environment
    869 MB × (50000 ÷ 2000) + 223 MB + ~15 GB ≈ 37 GB
    # ask for 100 GB. The SmolVLA run on the Jetson died at step 18,000
    # because the disk filled mid-write — that checkpoint dir is 12 KB.
  3. Choose a PyTorch + CUDA image, not a bare Ubuntu one. Every provider offers one. It saves an hour of driver debugging, and you are paying by the hour.
  4. Install LeRobot and log in.
    pip install -e ".[smolvla]"          # or plain lerobot for ACT
    huggingface-cli login               # paste a READ token for the dataset
    ⚠ Token hygiene: the dataset is private. Give the teammate a read-only token, and a separate write token only if they are pushing weights back. Never paste a token into a notebook that gets shared or committed.
  5. Launch, with output going to the Hub — not just to local disk.
    lerobot-train \
      --policy.path=lerobot/smolvla_base \
      --dataset.repo_id=Suyang99/xlerobot-cup-grasp-20260820-0230 \
      --batch_size=32 \
      --steps=50000 \
      --eval_freq=2000 \                # ← LeRobot defaults this OFF. Turn it on.
      --save_freq=5000 \                # ← fewer, larger gaps = less disk
      --output_dir=./run1
  6. ⚠ If you take a spot / interruptible instance, push checkpoints to the Hub as they land. Spot instances are much cheaper and can be killed at any moment. A checkpoint that only exists on the instance's disk dies with it. The repo already has upload_model.sh for this; run it on a timer or after each save.
  7. ⚠ Stop the instance when the run ends. Most providers bill for a running instance whether or not it is doing anything. An idle 4090 left up overnight costs more than the training did.

05 — Simulation

Also rentable — and this is where a big GPU actually changes the answer

 TrainingSimulation (ManiSkill)
Runs on the Jetson?yes, slowlyeffectively no
Why1.6 GB VRAM is fine; it is just slowSAPIEN rendering plus hundreds of parallel envs will not fit 8 GB shared memory
Gain from a real GPU10–30× fasterfrom "cannot run" to "runs"
Needs the robot?nono

The assets are already here: XLeRobot/simulation/Maniskill/xlerobot.urdf plus two runner scripts. Neither mani_skill nor sapien is installed on the Jetson, so this track is at zero — but nothing is missing except a machine to run it on.

!

Read the project's own simulation guide before starting this track

03-software/SIMULATION-TRACK-GUIDE.md opens by saying the objective is not "use simulation" — it is a robot that works on demo day, and simulation has to justify its engineering cost with measurable real-world benefit. It also pushes back on casual time estimates: importing a URDF is a 10–15 minute exercise, but a task-relevant twin needs scene setup, joint and zero validation, collision geometry, camera transforms and real-vs-sim validation — and that is where the unpredictable time goes.

The same logic that says "don't train a second model before the hardware test" applies here: with zero physical trials run, simulation cannot tell you whether the problem is calibration, cameras or the inference path. Simulation earns its place later — once the chain is known to work but the policy does not generalise, it is the cheap way to cover hundreds of cup positions.

If a teammate starts it in parallel, the first milestone should be matching the real kinematics. Ready-made reference numbers from the URDF: closed = 0°, fully open = 97.4°, pivot at (−20.2, −24.4) mm, tip gap at closure 7.25 mm.

06 — If you do buy: a Windows build

Spend everything on VRAM; almost nothing else matters

!

Two non-negotiables

NVIDIA only. LeRobot runs on CUDA. AMD and Intel Arc will cost more hours in ROCm/IPEX debugging than the training itself takes.

Install WSL2 and work inside Ubuntu. LeRobot's dependency chain — ffmpeg video decode, the Feetech servo SDK — is painful on native Windows and behaves exactly like Linux under WSL2.

PartSpecWhy
GPU — enough12–16 GB
RTX 3060 / 5060 Ti
An order of magnitude past the Jetson; batch ~16–24 with accumulation
GPU — sweet spot16 GB
RTX 4070 Ti Super / 5070 Ti
Best cost per useful hour
GPU — comfortable24 GB
used RTX 3090, or 4090
Batch 64 with no tricks — SmolVLA's documented recipe. A used 3090 is usually the cheapest way to 24 GB; this workload wants capacity, not the newest architecture
Storage1 TB NVMe minimumThe part this team keeps underestimating. Checkpoints are 869 MB (SmolVLA) / 198 MB (ACT) every 2,000 steps
RAM32 GBVideo decode and dataloader workers
CPUany modern 8-coreMeasured data_s = 0.005 s/step. It is not the bottleneck. Do not spend here
PSU / case / coolingwhatever the GPU needsNo performance effect on this workload

07 — One change that helps on any machine

Gradient accumulation decouples batch size from VRAM

batch_size = 8  +  accumulate 8 steps  =  effective batch 64

Memory is charged at 8; the gradient quality is that of 64. Both runs so far used batch 2, and a good part of why those curves wobble is gradient noise from a batch that small. This works on the Jetson today — it does not need new hardware, only more wall-clock time.