Data Collection

Overview

Qualia accepts three distinct structured robotics source formats: LeRobot 2.1 directories (lerobot_v2_1), LeRobot v3 directories (lerobot_v3), and ROS 2 MCAP sessions (mcap_ros2). It does not require a Hugging Face dataset import or Hub token; publish the local recording through the Qualia SDK, which validates and pairs its Git history with lakehouse rows.

Qualia supports several collection workflows for vision-language-action (VLA) models:

• Manual data collection - Record demonstrations manually
• Automated collection - Use scripts to gather data at scale
• Existing recordings - Upload a local LeRobot 2.1, LeRobot v3, or ROS 2 MCAP recording

LeRobot 2.1 and v3

LeRobotDataset v3.0 stores synchronized robot trajectories, camera video, and indexing metadata. Record with a LeRobot release that writes the v3 layout, then upload the local dataset directory:

from qualia import Qualia

client = Qualia()
result = client.data.upload(
    "/data/record-test",
    name="record-test",
    source_format="lerobot_v3",
)
print(result["dataset_id"], result["episode_count"])

For an existing LeRobot 2.1 directory, pass source_format="lerobot_v2_1". Qualia treats 2.1 and v3 as separate source contracts; it never aliases one literal or layout to the other.

ROS 2 MCAP

Lay out one directory per recording and upload with source_format="mcap_ros2". Every .mcap file is ingested as its own source, so a rosbag2 size split (bag_0.mcap, bag_1.mcap, …) is fine — each file yields its own episodes, and a take that straddles a split boundary becomes two. An optional episode.json sidecar per directory can supply task and recording-purpose fallbacks.

There is no fixed single-file ceiling on the current data plane: a 400 GiB bag is admitted at upload/begin like any other, and the limits that apply are negotiated there. Still split long sessions at the recorder (rosbag2’s own split size defaults to 2 GB): a bag is the unit of containment, so a damaged or underivable file costs only its own episodes, and a smaller file is a smaller re-upload when a recording has to be fixed.

Ingest degrades, it does not refuse

A source the platform can partly derive is published rather than rejected, so a successful upload can contain less than you recorded. The rules worth knowing before your first session:

• Bags must agree on their camera set. If they disagree, the dataset is degraded to the intersection and the cameras outside it are dropped dataset-wide — not just from the bags that lacked them. Group bags by camera set and upload each group as its own dataset if your rig changed mid-session.
• A camera that produced no decodable video anywhere is dropped the same way.
• A camera that dies partway through keeps every episode it did cover; the episodes it missed publish without it.
• A bag whose state/action dimensions disagree with the rest is skipped whole.
• A damaged bag costs its own episodes, not the upload. A bag with a broken envelope — a bad record length, a recorder killed mid-write — still uploads, at generic chunk boundaries so it deduplicates worse, and ingest reports it source_underivable at warn. Every other bag publishes; the dataset comes back DEGRADED with that finding in ingest_warnings. Fix the recording and re-upload: the good bags are already deduplicated.
• A row clock above the 240 Hz encode ceiling is re-elected onto a synthetic uniform grid at the slowest camera’s rate, floored at 30 Hz and capped at the ceiling (30 Hz when no camera is rateable). A 750 Hz /joint_states beside 30 fps cameras publishes at 30 Hz, beside 60 fps cameras at 60 Hz, unless you say otherwise — pin the cadence with [clock] fixed_hz = <rate> or name the channel that carries it with [clock] row_clock_topic = "<topic>". The lever is the slow camera: a grid faster than it would repeat frames it never sampled. The elected clock is recorded in the report’s row_clock.

Every one of these is recorded in meta/ingest_report.json, and the SDK prints them at the end of the upload and returns them as ingest_warnings:

done: kitchen generation 1 — 74 file(s), 41.2 GiB, 73 episode(s), 219000 frame(s)
  1 ingest finding(s) — this dataset was published DEGRADED:
    warn: camera "wrist_left" was dropped from the dataset: it is not present
    in every bag (first disagreement: raw/session/bag_31.mcap)

An empty ingest_warnings means the report was read and was clean. If it could not be read, ingest_warnings_unavailable says why — that is not the same as clean, and should be treated as “go look at the report”.

Declaring episode boundaries

Boundaries are resolved highest precedence first: anything you declare in meta/ingest.toml — spans, then an event vocabulary — then what can be read out of the bag: authored markers, an operator-verdict topic, a done-style boolean, and finally one episode per bag. A declaration always outranks a heuristic, so the two config blocks below are how you stop guessing what ingest will do. The report records which rule fired, as boundary_mechanism (e.g. ingest-toml-spans) and boundary_source (authored when you declared it).

A rig with its own marker vocabulary declares it in meta/ingest.toml instead of hoping for a heuristic:

schema = 1

[episodes]
event_topic = "/rig/marks"

[episodes.events]   # your values; unlisted ones are ignored
go   = "open"
halt = "close"
ok   = "label"      # a label describes a take, it does not cut one

A label describes the take it falls inside, and two rules decide what reaches the episode’s tags. The same value pressed repeatedly is one outcome, however many times it was pressed — a held or bounced pedal does not multiply a verdict. And a different value pressed within a second retracts the one before it: an operator who presses success, sees the mistake and immediately presses issue has corrected a mispress, not recorded two outcomes, so the take is tagged issue alone. Presses further apart are two real outcomes and both are kept.

Nothing is erased to do this. Every press stays on the episode’s marker timeline with its instant, so the dashboard still shows the correction that happened and a curation pass can disagree with it — the rule decides only what the take is tagged.

If your boundaries live outside the bag — a sidecar, a manifest, a database of take timestamps — compute them and declare the windows directly. This is the one strategy that reads nothing from the recording:

schema = 1

[episodes]
spans = [
  { start = "2026-08-12T09:00:00Z", end = "2026-08-12T09:02:30Z" },
  { start = "2026-08-12T09:05:00Z", end = "2026-08-12T09:07:10Z" },
]

Instants are RFC 3339 on the recorder’s clock (MCAP log_time). Windows are closed intervals, so they must be in time order and must neither overlap nor touch — start a back-to-back take one nanosecond later, or its boundary sample would belong to two episodes.

One list covers a whole multi-bag session: a window that names no message in a given bag simply contributes no episode to it. A bag that no window reaches is skipped and recorded in the ingest report at info — a session-wide list is expected to leave most of its bags out, so this never publishes the dataset degraded. A span list that misses a file costs you that file, not the upload.

You can declare event_topic alongside spans. They answer different questions: the windows decide where takes are cut, the vocabulary names what the operator pressed inside them. Under spans a value may only be label (or ignore) — mapping one to open/close/cut is an error, because the windows already decide the boundaries.

If a bag carries both the topic you declared and q-core’s own /recorder_event, the windows still decide every edge — neither topic moves one — and only the topic you declared is read for what its values mean. The report names the other one, so it is never silently unread.

Declaring windows changes only where takes are cut, not what the operator recorded inside them: a verdict topic in the bag is still carried through, it just no longer decides the boundaries.

A bad config costs you one second, not one transfer

meta/ingest.toml is validated at upload/begin, against the same parser the real ingest uses, before the repo exists and before a byte moves. A typo is refused in about a second with the line, the column and the accepted keys:

Error: /v1/datasets/upload/begin -> HTTP 400: invalid data: meta/ingest.toml: TOML parse error at line 5, column 1
  |
5 | event_topics = ["/recorder_event"]
  | ^^^^^^^^^^^^
unknown field `event_topics`, expected one of `cut_at_last_action`, `event_topic`, `events`, `spans`

Unknown keys are errors, not silently ignored, so a misspelled setting is caught here rather than discovered as a wrong result later.