Robotics · dataset curation

dataset-curation-dedup

Manifest-level curation with quality gates, supplied hashes and feature vectors, a reason for every rejection, and a human queue for borderline duplicates.

Public · synthetic demo
The problem

Automatic deduplication is dangerous when the threshold is treated as truth.

Confirmed duplicates need deterministic rejection reasons; borderline pairs need human review instead of automatic deletion.

The money shot

Every removal is explained; the borderline case remains

Measured curation result
50input records
45curated episodes
6total rejections
1borderline pair for review
stagemeasured result
corrupt ingest1
passed quality gate48
duplicate groups3
dropped as duplicate3
Human review boundary
ep_0003 ↔ ep_0046
metric: hamming
value: 6
verdict: review queue — not auto-dropped

The manifest also reports task coverage and imbalance; it does not reinterpret those counts as training utility.

Honest limitations

The tool operates on a manifest plus provided hashes and feature vectors, not raw pixels, video, point clouds, or sensor streams. Hamming and cosine thresholds are heuristics. Borderline duplicates are never auto-resolved.

This page reports the supplied sample output and its stated limits. Inspect the code and evidence ↗