Data Deduplication: Definition, Applications, and Training Data Requirements

Data deduplication is the process of detecting and removing duplicate or near-duplicate samples from a dataset to ensure that each training example provides unique learning signal. In the context of machine learning, deduplication operates at a spectrum of similarity thresholds: exact deduplication removes byte-identical copies, near-duplicate detection removes samples that differ only in superficial ways (compression artifacts, minor crops, resolution differences), and semantic deduplication removes samples that convey the same information despite visual or structural differences.

The rationale for deduplication is both statistical and computational. Statistically, duplicate samples distort the empirical distribution that the model learns from. If a particular object-environment combination appears 100 times while another appears 10 times, the model's learned policy will be biased toward the over-represented case — not because it is more important, but because the dataset collection process happened to capture it more often. This is distinct from intentional oversampling, where important or difficult cases are deliberately repeated to improve model performance on them. Accidental duplication provides no such benefit because the duplicated cases are typically easy, well-represented scenarios rather than informative edge cases.

Computationally, deduplication reduces training time and storage costs proportional to the fraction of data removed. For a dataset with 15% duplicates, deduplication cuts training time by roughly 15% with no loss in model performance — in fact, performance usually improves because the effective data distribution becomes more uniform. Large-scale studies have quantified this effect: the Scaling Data-Constrained Language Learners paper (Muennighoff et al., 2023) showed that deduplicating training data is equivalent to collecting 30-50% more unique data in terms of downstream task performance.

For robotics and physical AI, deduplication is complicated by the multi-modal, temporal nature of the data. A manipulation demonstration consists of a video stream, action commands, and optionally proprioceptive readings — comparing these multi-modal trajectories requires either reducing each to a fixed-size representation (e.g., averaging frame embeddings over the trajectory) or comparing modality by modality. Two demonstrations of the same task may look identical in RGB but differ in action timing or force profiles, meaning visual deduplication alone can remove trajectories that are actually distinct from a learning perspective. Effective robotics deduplication must consider the action modality alongside visual similarity.

Implementing deduplication for a robotics dataset follows a four-stage pipeline. First, compute representations: for each data sample (image, video clip, or trajectory), produce a fixed-size feature vector. For images, CLIP or DINOv2 embeddings (768-1024 dimensions) work well. For video clips, average per-frame embeddings or use a video encoder like I3D. For manipulation trajectories, concatenate the averaged visual embedding with a summary of the action sequence (mean, variance, and range of each action dimension). Second, index the representations: for datasets under 100,000 samples, brute-force cosine similarity with FAISS is fast enough. For larger datasets, use approximate nearest neighbor methods like MinHash LSH or FAISS IVF-PQ indexes. Third, cluster and select: group near-duplicates into clusters (connected components at a cosine similarity threshold, typically 0.90-0.95), then select one representative from each cluster — either the highest-quality sample (by a quality score) or the one most central in embedding space. Fourth, validate: spot-check removed samples to confirm they are genuine duplicates rather than valid data variation, and compare model performance with and without deduplication on a held-out test set.

The similarity threshold is the most consequential design choice. Too aggressive (threshold 0.80) removes legitimate data variation — two demonstrations of the same task in different lighting are semantically similar but visually distinct, and both provide useful training signal. Too conservative (threshold 0.98) misses near-duplicates that differ only in compression artifacts or frame alignment. For robotics video data, a cosine similarity threshold of 0.92-0.95 on DINOv2 embeddings balances duplicate removal with diversity preservation. This should be calibrated on a manually labeled sample of your specific dataset.

Deduplication interacts with dataset splitting. Always deduplicate before splitting into training and test sets — otherwise, near-duplicates may land in both splits, creating data leakage. Apply deduplication globally, then assign entire duplicate clusters to either training or test but never both.

Claru integrates deduplication into every data delivery pipeline. Our datasets undergo multi-level deduplication: exact hash matching removes literal copies, perceptual hash comparison removes visually identical frames, and embedding-based clustering removes near-duplicate clips and trajectories. Clients receive a deduplication report documenting the fraction of data removed at each level and the similarity thresholds used, so they can adjust thresholds if their use case requires more aggressive or more conservative deduplication.

Claru builds deduplication into every data pipeline, ensuring that clients receive datasets with verified uniqueness at the clip, trajectory, and scene level. Our multi-stage deduplication process uses exact hashing, perceptual hashing, and embedding-based semantic similarity to catch duplicates at every level — from literal file copies to demonstrations that are visually distinct but provide redundant training signal.

For clients aggregating data from multiple sources, Claru performs cross-dataset deduplication: merging new data collections with existing training corpora and identifying overlaps before delivery. Each dataset ships with a deduplication report showing the number of duplicates detected at each threshold, the duplicate clusters identified, and the selection criteria used to choose representatives. This transparency lets ML teams adjust deduplication aggressiveness based on their specific model architecture and training regime, rather than accepting opaque data processing decisions.