nemo-curator

GPU-accelerated data curation for LLM training. Supports text/image/video/audio. 16× faster fuzzy deduplication. 40% lower TCO vs CPU. Scalable across GPUs with RAPIDS.

• Tool for preparing high-quality LLM training data
• Supports text, image, video, and audio curation
• Performs fuzzy, exact, and semantic deduplication
• Filters low-quality content and toxic content
• Redacts PII and detects NSFW content
• Scales across GPU clusters with RAPIDS

• 16× faster fuzzy deduplication (GPU)
• 30+ quality heuristics for filtering
• Semantic deduplication with embeddings
• PII redaction and NSFW detection
• Near-linear scaling across GPU nodes
• 40% lower total cost of ownership vs CPU
• Multi-modal support (text/image/video/audio)
• Faster processing with GPU acceleration

1. 1.Install via uv pip with GPU support (CUDA 12)
2. 2.Load data into DocumentDataset or modality-specific datasets
3. 3.Apply quality filters (word count, URL ratio, etc.)
4. 4.Perform deduplication (exact, fuzzy, semantic)
5. 5.Apply PII redaction and NSFW detection
6. 6.Use classifier for quality scoring
7. 7.Save curated data to Parquet or other formats