Building and Red-Teaming an AI Content Moderation System

The AI lab needed a unified content moderation system across all products. Existing moderation generated excessive false positives creating user friction, while coverage gaps exposed the platform to safety and abuse risks. The system needed to align with internal content policy, reduce risk at scale, and maintain high-quality UX across consumer and enterprise products.

We decomposed the moderation pipeline into discrete visual and text classification models — NSFW detection, celebrity likeness recognition, and IP likeness detection — each with independent confidence thresholds. Rather than applying a single binary filter, we defined category-level rulings and conjunction-based logic: a piece of content could be flagged by one model but cleared by another depending on the product context and risk profile. Confidence thresholds were calibrated per category using labeled datasets, with separate configurations for consumer-facing products (stricter) and enterprise APIs (more permissive). Each threshold was validated through structured A/B testing against held-out evaluation sets to measure the trade-off between false positive rate and safety coverage.

The second phase focused on adversarial stress-testing. We built a red teaming dashboard that ran structured adversarial datasets against the live moderation system — including AI-generated pentesting prompts, known jailbreaking patterns, and edge-case inputs designed to exploit gaps between visual and text classifiers. The dashboard tracked block rates, false negative rates, and latency per model in real time. Each red teaming cycle produced a structured report identifying failure modes, which fed directly into threshold adjustments and model retraining priorities.

The result was a moderation system that could be tuned by product context — lighter filtering for enterprise customers who manage their own compliance, stricter enforcement for consumer-facing products — while maintaining a consistent safety floor across all endpoints.

The moderation system shipped to production with sub-2% rejection rates — meaning fewer than 2 in 100 user requests were blocked — while maintaining robust safety coverage across NSFW, celebrity likeness, and IP detection. Red teaming cycles surfaced and closed critical failure modes before launch, including edge cases where visual and text classifiers disagreed and conjunction logic produced incorrect rulings. The product-context architecture enabled the lab to offer enterprise customers permissive defaults while maintaining stricter enforcement for consumer products, without maintaining separate moderation stacks. The real-time dashboard became ongoing infrastructure, not just a launch tool — the team continues to run adversarial tests against each model update.