What I learned handling unsafe AI-generated content

I built a mobile app called Snap AI and initially thought its provider-level safety controls would be enough. After publishing it to Google Play, the app was suspended because generated content could still slip through those controls.

The first approach: filter the prompt

My first approach was prompt filtering. Before a user could submit a prompt, the app checked it for profanity. I adapted the bad-words package for Dart and shipped the update.

That only addressed the input. The model could still produce unsafe output from an apparently harmless prompt, and keyword lists are easy to work around.

Check the generated result

I then evaluated Google Cloud Vision to inspect the generated image itself.

const [result] = await imageAnnotator.safeSearchDetection(storageFilePath);
const detections = result.safeSearchAnnotation;

const isUnsafe = [
  detections?.adult,
  detections?.violence,
  detections?.racy,
].some((likelihood) => Number(likelihood) > 3);

When I tested previously missed images, the image-level check correctly identified the unsafe output.

The product lesson

The mobile listing was suspended before I completed the updated pipeline, but the experience changed how I think about safety in generative products.

Moderation has to cover the entire product flow. Input filtering is useful, but generated outputs need their own checks, failure handling, and clear product policy.