When the Model That Writes Erotica Also Moderates It: 83–92% of Blocks Were Wrong

The range in the second audit is honest uncertainty: nine blocks were borderline enough that two careful readers could disagree. Even the conservative count is 83%.

• Population: every block recorded in the moderation log for the period, not a sample. Each record holds the request, the layer that blocked it and the verdict.
• Re-judging: each request was read by a person against the written policy, in its original language, and classified as correctly blocked, wrongly blocked or borderline.
• Before/after tests: every change to a moderation prompt was run against a fixed set of cases before shipping — real false positives from production mixed with cases that must remain blocked. A change that let one prohibited case through was rejected regardless of how many false positives it fixed.
• Enforcement replay: changes to the strike rules were tested by replaying the real history of every sanctioned account through the new rules, to see who would and would not have been suspended.

The first classifier prompt opened by telling the model it was a strict content moderator, and said nothing about requests that describe nothing at all. Refused in production, among others: explicit adult requests of exactly the kind the service exists for, and “send me a photo of you” — refused alike in English, Italian, Arabic and Chinese.

The keyword layer had its own failure, one familiar to anyone who has built a multilingual blocklist: a pattern for one language matching an ordinary word in another. A German-language rule also matched the English word kind, so “you’re so kind” was flagged; so were “a minor detail” and “we met in high school”. Twelve of fourteen benign test phrases were blocked. On image requests there was no second check behind the keywords, and one such hit was enough for a 24-hour suspension.

The name classifier refused the Arabic word for Egypt as a companion name because it did not recognise it. The rewritten prompt says an unfamiliar name is not a reason to block.

After the rewrite, on an 18-case before/after test, the old prompt made 4 mistakes and the new one none, with all 8 prohibited cases still blocked.

Three weeks later the block rate was high again. The prompt had not changed; the model had. To reduce latency the moderation question had been routed to the same model that writes the conversation — a 24-billion-parameter model fine-tuned for roleplay. Asked to police the kind of text it was trained to produce, it refused almost everything explicit, and several things that were not: a single fictional creature name, a nude figure study, and in one case a file name.

Two plausible technical explanations were tested and ruled out: the verdict parser (the model answered cleanly with the single word) and the output length limit (nothing was being truncated). The decisive test was to send the same requests to a different, 14-billion-parameter model with the same prompt: it judged 13 of 14 correctly.

Finding: a model’s fitness as a judge is a property of the model, not only of the prompt. A model tuned to be permissive in generation is not therefore a lenient judge — in our data it was a severe and inconsistent one.

1. The first pass answers ALLOW or BLOCK, as before. ALLOW ends the check.
2. A BLOCK goes to a second pass that is not a yes/no question: it must name the prohibited category that applies, from a fixed list, or say that none does. Only an explicit “none” clears the request; any other answer, or an unrecognised one, confirms the block.

The first draft of the second pass was a softer question — roughly, “if you cannot name a category, allow it”. Replayed against production, it cleared a request describing incest, one that named a real person through a social-media handle, and one naming an anime character who is fifteen in canon. Requiring the category by name forced the model to look for one. That draft never shipped.

Two failure modes did not respond to any prompt wording and are now handled by deterministic rules instead: a common name for a sex position was read as animal content even when the instructions said explicitly that it is not, and requests naming real public figures could pass both passes. Real-person detection remains the weakest part of the system and relies on several layers, including human review of reports; we regard any list of names as incomplete by construction.

Result on the final test set: 28 of 28 correct — the 17 real false positives now allowed, the 11 prohibited cases still blocked.

A wrong block is an annoyance. A wrong block that counts toward a suspension is how a service loses the customers who were trying to pay. The audits found four separate mechanisms:

• Retries counted as repeat offences. When a request is refused, people press the button again. One user sent the same request three times in 62 seconds and was suspended on the third. Now an identical request within an hour is kept as evidence but carries no weight.
• Rephrasing counted as escalation. Nobody retries word for word; they reword. One user tried three variations of a creature name in 61 seconds and was suspended for 24 hours. Now further blocks in the same category within ten minutes count as one event.
• A stated adult age was read as a minor. A pattern meant to catch stated ages under 18 had been written without the number, so “a 19 year old girl” — and even 45 — was treated as a minor. One user was suspended for seven days for stating an adult age, which is exactly what the policy wants people to do. The number 1–17 is now required; 16 test cases, every minor blocked, every adult allowed.
• The user was punished for the AI’s words. A scene the companion herself had written tripped a keyword rule and suspended the man reading it. Text the AI wrote now carries no weight against the user; it still alerts a human.

Replaying every sanctioned account through the new rules: 10 suspended accounts became 6, all four released were bursts of classifier blocks, and both permanent bans stood. Wrongly issued strikes were cleared retroactively — 30 on September 2 and 33 on September 21 — and one suspension was lifted. The current enforcement rules are published in Safety, Moderation and Enforcement.

• Audit the blocks, not the block rate. A rate tells you nothing about whether the blocks are right; re-judging every one took hours and changed the system.
• Test a judge model on its own. Moving moderation to a cheaper or faster model is a model change and needs the same before/after test as a prompt change.
• Ask for a reason, not a verdict. A category the model must name is harder to produce by reflex than a yes or no.
• Keep deterministic rules for the cases a model gets wrong consistently, and test every multilingual pattern with a real sentence per language — see our note on word boundaries, which silently disabled rules in six languages.
• Separate detection from punishment. The filter can be strict; the strike system should punish a pattern of behaviour, never a single event or a retry.

• One service, one policy, three weeks of traffic. The absolute block counts are small (35 and 93); the rates are robust to a few disputed cases but should not be read as a general property of all moderation systems.
• Re-judging was done by the operators of the service, who have an interest in the outcome. The published enforcement rules and the appeal process at lovora.ai/appeal are the external check.
• Model names and prompts are not published: the prompts because publishing them would help people evade them, the model identities because they change and the finding is about model choice in general.