When “I don’t want your texts anymore” doesn’t say STOP

The problem

Customers who want to stop receiving texts don’t always type STOP. They write what they mean. The keyword system saw no keyword and kept them subscribed.

The opposite mistake has a cost too. “Don’t stop the sales emails” and “how do I cancel my order” contain keywords and are not opt-outs. A fix that removes people who still want messages would lose revenue.

What I designed

Keyword rules still run first because they are fast and exact. Everything else goes to a small, cheap model that returns a label and a confidence. The model never opts anyone out by itself. Its confidence decides what happens next.

The tradeoff behind the thresholds

A missed opt-out is worse than a wrongly removed subscriber: one is a compliance and complaint problem, the other is a lost sale that confirmation can often recover. So the automatic threshold is high, and uncertain cases get a question or a human instead of a guess. The reasoning and the failure-mode table are in the risk doc of the repo.

The prompt also treats each reply as untrusted data. A customer can’t instruct the classifier, and the output is limited to three labels.

How I tested it

Real Maestra data is confidential, so I generated 450 labeled replies in English, Russian and Spanish. The set includes the hard cases: refusals with no keyword (135), keyword traps like “cancel my order” (45), and ambiguous replies like “not interested” (42). The headline metric is recall on opt-outs with no keyword, since that is the gap in the old system. The baseline is the keyword rules. I split the data by phrase family into a dev set for tuning (275 messages) and a held-out test set (175), because many messages are small variations of one phrase and a random split would leak.

Results

Overall opt-out precision and recall went from 0.56 and 0.30 to 0.99 and 0.92. On the held-out test messages, recall on opt-outs with no keyword is 84% (42 of 50, plausible range 71% to 92%), so the headline number has real uncertainty at this sample size.

What routing did with the 195 true opt-outs

At the 0.90 threshold no non-opt-out reply was opted out automatically. The only false positive, “Quit it with the flash sales lol”, scored 0.75, so it triggered a confirmation message instead. The 15 human-review cases were borderline wording like “I’m done with these texts”, plus every “I never signed up for this garbage” reply, which the model scores as ambiguous. The next section covers the fix. The cost on the other side: 25 of 213 non-opt-outs, mostly deferrals like “Not now”, also go to review, which is wasted work for the queue. All figures in this section are from prompt version 1.

Choosing the automatic threshold

Share of true opt-outs handled automatically. The rest go to confirmation or review, so nobody is dropped. False opt-outs are 0.4% at thresholds up to 0.70 and 0% from 0.80 up. I set 0.90 in the design before running the test. The results support it: it sits well above the 0.70 level where the first false opt-out appears, and the cost is that about one in four opt-outs gets a confirmation text or a review first.

Reading the failures and fixing them

I read all 52 misses from version 1, wrote one sentence on each, and grouped them into causes. Most were not “the model is weak”:

• The prompt told the model to hedge. A rule calling vague dissatisfaction “ambiguous” pulled slang refusals like “get lost already” and “I’m done with these texts” down to 0.6, so they went to a person.
• The prompt lacked rules my labeling policy already had. Deferrals (“Not now”), “I never signed up” complaints and wrong-number replies had agreed answers that the model was never given. This explained 29 rows.
• A code bug, not a model error. Any message of three words or fewer containing a keyword skipped the model. So “Cancel my subscription” was opted out automatically at 0.99. The carrier convention is that the keyword must be the whole message, and the fix follows it.

I also wrote 104 harder messages (typos like “STPO”, sarcasm, look-alikes like “remove my card”, prompt-injection attempts, requests buried in long messages, threats). Then I rewrote the prompt against the dev split only and scored the held-out test split once. Version 2 was a dead end: one clause I added made the model over-cautious and the hard-set score fell from 59 to 54 correct. Version 3 removed it.

Cost is about $0.00009 per classified message (illustrative at 2026-11 list prices for the small model). Latency is about 1 second at the median and 2 seconds at the 95th percentile, which is fine because nobody is waiting on an opt-out.

Two open questions remain, flagged for a second labeler and compliance: “Enough already” messages that my labels call ambiguous but version 3 treats as requests to stop (some reach automatic opt-out), and whether “Delete my account” should remove SMS consent.

What I got wrong first

• A bug looked like a weak model. My first real run showed 35% recall. I had capped the model’s output length too low and a quarter of the answers were cut off and read as “ambiguous”. After the fix the number rose, and the lesson was to check the harness before judging the model.
• My labels were wrong. I had labeled angry replies like “Who the heck keeps texting me?” as opt-outs. The model called them ambiguous, and it was right: they don’t ask to stop. I relabeled them, moved the eight that explicitly asked (“delete my number”) back to opt-outs, and reran. Questioning the ground truth moved the headline metric more than any model change.
• The generator’s labels needed a human check. Claude wrote both the messages and their labels, so I labeled 70 of them blind and compared. We matched on every clear opt-out and non-opt-out (50 of 50) and disagreed on 11 messages, all in the generator’s “ambiguous” bucket: “Not now”, “Maybe later”, “Who is this?” and “I never signed up”. Agreement was 84% (kappa 0.76). I adopted my labels as a written policy, and the headline recall fell from 92% to 89% without any change to the model.

Try it

Pick a message to see what the classifier did with it. These are recorded answers from real runs of prompt version 3 on messages it had not been tuned on, including the ones it got wrong or routed to a person. Nothing here calls a model.

Limits and next steps

The rollout plan answers this by measuring on real traffic before acting on it:

• Shadow mode: the model classifies live replies but changes nothing; compare against support tickets and complaints.
• Human review: reviewers confirm the model’s calls and the thresholds get tuned on real data.
• Confirmation SMS, then automatic opt-out, enabled per client behind a feature flag with a kill switch. Each phase has exit criteria on recall and false opt-outs.

Not legal advice. A compliance review would need to confirm the opt-out rules for each market before launch.