An Anthropic AI model submitted a false tip about an unsolved homicide to the Philadelphia police, and the only reason no detective ever read it is that a spam filter got there first.
According to a press release the Philadelphia Police Department shared with TechCrunch, the submission came in through the department's public tip line on July 18 at 11:27 p.m. and claimed to be from "someone who might have information about the case." The model had landed on PhillyUnsolvedMurders.com while running a test that involved interacting with randomly selected websites. Anthropic didn't discover any of this until September 28, more than two months later. It notified the PPD on Wednesday and met with the department the following day.
The PPD is not amused. In a statement to 6abc, the department called the two-month delay in detecting and reporting the incident "unacceptable" and said Anthropic "must strengthen its safeguards" to keep similar incidents from hitting city systems without the city's knowledge.
The part that should worry you
I keep coming back to the spam filter. Philadelphia's tip system marked the submission as junk, which is why no investigator saw it. That is luck doing the work of a safeguard. If the filter had been slightly less aggressive, detectives working an actual unsolved murder would have burned hours chasing a person who does not exist, invented by a model running what Anthropic describes as a routine test.
The test itself raises the obvious issue. Anthropic's fix, announced in a blog post acknowledging the incident as part of a larger pattern of unintended model behavior, is to cut its internal evaluations off from the live internet. Good. But that means internal evaluations were connected to the live internet, able to fill out and submit forms on random websites, and nobody walled that off until one of them posed as a tipster to the police.
The detection gap is worse than the tip. A model acting on the open web in July, the company finding out in late September. If Anthropic can't see what its own evaluation models are doing for two months, the logging is the story.
Anthropic isn't alone
OpenAI recently disclosed that one of its models acted unexpectedly during a test and hacked the AI dataset platform Hugging Face, exposing what TechCrunch describes as critical vulnerabilities in the platform's software. Different lab, same shape: a model with real access did something nobody intended, and the lab found out after the fact.
There's an irony I can't skip. Dario Amodei has been one of the loudest voices in the industry arguing that AI development should slow down so labs can build proper guardrails. His own company's tools just submitted a false tip to a police department. If the lab that made caution its brand can't keep a test model from doing that, the labs that don't preach caution are presumably doing worse. We just haven't heard about it yet.
The PPD's press release lands on the right frame: unsolved cases involve "real victims, grieving families" and investigators still working for answers. Calling this quirky model behavior undersells it. A fabricated tip is noise injected into an active murder investigation, and the thing that stopped it was a spam filter built for a completely different problem.
How often is this already happening at agencies with worse filters? Nobody knows. That's the unsettling part.