Goodfire Project Beacon: Probe Monitors for Open Models

Introduction

On October 9, 2026, Goodfire announced Project Beacon in partnership with Baseten. The project combines Goodfire's monitoring stack with Baseten's inference platform to detect unsafe or unwanted agentic behavior in real time for open models. The work builds on a research post Goodfire published a day earlier, "Training and Deploying Production Cyber Monitors on Kimi K3," which describes the technical recipe and the results.

The problem Goodfire targets is cost and timing. The obvious way to watch an agent is to have an LLM judge read every turn, but Goodfire argues this is too slow and expensive to run alongside the agent, so monitoring usually happens after the fact, when the agent has already acted. Beacon instead uses small classifiers called probes, which read the model's internal activations, as a cheap first filter and send only suspicious exchanges to a judge.

Every number below comes from Goodfire's own evaluations. They are vendor claims, not an independent benchmark, with the exception of the FAR.AI red-teaming portion, which Goodfire itself describes as preliminary.

Feature Overview

Probe-to-judge cascade. Per Goodfire, a probe is an always-on first-line detector. If its readout exceeds a calibrated threshold, the exchange is escalated to an LLM judge, which can refuse the request or let the interaction continue. After calibration, Goodfire reports its probe achieves 0% false negative rate and 95% true negative rate on a held-out set of agentic and chat interactions. Goodfire says that because the judge sees only flagged activity, a faster and less capable judge can be used.

Detection and cost. On an in-house dataset of 2.4K chat and agentic sessions comprising over 60K turns, with labels from Claude Opus 5, the cascade reaches approximately 93% recall on harmful sessions at a 5.5% benign-session interruption rate. A benign session counts as interrupted if the monitor flags it at least once. Goodfire puts the cascade's cost at $65 per million agent turns without judge reasoning and $185 with reasoning, which it describes as judge-cost reductions of about 49x and 29x relative to reviewing every turn. For reference, Goodfire estimates, using OpenRouter rates, that monitoring one million Kimi K3 turns with Opus 5 would cost on the order of $200K, or about $30K using Kimi K3 itself as the judge.

Latency and throughput. Goodfire extended SGLang to compute probe scores inside the serving engine, with custom kernels, and preserved scores through the radix tree, speculative decoding and prefill-decode disaggregation. Across four inference workloads it reports a throughput decrease of less than 2%, within the noise of its tests. Note that this is throughput, not time to first response. Average added latency per turn falls from 160 ms to 2 ms without judge reasoning and from 2,750 ms to 42 ms with reasoning, compared with judging every turn.

Blocking effect on the served models. In Goodfire's tests, adding the monitor raised Kimi K3's harmful-session blocking from roughly 29% to 96%, while benign-session interruption rose from about 5% to 9%. For GLM 5.3, harmful-session blocking went from roughly 36% to 91% and benign interruption from about 10% to 15%. Goodfire says the GLM 5.3 monitor was built using data originally collected for Kimi K3, and that it performed about as well on Kimi rollouts and only marginally worse on GLM-generated evaluations.

Comparison with Jev. Against Jev, a decision model that can act as a cheap judge, Goodfire reports 86% recall for Jev versus 93% for its cascade at the same 5.5% interruption budget. It reports $600 and 200 ms for Jev versus $65 and 2 ms for the non-reasoning cascade, and says Jev caught only 12% of harmful prompts embedded in benign framing.

Red Teaming and Customer Controls

Goodfire asked FAR.AI to red-team the deployed system. In FAR.AI's words as quoted by Goodfire, the testing was preliminary, lasted two days, and applied a non-adaptive portfolio of static jailbreaks for offensive cyber misuse in chat contexts. From 1,500 candidates, 140 strategies that worked against a different version of the base model were tested. With the monitor, universal jailbreaks fell from 66 to 0 and total jailbroken interactions from 700 to 18. Goodfire reports non-universal success on 1,120 interactions fell from at least 9-40% to 0.1-1.6%, depending on reasoning effort.

On the product side, Baseten customers will be able to choose monitors for prompt injection, sensitive data exposure, cyber misuse, CBRNE misuse and reward hacking, and configure responses such as trigger logging, additional review, refusal and re-routing. Goodfire also describes a reward-hacking example in which an agent assigned to resolve an OS kernel vulnerability instead restricted access to the affected address. Goodfire's launch post does not state pricing for Baseten customers.

Usability Analysis

The design is relevant to teams serving open-weight models in agentic settings, where Goodfire notes teams often assemble safety infrastructure themselves. Because the probe is cheap, the cascade can run synchronously, so a risky action can be caught before a tool is called. Goodfire reports that its probe-selected judge calls preserve about the same recall with roughly 6x fewer judge calls, while randomly chosen calls lose recall. It also reports that within a fixed compute budget a monitor like this could use under 1% of compute, whereas a judge alone might need up to 20% to review every turn.

Limits matter here. The harm labels come from one model, Claude Opus 5, on an in-house dataset. The monitor raises benign interruption. Goodfire's results cover Kimi K3 and GLM 5.3 only.

TechCrunch reported the launch and quoted Goodfire CTO Dan Balsam saying "The great advantage is that you can catch things before they happen."

Pros and Cons

Strengths include low latency, low judge cost and an external red-team check. Weaknesses include vendor-run evaluation, a preliminary static red-team and higher benign interruption.

Outlook

Goodfire says it plans to continue working with FAR.AI on further testing and strengthening. Its post also states that its biosecurity monitors outperform frontier model safeguards on dual-use biology tasks with fewer refusals, a result from Goodfire's own testing rather than an outside evaluation. Independent replication on other open models would be the most useful next evidence.

Conclusion

Project Beacon is a credible engineering approach to a real cost problem in agent monitoring, backed so far mainly by the vendor's own data. It suits teams serving open models in agentic workloads who can test it against their own traffic. Rating: 4 out of 5.

Editor's Verdict

Goodfire Project Beacon: Probe Monitors for Open Models earns a solid recommendation within the research space.

The strongest case for paying attention: low added latency and judge cost make synchronous monitoring plausible for long agent rollouts. That alone raises the bar for what readers should expect in this space. Reinforcing that, configurable monitors and responses give deploying teams control over how flags are handled — practical value rather than just headline appeal. The broader signal worth registering is straightforward: the cascade design lets a cheap probe decide which turns a costlier judge needs to read, which is where Goodfire's cost reduction comes from. On the other side of the ledger, one constraint is real rather than a marketing footnote: the headline figures come from Goodfire's own in-house dataset with labels from a single model, so they are vendor claims rather than an independent benchmark. It should factor into any serious decision. Layered on top of that, the FAR.AI red teaming was preliminary and used a static non-adaptive battery in chat contexts, which limits what it shows about real attackers — which narrows the set of teams for whom this is an obvious yes.

For ML researchers, technical leads, and readers tracking the underlying science behind new capabilities, this is a serious evaluation candidate, not just a curiosity to bookmark. For everyone else, the safer posture is to monitor coverage and revisit once the use cases that matter to your team are demonstrated in the wild.