The Model Is Not the System

Poisoning a model is easy. Compromising an enterprise with one is a considerably longer architecture diagram.

The Headline Got Me

Frank Landymore’s Futurism article on Katie Paxton-Fear’s work is worth reading. The core finding is not fake and not silly: a model was intentionally fine-tuned with a very small set of malicious examples and then induced to produce vulnerable code. That is a legitimate AI supply-chain security concern.

It is also not the same thing as demonstrating an inexpensive, end-to-end enterprise compromise.

Congratulations. You modified a model you controlled and it behaved the way you trained it to behave. Interesting. Now let’s draw the rest of the architecture.

That distinction is not pedantry. It is the difference between a model artifact and a production AI capability. The research tells us something important near the top of the chain. Enterprise risk lives across the whole chain.

This is also why I do not read the story as anti-research or anti-open-source. A security researcher intentionally modifying a model to show a supply-chain hazard is doing useful work. The trouble starts when the finding is flattened into a simpler story: poisoned model exists, therefore enterprise compromise is cheap. That skips the architecture where most of the risk either becomes real or gets contained.

The Model Does Not Get the Final Word

A lot of model-risk commentary quietly assumes that model output flows straight from weights into consequence. That is sometimes true in toy demos and terrifyingly sloppy prototypes. It should not be true in a governed enterprise.

Modern AI systems tend to wrap models in approved catalogs, provenance checks, artifact validation, evaluation, inference platforms, identity, authorization, gateways, policy enforcement, tool permissions, guardrails, observability, human review, static analysis, dependency scanning, tests, and CI/CD controls.

None of that makes model poisoning harmless. An AI gateway is not a model MRI. It generally cannot stare into billions of learned parameters and announce, with confidence, that a behavioral backdoor is hiding behind token number 47,000 on a rainy Tuesday.

But a gateway is also not the only control. A poisoned model has to become the model someone discovers, trusts, approves, deploys, integrates, invokes, grants permissions to, and then lets influence a real downstream action. The architectural question is not merely, “Can the model generate malicious code?” It is, “Can that behavior survive the delivery, consumption, review, and deployment chain and produce an actual consequence?”

Agentic systems make this more important, not less. If the model only suggests code to a developer, the attack path runs through human judgment and the software-delivery pipeline. If the model is embedded in an agent with repository access, shell access, ticket access, cloud credentials, or deployment permissions, the path is shorter and much more interesting. The difference is not the model by itself. The difference is the authority the system wraps around the model.

That is where architecture should spend its paranoia budget. Models can say dangerous things. Systems decide whether dangerous things can act.

A Model Is an Artifact, Not the System

A model is usually a set of files and configuration: weights, tokenizer, config, adapters, metadata. It does not execute itself. A .safetensors file sitting on disk has never compromised production by itself. It needs a runtime, a serving path, prompts, context, tools, identity, data, network access, and output consumers.

That does not mean models cannot be poisoned. Data poisoning, model poisoning, malicious fine-tuning, and behavioral backdoors are real machine-learning security concepts. Semgrep’s underlying discussion is useful precisely because it emphasizes how hard models are to inspect compared with normal software artifacts.

The objection here is architectural, not semantic.

Traditional CVEs remain appropriate for vulnerabilities in inference engines, model loaders, libraries, frameworks, operating systems, APIs, serving platforms, orchestration systems, and agent frameworks. Those are software components with software failure modes.

A behavioral backdoor encoded into model weights is different. It may be better governed through provenance, lineage, artifact integrity, supply-chain trust, independent evaluation, behavioral testing, reproducibility, and model governance. A model can absolutely be malicious. So can a shell script. Neither becomes an enterprise architecture by sitting in a directory.

The Interesting Part Starts After the $100

The experiment tells us that a malicious actor with control over training or fine-tuning may be able to cheaply encode undesirable behavior. Security teams should care. Model lineage should not be a vibe check with a download button.

But the cost of creating a malicious artifact is not the cost of executing a successful enterprise attack.

An attacker still needs some combination of distribution, trust, adoption, deployment, useful trigger conditions, permissions, downstream acceptance, control bypass, and production consequence. In a coding-assistant scenario, the backdoored model may need to generate exploitable code, have that code accepted by a developer or agent, survive review, pass tests, avoid SAST and SCA signals, land in a deployable branch, pass CI policy, and reach production in a useful path.

Could that happen? Yes. Bad controls, rushed teams, shadow AI, overtrusted copilots, auto-merge workflows, and tool-rich agents can compress that chain in ugly ways.

But that is exactly the point. The meaningful enterprise risk depends on the architecture around the model. If your delivery path lets untrusted generated code move straight to production, your problem is larger than open weights. Your architecture is already asking to be taught manners.

The $100 number is still useful. It tells us the barrier to creating a suspicious artifact may be low. That should change how casually enterprises treat random model downloads, adapters, fine-tunes, and “just try this checkpoint” workflows. Cheap artifact creation raises the volume of possible poisoned artifacts.

It does not tell us the cost of distribution, trust-building, enterprise adoption, permission acquisition, control bypass, or consequence. Those are different ledgers. Collapsing them into one headline makes the threat sound cleaner than the work an attacker still has to do.

The Elephant in the GPU Room

Security concerns around open-weight AI are legitimate. They are also being argued at a moment when open weights challenge the economics of companies that monetize model access and inference.

Open-weight models can enable self-hosted inference, private deployment, model portability, local experimentation, fine-tuning, lower switching costs, alternate inference providers, enterprise sovereignty, potentially lower inference costs, and less dependence on a single API vendor.

Closed model-service architectures can be excellent. They can also concentrate control over the model, endpoint, access, pricing, upgrades, availability, and much of the operational dependency. That is not a moral failing. It is a business model.

So when “download the model” starts competing with “enter your credit card,” expect the philosophy of openness to become surprisingly complicated.

The economic stakes are no longer subtle. Business Insider reported on the current debate around Moonshot’s Kimi K3 and Dean Ball, OpenAI’s head of strategic futures, including his “full AI communism” framing for a world dominated by open-weight models. Ball later clarified, according to that reporting, that he was predicting regulatory pressure rather than recommending it. OpenAI has also separately said its official policy views should be judged by what it says publicly, not by personal political activity or outside commentary.

That context does not invalidate security research. It does mean architects should keep two thoughts in their heads at once: open-weight model risk is real, and the institutions warning about it may also have strong economic preferences about where inference happens.

Enterprises should care about that because architecture decisions are also dependency decisions. A closed API can reduce some operational burden and move some risk to a vendor. It can also reduce visibility, portability, and bargaining power. An open-weight deployment can increase local responsibility and supply-chain work. It can also give the enterprise a stable artifact it can hash, freeze, test, red-team, deploy privately, and move between runtimes.

Neither posture is automatically virtuous. Both can be lazy. Both can be governed well. The serious discussion is not open versus closed as a bumper sticker. It is which dependency model gives the organization enough control, evidence, and accountability for the use case.

The Research Is Not Wrong

Malicious models create real risk: unknown lineage, malicious fine-tunes, compromised repositories, untrusted adapters, targeted backdoors, model substitution, weak provenance, and difficult behavioral inspection. We cannot reliably inspect billions of learned parameters and fully predict behavior. That is a serious problem.

The answer is not “open weights bad.” Closed models do not magically remove questions about training data, fine-tuning, behavioral backdoors, lineage, alignment, and evaluation. Customers often have less direct visibility into those systems.

The answer is better architecture: model provenance, trusted or signed artifacts, approved registries, lineage tracking, independent evaluation, adversarial testing, behavioral regression testing, constrained deployment, least privilege, output validation, observability, and downstream software-security controls.

A mature program should make the happy path boring. Teams should know which models are approved, where they came from, what changed between versions, who can deploy them, which tools they can reach, what logs exist, and what tests must pass before the system earns more authority. That will not catch every backdoor. Nothing will. It does make the difference between “we downloaded a thing” and “we operate an AI system.”

Open weights do not automatically make a model safe. They can give enterprises the ability to possess the exact artifact, hash it, version it, evaluate it independently, constrain deployment, control upgrades, switch runtimes, and host privately.

The model is not the system. It is one artifact inside the system.

Security researchers are right to ask whether we can trust that artifact. Architects should ask the next question: what would it actually take for that artifact to hurt us?

That answer is considerably larger than a weights file, ten malicious examples, and a $100 GPU bill.