AI Is Now Building Software and Attacking It In the Same Cycle
Article

AI Is Now Building Software and Attacking It In the Same Cycle

Three recent stories show AI systems writing code, compromising developer pipelines, and auditing each other's work, collapsing the time developers have to react.

ManishankarOctober 22, 20265 min read

Photo: TechCrunch

The developer toolchain is being automated from both ends at once. AI systems are now writing production code, attacking the pipelines that ship it, and generating the infrastructure needed to verify what other AI systems did. The common thread across these three stories is a compression of the intervals developers have historically relied on: the time between research and release, between intrusion and detection, and between delegation and accountability.

For US engineering organizations, that compression is the story. Each of these developments shifts an assumption that developer workflows were quietly built on.

The model that wrote itself

Recursive's release of Flint, a free coding model sized to run on a laptop, is notable less for its footprint than for its provenance. As TechCrunch reported, the startup says the model was largely built by its own AI research system, framed as an early test of AI that improves itself.

On the Developer & Coding beat, the relevant detail is that the artifact under test is a coding model, not a general-purpose assistant. Coding models are unusually well suited to this kind of self-directed improvement loop because their output, source code, can be executed, tested, and scored without human judgment at every step. That is what makes the claim plausible enough to log, and also what makes it hard to evaluate from outside. A model that produces working code can generate its own training signal.

The distribution choice matters for US companies as well. A free model that runs locally on a laptop lowers the cost of entry for individual developers and small teams, but it also means the tooling escapes the vendor-mediated oversight that cloud-hosted coding assistants enforce. Firms that standardize on hosted models partly because of the telemetry and access controls they provide will find a different calculus with locally-run models that arrive with self-improvement claims attached.

Attackers no longer wait for a human

BleepingComputer's examination of real-world attacks, covering stolen GitHub tokens, malicious AI installers, and MFA phishing, makes a specific point that should be read as a developer-pipeline warning. Attackers can steal credentials and begin exfiltrating data within minutes, often before security teams can respond. The piece argues that machine speed attacks require machine speed detection and response, with automated containment as the practical answer.

Two of those three attack categories sit squarely inside the Developer & Coding remit. Stolen GitHub tokens are source-control credentials, which means a compromised token is a compromised pipeline. Malicious AI installers target developers who are accustomed to pulling packages and models from emerging, lightly-vetted sources, a habit that the local-model trend above will reinforce rather than curb.

The uncomfortable symmetry is worth stating plainly. The same period that produced a model reportedly built largely by AI produced documented attacks that exploit developer credentials and AI-themed installers. Automation on the build side and automation on the intrusion side are advancing on the same timeline, and the defensive advice, automated detection and containment, concedes that human-paced review cannot keep up at the front of the incident.

For US companies, this reframes a familiar investment question. Security budgets have generally treated detection and response as a layer above engineering. The BleepingComputer material suggests the pipeline itself is now the exposure surface, and that the response window is measured in minutes. Teams that still require human approval for containment actions at every step are, by design, slower than the attacks described.

Who audits the agents

Action State Group's launch, reported by SiliconANGLE, addresses the third side of the triangle. The company is releasing open-source tools, including EvidenceBook for tamper-evident records and Terms Packs for evaluating agent behavior, so businesses can independently check what AI agents did. The stated motivation is trust as automated systems begin buying, selling, and delegating work across corporate boundaries.

This is the accountability counterpart to the first two stories. If coding models can improve themselves and attackers can act at machine speed, the audit record of what an automated system actually did becomes the only durable artifact. Tamper-evident logging is not a novel idea in enterprise software, but applying it to agent behavior rather than human actions is a meaningful shift in what developers are being asked to build. Open-sourcing the verification layer is also a distribution strategy: records are only useful if the counterparty across a corporate boundary can check them with the same tooling.

What this means for US engineering teams

The three stories point at a single operational change. The review gates that US software organizations have relied on, code review before merge, human triage before containment, manual sign-off on delegated tasks, all assume that the volume and speed of the actors on either side is roughly human. The Recursive claim pushes on the first assumption, the Exaforce material on the second, and Action State's tools acknowledge that the third is already gone.

There is a market consequence too. Tooling that verifies agent behavior and tooling that contains incidents automatically are both positioned as infrastructure rather than features, and both are being offered, at least in part, as open source. For US buyers, that lowers the cost of adopting a verification layer but raises the question of who maintains it. For US vendors, it means the defensible position is less likely to be the logging format than the integration into existing pipelines.

The claims themselves deserve scrutiny. A self-improving coding model is difficult to verify externally, and a free local model with that framing will attract both enthusiasm and skepticism. The security material describes observed attacks, which carries more evidential weight than a vendor's argument for automation. The audit tooling is new and unproven in production across corporate boundaries.

What to watch

Three concrete things follow from the material above. First, whether Recursive publishes enough methodology for outside developers to assess the self-improvement claim, or whether Flint is judged purely on benchmark output. Second, whether US engineering organizations move automated containment earlier in the pipeline in response to the credential and installer attacks BleepingComputer documented. Third, whether EvidenceBook and Terms Packs see adoption across corporate boundaries, which is the only test that matters for a verification format whose value depends on the other party using it.

All three are unresolved. What is settled is that the intervals developers once counted on are shrinking, and the tooling to manage that is being built while the pressure mounts.

More on this beat: Software on TechManNews.

#AI coding models#developer security#agent verification#supply chain#open source tooling#enterprise software

Newsletter

Get Tech News in Your Inbox

The latest AI, gadgets, software and startup stories from TechManNews, delivered every morning - free.