Skip to content

Blast Radius Planning: Why Your Intelligence Pipeline Needs a Failure Budget

T. Holt T. Holt
/ / 4 min read

Every pipeline fails eventually. The question nobody wants to answer during the design phase is: when it fails, how much does it take down with it?

A woman coding on a laptop in a modern office environment with multiple monitors. Photo by Christina Morillo on Pexels.

Most intelligence engineering teams spend their budget on prevention. Harder access controls, more validation, tighter schemas. All good things. But prevention is a bet that you'll catch every failure mode before it happens, and if you've worked in this field for more than a year you know that bet loses eventually. Someone's credential gets phished. A dependency gets poisoned. A collection node gets burned by the target it was watching. The event itself is almost never the real disaster. The real disaster is what the event was connected to.

Blast radius planning asks a different question than threat modeling. Threat modeling asks what could go wrong. Blast radius planning asks: given that something will go wrong, what's the smallest version of wrong we can engineer for?

The coupling problem

Intelligence pipelines accumulate coupling the way old houses accumulate wiring nobody documented. A collection module gets wired into an enrichment service. That service feeds a scoring engine. The scoring engine writes to a shared knowledge graph that three other teams query. Nobody designed this as one system, but it behaves like one, and a compromise or corruption in any node now has a path to every other node.

The fix isn't more documentation of the coupling. It's reducing it. That means:

  • Hard boundaries between collection, enrichment, and analysis layers, enforced at the network and permission level, not just the org chart
  • Separate credential scopes per pipeline stage so a leaked token in collection can't touch the analyst-facing graph
  • Write-once, append-only logs between stages so a compromised downstream service can't retroactively poison upstream history
  • Rate limits and circuit breakers between every major hop, so a runaway or compromised node gets throttled instead of flooding everything behind it
graph TD
    A[Collection Node] -->|scoped token, rate limited| B[Enrichment Service]
    B -->|append-only write| C[(Knowledge Graph)]
    C -->|read-only, separate creds| D[Analyst Query Layer]
    E{Circuit Breaker} --> B
    E --> C

Setting a failure budget

Site reliability teams outside the intel world have used error budgets for years: you decide in advance how much downtime or degradation is acceptable, and you spend it deliberately instead of treating every incident as a five-alarm fire. Intelligence operations can borrow this, with a twist. The budget isn't just about uptime. It's about exposure.

Ask: if this node gets compromised today, how many downstream systems does the adversary reach? How many source identities get exposed? How much historical data becomes suspect? Put a number on it, even a rough one, and then treat any architecture change that increases that number as a regression, the same way you'd treat a performance regression in a normal software shop.

Teams that skip this step tend to discover their blast radius the hard way, usually during an actual incident, when someone realizes the compromised scraper had write access to the same database as the finished-product repository. That's not a hypothetical; it's the default state of a pipeline built incrementally without anyone explicitly deciding how far failure should be allowed to travel.

Testing the radius, not just the defense

Chaos engineering gets talked about in this space as a way to break things on purpose. Blast radius planning is the design discipline that makes chaos testing meaningful. There's not much point injecting a simulated compromise if your systems are so tightly coupled that every test produces the same answer: everything falls over.

Run the exercise differently. Pick a node, assume it's fully compromised right now, credentials and all, and trace every path an adversary could take from there. If the answer touches more than one or two adjacent systems, you don't have a resilience problem yet. You have a containment problem, and those get solved at the architecture table, not in the incident retro six months from now when it's too late to matter.

Get Intel DevOps in your inbox

New posts delivered directly. No spam.

No spam. Unsubscribe anytime.

Related Reading