Security · SIEM · Data Architecture

Centralize Like You Mean It, Federate Like You Have To

Not "is federation the future?" It is a future, not the future. The far more useful question: when, precisely, does federated SIEM actually work — and when does it blow up in your face at 2 AM?

Anton Chuvakin & Usman Chaudhary

Security Advisor, Office of the CISO, Google Cloud · Field CISO, Google Cloud · August 2026 · ~9 min read

This article also appears on Anton on Security as "SIEM: Centralize Like You Mean It, Federate Like You Have To".

A steampunk illustration of a Master Security Intelligence Engine, with pipes and conduits pulling in EDR signals, cloud VPC flows, remote log outposts, and SaaS API event stores

What's inside

Where the debate stands — three years of tracking centralize-vs-federate, in one paragraph
Why full centralization is breaking — the four pressures pushing logs out of the vault
What federation actually solves — the real wins, and where the pitch is honest
The seven hidden costs of federation — speed, resilience, compliance, and what the demo hides
The three real architecture choices — and which one most organizations should pick
How to decide, source by source — seven tests for whether a log can safely be federated
Which telemetry goes where — a concrete hot-core / federated / cold-archive mapping
What to do about it — five moves for security leaders
The bottom line — our recommendation, and how to choose what to federate

Prologue: Three Years After "The End Is Nigh"

Back in 2023, one of us wrote "Log Centralization: The End Is Nigh?" — an admittedly incomplete-thought blog with a scary premise: after 20+ years of yelling "centralize your logs!" (the earliest surviving deck is from 2003), we may be running out of places where centralizing all the logs is feasible, workable, or even worth the pain.

The conclusion then was cautious: centralize as long as you can, in as many places as you can, and augment with some form of centrally defined, lightly managed, highly distributed collection. By 2025 the position got more contrarian: the SIEM of 2027 will be roughly "90% centralized / 10% federated," and anybody promising you the inverse is selling a demo, not an architecture.

This post is the practical sequel. Not "is federation the future?" (it is a future, not the future), but the far more useful question: when, precisely, does the federated SIEM actually work — and when does it blow up in your face at 2 AM?

The Breaking Point of the Centralized Vault

For more than two decades, SIEM tools ran on a simple covenant: collect all telemetry into one repository, pay for the ingest and the storage, normalize everything upfront into one grand schema, and query from one console. In the era of predictable on-premises networks this worked — and, frankly, it still works for a lot of organizations.

But multi-cloud sprawl, ephemeral infrastructure, hundreds of SaaS applications and now AI agents have strained it in four specific ways:

The ingest and duplication burden. If you are present in multiple public clouds at scale, you are very likely not collecting logs into one place in one cloud. Egress fees, redundant storage, and pipeline sprawl make that a questionable decision. Add a few hundred SaaS apps, and "one vault" becomes a very expensive fantasy.
The fragile taxonomy tax. Forcing thousands of log sources into one rigid, deeply nested data model creates brittle pipelines. This was largely known since the mid 2000s when the first schema-on-read vendors appeared. A minor upstream vendor format change silently breaks parser mappings, and your detection rules go blind without so much as a warning.
The volume-to-value problem. Some log types are hugely useful in bulk and for investigations but almost never trigger a detection on their own — DHCP leases, VPC flow logs, the long tail of DNS resolution logs. Many organizations simply stop collecting them because they are "too costly to centralize." That is not a data decision; that is a billing decision masquerading as one. (Making that tradeoff an explicit engineering discipline — cost per detection, which telemetry earns full-fidelity treatment — is something we've written about in FinOps for SecOps.)
The volume curve just bent. Log growth was already relentless; AI made it vertical. Recent State of Log Management 2026 research found AI workloads drove a 93% increase in log and telemetry volume in a single year, with 1 in 5 organizations seeing growth above 150% — and organizations now exclude an average of 86% of their log data just to manage cost. Read that again: most enterprises are already discarding the vast majority of their telemetry, not by security design, but by budget necessity. Agents generating machine-speed telemetry will not slow this down.

Here is the honest framing, unchanged since 2023: the problem isn't that the distributed approach is easy. The problem is that the centralized approach is getting harder as volumes, source counts, and geographic sprawl go up. And, as we keep saying in the output-driven SIEM context: if you collect, you pay. Somebody has to own the hard drives.

There is also another driver that is neither cost nor architecture: data sovereignty. For multi-jurisdiction and sovereign-cloud organizations, some telemetry legally cannot cross borders — residency mandates make centralizing certain logs not expensive but impossible. For that class of organization, federation is not a temptation to resist; it is a compliance requirement to engineer for.

The Federated Temptation

Into this gap stepped two families of technology alternatives:

Federated query platforms. Leave the telemetry where it lives — cloud object stores, SaaS vendor event stores, edge repositories, your uncle's flooded basement — and push the compute to the data via distributed indexing and schema-on-read.
The open security lakehouse. Decouple storage from analytics: keep structured logs in open formats (Apache Iceberg, Parquet, etc.) on cheap cloud storage and query them through engines you already have (BigQuery, Snowflake, Databricks, and friends).

Plus the classic third option that predates both — tiering: dump the "less useful" logs into cheap storage and pray to the security gods you never have to search them at speed.

The pitch is intoxicating: stop paying egress! stop duplicating data! just federate the search! It is also, in specific and bounded conditions, correct. The trouble starts when "specific and bounded" quietly becomes "default."

Comparison: where federation genuinely shines vs. what the demo does not show you — What the demo shows you — and what it doesn't.

The Dark Side of Federation (Read This Before You Sign)

Federated search sounds magical until you are investigating a breach at 2 AM. Here are the costs and risks you actually have to swallow:

It is slower than centralized — unless you architect specifically for speed (and then pay for it)

A federated query across three clouds and forty SaaS APIs is bounded by the slowest source, the tightest rate limit, and the coldest object-storage tier. It looks fast on a demo dataset sitting in one bucket. Cross-source joins on read are expensive by construction. The vendors know this, which is why the serious ones build distributed indexes at the source — but indexes must be built, refreshed, stored, and paid for.

And here is the trap from the 2023 post: if you deploy big indexers in every cloud, you haven't eliminated centralization — you've just created several smaller islands of it. That isn't inherently bad, but let's be honest about what you're doing: you aren't escaping the architectural tax of centralization, you're just trading one big central bill and management domain for three smaller ones that will each grow their own operational headaches over time.

You trade cost for resilience

This one is rarely on the slide. A centralized SIEM is one thing to harden, one SLA, one on-call rotation. A federated platform is a query engine whose answer depends on N independent sources being up, reachable, authenticated, and under quota — at the exact moment you need them.

You pay less for storage, and in exchange the overall resilience of your detection-and-response platform goes down. Every added source is an added dependency, and dependencies fail at the least convenient time, by definition.

Centralized platforms fail too — but that risk is priced, contractually owned, and covered by one SLA. In federation, you self-insure across N sources. In theory, people assume that "distributed systems" are somehow more resilient. In practice and in this case, they are clearly less so.

No assurance the logs are even there

If you simply hope the logs will be there when your decentralized query tool reaches for them, you will be disappointed a lot. Sources get compromised, and attackers delete local logs. SaaS retention windows expire. A well-meaning admin "cleans up" a bucket. Then your IR consultant finishes the engagement and says: "Sorry, not sure what happened here — there were no logs — but here is the $100K bill for all the things we tried." Centralization has a cost, but once you pay it, you reliably own the logs. Federation gives you a pointer, not a possession.

Compliance did not get the memo

Many mandates directly require collection and centralization. PCI DSS v4 Requirement 10.3.3, for one, expects audit logs to be promptly backed up to a secure and central log server (or other media that is difficult to modify). Security people love to mock regulations as outdated for the cloud era; in this case they are a stabilizing force, perhaps.

Yes, you can mitigate this in a federated model — object lock, versioning, WORM buckets, immutable retention policies, documented evidence that every source enforces them. But note who does that work: you, the client. The federated search vendor gives you a query layer; it does not give you an audit trail your QSA will accept, at least not without a stressful argument. Budget the engineering time — and the assessor's skepticism — accordingly.

Federated search is workable; federated analytics mostly isn't

Detection is not the same as search. Continuous complex event processing — stateful detection windows, multi-event sequences, streaming IoC matches at line rate — needs data flowing through one high-speed engine, normalized to something. Mapping blast radius and lateral movement across users, assets, and service accounts needs a persistent entity graph, not a multi-table join fired off on read.

If your algorithms rely on normalized logs, you will wait a very long time for all logs to be normalized "naturally" wherever they sit (OCSF or no OCSF). We have barely made centralized analytics work well; decentralized analytics is a research project, not a product category. In the age of AI-speed attacks, speed matters again.

Operational toil, and nobody to scream at

A natively designed, integrated SIEM is simpler to run than a multi-component stack you assemble at home. A DIY lakehouse-plus-federated-search-plus-detection-layer is a data platform, and data platforms come with data platform engineers. If you do not employ them, you are not building a federated SIEM; you are building a science project with a SIEM logo. And when it breaks, you lose the underrated benefit of a "single face to scream at."

AI agents do not make it less messy (enough)

AI agents genuinely help here in one specific way: they are patient. An agent can fan out slow federated queries in the background without a human staring at a spinner. But "I didn't save any logs from X — hey agent, go get me the logs from X" does not work in real life. Worse, watch for the nastiest failure mode: an agent that reports "nothing found" when the truth is "source unreachable." In a centralized system that distinction is obvious. In a federated one it is a silent false negative, and automation bias will make sure nobody questions it. Always require explicit status reporting from agents so that absence of evidence does not become evidence of absence.

The Architectural Spectrum

Yes, there is a middle path — but it is much closer to the centralized end than the vendor decks suggest. Three honest positions:

The SIEM architecture spectrum: fully centralized, centralized plus cheap tier, and hybrid — Centralize like you mean it, federate like you have to.

Two observations. First, the "classic tiering" position is where many organizations already live comfortably and should probably stay. Second, the jump from "hybrid" to "federation-first" is not a matter of degree — it flips who bears the assurance, compliance, and resilience burden from the platform to your engineering team.

When Can Federated Actually Work? The Criteria

Federation is not a wand; it is a tool for specific conditions. The discipline that matters is deciding — in writing, ahead of time — which bucket each source falls into, rather than discovering the answer mid-incident. For a given log source, federated/decentralized handling works well when all of the following hold:

The use is largely asynchronous. Deep-dive forensics, threat hunting, post-incident review — situations where a query that takes 20 minutes costs you patience, not the company. If an active attacker is moving laterally, you cannot afford an hourglass spinner.
The source is reliable, managed, and tamper-resistant. A robust SaaS platform with documented retention, or a cloud store with versioning, object lock, and retention policies you control and can prove. If the attacker who compromised the host can delete the log, that log is not federated — it is gone.
The volume-to-value ratio is terrible. Petabytes of flows and DNS queries you rarely touch but desperately need when a specific IP shows up in an alert.
The queries are targeted, not fishing. "All DHCP leases for MAC X on date Y" — yes. "Show me anything weird across everything" — no, that is what your hot core is for.
No compliance mandate requires a central, immutable copy of this data — or you have already built and evidenced the equivalent controls at the source.
Nothing in your real-time detection depends on it. Federated data is for context and investigation. The moment a detection rule needs it, it belongs in the core.
Somebody owns the "is it still there?" question. A central catalog of what lives where, with retention, ownership, and health checks — centrally defined, lightly managed, highly distributed. Without this, you have not federated your logs; you have misplaced them.

If any criterion fails for a given source, the pragmatic answer is boring: centralize that source.

The Pragmatic Hybrid: Mapping Telemetry to Tiers

Rather than an all-or-nothing choice, modern architectures converge on an integrated high-speed core for continuous detection and graph correlation, coupled with open lakehouse federation for on-demand investigation.

Mapping telemetry and use cases to centralized, federated, and cold-storage tiers — What belongs in the core — and what earns federation.

Strategic Takeaways for Security Leaders

Avoid dogmatic extremes — but anchor on centralized. Pure centralization creates cost and schema bottlenecks; a fully disconnected DIY federated stack trades those for operational complexity, lost resilience, performance surprises, and toil. Look for platforms that integrate fast streaming detection with flexible storage options, with the center of gravity firmly in the integrated core.
Separate hot detections from cold investigations, and expect AI to widen the gap. Route identity, endpoint, and control-plane data through real-time detection; stow voluminous low-signal telemetry in cost-effective open lakehouses. AI-driven detection will increase the pull toward the hot core — models correlating across identity, endpoint, and cloud events need the data in one place, fresh, and normalized. The federated tier is where AI agents go for context, at their own pace, with a hard rule that "unreachable" is never reported as "nothing found."
Inventory before you federate. Know which logs exist, where, for how long, under whose control, and with what immutability guarantees. If you cannot answer those questions for a source, you are not ready to leave it there.
Build on open standards — but "open" is not "free." Open formats preserve agility and keep security telemetry aligned with the enterprise data architecture. Somebody still runs the lakehouse, and that somebody works for you.
Test the 2 AM query. Before production, run your worst realistic investigation query across every federated source during a simulated outage of one of them. The result tells you more than any vendor benchmark.

The Verdict

The centralized approach to logs will work as long as it can and in as many places as it can — that sentence has survived three years and two blog posts unchanged, and we see no reason to retire it. The physics of cloud-scale data means we will augment the centralized brain with centrally defined, lightly managed, highly distributed collection and federated analysis. Fine. Just remember what you are buying: cheaper storage in exchange for assurance, resilience, speed, and compliance work that lands on your desk.

Choose what you federate wisely. Or prepare to explain either your cloud storage bill to the CFO, or your missing logs to the regulator — and only one of those conversations ends with a budget adjustment. We aren't going back to the 1980s where you need to telnet to see logs. But we are entering an era where every log has to earn its place in the center.

Centralize like you mean it, federate like you have to.