Detection Is Broken Because Your Logs Are a Mess

Most of the time when a team says “our detection rules aren’t working,” it’s not the logic that’s broken — it’s the data.

I’ve seen this pattern across government, defence, financial services, and MSSPs: detection logic gets blamed, dashboards get reworked, but the real issue is buried in the telemetry pipeline.

If your logs are noisy, incomplete, or inconsistent — no amount of tuning will help.

🔍 The Real Problem: Your Logs Can’t Be Trusted

You can’t build detection rules on sand. And a surprising amount of modern SOCs are doing just that — relying on:

• Fields that don’t exist
• Timestamps that drift or break correlation
• Event types that vary depending on the ingestion route
• Vendors that overload message fields with junk
• CIM mappings that don’t match what the rule expects

And this isn’t theoretical — it’s the kind of issue I get called in to fix.

🧠 What This Looks Like in Practice

You’ve probably seen these symptoms before:

• Null fields in your SIEM when the parser fails silently
• Multiple field names for the same value across sources (src, source_ip, sourceAddress)
• Cribl pipelines that over-normalise or flatten data too early
• Over-indexed logs — bloated, unreadable, but full of “everything just in case”
• Rules that alert constantly, but no one trusts them enough to act

Sound familiar?

🔧 What You Can Do About It

Start small. Here’s what works when I get dropped into noisy environments:

1. Check Field Extractions Early

If you’re in Splunk, use extract reload=T and test with known events. In LogScale, inspect parse_* logic line by line.

2. Build Baselines with Outlier Logic

Don’t trust “this looks fine” — use stats to prove it. Look for volume drops, key field nulls, and field cardinality changes.

3. Align on What Good Logs Look Like

Build a mini log quality standard with your team. Pick a few sources and define: “This is good. This is what we trust.”

4. Custom Data Models Are Fine

Don’t force everything into CIM if it hurts more than it helps. I’ve built client-specific models that worked 10x better for their pipeline — and still mapped to ES when needed.

5. Review Detection Expectations

If a rule expects src_ip, but the log sends sourceAddress, either map it properly — or adjust the rule. Don’t let it drift.

🧭 The Point Is This:

Detection isn’t just about rules. It’s about trust.
You need to trust what your system is telling you — or your analysts will stop listening.

If you don’t trust your logs, you’ll never trust your alerts.
And if you don’t trust your alerts, you’ll never respond fast enough.

💬 Want Help Fixing the Mess?

This is what I do — dive into noisy pipelines, broken extractions, and detection logic that’s lost its edge.

📬 Get in touch — or explore more of my work.