{"id":341,"date":"2025-10-02T10:31:50","date_gmt":"2025-10-02T10:31:50","guid":{"rendered":"https:\/\/logsmith.io\/?p=341"},"modified":"2025-10-06T13:11:39","modified_gmt":"2025-10-06T13:11:39","slug":"detection-is-broken-because-your-logs-are-a-mess","status":"publish","type":"post","link":"https:\/\/logsmith.io\/index.php\/2025\/10\/02\/detection-is-broken-because-your-logs-are-a-mess\/","title":{"rendered":"Detection Is Broken Because Your Logs Are a Mess"},"content":{"rendered":"\n<p><strong>Most of the time when a team says &#8220;our detection rules aren&#8217;t working,&#8221; it&#8217;s not the logic that&#8217;s broken \u2014 it&#8217;s the data.<\/strong><\/p>\n\n\n\n<p>I&#8217;ve seen this pattern across government, defence, financial services, and MSSPs: detection logic gets blamed, dashboards get reworked, but the real issue is buried in the telemetry pipeline.<\/p>\n\n\n\n<p>If your logs are noisy, incomplete, or inconsistent \u2014 no amount of tuning will help.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">\ud83d\udd0d The Real Problem: Your Logs Can\u2019t Be Trusted<\/h3>\n\n\n\n<p>You can\u2019t build detection rules on sand. And a surprising amount of modern SOCs are doing just that \u2014 relying on:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Fields that don\u2019t exist<\/li>\n\n\n\n<li>Timestamps that drift or break correlation<\/li>\n\n\n\n<li>Event types that vary depending on the ingestion route<\/li>\n\n\n\n<li>Vendors that overload <code>message<\/code> fields with junk<\/li>\n\n\n\n<li>CIM mappings that don\u2019t match what the rule expects<\/li>\n<\/ul>\n\n\n\n<p>And this isn\u2019t theoretical \u2014 it\u2019s the kind of issue I get called in to fix.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">\ud83e\udde0 What This Looks Like in Practice<\/h3>\n\n\n\n<p>You\u2019ve probably seen these symptoms before:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Null fields<\/strong> in your SIEM when the parser fails silently<\/li>\n\n\n\n<li><strong>Multiple field names<\/strong> for the same value across sources (<code>src<\/code>, <code>source_ip<\/code>, <code>sourceAddress<\/code>)<\/li>\n\n\n\n<li><strong>Cribl pipelines<\/strong> that over-normalise or flatten data too early<\/li>\n\n\n\n<li><strong>Over-indexed logs<\/strong> \u2014 bloated, unreadable, but full of \u201ceverything just in case\u201d<\/li>\n\n\n\n<li><strong>Rules that alert constantly<\/strong>, but no one trusts them enough to act<\/li>\n<\/ul>\n\n\n\n<p>Sound familiar?<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">\ud83d\udd27 What You Can Do About It<\/h3>\n\n\n\n<p>Start small. Here\u2019s what works when I get dropped into noisy environments:<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">1. <strong>Check Field Extractions Early<\/strong><\/h4>\n\n\n\n<p>If you&#8217;re in Splunk, use <code>extract reload=T<\/code> and test with known events. In LogScale, inspect <code>parse_*<\/code> logic line by line.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">2. <strong>Build Baselines with Outlier Logic<\/strong><\/h4>\n\n\n\n<p>Don&#8217;t trust &#8220;this looks fine&#8221; \u2014 use stats to prove it. Look for volume drops, key field nulls, and field cardinality changes.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">3. <strong>Align on What Good Logs Look Like<\/strong><\/h4>\n\n\n\n<p>Build a mini log quality standard with your team. Pick a few sources and define: &#8220;This is good. This is what we trust.&#8221;<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">4. <strong>Custom Data Models Are Fine<\/strong><\/h4>\n\n\n\n<p>Don\u2019t force everything into CIM if it hurts more than it helps. I\u2019ve built client-specific models that worked 10x better for their pipeline \u2014 and still mapped to ES when needed.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">5. <strong>Review Detection Expectations<\/strong><\/h4>\n\n\n\n<p>If a rule expects <code>src_ip<\/code>, but the log sends <code>sourceAddress<\/code>, either map it properly \u2014 or adjust the rule. Don\u2019t let it drift.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">\ud83e\udded The Point Is This:<\/h3>\n\n\n\n<p><strong>Detection isn\u2019t just about rules. It\u2019s about trust.<\/strong><br>You need to trust what your system is telling you \u2014 or your analysts will stop listening.<\/p>\n\n\n\n<p>If you don\u2019t trust your logs, you\u2019ll never trust your alerts.<br>And if you don\u2019t trust your alerts, you\u2019ll never respond fast enough.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">\ud83d\udcac Want Help Fixing the Mess?<\/h3>\n\n\n\n<p>This is what I do \u2014 dive into noisy pipelines, broken extractions, and detection logic that\u2019s lost its edge.<\/p>\n\n\n\n<p>\ud83d\udcec <a href=\"https:\/\/logsmith.io\/index.php\/contact\/\" title=\"Contact\">Get in touch<\/a> \u2014 or explore more of my work.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Most of the time when a team says &#8220;our detection rules aren&#8217;t working,&#8221; it&#8217;s not the logic that&#8217;s broken \u2014&#8230; <\/p>\n<div class=\"art-el-more\"><a href=\"https:\/\/logsmith.io\/index.php\/2025\/10\/02\/detection-is-broken-because-your-logs-are-a-mess\/\" class=\"art-link art-color-link art-w-chevron\">Read more<\/a><\/div>\n","protected":false},"author":1,"featured_media":242,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"iawp_total_views":16,"footnotes":""},"categories":[27],"tags":[],"class_list":["post-341","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-insights"],"acf":[],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/logsmith.io\/index.php\/wp-json\/wp\/v2\/posts\/341","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/logsmith.io\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/logsmith.io\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/logsmith.io\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/logsmith.io\/index.php\/wp-json\/wp\/v2\/comments?post=341"}],"version-history":[{"count":2,"href":"https:\/\/logsmith.io\/index.php\/wp-json\/wp\/v2\/posts\/341\/revisions"}],"predecessor-version":[{"id":350,"href":"https:\/\/logsmith.io\/index.php\/wp-json\/wp\/v2\/posts\/341\/revisions\/350"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/logsmith.io\/index.php\/wp-json\/wp\/v2\/media\/242"}],"wp:attachment":[{"href":"https:\/\/logsmith.io\/index.php\/wp-json\/wp\/v2\/media?parent=341"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/logsmith.io\/index.php\/wp-json\/wp\/v2\/categories?post=341"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/logsmith.io\/index.php\/wp-json\/wp\/v2\/tags?post=341"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}