Choosing Sustainability Metrics That Survive a Multi-Year Red Team Campaign

When your red team campaign runs for years, not quarters, the metrics you pick on day one will either grow into a compass or rot into a distraction. I've watched teams start strong with elegant sustainability scores, only to see them abandoned after the first tool swap or leadership change. The problem isn't laziness—it's that most metric frameworks assume a stable baseline that rarely exists in real attack simulation.

So what does survive? This guide maps the choices that hold up under drift: decay-weighted scoring, adversary-agnostic indicators, and the uncomfortable truth that any metric needs recalibration before it hits year two. We'll cover foundations people confuse, patterns that work, anti-patterns that don't, and—most importantly—when to skip formal metrics altogether.

Where This Shows Up in Real Work

The slow burn of metric drift

A multi-year red team campaign is not one long battle—it's a hundred small skirmishes, each with its own rhythm. I have watched teams start strong, tracking mean time to detection with religious fervor. By month eight, that same metric collects dust. Why? Because the threat landscape shifted, the red team rotated operators, and the original baseline no longer matches reality. The tricky bit is—drift happens invisibly. No alarm rings when a metric becomes irrelevant. One quarter you're comparing apples to last year's oranges, and nobody notices until the annual review reveals an impossible trend line. The catch is that sustainability doesn't mean static repetition; it means building metrics that absorb personnel changes, tool swaps, and evolving adversary tactics without requiring a full reset.

Who actually reads these numbers?

Stakeholders talk past each other. The CISO wants a single number—are we safer than last quarter? The red team lead wants granular data—did that new detection rule actually fire during the credential dumping phase? The board wants cost-efficiency—how many dollars per finding? These three audiences can't share one dashboard. I have seen teams force a single "risk score" onto every slide deck, only to watch the board skip that page and the CISO ignore the underlying volatility. The gap between annual planning and multi-year reality is brutal. Annual plans assume linear improvement; multi-year campaigns produce jagged, non-linear returns. Good months followed by flat months. That hurts when you're defending a metric to a VP who expects a straight line upward.

'We spent eighteen months improving response time by forty percent, then a new phishing variant reset the clock. The metric was useless for a quarter.'

— former red team operator, private conversation

Why your planning horizon mismatches your metrics

Most teams design metrics for a 12-month cycle and then extend them to a 3-year campaign. Wrong order. A metric built for annual review can't survive the mid-campaign staff turnover, technology refresh, or shift in red team methodology. What usually breaks first is the denominator—number of attempts, active systems, user population—all drift. I once watched a team proudly report "detection rate held steady at 78%" while their user base silently doubled. Not malicious, just unnoticed. The metric held constant; the actual security posture degraded. Multi-year campaigns need metrics that track both numerator and denominator independently, and that requires discipline most teams skip during the sprint to launch. The odd part is—the teams that survive the full campaign are the ones who build slack into their measurement system: room to re-baseline, swap indicators, and absorb the inevitable bad quarter without declaring emergency.

Foundations Most Teams Get Wrong

Coverage vs. depth in long-term assessment

Most teams start with a sprawling dashboard. They track every login attempt, every phishing click, every patched endpoint. The board loves it. Then month eight hits, the red team goes quiet for six weeks, and the board sees green across the board. The seam blows out when the adversary returns with a credential they stole in month two — a stale backup token that never appeared on any coverage chart. You tracked breadth, not depth. A single credential pair sitting untouched for 200 days kills your posture, yet your risk register shows 'no alerts.' The trap is treating coverage as if it reveals adversary dwell time. It doesn't. Coverage tells you where you looked; depth tells you what you missed.

The fix is ugly and specific. Stop counting the number of detection rules. Start measuring the latency between initial access and triage for the three attack paths you actually dread. I have seen a SOC celebrate 95% coverage on phishing simulations — and lose control of a domain admin because the 5% blind spot was a silent mail-forwarding rule planted by a paid operator. That hurts. Depth says: can you replay how that token got exfiltrated, not just that you saw the login? Most teams skip this because it demands forensic narrative, not a bar chart.

Leading vs. lagging indicators under adversarial pressure

Standard wisdom says leading indicators predict failure; lagging indicators confirm it. In a multi-year campaign, that line dissolves. A leading indicator like 'mean time to detect' looks powerful — until the red team deliberately triggers a low-severity alert to calibrate your response speed while they exfiltrate logs through a cron job you never monitor. You celebrate the fast detection on the decoy. The lagging indicator — 'data lost' — arrives six months later, too late to matter. The odd part is: both metrics are technically correct. Both lie.

What holds up is a third category: friction indicators . Can the adversary re-authenticate after you rotate that token? Does a lateral move force them to touch a monitored service?

Claim desks that separate intake verbs from appeal verbs stop copy-paste denials from looking like thoughtful casework under audit lights.

The catch is friction indicators are harder to automate. You have to test them, not just graph them. A single rhetorical question for your team: if the adversary owns one domain-joined workstation today, how many hops can they take before you lock a door? If you can't answer within a week, your leading and lagging numbers are props.

‘We tracked detection time down to four minutes. The red team still owned us for nine months. The clock meant nothing because we never measured how fast they moved between rooms.’

— director of security operations, after a third-party exercise review

Flag this for penetration: shortcuts cost a day.

Flag this for penetration: shortcuts cost a day.

The myth of a single 'health score'

Every year, some vendor pitches a composite number. One score. Green. Red. Yellow. Executives love it. It's also the fastest way to kill a long-horizon simulation. A single score flattens two radically different realities: a network where the red team is inside your perimeter but hasn't moved, versus a network where the adversary is logging via a legitimate service account. Both can yield a '75 out of 100' on the dashboard. One is recoverable. The other is a quiet funeral. The problem is the average erases the line between 'we have time' and 'we're owned and don't know it yet.'

Teams revert to a single score because arguing about metric weight is exhausting. That said, the exhaustion is cheaper than the breach. I have watched a CISO drop a red team program because the 'health score' stayed green for two years — and the operator later revealed they had persistent command-line access for 14 of those 24 months. Wrong order. The score gave comfort, not truth. Fix this by banning composite scores from your simulation steering committee. Force every metric to carry its own raw data and its own caveat. Let the board squirm with uncertainty. That squirm is the only honest signal you have.

Patterns That Actually Hold Up

Decay-weighted scoring for aging findings

Most teams score findings the same on day one as on day 730. That's a quiet disaster. A weak TLS cipher found in week one matters less than the same cipher found after two years of patching cycles — unless the team never fixed it. I have watched campaigns where old findings buried new, more dangerous ones because severity scores stayed flat. The fix is decay-weighted scoring: every finding loses, say, fifteen percent of its severity weight per quarter unless re-validated or actively exploited. The odd part is — this forces honest triage. Old findings either get fixed, closed, or re-scoped. They don't sit in a zombie backlog poisoning the dashboard. The trade-off: you need tooling that supports custom weight curves. Without it, your team will revert to static scores within three months.

Adversary-agnostic baselines

Pick an adversary — any adversary — and your metrics will break when that actor fades. Ransomware groups pivot, state actors swap TTPs, and your carefully tuned baseline becomes a historical artifact. The pattern that holds up instead: adversary-agnostic baselines built from environmental friction. Measure how long it takes for a detected scan to trigger a response. Track credential churn rates across service accounts. Monitor the mean time to patch critical external infrastructure. These numbers don't care which group is knocking. That hurts when your CISO demands threat-specific reporting. But a baseline that survives tool swaps and actor rotations beats one that looks perfect for six months then quietly rots. The catch: these baselines require two to three quarters of clean data before they predict anything useful.

Composite indices that survive tool changes

'We swapped EDR vendors last quarter. Our entire risk score history became useless.'

— Platform engineering lead, after a twelve-month campaign

Single-vendor metrics are poison for multi-year work. The moment you change log aggregators, threat feeds, or scanning tools, your trend lines snap. Good composite indices solve this by blending three layers: environmental (patch latency, credential age, firewall rule count), behavioral (mean time to detect, incident re-occurrence rate), and operational (re-test cycle time, findings closed per sprint). No layer relies on a single data source. If your EDR vendor changes, the behavioral layer adjusts — you recompute the baseline against the new tool's data, but the environmental and operational layers stay intact. The pitfall: teams over-weight the behavioral layer because it feels real-time and flashy.

I have seen composites collapse because three out of four sub-scores came from the same SIEM query — one query change, and the whole index drifted. Build redundancy at the sub-index level. Accept that your composite will be less sensitive to short-term shifts than a single-vendor dashboard. That's the point. Sensitivity that decays inside eighteen months is noise, not signal.

Scale the weighting empirically. Start with equal weights across the three layers, then adjust quarterly based on which layer actually correlated with real incident severity in the previous cycle. Most teams skip this: they set weights once during planning and never touch them again. By month eight, the index is measuring something unrelated to the campaign's original threat model. Re-weighting is a chore. It's also how you keep the metric alive past year two.

Anti-Patterns and Why Teams Revert

Vanity metrics that look good on dashboards

I have watched teams plaster a wall with charts that never informed a single decision. Active alert count, ticketed findings per quarter, percentage of assets scanned — all green, all useless. The problem isn't the data; it's the absence of a forcing function. A dashboard that climbs upward feels like progress, but it measures throughput, not resilience. Most teams skip this: ask whether the metric would change what you do tomorrow. If the answer is 'no', you're decorating, not steering.

The catch is that vanity metrics are addictive. They give executives a number to cite in all-hands meetings. They make the security team look busy. Meanwhile, the real signal — how many hypotheses survived a four-month adversarial test — gets buried because it's ugly and it wiggles. One CISO told me, 'I need a chart that goes up and to the right.' That hurts. You can't sustain a multi-year red team campaign on a chart that never flatlines.

The odd part is—teams know this. They build the pretty dashboard first, then promise to add meaningful KPIs later. Later never arrives. The metric that matters most is often the one you can't put in a slide deck: 'Did we change our assumptions this quarter?'

'A metric that can't produce a fight is not a metric — it's a decoration.'

— red team lead, after his team's third dashboard redesign

Over-indexing on time-to-detect

Time-to-detect is a seductive number. Fast detection feels like control. But in a multi-year campaign, the obsessive focus on mean-time-to-detect (MTTD) creates a blind spot. Teams optimize for the easy stuff — obvious malware, noisy port scans — while the adversary shifts to low-and-slow exfiltration that sets off no alarms. Wrong order. You're celebrating your ability to catch the things you already know how to catch. That's not sustainability; that's pattern-matching.

Not every penetration checklist earns its ink.

Not every penetration checklist earns its ink.

What usually breaks first is the trade-off: pushing MTTD down often means raising false-positive rates until the SOC tunes out entirely. I have seen a team cut detection time by 40% while missing the one beacon that ran for eleven months. The metric went green. The network bled. The reversion comes when the noise drowns real signals, and operators quietly stop looking at the dashboard that made them look foolish.

A better question: 'How long does it take to understand what the attacker did after first contact?' Not detection — comprehension. That number rarely gets a dashboard slot because it's hard to automate and it makes leadership uncomfortable. But it survives a long campaign.

The silent slide back to finding counts

Teams revert to counting findings because findings are concrete. You found twelve new vulnerabilities this quarter — that's a story you can tell. The problem is that finding counts punish the wrong behavior. They incentivize low-effort scans, surface-level poking, and ignoring the deep structural weaknesses that take weeks to surface. I have watched a team abandon a promising sustainability framework because the boss asked, 'How many findings did you produce this month?' Three words, and three months of careful metric design collapsed.

The reversion is rarely dramatic. It creeps in. One team replaces a decision-quality metric with a volume metric for a single quarterly report. That report gets shared. The next quarter, the new metric is the standard. By the third quarter, nobody remembers the original purpose. The adversary, meanwhile, has learned exactly which seams the team stopped looking at. That's the real cost of metric abandonment — not the lost dashboard, but the blind spot you recreated.

Most teams skip this: build a contract with stakeholders upfront that finding counts will drop as the campaign matures. If you can't get that agreement, you will revert. Not because the old metric was better — because the organizational pressure to produce a rising number is stronger than the belief in a flat one.

Maintenance, Drift, and Long-Term Costs

Recalibration cycles and trigger events

Pick any metric on day one of a multi-year campaign and it will look stale by month eight. That's not a failure of design—it's the natural decay of relevance. I have seen teams set a 'time-to-detect' target of four hours, then watch the environment shift so their detection pipeline fires at ninety seconds for noise they never tuned. The threshold became meaningless. Recalibration should follow trigger events, not calendar dates: a major tool swap, a red team shift in TTPs, a false-positive spike that drowns real alerts. A quarterly review is fine for paperwork. Real maintenance happens when someone says, "This number stopped telling us anything useful three weeks ago." The odd part is—most teams keep reporting the old number anyway. Habit beats honesty.

The catch? Trigger-based recalibration imposes its own cost. Every adjustment introduces a break in data continuity. Compare month five against month seven and you might see a dip that's pure redefinition, not improved posture. That hurts cross-year trend lines. Some shops solve this by running dual calculations: the old metric for backward comparison and the new one for operational relevance. That doubles the mental overhead. Most teams skip this; they just re-anchor and pretend the past still lines up. Wrong order, but I have done it myself when a client needed a board deck in two hours.

Tool rotation and data continuity

Tools die mid-campaign. The EDR vendor gets acquired, the logging pipeline changes schema, someone rewrites the correlation engine from scratch. When the data source changes, the metric's foundation cracks. I fixed this once by keeping a raw log snapshot for six months so we could re-baseline after a sensor swap—but that required storage nobody budgeted for. The alternative is worse: you pretend the old tool still works and the new tool produces equivalent numbers. It never does. The acceptable drift band widens every time you paper over a tool change. After three rotations your 'time-to-detect' has no relation to actual response speed. It's a fiction held together by alignment charts and hope.

What usually breaks first is the denominator—the total attack surface or the volume of events. New tools count differently. One team I worked with switched from a signature-based IDS to a behavioral model. Their alert rate per host dropped 40% overnight. Not because the environment got safer. The new tool simply didn't fire on the same things. The metric screamed 'improvement.' It was artifact, not signal. You avoid this by explicitly mapping tool output to metric intent before the swap, but that means documenting intent while the campaign is running hard.

The hidden cost of metric debt

Metrics accrue debt just like code. Every skipped recalibration, every fudged threshold, every tool mismatch adds an invisible interest charge. Six months in, you have a dashboard that looks clean and decisions built on numbers that are quietly wrong. The cost shows up when an unexpected red team maneuver hits—your metrics said you were solid, but the seam blows out. Then you spend two weeks auditing the metric chain instead of fixing the vulnerability. That's the real price: not the time spent maintaining the metric, but the time spent rediscovering that the metric was never maintained.

Metric debt compounds faster than anyone admits. One skipped recalibration cycle doubles the effort to re-anchor later—especially if the original metric owner left the project. Documentation that seemed obvious in month one reads like archaeology by month fourteen. "Why did we set the alert threshold to 0.7 confidence?" Nobody remembers. The answer was probably 'felt right after coffee.'

“A metric you can't explain to a new hire is a metric that has already drifted beyond repair.”

— Observation from a SOC lead after rebuilding three years of detection baselines from scratch

The next action is concrete: before month six, schedule a metric audit where you delete the worst-performing indicator—the one nobody defends—and replace it with something simpler. Don't add. Swap. Debt reduction beats expansion every time.

Field note: penetration plans crack at handoff.

Field note: penetration plans crack at handoff.

When Not to Use This Approach

Short-Lived or Exploratory Campaigns

If your red team engagement runs for six weeks and disappears, don't build a sustainability metric suite. You'll spend more time designing the dashboard than actually running the operation. I once watched a team burn their first two weeks building a "long-term threat scoring matrix" for a proof-of-concept test that got cancelled in month three. Waste. Short campaigns need fast feedback loops—manual logs, daily standups, a shared spreadsheet. Formal sustainability metrics add ceremony without payoff. The catch is knowing when six weeks becomes six months: if there's any chance the campaign extends past quarter-end, add one lightweight tracker (mean-time-to-recover on credential rotation, say) and leave the rest alone. Otherwise you're decorating a tent that will be gone before breakfast.

Highly Volatile Environments

Some shops reorganise every two months. Teams merge, tooling swaps, the CISO leaves—three times in one year, I've seen it. In that chaos, sustainability metrics become moving targets you can't hit. You measure "time-to-detect lateral movement" in Q1; by Q2 the detection pipeline has been replaced, the data source is gone, and your baseline is meaningless. The odd part is—teams still try. They export last quarter's graphs to quarterly reviews, pretending the numbers track something real. They don't. What works instead: binary health checks (are we logging to this server? yes/no) and qualitative pulse surveys from operators. Save the multi-variable index for next year, if the org holds still that long.

Teams Without Dedicated Metric Ownership

Sustainability metrics die without a named keeper. Not a "we all own it" committee—a person who opens the dashboard on Tuesday mornings, recalibrates when a sensor breaks, and fights for the budget line. If your red team is four people running operations back-to-back with no slack, you can't absorb this role. The result? Metrics drift, then rot, then get ignored. A team leader told me once: "We built beautiful charts. Then nobody updated the data feed for two months. We just… stopped looking." That hurts because the time investment was real but the return vanished. Before you start, answer one question: who gets woken up when the metric pipeline breaks? If the answer is vague—don't start yet. Hire or reassign first.

The best sustainability framework is the one someone actually maintains. The second-best is no framework at all.

— red team lead, after scrapping three abandoned dashboards

What usually breaks first is not the metric concept—it's the feeding mechanism. A volunteer from another squad, an intern who leaves, a part-time analyst assigned to "help." I have seen three separate campaigns where the same pattern played out: month one, enthusiasm; month three, neglected scripts; month six, someone asks "what happened to that data?" Empty. If you can't assign a single accountable human with protected time, skip the formal approach. Run your campaign on gut checks and incident post-mortems instead. They're less precise but they stay alive.

Open Questions and FAQ

How often should metrics be recalibrated?

You will fight this question every quarter. The honest answer is 'as often as the attack surface changes — but no more than once per operational cycle.' I have watched teams schedule metric overhauls like clockwork, every ninety days, only to discover their adversary had already pivoted to a completely different breach path three weeks after the last recalibration. That hurts.

The pattern that holds: recalibrate after a significant simulation event, not on a calendar trigger. A major red-team breakthrough, a zero-day exploit in your stack, a shift in threat intelligence — those are your recalibration flags. The odd part is — most teams do the opposite. They polish metrics during quiet months and freeze them during actual campaigns. Wrong order. A metric that survives a multi-year engagement should be reviewed when the ground moves, not when the spreadsheet says so.

What usually breaks first is the threshold. You set a 'mean time to detect' target at four hours. Year two rolls around, your detection stack improves, but the red team starts using long-dwell impersonation — four hours becomes irrelevant. Time-to-respond metrics rot the same way. Recalibrate thresholds, not the whole framework. Swap the numbers, keep the structure, move on.

Can one metric survive a full campaign?

No. Absolutely not. I have seen teams try to anchor a three-year exercise on a single 'percentage of critical alerts reviewed within six hours' — and by month eight, the red team had learned to trigger triage fatigue by flooding that exact metric with low-noise alerts. The metric didn't fail; the adversary exploited its fixedness.

You need a metric portfolio, not a silver bullet. Three to five indicators that look at different layers: detection speed, containment accuracy, credential churn, signal clarity. One for the board (trend lines), one for the SOC (operational lag), one for the red team (coverage gaps). That mix survives drift because each metric checks the others. When the board sees green but the SOC sees flat containment times, you know something is off — and that tension is where real decisions happen.

The catch is — portfolios drift too. Not evenly. Some metrics become stale, others become noise. Plan to retire one metric per campaign phase. Replace it with something that targets the next likely adversarial shift. Not a full redesign. A swap.

'We kept the same three metrics for eighteen months. The red team stopped caring about two of them by month ten.'

— SOC director, after a long-dwell breach simulation

What if executive buy-in is weak?

Then your metrics are dead on arrival — not because they're wrong, but because nobody will act on them. The fast fix is to package one metric in terms of financial exposure. Remediation cost per incident over time. Mean dollars at risk per unpatched critical. Executives who tune out 'detection latency' snap to attention at 'dollars bleeding while we wait.'

But here is the trap: don't fake the translation. If your metric is technical, keep it technical and add a clear consequence line — 'every hour of undetected lateral movement adds $X in potential recovery cost.' No fabricated precision. Ballpark is fine. The goal is a functional shorthand, not a finance paper.

If buy-in still stalls, run a six-week mini-campaign. Show the delta before and after metric-driven action. Nothing convinces like a red team hitting a wall because someone actually used the alert-fidelity metric to tune a detection rule. Weak buy-in usually stems from abstract promises. Hand them a concrete win, then ask for the next quarter's commitment.

One more thing — never ask for permanent buy-in upfront. Ask for a trial. 'Give us two reporting cycles with this metric. If the signal is garbage, we drop it.' That lowers the stakes. Most executives say yes to a trial. Then the data talks.