Field Notes on Penetration Testing: What Works, What Breaks, and When to Walk Away

Penetration testing isn't a checklist activity. It's not running Nessus and pasting findings into a Word template. Real pen testing is messy, dependent on context, and often constrained by things outside your control — like the client who won't give you credentials because 'that's not realistic,' or the web application firewall that blocks your SQL injection attempts but also blocks legitimate traffic. These field notes capture what I've learned from hundreds of tests across startups, banks, and government agencies. The stuff that actually moves the needle — and the patterns that keep failing.

Where penetration testing actually shows up (and where it doesn't)

Regulatory drivers vs. genuine security need

Most penetration tests I see start with a compliance checkbox. Someone’s auditor demanded a report, so the test gets scheduled for the quietest quarter — preferably when no one deploys anything new. The result? A generic scan, a waterfall of findings that nobody triages, and a PDF that gathers digital dust. That kind of test shows up in board decks and auditor folders, but it rarely surfaces the vulnerabilities that keep security engineers awake at night. The catch is — compliance-driven tests aren’t useless. They catch low-hanging fruit. But they also breed a dangerous comfort: the belief that passing a test equals being secure. It doesn’t. A real security need test arrives when someone asks “What happens if we get hit tomorrow?” not “What does the auditor want to see?”

The tricky bit is distinguishing between the two. I once watched a client run a full external penetration test against a production environment that hadn’t changed in eighteen months. They passed. Meanwhile, their internal API — the one connecting their payment processor to customer data — had never been tested at all. Wrong order. The regulatory drivers pointed outward; the genuine risk lived inward. That gap between advertised scope and actual testing is where breaches happen.

Testing in DevOps vs. waterfall environments

Where pen testing actually shows up often depends on how fast the organization ships code. In waterfall shops, tests happen once a year, usually against a frozen snapshot. The report lands three weeks later, by which time the infrastructure team has already patched half the findings and introduced three new vulnerabilities. That pattern is frustrating but predictable. DevOps environments flip the problem entirely. Testing there happens in sprints, often automated or embedded in CI/CD pipelines. But here’s the pitfall: continuous delivery doesn’t mean continuous security review. I’ve seen teams run DAST scans every commit but skip manual testing entirely because “the pipeline catches everything.” It doesn’t. What usually breaks first is the business logic — the thing no scanner understands.

Most teams skip this: scoping a pen test for a DevOps pipeline means accepting that by the time the report lands, the code has changed. The useful output isn’t a list of bugs — it’s patterns. Where do developers keep making the same mistake? That’s the signal worth chasing. But regulators still want static reports. So you get two tests: one for compliance, one for safety. They rarely overlap.

The gap between advertised scope and actual testing

Scope creep isn’t just a project management headache — it’s a security blind spot. Clients love to list everything in the statement of work: “All external IPs, all internal subnets, all cloud accounts.” Then on day one, someone hands you a spreadsheet with 47 subnets, 200 cloud functions, and a Kubernetes cluster no one remembers configuring. The advertised scope looks thorough. The actual testing? You run out of hours before you touch the third of five critical applications.

“We tested everything in the scope document.” But the scope document missed the admin panel running on port 8443 behind a load balancer nobody mentioned.

— Lead consultant, post-mortem conversation

The fix is brutal but honest: scope the test around what matters, not what’s listed. One critical application, fully tested, beats ten systems half-scanned. That sounds obvious. It’s rarely how budgets get approved. The hidden cost of scope inflation isn’t overtime — it’s the false sense of coverage. A report that says “90% of assets tested” still leaves the 10% that probably contains the worst vulnerabilities. I have walked away from engagements where the scope was too wide to be useful. Better to test nothing than to test everything badly. That’s where pen testing actually shows up — in the honest call about what you can and can't cover.

Foundations that still trip up testers and clients

The difference between vulnerability scanning and penetration testing

Most teams I work with still treat these as interchangeable. They're not. A scanner rattles through a checklist of CVEs and spits out a PDF that looks official but rarely tells you what actually matters: can an attacker chain three low-severity findings into a domain admin session? The scanner says no. A penetration test says yes—or walks you through why it failed. I have watched clients burn three weeks arguing over a scanner's false positive while a real lateral-movement path sat open in the same subnet. The uncomfortable truth is that scanning gives you volume; testing gives you signal. The trade-off is cost—testing is slower, more expensive, and demands a human who can improvise when the exploit drops on step four.

The catch is that many clients sign up for a penetration test and then treat the results like a scan report, sorting by severity instead of reading the attack narrative. That misses the point. The narrative—how one foothold became ten—is where the actual risk lives.

What 'exploitation' actually means in contracts

Here is where the seam blows out. I have seen scopes that define exploitation as "demonstrate code execution or privilege escalation" and then penalize the tester for crashing a service during the attempt. You can't have both. Real exploitation is fragile—kernel exploits fail, heap sprays land sideways, and sometimes you trip a watchdog timer and the process dies. That's not incompetence; it's the nature of attacking a system that was designed to resist you. Contracts that demand proof of exploitation but prohibit any service degradation are asking for a demo, not a test. The fix is straightforward: scope language that says "attempt exploitation with a fallback to proof-of-concept when the attempt fails" and explicitly accepts a bounded service restart window. Without that, testers either fake the result or skip the interesting targets.

What usually breaks first is the reporting phase. The client sees a finding marked "not exploited" and asks why they paid for a penetration test. The answer—because the chain of conditions required for exploitation is worth documenting even when the final shell doesn't land—never feels satisfying in a post-meeting hallway conversation. But it's honest.

Authentication vs. authorization: why it matters for reporting

The odd part is—these two concepts get swapped in almost every root-cause analysis I read. A tester finds that User A can read User B's files. The report says "weak authentication." Wrong. Authentication verified that User A is User A. The failure is authorization: User A should not have permission to read User B's files, but the access control list allowed it. That distinction changes the fix. Patching authentication—say, adding MFA—does nothing if the real problem is a broken permission model. I have seen teams spend six months rolling out hardware tokens while the authorization flaw sat untouched, exactly as wide open as before. The fix is no harder once you name it correctly, but the mislabel delays the real work by redirecting budget to the wrong layer.

Flag this for penetration: shortcuts cost a day.

Flag this for penetration: shortcuts cost a day.

Most teams skip this: they report by severity instead of by root cause class. A high-severity authentication flaw gets fixed first; a medium-severity authorization flaw gets deferred. But if the authorization gap is the one that lets an attacker pivot laterally, the priority order is inverted. Reporting by root cause—not by CVSS score—would catch that. I have started asking clients to let me rename their finding categories before the report goes final. It's a small edit that reshapes the entire remediation plan.

'We tested authentication and it passed. The breach used an authorization gap that nobody tagged.'

— incident post-mortem notes, anonymous organization, 2023

Patterns that consistently find critical issues

Chaining low-severity findings into high-impact exploits

One finding is noise. Two findings, carefully connected, become a breach. I have watched teams spend days patching a medium-risk XSS in a forgotten admin panel while ignoring the real kill chain sitting in plain sight. The pattern that works: map the seams between flaws. A verbose error message on a forgotten API endpoint — something most scanners miss — paired with a session token that doesn't rotate on privilege escalation. Alone, neither hits the critical threshold. Chained? You get a full account takeover in three requests. The catch is that automation won't connect these dots. It requires a tester who reads log dumps like a detective reads case files, asking what can talk to what rather than what is broken.

Most teams skip the reconnaissance required to find these chains. They run a scanner, get a list, call it a day. Wrong order. The real yield comes from manual mapping of data flows — this form talks to that internal service, that service echoes user input back to an admin dashboard. I fixed a critical breach in a fintech app last year by noticing that a password reset link on staging still pointed to a live subdomain. Low-severity misconfiguration? Yes. Combined with a leaked debug key in the same codebase? Suddenly you control password resets for production accounts. That's the pattern: never dismiss a finding because it scores low alone. Ask instead what it touches.

'A 4.3 finding is a story half-told. The other half is always in another request.'

— field tester, after a 12-hour chain session

Manual review of business logic endpoints

Scanners hit login pages and parameter injection points. They never check if you can buy a $1000 item and apply a $2000 discount because the cart validation checks quantity but not price. Business logic flaws are where penetration testing earns its pay. The pattern: target every endpoint that mutates state in ways the developer didn't anticipate. I have seen a shopping cart that allowed negative quantities — the refund logic triggered automatically. That was not an injection or a misconfig; it was a missing greater than zero check. The fix took ten minutes. Finding it took three days of staring at transaction logs.

The tricky bit is that business logic varies per app. There is no generic payload. You need to understand the domain: if it's a payment system, test price overrides; if it's an HR portal, test whether a manager can promote someone outside their chain. One tester I worked with spent an hour clicking through a multi-step password reset flow. The third step had a hidden userId param that stepped down by one each iteration. He brute-forced it and took over twenty accounts. That's manual review — not fancy tooling, just curiosity applied to the right endpoints.

Using reconnaissance to prioritize attack paths

Half the battle is knowing where to look first. Most teams start scanning immediately. That's a trap. The stronger pattern: spend the first 40% of test time on reconnaissance — subdomain enumeration, certificate transparency logs, leaked API keys in public repos, and old JS bundles that still contain internal endpoints. I found an admin login page for a logistics company by scraping a minified JavaScript file that referenced /internal/v2/fleet-commands. That path was not in any scope document. The scanner never touched it. The team had forgotten to remove it from the build.

Prioritization emerges from this phase. You build a heat map: endpoints that accept file uploads rank higher than static pages; older subdomains running unpatched frameworks get probed before the shiny new microservice. The trade-off is that reconnaissance is slow and yields dead ends. But one live hit — a forgotten Wordpress instance on a subdomain that the client swore didn't exist — can collapse the entire perimeter. What usually breaks first is the thing they forgot existed. Find that, and you win. Not yet? Keep digging until the noise becomes signal.

Anti-patterns that teams keep falling back to

Over-reliance on automated scanners

Teams treat a scanner report like a finished map. It isn't. I have seen clients pay for a tool, watch it run overnight, then declare the network clean. The scanner found SQLi in three parameters — sure it did. It also missed a chained attack that required reading two config files and a custom API endpoint that wasn't in the crawl. The seductive lie of automation is coverage. You get noise instead of signal, and the noise becomes ammunition for false confidence. The real cost is not the false positives; it's the false negatives that nobody challenges. A scanner sees surface. It doesn't see privilege escalation paths that require chaining three mundane actions in the right order. You have to test those by hand.

Testing only in staging environments

Staging is a clean room. Production is a landfill of forgotten configs, stale session tokens, and that one dev account with domain admin that nobody remembers creating. The fix seems obvious: test prod, get permission, sign waivers. Yet teams keep falling back to staging because it's safe and scheduled. Safe for the environment, sure. Safe for the test results? No. I once watched a team spend two weeks breaking a staging replica that had different firewall rules, different logging, and a different load balancer. The production application had an exposed debug endpoint that granted shell access. Staging never showed it. The patterns that consistently find critical issues — lateral movement, data exfiltration, real-world latency exploits — only emerge under production load. Staging sanitizes the mess that attackers exploit.

Ignoring privilege escalation paths

Most test reports stop at 'low-priv shell acquired' and call it a win. That's where the real test begins. The anti-pattern is celebrating foothold without asking what happens next. I have seen teams hand over thirty pages of web app vulnerabilities and skip the kernel exploit that turned a guest account into root in four commands. Why? Because the scope said 'web layer only.' That boundary is artificial. Attackers don't respect scope lines. The odd part is — clients rarely push back on shallow findings because the report looks thick. But a thick report with no escalation path is a thin shield. You need to show the seam, not just the scratch.

'Nobody walks away from a pen test thinking about the gap between foothold and full compromise. That gap is where real damage lives.'

— senior tester reflecting on post-breach reviews

Not every penetration checklist earns its ink.

Not every penetration checklist earns its ink.

The fix is blunt: mandate at least one full privilege escalation chain per engagement. Even if it fails. Even if you end up stuck. The act of trying forces testers to think like attackers instead of checklist readers. Next time you scope a test, ask the team to show you one path from anonymous to local admin — or explain why they could not find it. That question alone shifts the focus from what the tool says to what the environment actually allows.

The hidden costs of maintaining a pen test program

Retesting and regression costs

The first test is the cheapest you will ever run. Sounds backward, but watch what happens after delivery: the client fixes the top five findings, requests a retest, and you spend three days re-proving that the same SQL injection still works on a different parameter. Regression testing eats calendar—I have watched teams burn 40 percent of their annual pen test budget just verifying patches that never held. The catch is that clients treat a retest as a courtesy, but your team treats it as a fresh engagement with half the joy and twice the political friction. That gap bleeds money.

What usually breaks first is the retest scope. The original report said "all user-facing inputs." The fix only covered the login form. Now you must decide: chase the remaining inputs for free, or write a change order that makes the client grumble. Neither path pays well. The smarter move is to quote retesting as a separate line item before the first scan starts—but most teams skip this to win the deal.

Tooling licenses vs. in-house skills

Burp Suite Pro, 450 dollars a year per seat. Nessus Professional, 3,800. Cobalt Strike, 3,500 per operator per year. A three-person team racks up five figures before touching a target. The trade-off is brutal: buy the license and your team depends on click-button automation, or build in-house and lose two months writing a custom proxy that still misses half the edge cases. I have seen shops spend 80 percent of their tool budget on one suite, then scramble when the license model changes mid-year. That hurts.

The odd part is that tool maintenance itself carries a hidden tax. Updates break your automation scripts. New API endpoints require re-learning. Nobody budgets for the week your lead tester spends upgrading the lab environment instead of finding bugs. Open-source alternatives reduce sticker shock but shift the cost to developer time—and developer time is often harder to defend on a spreadsheet.

Burnout among offensive security staff

Pen testers break things for a living. That sounds fun until the 18th consecutive report lands on a Friday afternoon with a turnaround of Monday morning. The emotional load is real: every finding you miss could become a breach headline, but every finding you report makes someone else defensive. I once watched a senior tester walk away mid-engagement because the client kept demanding "one more test" after midnight. He didn't come back.

'The industry sells the glamour of red teams. Nobody sells the 2 AM retest on a network that won't stay up.'

— field note from a pentest lead, 2023

Retention falls apart when the work is all peak, no plateau. Most shops lose their best operators within 18 months because the schedule never breathes. The fix is not more pay—it's enforced no-contact weeks and rotating the miserable assignments. But maintaining a program means maintaining people, and most organizations treat humans like licensing costs: replaceable until they're not.

When not to run a penetration test (and what to do instead)

Immature security programs: fix basics first

I once watched a client pay $80k for a penetration test on a web app that still had default admin credentials on the database server. The tester found it in under ten minutes. The client framed the report, called the test a success, and then did nothing about the other seventeen default accounts the same scan had turned up. That hurts. If your organisation can't reliably patch known vulnerabilities, enforce multi-factor authentication on external-facing systems, or inventory what it actually runs—a penetration test is a waste of money. You're renting a spotlight to shine on a house that already has no roof. The output will be a list of problems you already knew existed, plus a few obscure ones, and the team will burn its energy triaging the flashy findings while the basics rot. Fix the basics first. Run a vulnerability scan. Write an asset inventory. Lock down the perimeter with something that doesn't ship with password = password. A pen test will confirm the same truth—but it will cost you ten times as much to learn that lesson.

“We found sixty critical issues in the external test. Fifty-two were missing patches older than six months.”

— Lead tester, post-engagement debrief, 2023

Applications in active development

You're building version 2.0 of a product. The sprint is chaotic. Features get cut on Tuesday, re-added on Thursday. You want a penetration test next week. Stop. A static snapshot of something that will be rewritten in six weeks is not a security finding—it's archaeology. I have seen teams treat early-stage pen tests like a magic shield: “We tested it in April, so we're compliant.” Meanwhile the codebase tripled in size, the authentication flow was swapped twice, and every test finding from April now points to code that no longer exists. The real cost is not the test itself—it's the false confidence it creates.

What actually works: schedule a pen test only after a feature freeze, or at worst a documented code slush. Before that, lean on developer-led threat modelling and automated static analysis. A good SAST tool running in CI will catch injection flaws and crypto missteps the same day they're committed. The feedback loop is hours, not weeks. Pair that with manual code review of the high-risk areas—authentication, authorisation, payment flows. That combination costs a fraction of a full test and produces findings you can act on immediately. When the code finally settles, run the real test. The results will be thinner, but they will mean something.

Threat modeling as a cheaper alternative

Most teams reach for a penetration test because they don't know what else to do. The compliance checkbox looms, the stakeholders want a report with a red cover, and nobody has time to think. But threat modeling, done right, surfaces more actionable problems than any black-box test I have ever run—and it costs a tenth of the price. You sit down with the developers, the ops lead, and a whiteboard. You map data flows. You ask: where does the credit card number touch memory? What happens if the session token leaks? Who can escalate privileges from this endpoint? The conversation alone unearths design flaws that no scanner will find.

Field note: penetration plans crack at handoff.

Field note: penetration plans crack at handoff.

The catch is that threat modelling demands discipline. A single half-assed session is worse than none—it produces a laundry list of improbable risks and eats goodwill. But a structured approach—STRIDE, PASTA, even a simple attack tree—forces the team to think like an adversary before the code ships. I have watched a two-hour threat modelling session derail a feature because someone realised the token expiry logic was handled client-side. That fix cost one engineering day. A penetration test would have caught it four months later, after the feature was in production, after the breach window had been open for weeks. Choose the cheaper path first. Pen tests are the final referee, not the training camp.

Next time you hear “we need a pen test this quarter,” ask instead: what would we change if we found a critical issue tomorrow? If the answer is “panic,” fix the pipeline. Then test.

Open questions that still split the community

Should testers share zero-days with clients?

The honest answer bruises egos on both sides. I have sat in rooms where a researcher drops a fresh kernel exploit — and the client’s first move is legal, not technical. Some firms treat every novel finding as a liability, scrubbing reports until the edge is gone. Others demand full disclosure within 48 hours, then shelve the details because patching cycles run quarterly. The trade-off bites: share too early and the fix gets bodged under pressure; share too late and the window for exploitation widens in the wild. A handful of shops I respect enforce a responsible disclosure rider — tester keeps the method private for 90 days, client burns dev cycles on a real patch, not a workaround. That still breaks when the client ships no patch and the tester sits on a ticking time bomb. No clean answer exists. What I have seen work is a pre-engagement contract that names a specific disclosure timeline, with an escalation clause for active exploitation observed — but even that clause gets invoked maybe once in fifty tests.

Is certification worth the paper it’s printed on?

The CISSP costs a mortgage payment to maintain. The OSCP forces you through 24 hours of controlled suffering. Teams hire for both and then watch fresh cert-holders freeze on a real AD environment where the attack path is not a lab handout. I have interviewed people with three certs who could not explain why a wget pull from a non-standard port matters in detection evasion. Meanwhile, the best tester I ever worked with held zero credentials — he just broke things for a living and wrote notes that read like combat logs. The catch: HR filters on acronyms. So the cert is not a signal of skill; it's a ticket past the gatekeeper. That reality frustrates purists and rewards resume padders. What usually breaks first is the assumption that a passing score equals operational judgment. If I could rewrite hiring, I’d trade every paper credential for a single recorded test where the candidate explains why they skipped a scan or aborted a pivot.

“Certifications teach you to pass exams. Pen testing teaches you to survive failures you didn't expect.”

— Lead tester at a firm that dropped all cert requirements two years ago

How much automation is too much?

Scanners catch the low fruit — unpatched Tomcat, default creds, misconfigured S3 buckets. Run a Nessus pull and you look busy. But automation creates a blind monotony. I have reviewed reports where the tester ran nmap -A on every host, dumped the output into a report template, and called it a day. Missed? Business logic flaws, race conditions in cart checkout, a WAF bypass triggered by a specific HTTP header order. The machines can't smell those. The odd part is — teams that automate heavily rarely catch the cascade failures: a scanner reports a medium-severity XSS, but the tester never checked if that XSS feeds into a server-side template injection chain. Automation gives speed; it doesn't give context. The shops that balance it best use scripts to grind through the boring 80% and reserve human brain cycles for the weird edge cases — a custom protocol, a legacy app that crashes under a normal HEAD request, an API that accepts JSON and XML but sanitizes neither the same way. That balance is fragile. Shift too far toward automation and you produce volume with zero insight. Shift too far manual and your coverage gaps widen while the client bills by the hour. The question stays open because every environment rewards a different ratio.

Next up: What I would actually change if I ran my last five years of tests again — starting with the one engagement I should have walked away from on day one.

What I'd do differently next time

Invest more in client education upfront

The hardest battles I have fought in a pen test weren't against WAFs or obfuscated credentials—they were against mismatched expectations. You show up with a scope document, run the tests, hand over a report. That sounds clean. It isn’t. Clients often interpret 'critical' findings as immediate threats, then panic when the fix takes four weeks. Or worse—they treat 'informational' items as noise and ignore a slow data leak that later blows open. I should have spent the first thirty minutes of every kickoff walking through severity definitions, real impact timelines, and what a partial remediation looks like. Not a lecture. A calibration conversation. The catch is: most teams skip this because they assume the RFP or contract covers it. Wrong order. The contract covers liability; the conversation covers trust.

Spend less time on low-hanging fruit

Low-hanging fruit is addictive.

One XSS, three open S3 buckets, a misconfigured CORS policy—you burn half a day proving these out when everybody already knows the pattern. The real payoff lives in the gnarly intersections: the authentication bypass that only works because the developer used a custom session handler with a flawed expiry check. The business logic flaw that lets an attacker escalate privileges by manipulating a numeric role ID in a POST body. What breaks first? Usually, scope creep on easy stuff. I once spent four hours documenting a trivial SSRF in a staging environment while the client's production API endpoint had a race condition that allowed unlimited coupon generation. Priorities slipped. Next time I'd front-load the high-complexity items, confirm the architecture diagram early, and treat 'quick wins' as a one-liner in the summary, not a full section.

Build better templates for reporting

Most pen test reports read like autopsy notes—dry, passive, buried under boilerplate. That's a failure of craft, not of data. I should have invested in report templates that separate executive narrative from technical detail from reproduction steps, each with its own structure. A CEO doesn't care about the curl command. A DevOps lead doesn't need a three-page 'risk rating' preamble. The real fix: a one-page executive brief first, then a findings table sorted by business impact (not CVSS score alone), then a reproduction appendix. The odd part is—most teams spend weeks refining methodology but treat the report as an afterthought. That hurts. A clean report is a deliverable that gets actioned; a messy one gets filed. I would build a template library by engagement type—web, cloud, mobile—not by client vertical.

'The best report I ever read had seven words on the cover page: "Six critical issues. Two you can fix today." Everything else was backup.'

— paraphrased from a CISO I never met but whose style I stole

That single line changed how I present findings. Cut the noise. Lead with the one thing that keeps the client up at night, then hand them a path out.