Customer Snapshot
| Attribute | Details |
|---|---|
| Industry | Technology / SaaS |
| Cloud Environment | Multi-account AWS with production workloads |
| Team | SRE / DevOps on a rotating on-call schedule across time zones |
| Communication | Slack (incident coordination) |
| Problem | On-call engineers either carried standing prod admin or waited on approvals mid-incident |
| Desired Behaviour | Fast access during a verified incident; zero standing access otherwise |
| Cloudanix Scope | Cloud Console JIT with incident-aware auto-approval policies |
The Situation: 3 AM, Production Is Down, and the Clock Is the Enemy
Incident response has a brutal relationship with access control. When production breaks at 3 AM, the on-call engineer needs elevated access now — to read logs, inspect resources, restart a service, roll back a deploy. Every minute spent waiting on an access approval is a minute added to customer-facing downtime. So teams make a choice, and both common choices are bad.
The first choice is to give on-call engineers standing production access, so it’s always there when an incident hits. This solves the 3 AM latency problem by creating a far worse one: a rotating population of engineers who permanently hold production admin, most of the time for no active reason. That is standing privilege at its most dangerous — broad, production-scoped, and attached to credentials that live on laptops and in shells around the clock.
The second choice is to require approval for elevated access even during incidents. This keeps standing privilege low but turns every incident into a scramble to wake up an approver, who then rubber-stamps the request because of course the on-call engineer needs access during an outage. The approval is real friction that adds nothing but delay — a ceremony that slows the response without meaningfully gating it.
This SRE team lived the tension directly. Their on-call rotation spanned time zones, their incidents didn’t wait for business hours, and they were tired of choosing between “always-on prod admin” and “wake someone up to approve the obvious.”
The Core Challenge
On-call access has two requirements that appear to conflict: it must be immediate during a genuine incident, and it must be absent the rest of the time. Standing access satisfies the first and violates the second. Mandatory approval satisfies the second but adds latency to the first exactly when latency hurts most. The challenge was to make access fast when an incident is real and nonexistent when it isn’t — without asking a human to approve the obvious at 3 AM.
Where the Gaps Were
Standing On-Call Access: Convenience That Becomes the Attack Surface
Granting the on-call rotation standing production access feels pragmatic. In risk terms it’s the opposite:
- The privilege outlives the need by orders of magnitude. An engineer is actively responding to incidents for a tiny fraction of their on-call shift. The standing grant persists for the whole rotation — and often doesn’t get removed when the rotation ends.
- It’s the broadest grant you have. On-call access is production access, usually with wide scope so the engineer can handle whatever breaks. Broad plus standing plus production is the exact combination incident reviews dread.
- Rotation makes it sprawl. Every engineer who’s ever been on-call tends to accumulate the access. Over a year, “the on-call people have prod admin” quietly becomes “most of engineering has prod admin,” and nobody can confidently say otherwise.
- It’s indistinguishable from legitimate baseline. When an auditor asks “who can touch production and why,” standing on-call access is the hardest thing to explain, because its justification (“in case there’s an incident”) applies 24/7 but is true for minutes a month.
Mandatory Mid-Incident Approval: Friction Where It Costs Most
The disciplined alternative — require approval even during incidents — removes the standing-privilege problem but introduces a latency problem precisely when latency is expensive:
- The approver is a bottleneck during an outage. Someone has to be awake, reachable, and willing to click approve while production burns. Across time zones, that someone might be asleep.
- The approval adds no real control. Nobody is going to deny an on-call engineer access during a verified production incident. The approval is a formality — pure delay with no filtering value — which trains everyone to treat it as noise.
- It discourages the right behaviour. When “do it properly” means “add minutes to the incident,” engineers quietly drift back toward standing access so they never have to ask. The friction defeats the policy it was meant to enforce.
The Cloudanix Solution: Incident-Aware Auto-Approval, Time-Bound and Fully Audited
Cloudanix JIT resolves the conflict with policies that treat an active, verified incident as the approval. When the on-call engineer is responding to a real incident, access is granted fast and automatically. The rest of the time, there is no standing access at all — the engineer requests like anyone else, through normal policy.
How Incident-Driven Access Works
-
No standing production access by default. The on-call rotation carries no permanent production privilege. Between incidents, the attack surface simply isn’t there.
-
An incident becomes the trigger. When a genuine incident is active — surfaced through the team’s incident tooling and on-call signal — JIT policy recognises the responding on-call engineer as eligible for fast-tracked, auto-approved access to the relevant production scope. The incident context is the authorization, so no human has to be woken to approve the obvious.
-
Access is time-bound and scoped. The grant is for the window needed to respond, not open-ended, and scoped to what incident response requires. When the window ends, access auto-revokes — the engineer doesn’t have to remember to give it back, and it doesn’t linger past the incident.
-
Everything is identity-stamped and audited. Who got access, that it was tied to an active incident, what role and scope, when it started, and when it auto-revoked — all on one timeline. The incident review and the access record are the same story.
-
Non-incident access follows normal policy. When there’s no incident, elevated production access goes through standard JIT approval. Fast-tracking is a property of verified incidents, not a permanent perk of being on the rotation.
Why This Beats Both Bad Choices
The incident-aware model takes the best property of each failed approach and discards the cost:
- As fast as standing access, during incidents. The engineer isn’t waiting on anyone — the incident fast-tracks the grant. The 3 AM latency problem is solved.
- As clean as mandatory approval, the rest of the time. Between incidents there is zero standing production privilege. The audit answer to “who can touch production?” is “nobody standing — access is incident-bound and time-limited.”
- No pointless ceremony. Nobody is woken to approve the obvious. The approval that added only delay is replaced by the incident signal that actually justifies the access.
- Self-cleaning. Because grants are time-bound and auto-revoke, on-call access can’t sprawl across the rotation over time. The year-long drift from “on-call has prod admin” to “everyone has prod admin” doesn’t happen.
The Audit Trail Incident Reviews Actually Want
Post-incident reviews routinely ask: who accessed production during the incident, with what privileges, and when did that access end? With incident-driven JIT, that’s not a reconstruction exercise across CloudTrail and chat logs — it’s a single record. Each grant is stamped with the responder’s real identity, the incident it was tied to, the scope, and the auto-revocation time. The access story and the incident story are one timeline, which makes both the security review and the operational retro faster and more honest. For the broader discipline of complete access auditing, see closing the audit-trail gap.
Guarding Against the Obvious Abuse: What Stops a “Fake Incident”?
The sharpest objection to incident-driven auto-approval is the one a good security engineer raises immediately: if an active incident fast-tracks access, what stops someone from declaring a bogus incident to grab production access without approval? It’s the right question, and the answer is what makes the model trustworthy rather than a backdoor.
First, the incident signal comes from the team’s incident tooling, not from the requester simply asserting “there’s an incident.” Tying the fast-track to a verified, externally-visible incident — one that’s been opened in the on-call/incident system and is visible to the whole team — means a fabricated incident isn’t a quiet lie; it’s a loud, logged, falsifiable claim that others can see and will notice. The cost of faking an incident is that you’ve created a public record of an incident that didn’t happen, attached your name to the access it unlocked, and invited exactly the scrutiny you were trying to avoid.
Second, auto-approval is scoped by policy to specific roles and accounts — it isn’t a blanket “any incident unlocks everything.” The fast-track applies to the production scope that incident response legitimately needs, not to the entire estate. An engineer can’t use an incident on one service to auto-approve admin everywhere.
Third, every incident-driven grant is identity-stamped and reviewed. Because the access and the incident live on the same timeline, the post-incident review naturally surfaces any grant that doesn’t correspond to real response work. The deterrent isn’t a gate that blocks the request in the moment — it’s the certainty that an illegitimate grant will be visible and attributable afterward. For routine, non-incident access, standard approval still applies, so the fast-track is a narrow, observable exception rather than a general-purpose bypass.
The honest framing to give a security team is this: incident-aware auto-approval trades a preventive control (an approver’s click) for a detective-plus-deterrent control (verified incident, scoped policy, attributable audit) in the one situation where the preventive control added delay without adding judgement. That’s a sound trade when, and only when, the incident signal is trustworthy and the grants are fully attributable — both of which this model requires by design.
Platform Impact
| Dimension | Standing On-Call Access | Mandatory Mid-Incident Approval | Incident-Aware JIT |
|---|---|---|---|
| Access speed during incident | Instant | Slow (wake an approver) | Instant (incident fast-tracks) |
| Standing privilege between incidents | High, broad, production | None | None |
| Approval friction | None | High, adds pure delay | None for verified incidents |
| Risk of sprawl across rotation | High | Low | None (time-bound) |
| Audit answer for “who can touch prod” | “The whole rotation, always” | “Nobody standing” | “Nobody standing; incident-bound only” |
| Incident-review reconstruction | Hard | Hard | Single timeline |
Why On-Call Is the Perfect Case for Context-Aware Access
On-call access is where blanket policies fail hardest, because the need is intensely real for short bursts and completely absent the rest of the time. A policy that’s binary — always-on or always-ask — can’t fit a need that is itself conditional. The answer is to make the policy conditional too: let a verified incident be the thing that unlocks fast access, and let its absence mean no access at all.
That’s the shift incident-aware JIT delivers. It stops treating “the on-call engineer might need production access” as a reason to grant standing privilege, and starts treating “there is an active incident right now” as the authorization it actually is. The result is faster incident response, zero standing on-call privilege, and an audit trail that tells the whole truth in one place.
Tired of Choosing Between Standing Prod Access and 3 AM Approvals?
If your on-call engineers either carry permanent production access or lose minutes to approval chains during incidents, Cloudanix JIT offers a third option: fast, auto-approved, time-bound access the moment a verified incident is active — and absolutely no standing production privilege the rest of the time. Every grant tied to its incident, every revocation automatic, every access on one audit timeline.
Book a Free Assessment to see incident-aware JIT policies mapped to your on-call workflow.
Related Resources
- Auto-Approval Policies in JIT: Balancing Speed and Security for Developer Access
- Multi-Level Approval for JIT: Granular Policies for Production vs Development
- Closing the Audit Trail Gap: From Missing Logs to Complete JIT Access Auditing
- Approval Authority Delegation in JIT: Ensuring Global Teams Are Never Blocked
- Eliminating Standing Cloud Access for SOC2 and ISO27001
