Availability & Failure Handling
Identity & Access Control is on the critical path of almost every request. When the identity provider, policy engine, or their dependencies slow down or fail, the product must stay secure first and available second, and the trade-off between the two must be decided in advance, not during an incident.
Dependencies and Their Failure Modes
Rendering diagram…
Diagram source
flowchart LR req([Request]) --> verify[Token verification] verify --> check[Policy check] check --> app[Application] keys[Published keys] -.->|cached| verify idp[Identity provider] -.->|sign-in and refresh only| verify engine[Policy engine] -.-> check store[(Relationship store)] -.-> engine
Designs that verify tokens locally and cache keys survive an identity provider outage for existing sessions. The policy engine is harder: without it, no fine-grained decision can be made.
Failure Handling
| Failure | Behavior |
|---|---|
| Customer identity provider unavailable | Existing sessions continue. New SSO sign-ins for that organization fail with a clear message; no silent fallback to passwords. |
| Managed identity platform unavailable | Requests with valid tokens continue. Refreshes and new sign-ins fail until it recovers; token lifetimes are not extended automatically. |
| Policy engine slow | Requests use cached decisions where available, with a strict timeout on new checks. |
| Policy engine unreachable | Requests that need a decision are denied (fail closed) and the failure alerts immediately. |
| Signing key compromised or expiring | Keys rotate with overlap: new tokens use the new key while old ones verify until they expire. Emergency rotation revokes every session. |
| Credential stuffing or brute force | Rate limits per account and per source, progressive delays, and lockout with notification to the user. |
| Event delivery delayed | Events are delivered at least once from an outbox, so downstream systems catch up without loss. |
Fail Closed, Degrade Gracefully
Failing closed does not have to mean failing everything. The product can:
- Keep cached decisions for their normal lifetime, without extending them.
- Serve read-only or public content that needs no fine-grained decision.
- Run the engine close to the application, as a sidecar or replica per zone, so a network partition does not remove it.
- Isolate dependencies with timeouts and circuit breakers, so a slow engine does not exhaust the application's threads.
- Do expensive work at sign-in, not per request. Password hashing, MFA, and federation happen once per session; requests only verify a signature against cached keys, which refresh on an unknown key identifier.
Monitoring
- Sign-in success and failure rates per method and per customer identity provider.
- Authorization decisions per second, denial rate, and decision latency at the 95th and 99th percentiles.
- Token refresh errors and refresh-token reuse detections.
- Alerts on spikes in failed sign-ins, any policy engine unavailability, and certificates or keys close to expiry.
Watch For
- Sign-in spikes. Morning peaks and mass re-authentication after an incident concentrate load on the slowest part: password hashing and identity provider round trips. Size for them.
- Clock skew. Local token verification depends on time. Allow a small, explicit leeway and keep server clocks synchronized.
- Fail-open shortcuts. A temporary "allow on timeout" added during an incident tends to stay. Treat it as a security change requiring review.
- Untested recovery. Rehearse an identity provider outage and an engine outage in staging, and confirm the documented behavior actually happens.