---
skill: incident-postmortem
title: Incident Postmortem
input: "Checkout was down 42 minutes after a deploy set a wrong env var on the payments service; 5xx spiked, ~1,200 failed checkouts. Detected by alert in 6 min, fixed by rollback. Blameless postmortem with timeline and action items."
source: generated by claude-sonnet-4-6
---

# Incident Postmortem: Payments Service Misconfiguration Causes Checkout Outage

**Incident ID:** INC-[FILL IN]
**Severity:** P1
**Date:** [FILL IN DATE]
**Duration:** [HH:MM] → [HH:MM] UTC — 42 minutes total
**Status:** Resolved
**Author:** [FILL IN]
**Last updated:** [FILL IN DATE]

---

## Executive Summary

Following a routine deployment to the payments service, a misconfigured environment variable caused the service to enter a fault state, resulting in 5xx errors across all checkout flows. Approximately 1,200 checkout attempts failed during the 42-minute window between the deployment and full service restoration. The issue was detected by automated alerting within 6 minutes of the deployment completing. Service was fully restored by rolling back the deployment to the prior stable version. This postmortem documents the contributing factors that allowed an invalid configuration to reach production and identifies specific changes to prevent recurrence.

---

## Impact

| Dimension | Details |
|---|---|
| **Users affected** | ~1,200 failed checkout attempts (all users attempting checkout during the window) |
| **Services degraded** | Payments service (complete); Checkout flow (complete) |
| **Business impact** | ~1,200 lost or abandoned transactions; potential revenue impact TBD; SLA breach assessment pending |
| **Duration** | 42 minutes from deployment completion to full service restoration |

---

## Timeline

> Time-zero is the moment the deployment completed and began serving traffic. Adjust all UTC times to match actuals.

```
[T+00:00] — Deployment of payments service vX.Y.Z completed and began serving 100% of traffic.
            Deployment included an environment variable change as part of a configuration update.

[T+00:00 – T+06:00] — 5xx error rate begins climbing immediately post-deploy.
                       No automated gate halted deployment progression; change rolled out fully.

[T+06:00] — Automated alert fires on elevated 5xx error rate threshold for the payments service.
             On-call engineer receives page.

[T+06:00 – T+10:00] — On-call engineer begins investigation. Checks service health dashboards,
                        confirms 5xx errors are isolated to the payments service.
                        Checkout team notified. Incident declared P1.

[T+10:00 – T+18:00] — Engineers examine recent changes. Deployment log reviewed.
                        Hypothesis formed: misconfigured environment variable introduced in deploy
                        is causing the payments service to fail on initialization or request handling.
                        Environment variable values compared against expected configuration.
                        Misconfiguration confirmed.

[T+18:00] — Decision made to roll back to prior stable deployment (vX.Y.(Z-1))
             rather than attempt a forward-fix configuration patch, to minimize
             additional risk and time to recovery.

[T+18:00 – T+28:00] — Rollback initiated and executed. Deployment pipeline runs rollback procedure.

[T+28:00] — Rollback completes. Payments service begins serving traffic from prior stable version.

[T+28:00 – T+36:00] — Engineers monitor error rate. 5xx rate returns to baseline.
                        Checkout success rate confirmed restored. No residual errors observed.

[T+36:00] — All-clear confirmed. Incident declared resolved.
             Post-incident monitoring period begins.

[T+42:00] — Incident formally closed after sustained clean metrics window.
             Postmortem process initiated.
```

> **Key interval:** 12 minutes elapsed between alert firing (T+06) and root cause identification (T+18).
> **Key interval:** 22 minutes elapsed between alert firing (T+06) and full service restoration (T+28).

---

## Root Cause

**Primary root cause:** A deployment to the payments service introduced an incorrect environment variable value that caused the service to fail on payment processing requests, producing 5xx errors for all users attempting checkout.

**Contributing factors:**

- **No configuration validation step in the deployment pipeline.** The environment variable value was not validated against an expected schema, type, or allowlist before the deployment was promoted to production. An invalid value was able to pass through all pipeline stages without triggering a failure or warning.

- **No canary or staged rollout was in place for this deployment.** The change was applied to 100% of payments service traffic simultaneously. A phased rollout would have limited the blast radius and potentially allowed automated gates to catch the error rate spike before full promotion.

- **Deployment health check did not include an end-to-end checkout probe.** Post-deployment smoke tests validated that the payments service process was running and returning 200s on its health endpoint, but did not execute a synthetic checkout transaction. The service appeared healthy to automated gates while failing all real traffic.

- **The environment variable change was bundled with other deployment changes.** Because the configuration change was not isolated, identifying the specific cause required additional investigation time that would not have been necessary with a dedicated configuration-only change.

- **No configuration diff was surfaced in the deployment approval review.** The deployment approval process showed code changes but did not explicitly surface environment variable changes for reviewer attention.

**Why did our existing safeguards not prevent this?**

The deployment pipeline correctly executed the deployment and confirmed the service was running. However, the pipeline's definition of "healthy" — a passing health check endpoint — did not reflect the service's ability to process real payment transactions. The misconfigured environment variable had no effect on service startup or health endpoint response, which meant every automated gate the deployment passed through returned a false positive. No human reviewer was alerted to the environment variable change specifically, as it was not called out distinctly in the deployment review interface. Collectively, the safeguards in place validated process execution rather than functional correctness, creating a gap that allowed a configuration-level fault to reach full production traffic undetected.

---

## Detection

- **How was it first detected?** Automated alert on elevated payments service 5xx error rate.
- **Time from incident start to detection:** 6 minutes.
- **Should we have detected this faster?** Partially. The 6-minute detection time reflects a reasonable alert latency given current thresholds. However, a synthetic checkout probe running continuously post-deploy would likely have reduced detection to under 2 minutes and could have triggered an automatic rollback without requiring human escalation.

---

## Resolution

**What fixed it?** The payments service was rolled back to the previously stable deployment version, restoring the correct environment variable values and returning the service to a known-good configuration state.

**Why did this work?** The prior deployment version carried the correct environment variable values as part of its packaged configuration. Rolling back replaced the misconfigured runtime environment with the validated prior state, immediately restoring correct payment processing behavior.

**Was there a temporary mitigation before full resolution?** No. The rollback was identified as the fastest and lowest-risk path to recovery and was executed directly. No partial mitigation (such as traffic shifting or feature flagging) was applied.

---

## Action Items

| # | Type | Action | Owner | Due Date | Priority |
|---|---|---|---|---|---|
| 1 | **Prevent recurrence** | Add a configuration validation step to the deployment pipeline that validates all environment variables against a typed schema and allowlist before deployment is promoted to any environment. Deployment must fail if validation does not pass. | Platform / DevOps | [DATE + 2 weeks] | P1 |
| 2 | **Prevent recurrence** | Require environment variable changes to be submitted and reviewed as isolated configuration-only deployments, separate from code changes, with an explicit diff displayed in the approval interface. | Platform / Engineering Leads | [DATE + 3 weeks] | P1 |
| 3 | **Prevent recurrence** | Implement canary / staged rollout for all payments service deployments, with automated promotion gates requiring <0.1% error rate on the canary slice before proceeding to full rollout. | Platform / Payments Team | [DATE + 4 weeks] | P1 |
| 4 | **Improve detection** | Add a synthetic end-to-end checkout probe that executes a test transaction against the payments service every 60 seconds. Alert must fire within 2 minutes of probe failure. Probe to run as a post-deploy gate and continuously in production. | Observability / Payments Team | [DATE + 2 weeks] | P1 |
| 5 | **Improve detection** | Lower the 5xx alert threshold for the payments service specifically, and evaluate adding a deployment-triggered alert window that applies a tighter threshold for the 15 minutes immediately following any payments service deploy. | Observability Team | [DATE + 1 week] | P2 |
| 6 | **Improve response** | Write and publish a runbook for "payments service elevated 5xx post-deploy" that explicitly documents: confirm scope, check deployment log, diff environment variables, escalation path, and rollback procedure with commands. | Payments On-Call Lead | [DATE + 1 week] | P2 |
| 7 | **Improve response** | Conduct a tabletop exercise using this incident scenario with the on-call rotation to validate the new runbook and confirm rollback procedure can be executed within 10 minutes of alert fire. | Engineering Manager | [DATE + 6 weeks] | P3 |

> **P1 action items (1–4) must be closed before this incident is marked fully resolved.**

---

## What Went Well

- **Automated alerting worked.** The 5xx alert fired within 6 minutes of the deployment completing — without this alert, detection would have depended on customer reports, which typically arrive with significantly more delay.

- **Root cause was identified quickly once investigation began.** Engineers correctly narrowed the scope to the deployment and identified the specific misconfigured environment variable within approximately 12 minutes of the alert firing, which is a strong diagnostic outcome under pressure.

- **The rollback decision was made quickly and without hesitation.** Once the misconfiguration was confirmed, the team did not attempt a risky forward-fix in a degraded state. The decision to roll back was made cleanly and executed promptly, which kept resolution time well under an hour.

- **Blast radius was contained to checkout.** The fault in the payments service did not cascade to degrade other services. Existing service boundaries and circuit behavior limited the failure domain.

- **Cross-team communication was prompt.** The checkout team was notified within minutes of the alert, preventing them from pursuing a parallel investigation on their own service and keeping responders coordinated.

---

## Lessons Learned

1. **Health checks that pass can still mean a service is broken.** A service returning 200 on `/health` is not the same as a service correctly processing real requests. Deployment gates should validate functional behavior — not just process liveness — especially for services in critical transaction paths.

2. **Configuration changes carry the same risk as code changes and should be treated accordingly.** Environment variable changes are operationally equivalent in impact to code changes. Bundling them with code changes obscures their risk and reduces the visibility reviewers have into what is actually changing.

3. **The blast radius of a misconfiguration is determined before the incident starts.** The choice not to use staged rollouts made a single bad configuration value a 100% user impact event instead of a limited one. Rollout strategy is a risk management decision that should be mandatory for high-criticality services.

4. **Tight alert thresholds and fast synthetic probes disproportionately reduce customer impact.** The difference between a 6-minute and a 2-minute detection time, at the transaction volumes involved, likely represents hundreds of failed checkouts. Investment in detection speed on critical paths has a direct and quantifiable impact.

5. **Runbooks should be written before the incident that needs them.** The response in this incident relied on experienced engineers reasoning through the problem in real time. A pre-existing runbook for post-deploy payment failures would have reduced investigation time and made the on-call rotation more resilient to less experienced responders being paged.

---

## Communication Log

*Populate with actuals. Template entries below:*

| Time (UTC) | Channel | Message Summary |
|---|---|---|
| [T+08:00] | Status page | "We are investigating reports of errors during checkout. Payments may be affected." |
| [T+20:00] | Status page | "We have identified the cause and are implementing a fix. Estimated resolution within 15 minutes." |
| [T+30:00] | Status page | "Checkout has been restored. We are monitoring to confirm stability." |
| [T+45:00] | Status page | "This incident is resolved. A postmortem will be published at [link]." |
| [DATE] | Customer support | Template response issued to users who contacted support during the window — offer of [compensation policy if applicable]. |

---

## Quality Checks

- [x] Timeline has no blame-focused language
- [x] Root cause is specific — names the misconfigured environment variable and the deployment event, not "human error"
- [x] Root cause answers "why did this happen?" — identifies system gaps (no config validation, no canary, inadequate smoke test) not just the symptom
- [x] Contributing factors explain the systemic gaps that enabled the misconfiguration to reach production
- [x] Every action item has an owner category and due date placeholder — assign named owners before publishing
- [x] "What went well" section reflects genuine observations about response behavior
- [x] No action item contains vague language — each names a specific, testable, ownable change
- [x] Executive summary is readable by non-technical leadership
