Incident summary

A blameless write-up: what broke, why, and what stops it recurring.

Engineering · produces a Document · built in, ready to run

What you give it

Something like this:

Orders down 2026-02-18. Started ~09:40 when the morning deploy went out. Price list endpoint returned 500 for any store on the new tiered pricing. Stores saw 'price list unavailable' and could not order. Nobody noticed until Bilal called at 10:25 — our alerting only checks the homepage. Junaid found it at 10:40, the deploy included a migration that renamed price_tier to tier_id but the pricing cache was still serving old keys. Rolled back at 11:05, cache cleared 11:12, orders flowing 11:15. Roughly 90 minutes, about 60 stores affected, 140 orders delayed. The rollback was slow because nobody had run one before and the runbook was out of date.

What you get back

This is a real run of the skill on the input above, not a mock-up:

## **INCIDENT SUMMARY**
**Date:** 2026-02-18  
**Title:** Orders Service Unavailable – Price List Endpoint Failure  
**Duration:** 90 minutes (09:40–11:15)  
**Severity:** High

---

## **IMPACT**
- **Who:** 60 retail stores
- **What:** Customers unable to place orders; price list endpoint returned HTTP 500 errors
- **How long:** 90 minutes until full resolution
- **Business effect:** 140 orders delayed

---

## **TIMELINE**
| Time | Event |
|------|-------|
| 09:40 | Morning deploy released to production (included database migration: `price_tier` → `tier_id`) |
| 10:25 | Bilal receives customer complaints and escalates; incident detected |
| 10:40 | Junaid identifies root cause: price list endpoint returning 500s |
| 11:05 | Deploy rolled back to previous version |
| 11:12 | Cache cleared following rollback |
| 11:15 | Orders flowing normally; incident resolved |

---

## **ROOT CAUSE**
The morning deploy contained a database schema migration that renamed the `price_tier` column to `tier_id`. The pricing cache layer was not updated as part of the migration and continued serving responses with the old key names. When the application layer expected the new schema (`tier_id`), a mismatch occurred, causing the price list endpoint to fail with HTTP 500 errors for all stores.

---

## **WHAT MADE IT WORSE**
1. **Incomplete alerting coverage:** Production monitoring only checked the homepage; the price list endpoint was not instrumented for alerting, resulting in a 45-minute detection gap.
2. **Skipped smoke tests:** The deploy process did not execute the smoke test suite, which would have caught the price list endpoint failure immediately post-deployment.
3. **Stale runbook:** The rollback procedure documentation had not been maintained and contained outdated steps, causing confusion and delays during incident response.
4. **No prior rollback execution:** The team had never performed a production rollback before, adding uncertainty and manual troubleshooting time.

---

## **WHAT MADE IT BETTER**
1. **Fast manual escalation:** Customer reports via Bilal provided rapid external detection of the issue.
2. **Quick root cause identification:** Junaid's systematic investigation quickly identified the schema/cache mismatch and linked it to the deploy.
3. **Successful rollback execution:** Despite procedural friction, the team executed the rollback and cache clear successfully.

---

## **FOLLOW-UP ACTIONS**

| Action | Owner | Priority |
|--------|-------|----------|
| Add price list endpoint to production alerting; alert on 5xx errors or latency anomalies | Platform/Observability team | **Critical** |
| Enforce smoke test execution in deploy pipeline; block production deploys that skip tests | DevOps/Release Engineering | **Critical** |
| Audit and update rollback runbook with current procedures; include cache invalidation steps | On-call lead / SRE | **High** |
| Schedule and conduct team rollback drill to build muscle memory and reduce MTTR | Engineering lead | **High** |
| Add cache invalidation to database migration pre-flight checklist | Database/Schema team | **High** |

---

**Blameless principle applied:** This incident was caused by process and tooling gaps (incomplete monitoring, skipped automated testing, outdated procedures) rather than individual action. Follow-ups address the underlying systems to prevent recurrence.

The wording changes a little every run. The shape does not — that is what the procedure below is for.

Tools it uses

None of its own. A skill that only writes needs no tools, so this one inherits whatever your assistant can normally reach.

The procedure it follows

This is the skill, in full. Add it to your own skills to change any of it.

Write up the incident described below.

Sections: impact (who, how long, what could not be done), timeline, root cause, what made it worse, what made it better, and follow-up actions with owners.

Blameless: name systems and decisions, never people. "The deploy skipped the smoke suite", not "X forgot to run the tests".

How to run it

In chat, on the Skills page and in automations you do not have to add it first — a library skill runs as it is. Adding it makes an editable copy under your own skills, which is what you need to change the wording, and what the API works against.

New to skills? Start with what a skill is.