Knowledge Base / Cheat-sheets
What-If
Turning a vague worry into a falsifiable scenario, dependency and resource failure injection, data and state edge cases, detection and recovery drills, and a dedicated resilience tooling roundup - five sections you can filter and toggle as you verify them. Progress is saved on this device.
What this checklist covers
Framing the What-If
A what-if that nobody can answer is not a test, it is anxiety. The work is turning a vague worry into a specific, falsifiable question with a stated blast radius - and the assumptions you are quietly relying on are usually the best place to find them.
- The scenario names a specific trigger, not a mood - 'what if the payment provider returns 503 for 90 seconds' rather than 'what if payments break'.
- The expected behaviour under that trigger is written down before the test, or any observed outcome will be rationalised as intended.
- The assumptions the design rests on are listed explicitly - the network is reliable, latency is zero, the clock moves forward, the disk has room.
- Each listed assumption is turned into a what-if by negating it, which generates scenarios systematically rather than by imagination.
- The set of users, data and downstream systems a failure could reach is mapped before the scenario is run anywhere.
- Single points of failure are identified deliberately, including the shared ones - one auth service, one region, one certificate.
Failure & Chaos Scenarios
The classic what-ifs, and the ones distributed systems answer worst. Every dependency you do not control will eventually be slow, wrong, or absent, and the failure mode that hurts is almost always slow rather than absent.
- Each external dependency is tested unavailable, and separately tested slow - a hung dependency exhausts connection pools in a way a clean refusal never does.
- Malformed and unexpected responses are injected, not only error codes, since a 200 with the wrong shape bypasses most error handling.
- Behaviour at disk-full is tested, since it commonly surfaces as silent data loss rather than a clean error.
- Memory pressure and OOM-kill are exercised to confirm the process restarts cleanly and loses nothing it acknowledged.
- Each experiment starts from a written steady-state hypothesis, or it is an outage rather than an experiment.
- Experiments run in a pre-production environment until the team has demonstrated it can detect and stop them.
Data & State Edge Cases
The what-ifs that live inside your own boundary. These rarely announce themselves as outages - they show up months later as a reconciliation mismatch nobody can explain.
- Empty, single-element and maximum-size inputs are tested at every boundary, as these three catch a large share of collection-handling defects.
- Unicode is tested properly - combining characters, right-to-left text, emoji, four-byte characters - not just accented Latin.
- Simultaneous writes to the same record are tested, confirming the last-write-wins behaviour is the one that was intended.
- Double-submit and duplicate-message paths are tested against the idempotency guarantees the system claims.
- Daylight-saving transitions are tested in both directions, since the repeated hour and the missing hour break different code.
- Leap years, leap seconds and month-end boundaries are covered wherever scheduling or billing is involved.
Response, Recovery & Assurance
Surviving the scenario is half of it. The other half is whether anyone noticed, whether the documented recovery actually works, and whether the risk you chose to accept was written down where the next person will find it.
- Every scenario asks whether monitoring detected it, since an undetected failure is worse than a loud one.
- Time-to-detect is measured as its own metric alongside time-to-recover.
- Rollback is executed rather than documented, because an untested rollback is a plan and not a capability.
- Backup restoration is drilled end to end and timed, since a backup that has never been restored is an assumption.
- Scenarios considered and deliberately not mitigated are recorded with the reasoning, so the decision can be revisited rather than rediscovered.
- Accepted risks name the person accepting them and the date, since risk tolerance changes with scale and with staff.
Scenario & Resilience Tooling
Instruments for injecting the failure and observing what happened. Tooling makes scenarios repeatable; it does not tell you which scenarios are worth running.
- Chaos Mesh and LitmusChaos inject pod, network and I/O faults inside Kubernetes and express experiments as manifests you can version.
- AWS Fault Injection Service works at the cloud-provider layer, reaching AZ and instance failures that in-cluster tools cannot.
- Toxiproxy sits between service and dependency to add latency, bandwidth limits and connection drops - the cheapest way to test slow rather than absent.
- WireMock stubs HTTP dependencies with fault and delay responses, so third-party failure modes are reproducible in CI.
- Hypothesis (Python), fast-check (JS/TS) and jqwik (Java) generate inputs and shrink any failure to a minimal reproducing case.
- Properties are stated as invariants that must hold for all inputs, which is a different and more searching exercise than writing examples.
- Distributed tracing is in place before chaos work begins, or an injected failure produces a mystery rather than a finding.
- Grafana or an equivalent holds a dashboard defining steady state, so the hypothesis has something concrete to be measured against.