The team has shifted traffic from Green back to Blue. The overall error rate has fallen, but some customers still can’t open their orders. Would you declare the service restored? What would you check next?
This hypothetical incident picks up where my article on blue/green deployment on AWS left off. I’ll use it to look at what a team can do when a fix only partly delivers the result it expected. The improving graph shows that something recovered. The work customers still can’t do points to where the team should look next. The previous article on blue/green deployment on AWS
A mental model is a simplified explanation of how a system works. A useful model helps a team say what it expects to see and change its explanation when the evidence points elsewhere. Here, start with two separate questions. Which version is handling a request, and can that version still read the data it needs?
I separate the habits that keep a team open to evidence from the models that help it ask better questions. Say what you don’t know, distinguish observations from conclusions, and agree on what would change your mind. Then pick a model for the decision. If you’re losing track of how systems interact, draw a causal map. If several explanations fit, look for evidence that distinguishes them. If a release decision is coming up, ask which effects are hardest to undo and use a premortem to choose what to rehearse.
Match the method to the problem
To find out whether a request reached Blue or Green, the team can trace it from the Application Load Balancer (ALB) rule to the application logs. The job might be complicated and require someone who knows the system, but with enough evidence the team can reconstruct the path. In Cynthia Kurtz and Dave Snowden’s Cynefin framework, this resembles a context where analysis can establish cause and effect, called knowable in the original paper. Kurtz and Snowden’s original paper
Predicting how far the incident will spread may be more complex. Some customers may try again, the system may retry automatically, and background jobs may delay the effects of a change. When these factors interact, the team may need to run a limited experiment and observe the behavior before deciding what to do. Cynefin calls this probe, sense, respond. Knowing the ALB rules helps answer the first question. Knowing what customers experience takes a look at the whole workflow. Kurtz and Snowden’s original paper
Map what changed and what stayed behind
Start the map with two paths. Follow a request from the customer through the ALB to the app. Then follow the data Green wrote to the shared database. After the rollback, new requests go to Blue, and orders Green wrote are still there. If orders that fail to open cluster around the period when Green served traffic, the team has a reason to check the data format first. Timing alone leaves the cause open.
Add another arrow if the evidence shows that failed requests trigger retries. Retries may put more load on the app and database, slowing responses and triggering still more retries. That’s one example of a reinforcing feedback loop. Donella Meadows’s work directs attention to both feedback and the delay before an effect becomes visible. A quick recovery in the overall graph may leave open whether the queue has cleared or affected customers can finish their work. Donella Meadows’s Leverage Points
Use competing hypotheses to choose what to check
I’d start with a short observation: “After the rollback, some customers still can’t open their orders.” Then I’d write down more than one plausible explanation and say what I’d expect to see if each were true. That gives the team a way to choose evidence that distinguishes the explanations without trying to inspect every graph at once. Google SRE’s troubleshooting method
| Hypothesis | What to check |
|---|---|
| Some requests still reach Green | Trace the request IDs for failed requests and verify which app version responded. |
| Blue can’t read a value written by Green | Compare the responding version, write time, and data format for orders that open and ones that fail. Then test a de-identified example in an isolated environment. |
| Retries or queued jobs cause some requests to time out | Compare retry rate, queue age, and response time in the same window. Distinguish those failures from immediate errors when the app reads an order. |
If Blue fails to read a test order that Green wrote in an isolated environment but can still read an order in the old format, that supports the data-format hypothesis for the cases tested. If both open, ask how the test environment differs from production and which explanation is still worth investigating. A result carries weight within the scope of the test.
Limit the impact of an experiment and rehearse the way back
Review existing evidence before changing production. If it still leaves competing explanations open, reproduce the behavior with one test order in an isolated environment. Disable live email and payments. Decide in advance what result would make you stop or change your hypothesis. If harm is increasing, such as duplicate charges, use the incident procedure to stop the source of harm before continuing to investigate. Google SRE’s troubleshooting method
Before the next release, try Gary Klein’s premortem. Assume the team shipped, had to roll back, and still left some customers unable to use the service. Have each person write down possible reasons before discussing them, then choose which risk to rehearse. In this case, the team could have Green create an order and verify that Blue can open and continue processing it without charging or messaging anyone twice. Gary Klein’s Performing a Project Premortem
Ask how reversible each change is during that rehearsal. The ALB rule can point back if Blue is still ready. Data already written and work sent to other systems each need a recovery plan. Where an effect is hard to undo, move the experiment earlier or keep the new behavior behind a feature flag while Blue is still part of the recovery plan. AWS guidance on data and schema changes during blue/green deployments
Review how the team could spot the problem sooner
Afterward, record what the team knew when it decided, where the outcome differed from expectations, and which customers were still affected. Use the timeline to discuss the decision without blaming the person who shifted traffic. If the team couldn’t see order-read failures by cohort, add that signal or a pre-release test. Name an owner and decide how to check that the change helps. Google SRE’s postmortem guidance
Pick a problem your team is working on and write down the explanation you currently believe. Then answer one question: What evidence would make the team change its mind? That gives you a place to look next.