The content stays up when the commerce falls over.
A basket, a payment gateway and a stock service are the three things most likely to saturate on the busiest night of the year. None of them is the reason anybody came. So the site has a mode in which they are switched off and everything else keeps working.
What survives, and what is switched off.
| In read-only mode | Why | |
|---|---|---|
| Every content page | Up, and served from cache | It is static. It has no reason to fall over and it is what most people came for |
| The cameras | Up | They are a separate origin with a separate failure mode, and they are the second reason people came |
| The chapter archive | Up | Pure content, cached, and unaffected by anything commercial |
| Search and the palette | Up | The index is a static file. It works with the database on fire |
| The basket | Off, with an explanation | A basket that half works is worse than one that plainly does not |
| Checkout | Off, with an explanation | Taking a payment when the stock service is down is how you sell four hundred things you have thirty of |
| Account changes | Off | A write is a write. If writes are the problem, writes stop |
| Newsletter signup | Queued, not lost | It is one row. It can wait in a queue and be replayed |
| The seasonal feeds | Fall back to the server clock | Which is how they are built anyway — the season is known before a request is made |
Why degrade rather than fail.
Because the failure mode of a large content site with a shop attached is almost always the shop. The pages are static and cached and will serve a hundred times the normal load without noticing. The basket talks to a stock service, a payment gateway and a session store, and one of those three will be the thing that goes.
The default behaviour when that happens is that the whole site becomes an error page, because the error is thrown at a layer that does not know the difference between a chapter and a checkout. Which means the ninety-five per cent of visitors who came to read something get an outage caused by the five per cent who came to buy something.
Read-only mode is the refusal to do that. Commerce is a set of features that can be turned off independently, they are turned off deliberately when they start to struggle, and the site says so.
The load test that runs before every series.
The test
- Multiple of normal peak
- 100×Not fifty. A hundred, because a series release has produced a fifty-times spike and the test should exceed the event.
- Duration at peak
- 20 minutesThe realistic shape is a four-minute ramp and a twenty-minute plateau, not a spike.
- What is measured
- Time to first byte, error rate, and which service failed firstThe third one is the useful output. Everything eventually fails; the question is what fails first.
- Run
- Before every series release, without exceptionAnd after any significant change, which is the clause everybody skips.
What it has to prove
- Content stays up at 100×
- Pass or the release is not coveredIt is static and cached. If this fails, something has been made dynamic that should not have been.
- Commerce degrades rather than cascades
- PassThe basket may fall over. It may not take the content layer with it.
- Read-only engages automatically
- PassA mode that requires somebody to be awake is not a mode.
- The queue holds
- PassAnd the waiting room page itself is static, on a separate origin, for obvious reasons.
What it costs
- A day of engineering time, twice a year
- RoughlyPlus the cost of the load generation itself, which is small.
- An argument, every time
- ReliablyBecause it is scheduled in the fortnight when everybody is busiest, which is precisely why it has to be scheduled.
What actually happens on the night.
Traffic ramps
Fifty to a hundred times normal, over about four minutes, from a push notification and a social post.
The queue engages at the edge
Before anything reaches an application server. The waiting room is static and on a separate origin.
Commerce starts to struggle
Stock lookups slow first, every time. It is the most contended service in the stack.
Read-only engages automatically
On a latency threshold rather than on somebody noticing. The basket goes off and says why.
Content and cameras carry on
Which is what the overwhelming majority of the traffic came for anyway.
It comes back in order
Reads, then writes, then payment last. Payment is the last thing on and the first thing off.
This build has no commerce to degrade
Every checkout control on this site is a placeholder, no card details are collected anywhere and no payment is taken. Read-only degradation is described here as it would be built; there is nothing behind it to switch off.
When it breaks, questioned.
Because a series release has produced a fifty-times spike, and a test that only just covers the known event covers nothing at all. The margin is the point.
Because a mode that needs a person to notice and act is a mode that engages fifteen minutes late, on the one night when fifteen minutes is the whole event.
A named state with a plain explanation of what is off and roughly when it will be back, rather than a spinner or a generic error. Every state on this site is named and has a page.
And the queue that holds them at the door.
An edge waiting room for a spike of fifty to a hundred times in four minutes.
