Making sure that our 600,000+ merchants have a reliable online store that can deal with jumps in demand during one of the biggest shopping holidays is extremely important to us. Things like payment processing, app functionality, and order processing need to be buttoned up before the influx of holiday traffic during Black Friday and Cyber Monday (BFCM).

Here are a few examples of how our Production Engineering team prepares for BFCM:

• Game days: A technique to test the resilience of the platform by degrading platform dependencies in controlled environments. Basically, we simulate BFCM scenarios ahead of time so we’re ready for anything when the real thing comes.
• Load testing exercises: We create volumes similar to what we expect during BFCM and send the load to different applications to ensure the system and its components behave as expected.
• When it’s business as usual, we deploy around 40 new versions of Shopify each day. Leading up to BFCM, we manage our normal cadence of deploys by establishing both a feature freeze and a code freeze.
• • Feature freeze: We’re making changes, but most user-facing features won’t be touched to ensure there aren’t any surprises for our users.
  • Code freeze: Basically we’re keeping everything as-is: Only critical fixes can be deployed.

All of this preparation needs to happen without an interruption in service.

We’re proud to say that the effort we put in to deploying, scaling, and launching new projects on a daily basis gives our merchants access to a platform with 99.98% uptime.

Did all of our preparations work out? With BFCM in the rearview mirror, we can say, yes. Most system metrics followed a pattern of 1.8 times what they were in 2017, with  BFCM 2018 being a record breaker for Shopify. We saw nearly 11,000 orders created per minute and around 100,000 requests per second being served for extended periods during the weekend.