Incident Time (UTC):

1. 07:39 – 08:50 – High error rates and latency in all APIs. Peaks up to 1 min in 0.5 percentile. Latency of 1 min in 0.9 percentile. Success rate drop for 2 min to 0.77
2. 09:50 – 10:33 – Peaks of latency in Payments API in the 0.9 percentile of up to 1 min. Success rate 100%
3. 10:33 - 10:55 - Success rate drops to 98% in Payments API and latency peaks in Tokenization API. Success rate drop is related to specific integrations (PayUPoland, Safecharge, Daleynes, PayuCitrus ,Worldpay, RSB, Sberbank, Wirecard) and/or API version 1.3
4. 10:55 - 12:40 - Success rate drops to 98% in Payments API and latency peaks in Tokenization API. Most integrations recover except for Safecharge, Daleynes, PayuCitrus and/or API version 1.3.

Incident Details (step by step):

07:39 – 08:28:

• Zooz on-call team gets alerts and detects issues with services originating from some DC workers and Cassandra nodes down.
• eu-central-1c AZ is marked as the zone from which all problematic services and nodes originated.
• Official recognition from AWS regarding the outage.
• After considering stopping problematic nodes, clarification received in a discussion with AWS support claims that stopping instances isn’t currently feasible due to the AZ outage.
• Latency issue is seen in the system due to timeouts to Cassandra as the nodes were flapping and could not be stopped (also due to AWS outage).
• All Cassandra clusters self recovered

08:29 - 09:50:

• System returns to normal latency and 100% success rate.

09:50 – 10:33:

• One Cassandra node in one of clusters is flapping.
• Latency issues start again in the system.
• Zooz team stops the node as the system should be able to operate properly with one node down.
• Latency continues due to a bug in our API gateway that continues trying to reach the stopped node.
• The latency was partial and did not affect the majority of API calls. Success rate remains 100%.

10:33 - 10:55:

• To decrease latency ZOOZ team starts the problematic node. Latency is back to normal.
• Notification received from AWS about issues with EBS volumes in the same AZ - “Some EBS volumes are experiencing degraded performance in a single Availability Zone in the EU-CENTRAL-1 Region. We are working to resolve this issue. Network connectivity errors and elevated API error rates in the Availability Zone have been resolved”
• After the Cassandra node joined again ZOOZ team begins to see permission errors from various services due to corruption of sys_auth keyspace.
• Success rate drops to 98%.

• `specific services in the system experience full/partial outage:

  • PayUPoland integration - partial
  • SafeCharge - full
  • Payments API version 1.3 - partial
  • Dalenys - partial
  • PayUCitrus - full
  • WorldPay - partial
  • RSB - partial
  • Sberbank - partial
  • Wirecard - partial

10:55 – 12:40:

• To handle the corruption as soon as possible the problematic node is removed and the team starts working on provisioning a replacement node.
• Spikes of latency appear in the system again in the 0.9 percentile

• Some services remain in partial/full outage:

  • SafeCharge - full
  • Payments API version 1.3 - partial
  • Dalenys - partial
  • PayUCitrus - full

• Cassandra node preparation took more time than expected due to the complexity of the terraform and Ansible playbook failures.

12:40:

• New Cassandra node replaces the old damaged one.
• A repair of the sys_auth table is done.
• System returns to normal.

14:23 - Resolution notification from AWS that all issues in the problematic AZ are resolved

Damage Assessment:

• Partial outage and drop in success rate of the system.
• High latency in the 0.9 percentile.
• 4000 5XX errors were returned from the system during the whole time period.

Incident investigation and findings:

• Our strategy to spread across AZs proved itself in our new DC. We need to speedup migration.
• Our Cassandra automation is not good enough and involves manual intervention; it takes far too long to provision a new node.
• Our API gateway does not handle well Cassandra node down compared with other services in the system.

Action Plan:

• Continue with accelerated migration of the system to our new DC with the strategy of spreading instances across AZs - in process.
• Improve Cassandra node replacement - reduce time to 5 min - in process.
• Upgrade our API gateway - the newly released version includes a fix to the node refresh mechanism - in process.
• Add chaos tests for full AZ failure to be able to turn off an AZ if a similar incident in AWS would happen again - planned for Jan