Skip to main content

Incidents

Open incidents

3

Median time to acknowledge

3m 12s

Median time to resolve

41m

Error budget left (30 days)

62%

SEV1INC-2041Started 14:02 UTC · 12m ago

Checkout requests failing for card payments

Relay summary

Error rate on checkout-api rose from 0.2% to 18% at 14:02, four minutes after deploy v2.42.0. Failures are limited to card payments; wallet payments are unaffected. The new build raised the payment client's timeout retries, and payments-worker is now saturated.

Drafted from alerts and recent deploys. Review before sharing.

Affected services

  • checkout-api
  • payments-worker
  • web-storefront

Suggested next steps

  • Roll back checkout-api to v2.41.0
    The last healthy build. Estimated 3 minutes.

  • Scale payments-worker from 6 to 12 replicas
    Relieves the queue while the rollback completes.

  • Draft a status-page update
    Opens a draft for review. Nothing is published.

Timeline

  1. 14:06Paged Ana Kowal (primary on-call)
  2. 14:03Alert: checkout-api 5xx rate above 5% for 60s
  3. 13:58Deploy v2.42.0 to production by CI