Infosoft - Checkout reporting 500 Internal Server Error – Incident details

All systems operational

Checkout reporting 500 Internal Server Error

Resolved
Major outage
Started 1 day agoLasted about 1 hour

Affected

INFO-Subscription

Major outage from 11:29 AM to 12:16 PM, Partial outage from 12:16 PM to 12:45 PM

Self Service

Major outage from 11:29 AM to 12:16 PM, Partial outage from 12:16 PM to 12:45 PM

Updates
  • Resolved
    Resolved

    This incident has been resolved. We are continuing to monitor the setup and we will start investigating how to prevent the issue in the future.

  • Monitoring
    Monitoring

    We have identified the source of the traffic and taken steps to isolate the issue so it does not affect all tenants. We are continuing to monitor the issue.

  • Identified
    Identified

    We have identified a 80 fold increase in traffic to our checkout applications, which in turn is causing the upstream authentication service to throttle requests.

    We are still investigating how to mitigate the issue.

  • Investigating
    Investigating

    We are currently investigating an issue where the Checkout Application is reporting intermediate 500 Internal Server errors.

    Not all tenants are affected at the current time, but the issue seems to be related to the authentication environment.