Infosoft - Intermediate errors – Incident details

All systems operational

Intermediate errors

Resolved
Partial outage 50 %
Started about 1 year agoLasted about 2 hours

Affected

INFO-Subscription

Partial outage from 8:31 AM to 9:23 AM, Degraded performance from 9:23 AM to 10:15 AM

API

Partial outage from 8:31 AM to 9:23 AM, Degraded performance from 9:23 AM to 10:15 AM

Updates
  • Postmortem
    UTC
    Postmortem

    The underlying issue was caused by a deployment of an updated security certificate.
    Unfortunately the deployment also triggered a permission switch, replacing an identity with a new one, but the old identity was removed before the new one was deployed, causing some elevated operations to fail.

  • Resolved
    UTC
    Resolved

    The issue has been resolved an all services are operating at normal capacity.

    The underlying issue was caused by a deployment of an updated security certificate.
    Unfortunately the deployment also triggered a permission switch, replacing an identity with a new one, but the old identity was removed before the new one was deployed, causing some elevated operations to fail.

  • Update
    UTC
    Update

    All operations are now operational, but with reduced capacity/performance until the fix has propagated to all nodes.
    We are continuing to monitor the results.

  • Monitoring
    UTC
    Monitoring

    We implemented a fix and are currently monitoring the result.
    Most operations are performing as expected, but there are still intermediate errors as the fix continues to propagate through the platform.

  • Identified
    UTC
    Identified

    We have identified the cause of the issue and apply a fix to bring back the service to full capacity.

  • Investigating
    UTC
    Investigating

    We are currently investigating an issue with the platform.
    API requests are taking a long time, and in some cases generating error responses.