Infosoft - Intermediate errors – Incident details

Intermediate errors

ResolvedPartial outage 50%1 hour 44 minutes

Affected

INFO-Subscription

Partial outage from 8:31 AM to 9:23 AM, Degraded performance from 9:23 AM to 10:15 AM

API

Partial outage from 8:31 AM to 9:23 AM, Degraded performance from 9:23 AM to 10:15 AM

  • PostmortemUTC

    The underlying issue was caused by a deployment of an updated security certificate.
    Unfortunately the deployment also triggered a permission switch, replacing an identity with a new one, but the old identity was removed before the new one was deployed, causing some elevated operations to fail.

  • ResolvedUTC

    The issue has been resolved an all services are operating at normal capacity.

    The underlying issue was caused by a deployment of an updated security certificate.
    Unfortunately the deployment also triggered a permission switch, replacing an identity with a new one, but the old identity was removed before the new one was deployed, causing some elevated operations to fail.

  • UpdateUTC

    All operations are now operational, but with reduced capacity/performance until the fix has propagated to all nodes.
    We are continuing to monitor the results.

  • MonitoringUTC

    We implemented a fix and are currently monitoring the result.
    Most operations are performing as expected, but there are still intermediate errors as the fix continues to propagate through the platform.

  • IdentifiedUTC

    We have identified the cause of the issue and apply a fix to bring back the service to full capacity.

  • InvestigatingUTC

    We are currently investigating an issue with the platform.
    API requests are taking a long time, and in some cases generating error responses.