Duration: 1 hour and 16 minutes

Affected Components: Self Service

May 19, 11:29:28 GMT+0
Investigating - We are currently investigating an issue where the Checkout Application is reporting intermediate 500 Internal Server errors. Not all tenants are affected at the current time, but the issue seems to be related to the authentication environment..

May 19, 11:53:14 GMT+0
Identified - We have identified a 80 fold increase in traffic to our checkout applications, which in turn is causing the upstream authentication service to throttle requests. We are still investigating how to mitigate the issue..

May 19, 12:16:34 GMT+0
Monitoring - We have identified the source of the traffic and taken steps to isolate the issue so it does not affect all tenants. We are continuing to monitor the issue..

May 19, 12:45:26 GMT+0
Resolved - This incident has been resolved. We are continuing to monitor the setup and we will start investigating how to prevent the issue in the future..

Duration: 27 minutes

Affected Components: Self Service

Aug 7, 10:40:46 GMT+0
Investigating - We are currently investigating an incident causing checkout processes to be non-functional..

Aug 7, 10:52:57 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Aug 7, 11:07:41 GMT+0
Resolved - This incident has been resolved. All checkout and order api related operations are functioning as expected..

Aug 7, 11:12:36 GMT+0
Postmortem - A reconfiguration of the main API related to security hardening was done on the 6th of August around 9.00 CEST. This reconfiguration cuased the Orders API to only be granted access to a few tenants, due to a human configuration error of the permissions. Adding the required permission to affected tenants immediately resolved the issue and allowed the Order API to communcate with the main API as expected. Unfortunately the error was not discovered beforehand, because the tenants used in surveillance testing were allowed access. Additionally the type of API error emitted is a common error type when consumer requests data without sufficient permissions. We will investigate how to avoid such issues in the future. Sorry for any inconveniences this may have caused..

Duration: 1 hour and 44 minutes

Affected Components: API

Jul 1, 08:31:03 GMT+0
Investigating - We are currently investigating an issue with the platform. API requests are taking a long time, and in some cases generating error responses..

Jul 1, 08:41:01 GMT+0
Identified - We have identified the cause of the issue and apply a fix to bring back the service to full capacity..

Jul 1, 08:49:24 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result. Most operations are performing as expected, but there are still intermediate errors as the fix continues to propagate through the platform. .

Jul 1, 09:23:53 GMT+0
Monitoring - All operations are now operational, but with reduced capacity/performance until the fix has propagated to all nodes. We are continuing to monitor the results..

Jul 1, 10:15:24 GMT+0
Resolved - The issue has been resolved an all services are operating at normal capacity. The underlying issue was caused by a deployment of an updated security certificate. Unfortunately the deployment also triggered a permission switch, replacing an identity with a new one, but the old identity was removed before the new one was deployed, causing some elevated operations to fail..

Jul 1, 10:15:42 GMT+0
Postmortem - The underlying issue was caused by a deployment of an updated security certificate. Unfortunately the deployment also triggered a permission switch, replacing an identity with a new one, but the old identity was removed before the new one was deployed, causing some elevated operations to fail..

Duration: 2 hours and 13 minutes

Affected Components: Merchant, API

Jun 18, 05:58:26 GMT+0
Monitoring - We have identified the issued and implemented a fix and are currently monitoring the result. Most tenants have become immediately operational, but data needs to be processed so some tenants may still experience some delays in the analytics data..

Jun 18, 05:33:14 GMT+0
Identified - We are currently investigating an incident related to an upgrade where economy related data is not being processed for reporting/analytics. There are no indications that data is lost, just that it is queue for processing and not being processed in a timely manner..

Jun 18, 07:46:30 GMT+0
Resolved - Everything is operation as normal again and new data is being processed in a timely manner. Some data generated during the degradation period still needs to be processed completely, it is currently being processed steadily..

Duration: 4 hours and 42 minutes

Affected Components: Merchant

Apr 2, 02:15:00 GMT+0
Investigating - A deployment during the night has left the administrative merchant client unavailable. We are currently investigating the issue..

Apr 2, 05:16:34 GMT+0
Monitoring - We have rolled back the deployment and the merchant is now back online. We are still investigating the cause of the deployment error..

Apr 2, 06:56:58 GMT+0
Resolved - Merchant is running steadily with no errors. We are still investigating the underlying cause for the deployment error to avoid a repetition..

Duration: 2 hours and 38 minutes

Affected Components: Merchant

Nov 20, 08:00:00 GMT+0
Monitoring - From around 09:00 CET today there has been a slow down for some tenants when navigating within the Merchant application. Especially the dashboard and areas related to search and filtering has been affected. The issue is due to a limit in the automatic resource allocation for some applications. We have raised the limit and the backlog is being processed, and already showing clear signs of improvement. An update will be posted once we can confirm the issue has been solved..

Nov 20, 10:37:37 GMT+0
Resolved - Performance are back to normal numbers and have been steady for the last 30 minutes..

Duration: 12 hours and 27 minutes

Affected Components: Self Service, Merchant, API

Oct 14, 06:18:50 GMT+0
Investigating - We are currently investigating an issue with logging into the merchant administrative portal. Updates will be sent out as we get more information about the issue. Self-services, salesposter and API operations appear to be unaffected, but we are investigating if there should be any unforseen side effects..

Oct 14, 06:26:47 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Oct 14, 18:46:00 GMT+0
Resolved - This incident has been resolved..

Duration: 1 hour and 16 minutes

Affected Components: Self Service, Merchant, Third Party: Vipps MobilePay → Recurring API, API, Mastercard Payment Services

Aug 30, 09:00:00 GMT+0
Monitoring - This morning at roughly 11:00 CET, the certificate used to encrypt traffic to/from the API for INFO-Subscription expired. This caused interactions between our UI applications (Self-Service and Merchant) to be affected, as well as any third party integrations that verify the certificate properly when interacting with the API. The certificate was replaced with a valid one around 11:45 CET and everything is back to normal. Ordinarily the certificate is automatically replaced with a working certificate a month prior to the expiration. This time around, the tooling we rely on for replacing the certificate did not replace it. The altert for the missing new certificate was handled, but unfortunately we were not aware of extra actions needed to replace the certificate at that time. We apologize for any issues this may have caused..

Aug 30, 10:15:34 GMT+0
Resolved - This incident has been resolved..

Duration: 21 hours and 36 minutes

Affected Components: Self Service, Merchant, API

May 5, 14:53:40 GMT+0
Investigating - We are currently investigating an issue with obtaining authentication across the platform. Initial log investigation points to Azure ADB2C infrastructure being unavailable for token generation. Regular authentication works, but the underlying API authorization fails across the board. We have yet to get any confirmation from Microsoft about the issue and we are working on the problem. Unfortunately, when unable to authenticate no other actions are allowed by the platform..

May 5, 15:14:38 GMT+0
Monitoring - ADB2C seems to now be issuing tokens again and authentication works. There is already a drop in the error rate on our metrics. We will continue to monitor this, and will provide more information if and when we receive it. For now the systems are back again, but we are unsure if they will remain available..

May 5, 15:51:58 GMT+0
Monitoring - Microsoft Azure Support has confirmed an issue in their ADB2C infrastructure in Europe. The issue should be fixed now, but we will continue to monitor closely..

May 5, 16:41:22 GMT+0
Identified - Unfortunately the issue has not been solved entirely, and we are now experiencing timeouts, but the occasional token request makes it through. We have informed Microsoft about the re-occurrence and they are actively working on asolution..

May 5, 18:36:31 GMT+0
Identified - The Microsoft Azure support team has now shared a status update on the incident. The issue continues to affect the entire platform. They are still trying to figure out the underlying cause..

May 6, 12:30:00 GMT+0
Resolved - During the night the issue has been resolved by Microsoft and everything is operational again. Specifically the underlying service was recovered at 00:27 UTC (02:27 CET). Our metrics indicates everything is operational again..

Duration: 21 hours

Affected Components: Mastercard Payment Services

Apr 13, 22:00:00 GMT+0
Identified - We have identified an issue with communications to Mastercard Payment Services (MPS), affecting everything related to AvtaleGiro and eFaktura. The effect is that claims and invoices are not submitted to MPS, nor can we retrieve new mandates and agreements. We have notified Mastercard Payment Services, and are working towards a resolution..

Apr 15, 07:54:49 GMT+0
Identified - we have identified the issue as being due to an expired certificate. We are working with MPS to resolve the issue..

Apr 16, 10:45:00 GMT+0
Resolved - The issue has been resolved, all queued up operations have been processed and we have monitored that everything is running as it should with continous eFaktura scanning, AvtaleGiro mandates and related operations. Internally we have updated our monitoring procedures to alert us to similar cases, so that in the future we may avoid the disruption..

Duration: 1 hour and 30 minutes

Affected Components: API

Mar 12, 07:16:45 GMT+0
Identified - We have identified an issue with the AvtaleGiro Claims related API from Mastercard Payment Services. The effect is that some claims are not generated, and obtaining claims status fails roughly a third of the time. We have notified Mastercard Payment Services, and are working towards a resolution. The consequences are currently a delay in claims processing, as soon as the incident is resolved by Mastercard we will continue to process outstanding claims..

Mar 12, 08:47:08 GMT+0
Resolved - Mastercard Payment Services has reported that everything is operational again. We have verified this on our end, and all outstanding claims have now been processed..

Duration: 20 hours and 46 minutes

Affected Components: Self Service

Jan 16, 06:24:22 GMT+0
Identified - We are continuing to work on a fix for this incident. We are in contact with Microsoft to have them revert the configuration (or adjust it), at the same time we are working towards an application fix to circumvent the issue entirely..

Jan 15, 12:31:50 GMT+0
Investigating - We are currently investigating an incident where Vipps orders returns an error reporting that the resource was not found or moved. Orders using other types of payment providers still work as expected..

Jan 15, 14:40:24 GMT+0
Identified - We have identified the issue as an undocumented change in underlying Azure infrastructure. The change has restricted URLs to 2048 characters in length. For tenants with long product and package names and descriptions will end up with a Vipps order confirmation URL that is longer than those 2048 characters. A temporary workaround is to reduce the Name and Description as much as possible on the subscription plan. We are working with Microsoft to increase the limit, and at the same time have started work on an application fix to reduce the length for our generated URLs..

Jan 16, 06:52:20 GMT+0
Monitoring - We implemented a temporary configuration fix and are currently monitoring the effects..

Jan 16, 09:18:00 GMT+0
Resolved - No new occurence of the issue has been observered since the fix was deployed. Everything is operational again. We will deploy an application fix at a later time as a more permanent solution. Updated Analysis/Description of the Issue: The initial assumption that the cause was due to an underlying Azure reconfiguration was only partially correct. The Azure reconfiguration happened 3 months ago, enforcing the URL limit to become 2048 chars (which is documented default, so basically they fixed a bug). This did not immediately have any effect. On January 9th, Vipps deployed a change causing their order/agreement draft response to contain additional characters. For some tenants, this tipped the generated URLs above the 2048 character limit. Our current solution has been to reconfigure the limit, but we are working on a more permanent solution where such changes to third parties will not interrupt the workflow again. Sorry for any inconveniences this may have caused..

Jan 18, 07:44:27 GMT+0
Resolved - We are in the process of rolling out a more permanent solution to the previous incident. It will be deployed throughout the day for all tenants..

Duration: 1 hour and 27 minutes

Affected Components: Self Service

Nov 29, 12:14:41 GMT+0
Monitoring - Configuration changes are currently being applied. We are monitoring and going through all tenants to ensure the sales poster is up and running again..

Nov 29, 13:14:39 GMT+0
Resolved - This incident has been resolved. All salesposters are now up and running again..

Nov 29, 11:47:35 GMT+0
Investigating - We are currently investigating an incident affecting multiple tenants, resulting in salesposter error messages. Not all tenants are affected..

Nov 29, 11:53:09 GMT+0
Identified - We have identified the problem. An invalid configuration option for the affected tenants coupled with a change in the newly released version is causing the issue. We have fixed the configuration issue and are applying the configuration changes across the affected tenants..

Duration: 8 hours and 54 minutes

Affected Components: API

Nov 1, 08:33:58 GMT+0
Resolved - This incident has been resolved. All queued work has been processed and new work is being processed as normal. We are sorry for any inconveniences this may have caused..

Oct 31, 23:40:00 GMT+0
Monitoring - Due to a failed processing replica, billing is slightly delayed. We have fixed the processing issue and are currently processing the queue that has build up. We expect everything to be back to normal within an hour or two. The consequences are a few hours delays in generation of some of the invoices. Most of the build up queue relates to preliminary billing (drafts)..

Duration: 3 hours and 1 minute

Affected Components: Self Service, Merchant

Jun 16, 05:52:24 GMT+0
Investigating - Our primary hosting platform, Microsoft Azure, is experiencing general network connectivity issues across Europe. Unfortunately this is impacting our services, meaning you will experience degraded performance. Investigation shows no significant impact except for some delays in the UI services. We will continue to monitor and provide an update once things are running as expected. Since this is an upstream issue with multiple failures there is little we can do to mitigate the problem immediately. For the underlying status issue have a look at https://status.azure.com/en-us/status.

Jun 16, 06:51:01 GMT+0
Monitoring - Microsoft reports a mitigation has been deployed to their infrastructure. We are already seeing improvements across the board, but we will continue to monitor. Microsoft expects the error to be fully resolved around 10.00 CEST..

Jun 16, 07:22:59 GMT+0
Monitoring - All services seems to be running at regular levels. We have an increase in number of succesfully processed requests, and a reduction in response times that are consistent with the normal operating range. We will continue to monitor the progress. Note: that other services are also affected by this outage, we are currently observing reduced response times and timeouts from Vipps settlement processing. This have a limited impact on our services..

Jun 16, 08:53:29 GMT+0
Resolved - This incident has been resolved. Now known issues persists..

Duration: 2 days and 2 hours

Affected Components: Merchant, API

Jun 6, 09:50:00 GMT+0
Resolved - During a semi-automated upgrade of the platform runtime environment, the reporting and analytics component stopped responding to queries because it was being moved around. The Merchant client uses this component for query/search of Subscribers and was unable to produce any results while the component was unresponsive. The reporting and analytics component is once more up and running and everything should proceed as normal. All other operations was working as normal during the outage. Root cause: The reporting and analytics component was misconfigured causing only one instance to be running at any given time. The automated upgrade caused long responsetimes and frequent restarts of the component. The issue has now been mitigated and configuration adjusted to allow multiple concurrent instances to prevent outages similar to the current one..

Duration: 3 hours and 23 minutes

Affected Components: API

Apr 19, 04:00:00 GMT+0
Monitoring - There is an issue with Swedbank Pay, causing new agreement registration to timeout. The effect is that new orders and attempt to create a new Swedbank Pay agreement will produce an error message instead of the Swedbank Pay terminal. We have identified the problem and rolled out a fix. We are continuing to monitor the situation and will update this incident once we have confirmed the issue is rectified..

Apr 19, 07:22:51 GMT+0
Resolved - This incident has been resolved. All agreement registrations are now being processed in a timely fashion. The root cause was a misconfiguration in an internal component responsible for Swedbank Pay processing. .

Duration: 1 hour and 51 minutes

Affected Components: API

Feb 16, 19:00:01 GMT+0
Identified - Maintenance is now in progress.

Feb 16, 19:00:00 GMT+0
Identified - We are planning to do internal maintenance to a small part of the service. The maintenance should be transparent to both subscribers and admins, but in this time window you may experience small delays in queries and new registrations related to: - Prices - Subscription Plans (Template Packages) - Organizations We do not expect that will have any impact on business areas such as new orders, authorization and payments. If possible please refrain from registering new prices, organizations or subscription plans on this date. This in order to reduce possible friction with the maintenance..

Feb 16, 20:51:16 GMT+0
Completed - Maintenance has completed successfully. No downtime or errors detected during the process..

Duration: 3 hours and 2 minutes

Affected Components: Self Service, API, Merchant

Jan 25, 08:45:22 GMT+0
Identified - Our primary hosting platform, Microsoft Azure, is experiencing general network connectivity issues across multiple regions. Unfortunately this is impacting our services, meaning you will experience degraded performance. Investigation shows no significant impact except for some delays in processing actions. We will continue to monitor and provide an update once things are running as expected. Since this is an upstream issue with multiple failures there is little we can do to mitigate the problem immediately. For the underlying status issue have a look at https://status.azure.com/en-us/status.

Jan 25, 10:17:14 GMT+0
Monitoring - Azure is now recovering, and we are currently not observing any delays or timeouts due to the Azure networking issue. We will continue to monitor throughout the day..

Jan 25, 11:47:16 GMT+0
Resolved - The underlying azure issue is resolved. As mentioned in the previous update everything is running as it should and the issue has been mitigated/resolved..

Duration: 2 hours and 57 minutes

Affected Components: Self Service, API, Merchant

Jan 18, 08:29:26 GMT+0
Investigating - We are currently investigating an incident where the underlying resources used by the platform are unavailable, slow to respond and/or moving. The effect is primarly related to our backend resources, indirectly affecting the API and the Merchant/Self-Service. We are monitoring the issue, and have started investigations with Microsoft Support, as well as provisioned additional resources to mitigate the issue..

Jan 18, 08:56:41 GMT+0
Identified - In an attempt to reduce pressure we have disabled all the recurring/schedule automations for billing. The jobs will be re-enabled once we have determined that everything is available again. The side effect: Some Payment Demands/Invoices and Payment Request Processing (CreditCard, Vipps, AvtaleGiro, eFaktura) will be delayed a bit relative to what you are used to..

Jan 18, 09:35:03 GMT+0
Monitoring - We have implemented some mitigations and everything seems to be running at a reduced pace, but running. The underlying issue in Azure is still not resolved, but the effects have apparently been mitigated by our changes. We will continue to monitor the status..

Jan 18, 11:26:40 GMT+0
Resolved - The services have recovered and all previously disabled automations are now toggled on again. We are continuing our increased monitoring efforts..