Incident: Websites are Inaccessible

Incident Report for RevolutionParts, Inc

Postmortem

Starting on May 2nd, 2022 at 5:23 AM PST, RevolutionParts experienced a disruption of service.

Impact: Slow page load times and/or errors on webstore, plugins, and the RP Platform
No data was compromised
Resolved: May 2nd, 2022 at 6:20 PM PST

We understand that you rely on our service to run your business. Please read below for additional details and what we have done to prevent a similar event in the future

What Happened

We determined that a part of our infrastructure slowed down and caused a ripple effect that slowed/impacted all traffic to our systems. We restarted, replaced, and added capacity to this area of our system, which temporarily helped, but the issue ultimately returned which prolonged the length of this incident.

The root cause was found to be a large amount of data that was saved into parts of the infrastructure. Once this was removed all system metrics went back to normal.

Next Steps

We now have additional guards and monitoring in place to prevent this from happening again.

Posted May 04, 2022 - 15:13 MST

Resolved

All services impacted by the issue with RP systems, websites, and plugins have recovered and are operating as intended. This issue has been resolved.

Posted May 03, 2022 - 10:03 MST

Update

We continue to see systems operating as normal and will continue to monitor. We will provide another update in two hours or if we see any issues with performance.

Posted May 03, 2022 - 08:05 MST

Monitoring

Our systems are no longer experiencing accessibility issues, and load times have returned to a healthy state. We will be monitoring all services to ensure a full recovery and provide another update as soon as more information becomes available.

Posted May 03, 2022 - 05:34 MST

Update

Overnight our engineering teams continued to work toward a solution for this incident. Our systems are operational and we will continue to improve and monitor performance.

We will provide another update later this morning.

Posted May 03, 2022 - 03:36 MST

Update

Our systems continue to be operational, but the root cause of the issue has yet to be identified. Our engineers are continuing to investigate and monitor current performance.

The next update will be given tomorrow morning, or as soon as the issue has been resolved.

Posted May 02, 2022 - 20:40 MST

Update

Our systems are currently operational, but the root cause of our earlier issue has yet to be identified. Our engineers are continuing to investigate the issue and monitor current performance. We will provide an update as soon as one becomes available, or in an hour.

Posted May 02, 2022 - 19:33 MST

Update

We are still experiencing intermittent outages. Our engineering team is actively working on a resolution. We'll provide another update in 1 hour or as soon as more information becomes available.

Posted May 02, 2022 - 18:26 MST

Update

We are still experiencing intermittent outages, and our engineers are still working hard to get all websites to be accessible. We'll provide another update in 1 hour or as soon as more information becomes available.

Posted May 02, 2022 - 17:32 MST

Update

You may be experiencing a mix of slow site load times or 504 errors. 504 errors appear when the request to load a page has timed out. When you do not get an error, you may still experience slow loading times when accessing our back-end or browsing sites. Both issues stem from the same root cause, which is actively being worked on by our engineering team. We expect to provide another update in 1 hour or as soon as more information becomes available.

Posted May 02, 2022 - 16:33 MST

Update

We are still experiencing intermittent outages and our engineers are still working hard to get all websites to be accessible. We'll provide another update in 1 hour or as soon as more information becomes available.

Posted May 02, 2022 - 15:35 MST

Update

We are still experiencing intermittent outages, but we continue to see progress on accessibility. Our engineers are still working hard to resolve this issue. We will provide another update in the next hour.

Posted May 02, 2022 - 14:36 MST

Update

We have seen some improvements in the last hour, however we are still experiencing intermittent outages. We continue to work to resolve this issue and will provide another update in an hour or when we have a resolution. We apologize for the inconvenience this is creating for you and your customers.

Posted May 02, 2022 - 13:40 MST

Update

We are continuing to investigate this issue. As soon as there is an update, we will let you know.

Posted May 02, 2022 - 12:33 MST

Update

We are continuing to investigate this issue. As soon as there is an update, we will let you know.

Posted May 02, 2022 - 11:33 MST

Investigating

The fix applied to this issue seemed to not have resolved the problem. We are setting the incident back to Investigating and trying a different approach.

We will send an update as soon as one becomes available or in one hour.

Posted May 02, 2022 - 10:28 MST

Update

We are continuing to monitor for any further issues.

Posted May 02, 2022 - 09:44 MST

Monitoring

The issue was identified in one of our in-memory data structure stores. The team has replaced it, and the sites are now operational. We are switching the incident to "Monitoring" and will be providing another update in 30 minutes.

Thank you for your patience.

Posted May 02, 2022 - 09:45 MST

Update

We are continuing to investigate the known issue impacting webstores, plugins, and the RP Platform. We will provide another update in one hour.

For customers selling on Marketplaces, we recommend logging directly into your accounts (Amazon.com, eBay.com and/or Walmart.com) to process orders to maintain seller ratings.

We hope to have a resolution in place shortly.

Posted May 02, 2022 - 08:14 MST

Investigating

This Monday morning we encountered a problem with our websites not loading correctly. Our Engineering team is aware of the situation and is investigating the problem.
As soon as there is an update, we will update the Status Page.

We truly apologize for the inconvenience.

Posted May 02, 2022 - 06:24 MST

This incident affected: RevolutionParts Platform.