Resolved
Service Disruption affecting DealerTeam Platform users
Update [14:30 PM PDT] - The severity of the outage has been downgraded from disruption to impacted. Our internal monitoring suggests services are restored. Performance reporting has returned no failures or error codes in the last 60 minutes. We will continue to resolve the services.
Update [13:00 PM PDT] - Summary from AWS and third party vendor
Service recovery across all AWS services continues to improve. We continue to reduce throttles for new EC2 Instance launches in the US-EAST-1 Region that were put in place to help mitigate impact. Lambda invocation errors have fully recovered and function errors continue to improve. We have scaled up the rate of polling SQS queues via Lambda Event Source Mappings to pre-event levels.
Update [12:15 PM PDT] - Summary from AWS Vendor
We continue to observe recovery across all AWS services, and instance launches are succeeding across multiple Availability Zones in the US-EAST-1 Regions. For Lambda, customers may face intermittent function errors for functions making network requests to other services or systems as we work to address residual network connectivity issues. To recover Lambda’s invocation errors, we slowed down the rate of SQS polling via Lambda Event Source Mappings. We are now increasing the rate of SQS polling as we experience more successful invocations and reduced function errors. We will provide another update by 1:00 PM PDT.
We are currently investigating a service interruption affecting a wide variety of our services. This issue appears to be caused by a widespread outage at Amazon Web Services (AWS), one of our infrastructure providers.
Our team is monitoring the AWS resolution efforts and will restore service as soon as possible.
We apologize for the inconvenience. We will post our next update in 60 minutes or sooner if we have a significant development.
Resolved
·
20 Oct at 09:13am PDT
