28th December 2021
What happened?
There were problems with logins and queue processing for https://app.contrastsecurity.com.
Timeline
Shortly after 3:17am UTC, our platform monitoring detected connection issues to Amazon RDS causing intermittent availability. The service continued to be unstable until ~4:30am UTC. The incident response team discovered failures when attempting to write Contrast Protect event details to the database. A ticket was filed with AWS.
At 8:32am UTC AWS restored the database to a few minutes before the incident in order to restore operations. The service was restarted with Protect message processing disabled.
Protect messaging was successfully reenabled at 2:58pm UTC.
Impact
During the course of the incident:
- Users initially saw errors when attempting to login to the Contrast UI. This has now been resolved.
- Assess traffic between 3:12am UTC and 8:32am UTC was lost when the database was restored.
- Protect traffic between 3:12am UTC and 2:58pm UTC was not processed.
- While Protect traffic to the system was disrupted, Contrast agents running in Protect mode continued to block malicious traffic and write events to their local security.log and any configured syslog integrations.
Status
Service was fully restored at 2:58pm UTC.
If you have any questions, concerns, or would like to discuss this issue further, please don’t hesitate to reach out to us at support@contrastsecurity.com.
<%= heading %>