Awell - Notice history

Early this morning, our Sandbox environment experienced an outage due to a new data replication feature intended to improve the availability of the database cluster. During a routine maintenance operation, a database server failed to shut down gracefully because of the data replication configuration.

While the server eventually restarted, it was not able to synchronize with the other servers, leading to a continuous loop of failed synchronization attempts which eventually made the cluster unresponsive.

The Production environments have no risk of being impacted by this issue as they use a different configuration for data replication. The configuration of the Sandbox environment has been aligned with the Production environments to eliminate the risk of this occurring again.

The issue has now been fully resolved. We apologize for any inconvenience caused and appreciate your understanding.

The database backup has been successfully restored. All services are back online. We will keep investigating the root cause of this issue and will post updates as we find more information.

Our database cluster started experiencing issues around 4:15 AM UTC. Despite our best attempt we were not able to restore it to a healthy state. In order to restore service we decided to restore data from the latest backup. This operation is ongoing and should complete within the next hour. More information will be posted once service is restored.

[EU] Orchestration - GraphQL API is now operational! This update was created by an automated monitoring service.

[Sandbox] Orchestration - GraphQL API is now operational! This update was created by an automated monitoring service.

[EU] Orchestration - GraphQL API is now operational! This update was created by an automated monitoring service.

[EU] Orchestration - GraphQL API is now operational! This update was created by an automated monitoring service.

[EU] Orchestration - GraphQL API is now operational! This update was created by an automated monitoring service.

[US] Orchestration - GraphQL API is now operational! This update was created by an automated monitoring service.

We’re happy to let you know that we’ve wrapped up all the steps to fix the recent outage. Our team has identified and resumed 26 stuck care flows.

Here’s what we’ve done:

1. Fixing Bottlenecks: We found the root causes of the bottlenecks and made the necessary changes to sort them out.

2. Better Alerts: We’ve set up new alerting policies that will notify us proactively if something similar happens again.

3. Improved Recovery: We’ve added more recovery strategies to make sure care flows don’t get stuck after incidents like this.
   

Thank you for your patience and understanding as we worked through this. We’re committed to providing you with reliable service and will continue to improve our systems.

An unexpected increase in the usage of the product put two key systems under stress (Message broker and Application database). We have identified what caused the bottleneck in each system (memory leak in the message broker, throttled CPU in the application database) and made the necessary changes to remove them.

In addition, we've created new alerting policies that will proactively inform us should a similar scenario play out. This will enable us to take mitigation actions early enough to prevent system failures.

We are still investigating the impact of this incident and will post another update when it has been identified.

The team is investigating the root cause to ensure the issue does not reoccur. Further updates will be provided once the investigation is complete.

[US] Orchestration - GraphQL API is now operational! This update was created by an automated monitoring service.

[Sandbox] Orchestration - GraphQL API is now operational! This update was created by an automated monitoring service.

[Sandbox] Orchestration - GraphQL API is now operational! This update was created by an automated monitoring service.