This incident has been resolved and we have not seen recurrence for the last several hours. We are concluding enhanced monitoring at this time.
Posted Aug 08, 2023 - 22:12 UTC
Update
Further problematic queries were found and those deployed graphs using those queries have now beed paused. We are now back to normal running.
We'll be monitoring the situation for the next couple of hours and will be marking this incident as resolved if no further issues arise.
Posted Aug 08, 2023 - 15:59 UTC
Update
We are continuing to monitor for any further issues.
Posted Aug 08, 2023 - 15:56 UTC
Monitoring
We have now terminated the initial problem query and that allowed us to identify a secondary cause that was running without such widespread impact. That is connected to a Graph Listener for a project which needs to be paused.
We have paused the Listener for that graph so that we can stop issuing the problem query.
Posted Aug 08, 2023 - 15:55 UTC
Identified
It appears the shared MySQL database was overloaded by a query to ALTER a table issued by a user who didn't clear the query properly. The user thought they killed the query after about 10 minutes, but it simply disconnected their UI. The query ran for 3 hours. until terminated by our Operations Team.
We terminated the impacting query and have been monitoring since.
Posted Aug 08, 2023 - 15:53 UTC
Investigating
We are currently investigating this issue. We will keep you posted.
Posted Aug 08, 2023 - 15:50 UTC
This incident affected: CSG (US East (N. Virginia) Engine, US East (N. Virginia) Graph API) and Kitewheel (US East (N. Virginia) Engine, US East (N. Virginia) Graph API).