Upstream Outage

Incident Report for Circle

Postmortem

This incident was caused by an outage at our primary compute provider, GalaxyGate.

Circle hosts our critical infrastructure (database, session management, gateway [in dev], and cache management) on several other hosting providers on the East Coast of the US.
We use GalaxyGate for our main services like the bots and API, which is why this outage caused all of our services to drop offline. We were able to quickly restore Premium services by enabling a backup server at Digital Ocean, regular Circle has more moving parts and is not as easy to startup on different infrastructure on demand. By the time we were beginning to setup regular infrastructure on Digital Ocean, the GalaxyGate issue was resolved.

This was GalaxyGate’s first issue that affected us since 2022 and we do not have plans to move our infrastructure away at this time.

Posted Dec 02, 2023 - 16:56 PST

Resolved

This incident has been resolved, all systems are operating normally.

Posted Nov 16, 2023 - 18:40 PST

Monitoring

All systems are showing as online and operating normally, we're monitoring to ensure everything is healthy.

Posted Nov 16, 2023 - 18:30 PST

Identified

We're observing abnormal latency issues with regular Circle & the website again

Posted Nov 16, 2023 - 18:26 PST

Monitoring

Some servers are able to use Regular Circle. The dashboard is still unavailable.

Posted Nov 16, 2023 - 18:24 PST

Update

Circle Premium has been temporarily setup on our staging/backup infrastructure. We do not have an ETA on when the website and main bot will be online.

Posted Nov 16, 2023 - 18:04 PST

Identified

We are aware of an issue with all Circle services caused by an outage at our upstream provider.

Posted Nov 16, 2023 - 17:49 PST

This incident affected: Circle, Circle Premium, and Website.