USA on Opalstack Status

[Resolved] DNS outage (resolved)

Fri, 31 May 2024 11:45:00 +0000

About 1 hour ago a failed DNS sync with our alternate DNS provider caused an outage in our DNS servers. The outage lasted approximately 1 hour during which customers may have experience trouble accessing Opalstack systems.

The problem has been resolved and all DNS services are working normally at this time, however there may be lingering effects due to global DNS propagation. These issues will resolve themselves over the next few hours. You may be able to resolve them sooner for yourself by flushing your DNS and/or by power-cycling your devices. (11:45 UTC — May 31)

[Resolved] Partial DNS outage.

Fri, 30 Jun 2023 11:19:36 +0000

Fixed - The DNS issue is fully resolved now.

Note that while this issue affected only the Opalstack US name servers and only for the domain opalstack.com. Since all of our regional services use the opalstack.com domain, effects of this issue may have been seen by customers outside of the US.

If you’re still having trouble connecting to Opalstack services then please try flushing your local DNS cache (13:00 UTC — Jun 30)

During a routine DNS update our Secondary DNS provider captured an incomplete zone update and propagated it. This means that various DNS services for opalstack.com services (DNS, Mail, and Web) are inconsistently available.

We have removed the Secondary provider and we expect this to resolve over the next hour as the DNS TTL caches time out. (11:19 UTC — Jun 30)

[Resolved] US DNS under attack

Tue, 02 May 2023 04:48:43 +0000

Fixed - There have been no further issues since our last update. (13:44 UTC — May 2)

Watching - We’ve mitigated the worst of the attack and all services are online at this time. DNS lookups are still a bit slow so our troubleshooting is ongoing. (05:33 UTC — May 2)

Our US DNS infrastructure under attack, with impacts throughout our hosting services in all regions. We’re working to resolve the issue at this time. (04:48 UTC — May 2)

[Resolved] Ongoing DNS DDOS Attack

Thu, 29 Dec 2022 12:00:00 +0000

Fixed - Post-mortem: this was a DNS flood attack which targeted all 9 of our regional name servers. The attack originated from several cloud provider subnets, including Google and Cloudflare.

To mitigate the attack we had to temporarily block the provider subnets. This had some unintended and undesirable side effects:

The subnets included public DNS resolver infrastructure run by Google and Cloudflare, so blocking them prevented those resolvers from looking up DNS records on Opalstack name servers. As a result, people who use those resolvers were temporarily unable to access opalstack.com (including this status page) and customer domains that use our name servers.
The block also prevented Google from looking up DNS records related to email such as SPF and DKIM. As a result, mail sent (or forwarded) to Gmail from Opalstack’s mail system was rejected by Google while the block was in place.

The block was lifted over 24 hours ago and there have been no further issues since that time.

Going forward, we’ll refine our mitigation techniques to avoid blocking major providers when possible. We’ll also move this status page to an externally-hosted domain to ensure that system status updates will be available regardless of the state of our infrastructure. (16:40 UTC — Dec 30)

Fixed - We’ve seen no further issues in the past several hours and consider this to be resolved. (21:18 UTC — Dec 29)

Watching - The attack has subsided, we’ll continue to monitor. (14:27 UTC — Dec 29)

Over the last few hours we’ve identified and mitigated an ongoing DDOS attack against our DNS infrastructure. This attack has also effected our Singapore and German DNS servers.

We’re continuing to monitor the attack. (12:00 UTC — Dec 29)

[Resolved] Service disruption caused by firewall issue

Thu, 10 Feb 2022 19:43:00 +0000

About an hour ago we deployed a firewall update that had the unintended side effect of blocking our resolving DNS servers.

This caused various issues on customer sites on shared hosting, usually showing up as a “504 Gateway Timeout” error.

The issue has been resolved and all customer sites should be working now as they were prior to the change. (19:43 UTC — Feb 10)