Opalstack - Notice history

Opal4 is back online and operating normally.

Opal4 is not responding at this time.

Opal4 is back online and is operating normally. We've confirmed another network attack was the cause.

Linode reports the incident is resolved as of Dec 18, 2020 20:01 UTC.

All Singapore services are back online, we'll continue to monitor while Linode wraps this up.

Linode has identified the cause of the outage and is implementing a fix, we're seeing the Singapore servers coming back online at this time.

If your domain is using our Singapore name servers then as an interim fix please switch your domain's NS to one of our other regions: https://help.opalstack.com/article/48/what-are-your-name-servers

All pending mail imports have been completed and the IMAP servers are operating normally.

The US IMAP service is currently under heavy load while importing mail for new customers.

The Opal6 Apache httpd and PHP-FPM services were hitting an open files limit. We've raised the limit and the services are back online.

AT&T are no longer blocking mail forwarded from our US MX servers.

AT&T says that our US MX servers will be unblocked in 24-48 hours.

Opal4 is operating normally.

Opal4 is currently under high load and not responding to most requests, we're looking into it.

Opal4 just experienced another load spike due to a network attack. We'll continue to monitor.

Opal4 has been stable since our last update.

The system was once again under heavy load due to a network attack. We'll continue to monitor.

Opal4 is responding to ping but the system is under heavy load, we will continue to investigate.

Opal4 is back online. The cause was a brief network attack.

Newark VPS services are back online.

This is an upstream outage in our provider's network. They are working to restore service at this time.

Microsoft is no longer blocking our US SMTP.

Microsoft has responded saying that our US SMTP will be removed from the blacklist within the next 24-48 hours.

Microsoft have responded to our inquiry. It appears that the recent influx of WebFaction customers 🎉 has caused a sudden increase our outgoing mail volume and that this is what triggered the block.

We've verified via Microsoft's own postmaster tools that there have been zero complaints regarding mail from our US SMTP and have provided this data to Microsoft for consideration.

We hope to follow up with a resolution to this issue as soon as possible. Thank you for your patience.

We've made some changes to the way Let's Encrypt works with the shared opalstacked.com domain and as a result the domain is no longer rate-limited. We'll be posting updated documentation with the details later this week.

Looks like we've hit our new limit, we'll get this corrected ASAP.

We've confirmed that our increased rate limits are in effect, Calling this fixed, thanks for your patience :-)

The LE service has increased our limit and at this time we are not seeing any rate-limiting for new LE certificates and renewals on any domains. We'll continue to monitor.

The issue was caused by a spike in system load due to temporary high network usage. The server is now stable.

We've rebuilt most of our PHP stack and this issue is now resolved. Updated documentation and maybe a blog post are on the way.

Over the last few weeks we’ve dealt with various PHP-FPM related problems. After trying various combinations of fixes to the server, apache, and PHP-FPM itself we’ve developed a new, more stable PHP-CGI stack to replace PHP-FPM. With PHP and WordPress powering large parts of the Internet (and businesses) having a stable, reliable stack for them is critical.

We rolled the new stack out to all servers yesterday afternoon and found some edge cases we hadn’t found in our previous testing.

We’ll be rolling those changes out again tonight. While we do there may be a brief downtime of less than 10 minutes across PHP based sites and the update happens. Once the sites are updated we’ll begin monitoring for any breakages that didn’t show up in our testing and fix them ASAP.

We’ll follow up with a blog post on exactly what we’ve changed and how things are working in the future.

Our troubleshooting is going but we've improved the stability of PHP-FPM quite a bit over the past few days and are seeing no customer facing problems at this time. We'll leave this incident open until we're satisfied that we've got this issue nailed down.

There have been intermittent PHP-FPM outages on Opal5 in the past several hours. Troubleshooting is ongoing.

PHP-FPM for PHP 7.3 on Opal5 was briefly disrupted about 15 minutes ago. We'll continue to troubleshoot and monitor.

We've just deployed a potential fix and will continue to monitor.

The data center has reported that the issue is resolved.

Opal4 is back online. We will continue to monitor.

The data center reports "an outage related to a networking maintenance that was intended to be non-impacting in our DAL-12 datacenter".

We're awaiting further updates at this time.

The outage appears to be upstream in our data center. We've contacted them and are awaiting their response.