Shared: Dallas (USA) on Opalstack Status

[Resolved] Unexpected reboot of opal4

Mon, 01 Sep 2025 23:40:13 +0000

Fixed - The cause of the reboot was determined to be a deadlocked kernel process. We’ve seen no further issues following the reboot. (20:21 UTC — Sep 4)

A short while ago opal4.opalstack.com (Dallas shared hosting) rebooted for an unknown reason. The server is back online at this time and we’re now investigating the cause of the reboot. (23:40 UTC — Sep 1)

[Resolved] unexpected reboot of opal4

Fri, 06 Jun 2025 12:46:27 +0000

Fixed - Opal4 has remained stable for the past several hours. We haven’t been able to determine the precise cause of the unexpected reboots earlier today but we suspect it was a possible fault in the server’s remote management interface. We’ve reset that system and are seeing further no problems at this time. (22:59 UTC — Jun 6)

Investigating - opal4 is back online at this time. Our investigation is still in progress. (16:46 UTC — Jun 6)

Investigating - opal4 is once again offline for troubleshooting and maintenance. During this time, the server may be placed into a rescue mode where it will respond to ping, but SSH connections will fail and other services will not be available. (16:01 UTC — Jun 6)

Investigating - opal4 is experiencing heavy load at this time. (15:21 UTC — Jun 6)

Investigating - Opal4 is back online at this time, the investigation is ongoing. (14:46 UTC — Jun 6)

Investigating - opal4 is now being taken offline for emergency maintenance. (13:59 UTC — Jun 6)

Investigating - The server was rebooted again unexpectedly, we’re investigating the cause. (13:29 UTC — Jun 6)

Investigating - We’ve lost connectivity to opal4 and suspect another reboot, we’re investigating further. (13:21 UTC — Jun 6)

opal4.opalstack.com (Dallas shared hosting) rebooted unexpectedly a few moments ago as around 12:25 UTC. The server is back online and operating normally at this time. (12:46 UTC — Jun 6)

[Resolved] heavy load on opal4

Wed, 04 Jun 2025 14:25:43 +0000

Fixed - The cause of the high load was a compromised customer website. We’ve locked down the affected site and have notified the site operator of the issue. (15:07 UTC — Jun 5)

Investigating - opal4 is back online. We’re still investigating the cause of the high load. (14:43 UTC — Jun 4)

Investigating - We’re rebooting opal4 at this time. (14:34 UTC — Jun 4)

opal4.opalstack.com (Dallas shared hosting) is currently experiencing heavy load resulting in very poor performance. We’re investigating at this time. (14:25 UTC — Jun 4)

[Resolved] OPAL4 MariaDB Outage

Sun, 09 Mar 2025 08:12:00 +0000

Fixed - We’ve seen no further problems since our last update. (19:11 UTC — Mar 9)

Watching - The issue with database permissions is resolved and the MariaDB service is back online and operating normally. We will continue to monitor. (14:22 UTC — Mar 9)

Identified - We’ve discovered that there is still a problem with database permissions. We’re working to resolve that at this time. (13:29 UTC — Mar 9)

Fixed - The data restoration has completed and we are continuing to monitor the server as usage increases. (12:29 UTC — Mar 9)

Identified - The restoration is approximately 75% complete. (11:24 UTC — Mar 9)

Identified - The data restoration is approximately 50% complete. (10:32 UTC — Mar 9)

Identified - The MariaDB restore process is still running. (09:41 UTC — Mar 9)

Identified - The database corruption is unrecoverable so we will proceed with restoring from backups. The most recent valid backups were from 1.5 hours before the server crashed so we’ll proceed with the restore from there.

The restoration process varies in completion time. It may take up to 6 hours or more. We’ll keep this post updated. (08:37 UTC — Mar 9)

MariaDB on OPAL4 has crashed, we’re working to identify the cause and solution now. (08:12 UTC — Mar 9)

[Resolved] OPAL4 - MariaDB has crashed.

Wed, 31 Jul 2024 10:19:00 +0000

Fixed - We found and corrected a problem with database permissions and all database users should be able to connect now. (17:00 UTC — Jul 31)

Investigating - We’re getting reports that some users still cannot connect to their databases so we are investigating that now. (16:34 UTC — Jul 31)

Fixed - Databases have been restored and should be fully operational. (16:15 UTC — Jul 31)

Identified - The restore process is at 99% and the last databases are being restored. (15:25 UTC — Jul 31)

Identified - The database restores are 66% complete. (14:45 UTC — Jul 31)

Identified - The database restores are 51% complete. (13:58 UTC — Jul 31)

Identified - The database restores are in progress and are ~25% complete. (13:21 UTC — Jul 31)

Identified - The InnoDB data corruption is too bad to recover data from the database server. Tables are unreadable causing data dumps or repairs to the tables to be impossible.

We have started the database repair process from our backups that were 3-5 hours old when the server went offline. We will update this post with our progress. (12:26 UTC — Jul 31)

Identified - The data corruption on the server is in the innodb data which means there is potentially corrupted data across multiple databases due to how MySQL/MariaDB store data. We are still trying to recover and repair the data in the MariaDB server. If the data is too corrupt to repair we will restore the data from the latest backups. The latest backups were taken between 3 and 5 hours (depending on the dump time) before the database server crashed. (12:07 UTC — Jul 31)

MariaDB has crashed on OPAL4. We are investigating the cause of the crash and working to restore services. (10:19 UTC — Jul 31)

[Resolved] Emergency MariaDB maintenance for opal4, opal7, and opal9

Sun, 03 Dec 2023 07:23:00 +0000

Fixed - The MariaDB recovery on opal9 is complete. Databases with names starting with tro through zzz are being restored from our 2 December 2023 backup.

Our plan now is to test the recovery process on a test server to determine why we were not able to import the dump files and then refine our maintenance procedures based on those findings. (22:21 UTC — Dec 3)

Watching - The MariaDB recovery on opal9 is still in progress. We expect the recovery to be complete in around 90 minutes. (21:21 UTC — Dec 3)

Watching - The MariaDB recovery on opal7 is complete.

The MariaDB recovery on opal9 is still in progress. We’ll update with an ETA as soon as possible. (21:12 UTC — Dec 3)

Watching - The MariaDB recovery on opal7 and opal9 is still in progress. We expect that it will take at least a couple of hours more to complete. (19:39 UTC — Dec 3)

Watching - The MariaDB recovery on opal7 and opal9 is still in progress. We expect that it will take at least a few more hours to complete.

The MariaDB recovery on opal4 is now complete, however we were not able to use the most recent backup for about 50% of the recovered databases. The remaining databases were recovered as follows:

Databases with names starting with lsa through shn were restored from our 2 December 2023 backup.
Databases with names starting with sho through zzz were restored from our 1 December 2023 backup. (16:56 UTC — Dec 3)

Watching - It seems some opal4 databases were not restored. We’re restoring those databases now.

The recovery on opal7 and opal9 is still in progress. (14:55 UTC — Dec 3)

Watching - The MariaDB recovery on opal4 is complete. Recovery on opal7 and opal9 is still in progress. (14:33 UTC — Dec 3)

Watching - The database restore is still in progress. We expect that it will take at least a few more hours. (13:47 UTC — Dec 3)

Watching - What happened?

We took the MariaDB database down to stop the ibdatafile from growing exponentially. The only way to do this is to dump all databases, delete the log files, update the server configuration, and restore the data.

The dump process worked without any errors. The clean up and server configuration completed without any errors. The restore process is where things failed.

Because of errors in 1 or more database(s) transactions failed to complete causing the restore to never get past a certain point. The amount of data in the mysql data directory would increase up to a certain point and then the data size would drop by half before repeating. After running the restore process twice and having errors in different spots we decided to extract each database from the monolithic backup we had taken previously. Why a monolithic backup? It’s usually faster to dump and restore, except in this case.

The extraction process is painfully slow compared to just running a working dump restore. That’s why this process is taking so long.

What didn’t we do?

We could have restored from the latest backup which would have been 24 hours or less old, however, that option comes with the significant risk of data loss. Rather than risk losing 24 hours of data we went with the slower, safer process.

The current restore process for each server is now past the previous failure point.

We’d sincerely apologize for the downtime this has caused for your apps that use MariaDB. In the tests we performed before the actual event we did not run into any of these errors. (12:15 UTC — Dec 3)

Watching - The data restoration process is still ongoing. There is currently no ETA we can provide. (10:29 UTC — Dec 3)

Watching - Opal4, Opal7, Opal9: We are still restoring data to the MariaDB databases. The exact amount of time left in the restoration process is unknown but we will update every hour until all of the restores have finished. (08:42 UTC — Dec 3)

On Sunday, 03 December 2023 at 0500 UTC we’ll be taking the managed MariaDB database service offline for maintenance on the following servers:

opal4.opalstack.com (Dallas)
opal7.opalstack.com (Phoenix)
opal9.opalstack.com (Frankfurt)

The maintenance window is 2 hours. During the maintenance sites and applications which use MariaDB (including WordPress sites) will not function.

We apologize for the short notice and any inconvenience. If you have any questions or concerns regarding the maintenance then please contact our support team. (07:23 UTC — Dec 3)

[Resolved] Opal4 offline

Fri, 20 May 2022 15:45:00 +0000

Fixed - The issue is resolved and opal4 is back online. (16:31 UTC — May 20)

Opal4 (Dallas shared hosting) is currently offline due to a network issue. We’re working to restore service at this time. (15:45 UTC — May 20)

[Resolved] Service disruption caused by firewall issue

Thu, 10 Feb 2022 19:43:00 +0000

About an hour ago we deployed a firewall update that had the unintended side effect of blocking our resolving DNS servers.

This caused various issues on customer sites on shared hosting, usually showing up as a “504 Gateway Timeout” error.

The issue has been resolved and all customer sites should be working now as they were prior to the change. (19:43 UTC — Feb 10)

[Resolved] Opal4 not responding

Tue, 28 Dec 2021 18:50:52 +0000

Fixed - Opal4 is back online. We’ve reached out to our network provider for details regarding the outage. (19:10 UTC — Dec 28)

opal4.opalstack.com (Dallas shared hosting) is not responding. We’re looking into it. (18:50 UTC — Dec 28)

[Resolved] Opal4 not responding

Mon, 28 Dec 2020 18:46:39 +0000

Fixed - Opal4 is back online and operating normally. (01:43 UTC — Dec 29)

Investigating - Opal4 is not responding at this time. (01:31 UTC — Dec 29)

Fixed - Opal4 is back online and is operating normally. We’ve confirmed another network attack was the cause. (19:07 UTC — Dec 28)

Opal4 is currently not responding to ping. We suspect this is due to another attack. (18:46 UTC — Dec 28)

[Resolved] Recurring attacks on opal4

Fri, 11 Dec 2020 19:17:00 +0000

Watching - Opal4 is stable at this time. (18:50 UTC — Dec 16)

Investigating - Opal4 is not responding, likely another attack. We’re investigating. (18:31 UTC — Dec 16)

Fixed - Opal4 is back online. The disruption was caused by another network attack. The replacement server will be set up soon and we’ll start scheduling the transition. (04:40 UTC — Dec 16)

Investigating - Opal4 is currently not responding to ping, investigating. (04:32 UTC — Dec 16)

Fixed - Opal4 is stable at this time. (21:30 UTC — Dec 11)

Watching - Opal4 is back online.

The earlier assessment of an external network problem was incorrect, this appears to have been another network attack on Opal4.

The attacks that have taking opal4 down haven’t seemed to be particularly massive - similar attacks happen on other shared hosts in our infrastructure and those machines are fine. Because of this we’re planning to replace opal4 in the coming weeks. We’ll contact all affected customers individually with the details as we finalize the plan. (21:08 UTC — Dec 11)

Identified - Opal4 is offline due to a network problem in the Leaseweb Dallas TX data center. We’re awaiting further updates from them at this time. (19:43 UTC — Dec 11)

opal4.opalstack.com in Dallas TX USA is currently not responding. We’re looking into it. (19:17 UTC — Dec 11)

[Resolved] opal4 not responding

Wed, 11 Nov 2020 19:24:00 +0000

Fixed - Opal4 is operating normally. (19:44 UTC — Nov 28)

Investigating - Opal4 is currently under high load and not responding to most requests, we’re looking into it. (19:36 UTC — Nov 28)

Watching - Opal4 just experienced another load spike due to a network attack. We’ll continue to monitor. (04:49 UTC — Nov 15)

Fixed - Opal4 has been stable since our last update. (18:58 UTC — Nov 13)

Watching - The system was once again under heavy load due to a network attack. We’ll continue to monitor. (20:06 UTC — Nov 11)

Investigating - Opal4 is responding to ping but the system is under heavy load, we will continue to investigate. (20:00 UTC — Nov 11)

Fixed - Opal4 is back online. The cause was a brief network attack. (19:42 UTC — Nov 11)

Opal4 in Dallas TX USA is currently not responding to ping, we’re looking into it. (19:24 UTC — Nov 11)

[Resolved] Let's Encrypt rate limit for opalstacked.com subdomains

Wed, 21 Oct 2020 17:40:00 +0000

Fixed - We’ve made some changes to the way Let’s Encrypt works with the shared opalstacked.com domain and as a result the domain is no longer rate-limited. We’ll be posting updated documentation with the details later this week. (19:34 UTC — Nov 3)

Watching - Looks like we’ve hit our new limit, we’ll get this corrected ASAP. (20:56 UTC — Oct 27)

Fixed - We’ve confirmed that our increased rate limits are in effect, Calling this fixed, thanks for your patience :-) (16:36 UTC — Oct 25)

Watching - The LE service has increased our limit and at this time we are not seeing any rate-limiting for new LE certificates and renewals on any domains. We’ll continue to monitor. (16:03 UTC — Oct 24)

At this time the Let’s Encrypt service is rate-limiting certificate issue and renewal requests for the entire *.opalstacked.com domain. This is the domain that we use to provide free subdomains to our hosting customers.

We’re working with the LE team to get a higher limit. In the meantime if you are attempting to issue a LE cert for a site that uses your opalstacked.com subdomain then it may fail. If this happens you’ll receive a failure notice via email.

Until the rate limit is raised then the alternative is to use one of your own registered domains on your site instead of your opalstacked.com subdomain. If you’re not ready to point your domain at Opalstack, then you can create a testing subdomain and A record via your NS provider to point that subdomain at your Opalstack server.

We’ll update this issue as soon as the rate limit has been raised. (17:40 UTC — Oct 21)

[Resolved] Opal4 offline

Mon, 19 Oct 2020 20:40:41 +0000

Fixed - The issue was caused by a spike in system load due to temporary high network usage. The server is now stable. (20:44 UTC — Oct 19)

Opal4 (Dallas shared hosting) is currently not responding to ping. We’re looking into it at this time. (20:40 UTC — Oct 19)

[Resolved] Intermittent PHP-FPM failures on Opal4 and Opal5

Sun, 18 Oct 2020 15:34:00 +0000

Fixed - We’ve rebuilt most of our PHP stack and this issue is now resolved. Updated documentation and maybe a blog post are on the way. (02:54 UTC — Oct 26)

Watching - Over the last few weeks we’ve dealt with various PHP-FPM related problems. After trying various combinations of fixes to the server, apache, and PHP-FPM itself we’ve developed a new, more stable PHP-CGI stack to replace PHP-FPM. With PHP and WordPress powering large parts of the Internet (and businesses) having a stable, reliable stack for them is critical.

We rolled the new stack out to all servers yesterday afternoon and found some edge cases we hadn’t found in our previous testing.

We’ll be rolling those changes out again tonight. While we do there may be a brief downtime of less than 10 minutes across PHP based sites and the update happens. Once the sites are updated we’ll begin monitoring for any breakages that didn’t show up in our testing and fix them ASAP.

We’ll follow up with a blog post on exactly what we’ve changed and how things are working in the future. (04:21 UTC — Oct 23)

Watching - Our troubleshooting is going but we’ve improved the stability of PHP-FPM quite a bit over the past few days and are seeing no customer facing problems at this time. We’ll leave this incident open until we’re satisfied that we’ve got this issue nailed down. (17:32 UTC — Oct 21)

Watching - There have been intermittent PHP-FPM outages on Opal5 in the past several hours. Troubleshooting is ongoing. (20:45 UTC — Oct 19)

Watching - PHP-FPM for PHP 7.3 on Opal5 was briefly disrupted about 15 minutes ago. We’ll continue to troubleshoot and monitor. (23:24 UTC — Oct 18)

Watching - We’ve just deployed a potential fix and will continue to monitor. (19:01 UTC — Oct 18)

We’ve been investigating intermittent failures of PHP sites hosted on Opal4 (Dallas) and Opal5 (Frankfurt) which result in “503 Service Unavailable” responses on websites. We’ll post more details here as they become available. (15:34 UTC — Oct 18)

[Resolved] Opal4 (Dallas) offline

Sat, 17 Oct 2020 05:48:00 +0000

Fixed - The data center has reported that the issue is resolved. (08:38 UTC — Oct 17)

Watching - Opal4 is back online. We will continue to monitor. (07:37 UTC — Oct 17)

Identified - The data center reports “an outage related to a networking maintenance that was intended to be non-impacting in our DAL-12 datacenter”.

We’re awaiting further updates at this time. (06:52 UTC — Oct 17)

Investigating - The outage appears to be upstream in our data center. We’ve contacted them and are awaiting their response. (06:02 UTC — Oct 17)

opal4.opalstack.com located in Dallas TX is currently not responding to ping. We’re looking into it and will follow up on this post with more information as it becomes available. (05:48 UTC — Oct 17)

[Resolved] intermittent connectivity problems on opal4

Thu, 23 Jul 2020 19:30:00 +0000

Fixed - Opal4 is stable. (20:29 UTC — Jul 23)

Watching - Opal4 is back online. The data center team believed that the connectivity issue was caused by a configuration problem which they corrected

We will continue to monitor. (19:50 UTC — Jul 23)

Investigating - Opal4 is being rebooted at this time. (19:40 UTC — Jul 23)

Opal4 is experiencing intermittent connectivity problems following this morning’s scheduled maintenance. The data center team is investigating. (19:30 UTC — Jul 23)

[Resolved] Opal4 network was disrupted

Mon, 13 Jul 2020 20:11:00 +0000

Network connectivity to opal4 was disrupted for 1 minute due to a short network attack.

The server is online and operating normally at this time. (20:11 UTC — Jul 13)

[Resolved] Opal4 under high load

Tue, 07 Jul 2020 22:54:08 +0000

Fixed - Opal4 is back online following a reboot. The high load was caused by a network attack. We’ve identified the site targeted by the attack and are working with the site owner to isolate the site from other customers on opal4. (23:13 UTC — Jul 7)

Opal4 is presently under high load and is mostly unresponsive. We’re rebooting it at this time. (22:54 UTC — Jul 7)

[Resolved] Opal4 not responding

Thu, 30 Apr 2020 21:48:36 +0000

Fixed - We believe this latest issue was caused by a DNS reflection attack.

We’re consulting with our upstream provider to see what they can do to protect the server and network from future attacks. (22:58 UTC — Apr 30)

Investigating - Opal4 is back online, investigation is ongoing. (21:56 UTC — Apr 30)

Opal4 is not responding. We’re looking into the cause. (21:48 UTC — Apr 30)

[Resolved] Apache was not responding on Opal4

Thu, 30 Apr 2020 15:10:00 +0000

Apache was once again not responding on Opal4 after hitting its configured ServerLimit. The underlying cause appears to have been a high number of PHP-FPM processes being re-spawned for one customer site. We’ve adjusted the PHP-FPM configuration for that site to avoid the problem in the future. (15:10 UTC — Apr 30)

[Resolved] Apache was not responding on Opal4

Fri, 24 Apr 2020 18:23:00 +0000

Fixed - The root cause appears to have been a misconfigured customer application. We’ve corrected the configuration and have updated our monitoring system to alert us when Apache approaches its configured limits. (17:22 UTC — Apr 28)

The Apache service on Opal4 was not responding for approximately 1 hour today beginning 18:23 UTC.

The problem was that the Apache service had hit its configured ServerLimit which meant that no further Apache processes could spawn to handle requests. That has been corrected and the service is online now.

We’re now investigating the root cause and also looking into why our own monitoring systems did not alert us to this condition. (18:23 UTC — Apr 24)

[Resolved] Opal4 load issues

Fri, 10 Apr 2020 23:40:00 +0000

Fixed - We’ve seen no further load issues since our last update. (20:00 UTC — Apr 24)

Watching - Opal4 has experienced two load spikes in the past 24 hours.

During the most recent spike we were able to positively identify one of the sites causing the problem and have taken steps to prevent it from happening again.

We’ll continue to monitor. (15:20 UTC — Apr 14)

Opal4 experienced a brief spike in system load between 23:40 and 23:50 UTC on Friday 10 April 2020.

The cause appears to have been high CPU usage due to a SYN flood attack.

The server is operating normally at this time. We’ll continue to monitor. (23:40 UTC — Apr 10)

[Resolved] Opal4 not responding

Wed, 26 Feb 2020 22:55:25 +0000

Fixed - We’ve successfully mitigated the attack and Opal4 has been stable since our last update. (16:02 UTC — Feb 28)

Watching - We’ve identified an ongoing attack pattern against sites on opal4 and have taken steps to mitigate it platform-wide. We’ll continue to monitor. (16:08 UTC — Feb 27)

Investigating - Opal4 is back online.

The problem was caused by high load following heavy RAM usage by the system `httpd` service. We’re still investigating the cause of the RAM usage. (23:22 UTC — Feb 26)

Opal4 is currently unresponsive. We’ll update this incident when more info is available. (22:55 UTC — Feb 26)

[Resolved] Intermittent outages on opal4

Wed, 08 Jan 2020 00:36:00 +0000

Fixed - During last week’s maintenance our upstream provider updated the server firmware on opal4. We’ve seen no further issues since that time. (16:03 UTC — Apr 6)

Watching - The maintenance is complete and Opal4 is back online. We’ll continue to monitor. (02:58 UTC — Apr 2)

Identified - Opal4 is down for the emergency maintenance scheduled earlier today. The total expected downtime is less than one hour. (02:05 UTC — Apr 2)

Investigating - Opal4 will be going down this evening at 9PM US Central (2020-04-01 21:00 UTC-5) for emergency maintenance. The total downtime should be less than one hour.

The ongoing issue on opal4 has been high system load caused by two separate types of attacks:

High Apache RAM usage due to attacks against common targets like Wordpress sites
High CPU usage by the system firewall under a high volume of SYN packets

We’ve resolved the first problem by tuning our application firewall rules and by working with a couple of specific site owners that were receiving the brunt of the attacks.

The second problem is part of what we’re troubleshooting this evening.

At first glance the SYN packet issue would seem like a common SYN flood attack but other shared servers in our infrastructure receive similar amounts of traffic and don’t have any problems mitigating it.

In the past 24 hours we’ve discovered key differences in Opal4’s hardware compared to the other servers, so we’re working with our upstream provider to sort that out.

Tonight’s downtime is at their request to allow them to perform hardware diagnostics in support of that investigation. (23:24 UTC — Apr 1)

Watching - Opal4 just went through a brief spike in system load during which performance was degraded, but the system is back to normal at this time. (23:20 UTC — Mar 29)

Identified - We’ve identified two distinct attack patterns responsible for the intermittent outages on opal4 and are putting measures in place to mitigate them. (23:01 UTC — Mar 25)

Watching - This intermittent issue on Opal4 has started up again - we will continue to monitor as we work to resolve it. (19:05 UTC — Mar 25)

Fixed - We’ve seen no further issues in the past several hours. (16:38 UTC — Jan 8)

Watching - The problem appears to have been caused by temporary high load due to high CPU usage by the system firewall.

opal4 is stable at this time. We’ll continue to monitor. (01:21 UTC — Jan 8)

opal4.opalstack.com is experiencing intermittent outages. We’re looking into it and will update this item when we have more information. (00:36 UTC — Jan 8)