Shared: Frankfurt (DE) on Opalstack Status

[Resolved] Opal17 server connectivity outage

Sun, 26 Jul 2026 23:00:00 +0000

July 27 01:27 UTC: The problem was caused by a failed network device in the server rack where opal17 is located. The failed component has been replaced, and connectivity to opal17 has been fully restored.

July 27 00:50 UTC: Connectivity to opal17 was recently disrupted by local network hardware issues, techs on site working to restore connectivity. (23:00 UTC — Jul 26)

[Resolved] MariaDB disruption on opal9

Thu, 23 Apr 2026 04:27:00 +0000

Fixed - We’ve seen no further issues since our previous update. (06:53 UTC — Apr 23)

A short while ago the MariaDB service on opal9 (Frankfurt shared hosting) went offline due to a problem with its InnoDB storage. The problem has been corrected and the service is operating normally at this time. We’ll continue to monitor. (04:27 UTC — Apr 23)

[Resolved] MariaDB disruption on opal16

Wed, 22 Apr 2026 07:43:00 +0000

Fixed - We’ve seen no further issues since our previous update. (16:50 UTC — Apr 22)

A short while ago the MariaDB service on opal16 (Frankfurt shared hosting) went offline due to a problem with its InnoDB storage. The problem has been corrected and the service is operating normally at this time. We’ll continue to monitor. (07:43 UTC — Apr 22)

[Resolved] MariaDB disruption on opal15

Tue, 17 Mar 2026 15:54:17 +0000

Fixed - We’ve seen no further issues since our previous update. (14:21 UTC — Mar 20)

A short while ago the MariaDB service on opal15 (Frankfurt shared hosting) went offline due to a problem with its InnoDB storage. The problem has been corrected and the service is operating normally at this time. We’ll continue to monitor. (15:54 UTC — Mar 17)

[Resolved] MariaDB disruption on opal5

Sat, 21 Feb 2026 14:11:29 +0000

Fixed - We’ve seen no further issues. (13:30 UTC — Feb 23)

The MariaDB service on opal5 went down approximately 3 hours ago due to a problem with the InnoDB store. We’ve corrected that and the service is now operational. We’ll continue to monitor for problems. (14:11 UTC — Feb 21)

[Resolved] Unexpected reboot of opal9

Tue, 09 Sep 2025 07:55:00 +0000

Fixed - The cause of the reboot was caused by a deadlocked kernel process. We’ve seen no further issues following the reboot. (14:22 UTC — Sep 9)

A short while ago opal9.opalstack.com (Frankfurt shared hosting) rebooted for an unknown reason. The server is back online at this time and we’re investigating the cause of the reboot. (07:55 UTC — Sep 9)

[Resolved] opal11 not responding

Tue, 15 Jul 2025 18:52:47 +0000

Fixed - We’re continuing to monitor the server to make sure there are no additional issues. (00:10 UTC — Jul 16)

Fixed - After some difficulty with the chassis swap and subsequent firmware updates, the data center team has completed repairs and opal11 is now back online. (23:52 UTC — Jul 15)

Identified - The data center team determined that a full chassis swap is required to repair the server. They are proceeding with that at this time. (20:12 UTC — Jul 15)

Identified - The cause of the outage appears to be a failed memory module. The data center team is working to repair it at this time. (19:02 UTC — Jul 15)

opal11 (Frankfurt DE shared hosting) is currently not responding. We’re investigating at this time. (18:52 UTC — Jul 15)

[Resolved] opal5 not responding

Sat, 31 May 2025 16:24:02 +0000

Fixed - The reboot is complete and opal5 is back online. The cause of the outage was high load due to malicious traffic from an abusive network. The network has been blocked. (16:34 UTC — May 31)

Opal5 (Frankfurt shared hosting) is not responding, so we’re rebooting the machine to restore service. (16:24 UTC — May 31)

[Resolved] High load on opal15

Fri, 17 Jan 2025 17:14:19 +0000

Fixed - We’ve identified the cause of the problem and have taken measures to mitigate it as of about 3 hours ago. The server has been stable since that time. (21:11 UTC — Jan 17)

Watching - The high load was caused by heavy CPU usage from the MariaDB service. We’re still working to identify the exact cause but the server is stable at this time. (18:38 UTC — Jan 17)

opal15.opalstack.com (Frankfurt shared hosting) is currently experiencing heavy load. We’re working to resolve it at this time. (17:14 UTC — Jan 17)

[Resolved] opal6 under heavy load

Thu, 19 Sep 2024 18:01:38 +0000

Fixed - The server is back online following the reboot. The cause was a runaway Apache httpd process consuming all of the server’s available RAM.

We’ll identify the site that caused the problem and will work with the site owner to prevent the problem from occurring again. (18:19 UTC — Sep 19)

Investigating - We’re unable to troubleshoot due to the extremely high load, so we are rebooting the server at this time. (18:06 UTC — Sep 19)

opal6.opalstack.com (Fraankfurt shared hosting) is currently under heavy load and is responding very slowly. We’re attempting to identify and correct the cause at this time and will follow up when more info is available. (18:01 UTC — Sep 19)

[Resolved] OPAL6 PostgreSQL upgrade issues.

Fri, 14 Jun 2024 09:34:00 +0000

Fixed - The database upgrade has been reverted and OPAL6 is back running on postgresql-11. (10:33 UTC — Jun 14)

After finishing the upgrade of PostgreSQL on Opal6 we discovered some inconsistencies and we’re investigating further. (09:34 UTC — Jun 14)

[Resolved] Control panel delays for MariaDB items on opal16

Sun, 24 Mar 2024 21:01:08 +0000

Fixed - There have been no further delays. (14:01 UTC — Mar 25)

We are importing a large MariaDB database for a customer on opal16 at this time. The large import is putting heavy load on MariaDB which may result in a delay when creating/deleting new MariaDB databases/users and when changing MariaDB passwords.

We expect the import to be complete by 0300 UTC on Monday 25 March 2024 but hopefully sooner. (21:01 UTC — Mar 24)

[Resolved] Recent disruptions to Apache service on opal16

Fri, 09 Feb 2024 17:32:55 +0000

Fixed - We’ve seen no further issues and are marking this as resolved. (16:39 UTC — Feb 10)

Watching - We believe we’ve mitigated this and will continue to monitor the server. (00:43 UTC — Feb 10)

We’ve seen some recent disruptions on the Apache httpd service on opal16 which were caused by high traffic to various customer sites.

We’re working to mitigate that and will update this post when it has been resolved. (17:32 UTC — Feb 9)

[Resolved] Emergency MariaDB maintenance for opal4, opal7, and opal9

Sun, 03 Dec 2023 07:23:00 +0000

Fixed - The MariaDB recovery on opal9 is complete. Databases with names starting with tro through zzz are being restored from our 2 December 2023 backup.

Our plan now is to test the recovery process on a test server to determine why we were not able to import the dump files and then refine our maintenance procedures based on those findings. (22:21 UTC — Dec 3)

Watching - The MariaDB recovery on opal9 is still in progress. We expect the recovery to be complete in around 90 minutes. (21:21 UTC — Dec 3)

Watching - The MariaDB recovery on opal7 is complete.

The MariaDB recovery on opal9 is still in progress. We’ll update with an ETA as soon as possible. (21:12 UTC — Dec 3)

Watching - The MariaDB recovery on opal7 and opal9 is still in progress. We expect that it will take at least a couple of hours more to complete. (19:39 UTC — Dec 3)

Watching - The MariaDB recovery on opal7 and opal9 is still in progress. We expect that it will take at least a few more hours to complete.

The MariaDB recovery on opal4 is now complete, however we were not able to use the most recent backup for about 50% of the recovered databases. The remaining databases were recovered as follows:

Databases with names starting with lsa through shn were restored from our 2 December 2023 backup.
Databases with names starting with sho through zzz were restored from our 1 December 2023 backup. (16:56 UTC — Dec 3)

Watching - It seems some opal4 databases were not restored. We’re restoring those databases now.

The recovery on opal7 and opal9 is still in progress. (14:55 UTC — Dec 3)

Watching - The MariaDB recovery on opal4 is complete. Recovery on opal7 and opal9 is still in progress. (14:33 UTC — Dec 3)

Watching - The database restore is still in progress. We expect that it will take at least a few more hours. (13:47 UTC — Dec 3)

Watching - What happened?

We took the MariaDB database down to stop the ibdatafile from growing exponentially. The only way to do this is to dump all databases, delete the log files, update the server configuration, and restore the data.

The dump process worked without any errors. The clean up and server configuration completed without any errors. The restore process is where things failed.

Because of errors in 1 or more database(s) transactions failed to complete causing the restore to never get past a certain point. The amount of data in the mysql data directory would increase up to a certain point and then the data size would drop by half before repeating. After running the restore process twice and having errors in different spots we decided to extract each database from the monolithic backup we had taken previously. Why a monolithic backup? It’s usually faster to dump and restore, except in this case.

The extraction process is painfully slow compared to just running a working dump restore. That’s why this process is taking so long.

What didn’t we do?

We could have restored from the latest backup which would have been 24 hours or less old, however, that option comes with the significant risk of data loss. Rather than risk losing 24 hours of data we went with the slower, safer process.

The current restore process for each server is now past the previous failure point.

We’d sincerely apologize for the downtime this has caused for your apps that use MariaDB. In the tests we performed before the actual event we did not run into any of these errors. (12:15 UTC — Dec 3)

Watching - The data restoration process is still ongoing. There is currently no ETA we can provide. (10:29 UTC — Dec 3)

Watching - Opal4, Opal7, Opal9: We are still restoring data to the MariaDB databases. The exact amount of time left in the restoration process is unknown but we will update every hour until all of the restores have finished. (08:42 UTC — Dec 3)

On Sunday, 03 December 2023 at 0500 UTC we’ll be taking the managed MariaDB database service offline for maintenance on the following servers:

opal4.opalstack.com (Dallas)
opal7.opalstack.com (Phoenix)
opal9.opalstack.com (Frankfurt)

The maintenance window is 2 hours. During the maintenance sites and applications which use MariaDB (including WordPress sites) will not function.

We apologize for the short notice and any inconvenience. If you have any questions or concerns regarding the maintenance then please contact our support team. (07:23 UTC — Dec 3)

[Resolved] Opal9 offline for scheduled maintenance

Sun, 28 May 2023 04:04:39 +0000

Fixed - The maintenance is complete and opal9 is back online. (04:23 UTC — May 28)

opal9.opalstack.com is offline for scheduled maintenance. (04:04 UTC — May 28)

[Resolved] Opal9 not responding

Wed, 01 Feb 2023 04:27:20 +0000

Fixed - The relocation is complete. (13:19 UTC — Feb 16)

Watching - The servers are now offline and being moved between datacenters. (09:27 UTC — Feb 16)

Watching - The servers are now offline and being moved between datacenters. (09:25 UTC — Feb 16)

Watching - The servers are now offline and being moved between datacenters. (09:21 UTC — Feb 16)

Fixed - OPAL9 unexpectedly lost network connection and was rebooted. After the reboot the boot process hung on a file system corruption. We’ve run the file system check on all file systems, updated the boot loader, system bios, and kernel versions and the server is now back online and functioning normally. (08:55 UTC — Feb 1)

Opal9 is currently not responding, we’re investigating. (04:27 UTC — Feb 1)

[Resolved] MariaDB disrupted on opal11

Fri, 18 Nov 2022 14:14:45 +0000

The MariaDB service on Opal11 (Frankfurt shared hosting) was briefly disrupted due to a service restart. The service is operating normally at this time. (14:14 UTC — Nov 18)

[Resolved] Opal11 offline for scheduled relocation

Thu, 15 Sep 2022 07:00:40 +0000

Fixed - The relocation was completed ahead of schedule as of 10:30 UTC. (13:54 UTC — Sep 15)

As previously announced, opal11 has been shut down for physical relocation to a new data center facility. The expected downtime is approximately 8 hours.

We’ll update this post when the relocation is complete. (07:00 UTC — Sep 15)

[Resolved] opal5 not responding

Mon, 18 Apr 2022 17:31:00 +0000

Fixed - Opal5 is stable following the reboot. We’ll continue to monitor. (18:44 UTC — Apr 18)

Investigating - Opal5 is back online, we’re still investigating. (18:30 UTC — Apr 18)

Investigating - We’re rebooting opal5 at this time. (18:22 UTC — Apr 18)

Investigating - The high load condition has returned so we’re continuing our investigation. (18:11 UTC — Apr 18)

Fixed - Opal5 is back online. The problem was high load caused by heavy disk I/O activity due to high swap usage by a customer process. We’ve increased available swap and are working with the customer to resolve the issue. (17:51 UTC — Apr 18)

Opal5 (shared hosting in Frankfurt DE) is currently not responding to web and SSH requests. We’re investigating. (17:31 UTC — Apr 18)

[Resolved] Service disruption caused by firewall issue

Thu, 10 Feb 2022 19:43:00 +0000

About an hour ago we deployed a firewall update that had the unintended side effect of blocking our resolving DNS servers.

This caused various issues on customer sites on shared hosting, usually showing up as a “504 Gateway Timeout” error.

The issue has been resolved and all customer sites should be working now as they were prior to the change. (19:43 UTC — Feb 10)

[Resolved] opal6 offline

Tue, 08 Feb 2022 13:04:00 +0000

Fixed - The maintenance is complete and opal6 is back online. (22:56 UTC — Feb 8)

Watching - Opal6 is now offline for scheduled emergency maintenance. The expected downtime is 30 minutes. (22:40 UTC — Feb 8)

Watching - The data center team suspects that a fault in the server’s physical power switch caused the server to power down unexpectedly.

We’ve scheduled emergency maintenance for 2022-02-08 22:30 UTC (tonight) to investigate and replace the switch if necessary. (17:33 UTC — Feb 8)

Watching - Opal6 is back online. We’re still awaiting word from the data center to determine why the machine had been powered off. (14:09 UTC — Feb 8)

Identified - We discovered that opal6 had been powered down unexpectedly. The machine is rebooting at this time and we’re awaiting an update from the data center as to why the machine was powered down. (13:36 UTC — Feb 8)

Opal6 is currently offline, we’re looking into it. (13:04 UTC — Feb 8)

[Resolved] Intermittent high load on opal11

Thu, 16 Sep 2021 22:31:00 +0000

Fixed - There have been no further issues on opal11 in the past 24 hours so we consider this to be resolved. (14:00 UTC — Oct 2)

Investigating - We’ve mitigated most of the abusive traffic that’s been driving up high load on opal11 and performance should be much better at this time.

We are still seeing occasional spikes and will continue to investigate. (16:02 UTC — Oct 1)

Investigating - Our investigation into this issue is ongoing. We’ll provide more info when it is available. (15:35 UTC — Sep 30)

Our shared hosting server opal11.opalstack.com in Frankfurt Germany is experiencing an intermittent high load condition which is impacting system performance. We’re working to resolve this and will update this post when the issue is resolved. (22:31 UTC — Sep 16)

[Resolved] opal5 offline

Mon, 12 Jul 2021 09:32:04 +0000

Fixed - The null route has been removed and opal5 is back online. (09:53 UTC — Jul 12)

Identified - The problem seems to be a network null route due to an abuse issue that has been mis-handled by an upstream provider. We’re working on restoring service now. (09:40 UTC — Jul 12)

Opal5 (shared hosting in Frankfurt) is offline. We’re investigating at this time. (09:32 UTC — Jul 12)

[Resolved] Apache was briefly down on opal5

Thu, 10 Jun 2021 02:00:00 +0000

The Apache httpd service on opal5 was briefly down because the system /tmp directory was full.

The service was down for 2-3 minutes. We’ve identified the customer responsible for the problem and will work with them to ensure it does not happen again. (02:00 UTC — Jun 10)

[Resolved] MariaDB down on opal9

Fri, 30 Apr 2021 15:48:02 +0000

Fixed - The MariaDB service on Opal9 is back online. The problem was due to an error in a monitoring check that has been corrected. (15:57 UTC — Apr 30)

The MariaDB (mysql) service on Opal9 is currently offline, we’re working to restore service at this time. (15:48 UTC — Apr 30)

[Resolved] Apache was down on Opal6

Mon, 29 Mar 2021 11:30:00 +0000

The Apache service on Opal6 was offline for approximately 2 hours after the server ran out of available semaphores. The problem has been corrected as of around 13:30 UTC (~1 hour ago) and we’ll be adjusting our monitoring system to detect and correct this condition when it occurs. (11:30 UTC — Mar 29)

[Resolved] opal5 was offline

Thu, 07 Jan 2021 02:03:54 +0000

Opal5 was temporarily offline after running out of disk space. The server is online at this time and we’re working to make more storage available.

Additionally at the time of this incident our status page (this site) was failing save our updates. We’re working to correct that. (02:03 UTC — Jan 7)

[Resolved] Apache down on opal6

Mon, 16 Nov 2020 11:30:00 +0000

Fixed - The Opal6 Apache httpd and PHP-FPM services were hitting an open files limit. We’ve raised the limit and the services are back online. (15:46 UTC — Nov 16)

The Apache httpd service on opal6 is currently down, we’re looking into it. (11:30 UTC — Nov 16)

[Resolved] Let's Encrypt rate limit for opalstacked.com subdomains

Wed, 21 Oct 2020 17:40:00 +0000

Fixed - We’ve made some changes to the way Let’s Encrypt works with the shared opalstacked.com domain and as a result the domain is no longer rate-limited. We’ll be posting updated documentation with the details later this week. (19:34 UTC — Nov 3)

Watching - Looks like we’ve hit our new limit, we’ll get this corrected ASAP. (20:56 UTC — Oct 27)

Fixed - We’ve confirmed that our increased rate limits are in effect, Calling this fixed, thanks for your patience :-) (16:36 UTC — Oct 25)

Watching - The LE service has increased our limit and at this time we are not seeing any rate-limiting for new LE certificates and renewals on any domains. We’ll continue to monitor. (16:03 UTC — Oct 24)

At this time the Let’s Encrypt service is rate-limiting certificate issue and renewal requests for the entire *.opalstacked.com domain. This is the domain that we use to provide free subdomains to our hosting customers.

We’re working with the LE team to get a higher limit. In the meantime if you are attempting to issue a LE cert for a site that uses your opalstacked.com subdomain then it may fail. If this happens you’ll receive a failure notice via email.

Until the rate limit is raised then the alternative is to use one of your own registered domains on your site instead of your opalstacked.com subdomain. If you’re not ready to point your domain at Opalstack, then you can create a testing subdomain and A record via your NS provider to point that subdomain at your Opalstack server.

We’ll update this issue as soon as the rate limit has been raised. (17:40 UTC — Oct 21)

[Resolved] Intermittent PHP-FPM failures on Opal4 and Opal5

Sun, 18 Oct 2020 15:34:00 +0000

Fixed - We’ve rebuilt most of our PHP stack and this issue is now resolved. Updated documentation and maybe a blog post are on the way. (02:54 UTC — Oct 26)

Watching - Over the last few weeks we’ve dealt with various PHP-FPM related problems. After trying various combinations of fixes to the server, apache, and PHP-FPM itself we’ve developed a new, more stable PHP-CGI stack to replace PHP-FPM. With PHP and WordPress powering large parts of the Internet (and businesses) having a stable, reliable stack for them is critical.

We rolled the new stack out to all servers yesterday afternoon and found some edge cases we hadn’t found in our previous testing.

We’ll be rolling those changes out again tonight. While we do there may be a brief downtime of less than 10 minutes across PHP based sites and the update happens. Once the sites are updated we’ll begin monitoring for any breakages that didn’t show up in our testing and fix them ASAP.

We’ll follow up with a blog post on exactly what we’ve changed and how things are working in the future. (04:21 UTC — Oct 23)

Watching - Our troubleshooting is going but we’ve improved the stability of PHP-FPM quite a bit over the past few days and are seeing no customer facing problems at this time. We’ll leave this incident open until we’re satisfied that we’ve got this issue nailed down. (17:32 UTC — Oct 21)

Watching - There have been intermittent PHP-FPM outages on Opal5 in the past several hours. Troubleshooting is ongoing. (20:45 UTC — Oct 19)

Watching - PHP-FPM for PHP 7.3 on Opal5 was briefly disrupted about 15 minutes ago. We’ll continue to troubleshoot and monitor. (23:24 UTC — Oct 18)

Watching - We’ve just deployed a potential fix and will continue to monitor. (19:01 UTC — Oct 18)

We’ve been investigating intermittent failures of PHP sites hosted on Opal4 (Dallas) and Opal5 (Frankfurt) which result in “503 Service Unavailable” responses on websites. We’ll post more details here as they become available. (15:34 UTC — Oct 18)

[Resolved] Connectivity issues in Frankfurt

Thu, 16 Apr 2020 19:00:00 +0000

Fixed - The data center team has reported that the issue has been resolved. (20:37 UTC — Apr 16)

Several servers in our Frankfurt DE datacenter have experienced network connectivity issues.

The data center team reports that the problem is due to high temperatures in the DC. Their investigation into the root cause is ongoing.

At this time all servers are online. We’ll update this post when more information is available. (19:00 UTC — Apr 16)