Shared: Phoenix (USA) on Opalstack Status

[Resolved] unexpected reboot of opal13

Wed, 05 Nov 2025 17:57:32 +0000

Fixed - The firmware update is complete and the server is back online. (07:21 UTC — Nov 6)

Identified - opal13.opalstack.com is currently offline for a critical firmware update (06:33 UTC — Nov 6)

Identified - The data center team has determined that the reboot was caused by a firmware bug. We’ll be taking the server offline later this evening to update the firmware.

The server remains online and operating normally at this time. (21:18 UTC — Nov 5)

opal13.opalstack.com (Phoenix shared hosting) was offline for a few minutes following an unexpected reboot. The server is online and operating normally at this time. The cause of the reboot is under investigation. (17:57 UTC — Nov 5)

[Resolved] OPAL7 MariaDB Server Offline

Sat, 07 Dec 2024 10:16:00 +0000

Fixed - The MariaDB recovery is complete and by all indications customer sites should be working normally at this time. If you have a site hosted on opal7 and notice any problems then please contact our support team for further assistance. (15:13 UTC — Dec 7)

Identified - At this time we are rebooting opal7. The recovery is nearly complete. (14:29 UTC — Dec 7)

Identified - The restoration process is 80% complete. (13:37 UTC — Dec 7)

Identified - Databases are still restoring. (13:15 UTC — Dec 7)

Identified - Databases are continuing to be restored. We do not have an ETA on when the full restore will be finished but we’ll keep this post updated. (12:11 UTC — Dec 7)

Identified - The recovery of MariaDB has been unsuccessful through all of the recovery methods. We will begin restoring the databases from latest backup.

The latest backup was taken ~4 hours before the MariaDB server went offline. (11:13 UTC — Dec 7)

Identified - The InnoDB transaction log has been corrupted. We’re working to restore service by running in recovery mode. (10:54 UTC — Dec 7)

The MariaDB server on OPAL7 has stopped responding to queries. We’re investigating this failure now. (10:16 UTC — Dec 7)

[Resolved] High load on opal13

Mon, 28 Oct 2024 20:02:00 +0000

Watching - The cause appears to have been an inbound attack from several IP addresses. We’ve blocked the IPs and server load has returned to normal levels. We will continue to monitor. (20:30 UTC — Oct 28)

At this time we’re tracking an intermittent high load condition on opal13.opalstack.com (San Francisco shared hosting). We’re working to identify the cause and will update this post when more information is available. (20:02 UTC — Oct 28)

[Resolved] MariaDB down on opal7

Thu, 19 Sep 2024 18:16:11 +0000

Fixed - The recovery and repair is complete and the MariaDB service is operating normally. (20:58 UTC — Sep 19)

Watching - The cause of the MariaDB outage was corruption in the service’s InnoDB store.

The service is now operational while running recovery and repair routines in the background. We expect the recovery and repair to take several hours but the service should remain operational in the interim. (18:45 UTC — Sep 19)

Investigating - The MariaDB service is back online at this time but service may be intermittent as we continue to troubleshoot. (18:36 UTC — Sep 19)

The MariaDB service on opal7.opalstack.com (Phoenix AZ shared hosting) is currently down. We’re attempting to restore service and will follow up as soon as possible. (18:16 UTC — Sep 19)

[Resolved] Emergency MariaDB maintenance for opal4, opal7, and opal9

Sun, 03 Dec 2023 07:23:00 +0000

Fixed - The MariaDB recovery on opal9 is complete. Databases with names starting with tro through zzz are being restored from our 2 December 2023 backup.

Our plan now is to test the recovery process on a test server to determine why we were not able to import the dump files and then refine our maintenance procedures based on those findings. (22:21 UTC — Dec 3)

Watching - The MariaDB recovery on opal9 is still in progress. We expect the recovery to be complete in around 90 minutes. (21:21 UTC — Dec 3)

Watching - The MariaDB recovery on opal7 is complete.

The MariaDB recovery on opal9 is still in progress. We’ll update with an ETA as soon as possible. (21:12 UTC — Dec 3)

Watching - The MariaDB recovery on opal7 and opal9 is still in progress. We expect that it will take at least a couple of hours more to complete. (19:39 UTC — Dec 3)

Watching - The MariaDB recovery on opal7 and opal9 is still in progress. We expect that it will take at least a few more hours to complete.

The MariaDB recovery on opal4 is now complete, however we were not able to use the most recent backup for about 50% of the recovered databases. The remaining databases were recovered as follows:

Databases with names starting with lsa through shn were restored from our 2 December 2023 backup.
Databases with names starting with sho through zzz were restored from our 1 December 2023 backup. (16:56 UTC — Dec 3)

Watching - It seems some opal4 databases were not restored. We’re restoring those databases now.

The recovery on opal7 and opal9 is still in progress. (14:55 UTC — Dec 3)

Watching - The MariaDB recovery on opal4 is complete. Recovery on opal7 and opal9 is still in progress. (14:33 UTC — Dec 3)

Watching - The database restore is still in progress. We expect that it will take at least a few more hours. (13:47 UTC — Dec 3)

Watching - What happened?

We took the MariaDB database down to stop the ibdatafile from growing exponentially. The only way to do this is to dump all databases, delete the log files, update the server configuration, and restore the data.

The dump process worked without any errors. The clean up and server configuration completed without any errors. The restore process is where things failed.

Because of errors in 1 or more database(s) transactions failed to complete causing the restore to never get past a certain point. The amount of data in the mysql data directory would increase up to a certain point and then the data size would drop by half before repeating. After running the restore process twice and having errors in different spots we decided to extract each database from the monolithic backup we had taken previously. Why a monolithic backup? It’s usually faster to dump and restore, except in this case.

The extraction process is painfully slow compared to just running a working dump restore. That’s why this process is taking so long.

What didn’t we do?

We could have restored from the latest backup which would have been 24 hours or less old, however, that option comes with the significant risk of data loss. Rather than risk losing 24 hours of data we went with the slower, safer process.

The current restore process for each server is now past the previous failure point.

We’d sincerely apologize for the downtime this has caused for your apps that use MariaDB. In the tests we performed before the actual event we did not run into any of these errors. (12:15 UTC — Dec 3)

Watching - The data restoration process is still ongoing. There is currently no ETA we can provide. (10:29 UTC — Dec 3)

Watching - Opal4, Opal7, Opal9: We are still restoring data to the MariaDB databases. The exact amount of time left in the restoration process is unknown but we will update every hour until all of the restores have finished. (08:42 UTC — Dec 3)

On Sunday, 03 December 2023 at 0500 UTC we’ll be taking the managed MariaDB database service offline for maintenance on the following servers:

opal4.opalstack.com (Dallas)
opal7.opalstack.com (Phoenix)
opal9.opalstack.com (Frankfurt)

The maintenance window is 2 hours. During the maintenance sites and applications which use MariaDB (including WordPress sites) will not function.

We apologize for the short notice and any inconvenience. If you have any questions or concerns regarding the maintenance then please contact our support team. (07:23 UTC — Dec 3)

[Resolved] opal13 not responding

Thu, 20 Oct 2022 15:41:15 +0000

Fixed - The unexpected reboot was caused by human error. The server has been stable since coming back online. (20:19 UTC — Oct 20)

Investigating - Opal13 is back online. The outage was due to an unexpected reboot. We are now investigating the cause. (15:48 UTC — Oct 20)

Opal13 (shared hosting in Phoenix AZ) is currently not responding. We’re investigating now and will follow up when more info is available. (15:41 UTC — Oct 20)

[Resolved] MariaDB down on opal7

Fri, 04 Mar 2022 16:35:48 +0000

Fixed - The issue was due to a problem with the server’s /tmp filesystem. The problem has been corrected and all Opal7 services are now working normally. (16:49 UTC — Mar 4)

The MariaDB service on Opal7 is currently down, we’re working to restore service at this time. (16:35 UTC — Mar 4)

[Resolved] Service disruption caused by firewall issue

Thu, 10 Feb 2022 19:43:00 +0000

About an hour ago we deployed a firewall update that had the unintended side effect of blocking our resolving DNS servers.

This caused various issues on customer sites on shared hosting, usually showing up as a “504 Gateway Timeout” error.

The issue has been resolved and all customer sites should be working now as they were prior to the change. (19:43 UTC — Feb 10)

[Resolved] Opal7 rebooted unexpectedly

Sat, 12 Jun 2021 02:50:00 +0000

Fixed - The maintenance was confirmed completed on Saturday shortly after our last update. (15:11 UTC — Jun 14)

Watching - Opal7 is back online and operating normally. We’re awaiting confirmation from the data center that the maintenance is complete. (17:04 UTC — Jun 12)

Identified - Opal7 is offline for emergency RAM replacement at this time. (16:51 UTC — Jun 12)

Identified - The cause of the reboot was a faulty memory module. We’re scheduling emergency maintenance at this time. (13:44 UTC — Jun 12)

Opal7 (shared hosting in Phoenix) was unexpectedly rebooted. We’re investigating the cause.

The server is online and operating normally at this time. (02:50 UTC — Jun 12)

[Resolved] Opal7 server relocation

Wed, 14 Apr 2021 07:03:16 +0000

Fixed - The maintenance is now complete and opal7 is online. (13:39 UTC — Apr 14)

Watching - Opal7 is back online. We’ll continue to monitor. (13:09 UTC — Apr 14)

Investigating - Opal7 is offline again, the maintenance may be ongoing. Awaiting confirmation from the data center. (12:54 UTC — Apr 14)

Fixed - The migration is complete and Opal7 is back online. (12:33 UTC — Apr 14)

Watching - Opal7 is offline for the migration at this time. (08:00 UTC — Apr 14)

Our data center team will be moving opal7.opalstack.com to a new facility within the next 6 hours. During the move the server will be offline. (07:03 UTC — Apr 14)

[Resolved] Let's Encrypt rate limit for opalstacked.com subdomains

Wed, 21 Oct 2020 17:40:00 +0000

Fixed - We’ve made some changes to the way Let’s Encrypt works with the shared opalstacked.com domain and as a result the domain is no longer rate-limited. We’ll be posting updated documentation with the details later this week. (19:34 UTC — Nov 3)

Watching - Looks like we’ve hit our new limit, we’ll get this corrected ASAP. (20:56 UTC — Oct 27)

Fixed - We’ve confirmed that our increased rate limits are in effect, Calling this fixed, thanks for your patience :-) (16:36 UTC — Oct 25)

Watching - The LE service has increased our limit and at this time we are not seeing any rate-limiting for new LE certificates and renewals on any domains. We’ll continue to monitor. (16:03 UTC — Oct 24)

At this time the Let’s Encrypt service is rate-limiting certificate issue and renewal requests for the entire *.opalstacked.com domain. This is the domain that we use to provide free subdomains to our hosting customers.

We’re working with the LE team to get a higher limit. In the meantime if you are attempting to issue a LE cert for a site that uses your opalstacked.com subdomain then it may fail. If this happens you’ll receive a failure notice via email.

Until the rate limit is raised then the alternative is to use one of your own registered domains on your site instead of your opalstacked.com subdomain. If you’re not ready to point your domain at Opalstack, then you can create a testing subdomain and A record via your NS provider to point that subdomain at your Opalstack server.

We’ll update this issue as soon as the rate limit has been raised. (17:40 UTC — Oct 21)