Shared: Washington D.C. (USA) on Opalstack Status

[Resolved] opal12 under attack

Fri, 03 Apr 2026 13:25:06 +0000

Fixed - The reboot is complete and server conditions are normal. (13:38 UTC — Apr 3)

opal12 (WDC shared hosting) is currently experiencing a DDOS attack against a customer site. We’ve identified the site and have taken measures to mitigate that attack, but we need to reboot the server in order to restore service.

We’re doing that now and will follow up when the server is back online. (13:25 UTC — Apr 3)

[Resolved] MariaDB offline on opal1

Mon, 18 Aug 2025 14:49:47 +0000

Fixed - We’ve seen no further issues with MariaDB on opal1. (18:22 UTC — Aug 19)

Watching - The restore is 100% complete and the server should be online and functioning. We will continue to monitor. (20:36 UTC — Aug 18)

Watching - The restore is 50% complete. (19:15 UTC — Aug 18)

Watching - Following the integrity checks the database service remains unstable, so we’re now rebuilding the system via a full dump/restore of all data. The database service will be unavailable until the restore is complete. (17:41 UTC — Aug 18)

Watching - The database service is back online. We’re now running integrity checks on customer databases. (14:58 UTC — Aug 18)

The MariaDB database service on opal1.opalstack.com (Washington DC shared hosting) is currently offline. We’re working to restore service at this time. (14:49 UTC — Aug 18)

[Resolved] Unexpected reboot of opal12

Sun, 01 Jun 2025 11:20:00 +0000

opal12.opalstack.com (Washington DC shared hosting) was rebooted unexpectedly on Sunday, 1 June 2025 at around 11:20 UTC. The reboot was caused by a power issue. The server was back online at 11:27 UTC and has been operating normally since (11:20 UTC — Jun 1)

[Resolved] Inadvertent shutdown of opal1

Sat, 17 May 2025 13:04:47 +0000

At approximately 04:00 UTC today, opal1.opalstack.com (Washington DC shared hosting was inadvertently shut down by the data center. The server was rebooted shortly after the shutdown and is online and operating normally at this time. (13:04 UTC — May 17)

[Resolved] Inadvertent shutdown of opal12

Wed, 19 Feb 2025 06:00:00 +0000

At approximately 05:20 UTC today, opal12.opalstack.com (Washington DC shared hosting was inadvertently shut down by the data center. The server was rebooted shortly after the shutdown and is online and operating normally at this time. (06:00 UTC — Feb 19)

[Resolved] opal12 not responding

Tue, 21 Jan 2025 04:09:43 +0000

Fixed - The outage was caused by a very heavy spike in system load, itself caused by a brief (but intense) attack from an abusive IP address. The problem has been mitigated and the server has been stable since our previous update. (15:00 UTC — Jan 21)

Watching - opal12 is back online and stable at this time. The cause of the outage is still under investigation. (04:30 UTC — Jan 21)

opal12.opalstack.com (washington DC shared hosting) is currenty not responding. We’re investigating at this time. (04:09 UTC — Jan 21)

[Resolved] MariaDB down on opal10

Thu, 16 Jan 2025 13:34:02 +0000

Fixed - The recovery is complete and the database service is back online. (19:00 UTC — Jan 16)

Identified - We’ve been unable to recover the MariaDB service in-place, so we’re restoring the service from the most recent backup which was completed about 13 hours ago at 04:21 UTC.

The estimated recovery time is 4 hours or less. We’ll update this post when more information is available. (17:25 UTC — Jan 16)

Identified - The MariaDB InnoDB store on opal10 is corrupted. We’re still working to recover but do not have an ETA at this time. (16:20 UTC — Jan 16)

At this time the MariaDB service on opal10 is offline. We’re working to restore service and will update this post when more information is available. (13:34 UTC — Jan 16)

[Resolved] Intermittent network connectivity to opal12

Sat, 14 Sep 2024 18:41:20 +0000

Fixed - We’ve seen no further issues. (15:03 UTC — Sep 16)

Watching - We’ve identified and blocked an abusive IP address that was saturating the network. Connectivity is stable at this time. We will continue to monitor. (19:25 UTC — Sep 14)

Investigating - Connectivity is failing again, we’ll continue to troubleshoot. (18:56 UTC — Sep 14)

Watching - opal12 is back online at this time. We suspect that a network issue upstream in the data center was the cause. We’re awaiting confirmation and will continue to monitor. (18:50 UTC — Sep 14)

At this time network connectivity to opal12.opalstack.com is intermittent and customer sites may be inaccessible. We’re investigating at this time and will update when more information is available. (18:41 UTC — Sep 14)

[Resolved] opal12 blocked by Malwarebytes

Thu, 12 Sep 2024 15:46:00 +0000

Fixed - opal12.opalstack.com is no longer blocked by Malwarebytes. (20:31 UTC — Oct 1)

At this time opal12.opalstack.com (Washington DC shared hosting) is being blocked by Malwarebytes. The cause was an outbound SSH attack from a compromised customer shell account which caused the server to be listed in a major block list. As a result visitors who use Malwarebytes may see a warning when visiting sites hosted on opal12.

We’re working with the block list operator to have the server delisted and expect to have this resolved within the next couple of days.

In the meantime, you may be able to work around the problem by adding your site domain and/or server IP 207.244.121.250 to your Malwarebytes exception list. (15:46 UTC — Sep 12)

[Resolved] opal10 rebooted unexpectedly

Tue, 20 Aug 2024 18:02:51 +0000

Fixed - The cause of the reboot was a faulty power cable which has since been replaced. (20:47 UTC — Aug 28)

At approximately 17:36 UTC opal10.opalstack.com (Washington DC shared hosting) rebooted unexpectedly. The server was back online at approximately 17:42 UTC.

The server is operating normally at this time. We’re investigating the cause of the reboot and will update this post when more information is available. (18:02 UTC — Aug 20)

[Resolved] OPAL12 unresponsive.

Thu, 23 May 2024 19:39:00 +0000

Fixed - Everything looks normal and service has been restored. (21:16 UTC — May 23)

Watching - The server is now responding and we are carefully monitoring the server. (19:46 UTC — May 23)

OPAL12 became unresponsive, we’re investigating the cause and working with the datacenter to get the server back online. (19:39 UTC — May 23)

[Resolved] Intermittent network connectivity in Washington DC

Wed, 14 Jun 2023 16:01:15 +0000

Fixed - The issue has been resolved. (17:50 UTC — Jun 15)

There have been intermittent network connectivity issues this morning in our Washington DC shared hosting facility, typically presenting as a network dropout lasting 1-2 seconds. The issues could potentially affect the opal1, opal10, and opal12 shared hosting servers.

The data center team believes they’ve identified the issue and are taking corrective action. We’ll update this post when more information is available. (16:01 UTC — Jun 14)

[Resolved] opal10 not responding

Thu, 13 Oct 2022 11:30:50 +0000

Fixed - The cause of the unexpected reboot was a power cable fault. The cable has been been replaced and the server is operating normally at this time. (16:01 UTC — Oct 13)

Investigating - Opal10 is back online after an unexpected reboot, we’ll continue to investigate. (11:33 UTC — Oct 13)

Opal10 (Washington DC shared hosting) is currently not responding, we’re investigating at this time. (11:30 UTC — Oct 13)

[Resolved] Opal10 not responding

Sat, 27 Aug 2022 04:46:09 +0000

Fixed - Opal10 is back online. (04:54 UTC — Aug 28)

Watching - Opal10 became non-responsive again so we’re rebooting it once more. (04:45 UTC — Aug 28)

Fixed - Opal10 is back online. The outage was caused by a runaway user process consuming too much memory. We’ll work with the related customer to resolve the issue. (05:08 UTC — Aug 27)

Investigating - We’re rebooting opal10 at this time. (04:54 UTC — Aug 27)

Opal10 (shared hosting in Washington DC) is currently not responding. We’re investigating. (04:46 UTC — Aug 27)

[Resolved] Opal1 not responding

Fri, 08 Jul 2022 21:40:47 +0000

Fixed - Opal1 is back online and stable online following a reboot.

The issue was caused by high RAM usage by the server’s web-based database admin site. (21:58 UTC — Jul 8)

Opal1 (Washington DC shared hosting) is currently not responding to requests.

We suspect the issue is heavy server load, but we’re unable to log in to confirm so to restore service we’re now rebooting the server. (21:40 UTC — Jul 8)

[Resolved] Opal12 was offline

Sat, 07 May 2022 18:53:00 +0000

Fixed - We’ve confirmed that the maintenance is complete and there are no further issues at this time. (02:35 UTC — May 14)

Watching - Opal12 is back online and we’re awaiting confirmation from the data center that the maintenance is complete. (02:20 UTC — May 14)

Watching - Opal12 is now offline for scheduled maintenance. (01:00 UTC — May 14)

Watching - Opal12 will be going offline for troubleshooting and repair in approximately 50 minutes at 9PM US/Eastern time. (00:12 UTC — May 14)

Watching - We’ve scheduled maintenance downtime on opal12 to correct the issue with the network controller. Details are at https://status.opalstack.com/schedules/36. (14:13 UTC — May 9)

Watching - We suspect that the server’s network controller is failing. We’re now working with the data center team to have the problem corrected. (19:54 UTC — May 7)

Opal12 (Washington DC shared hosting) was offline for approximately 25 minutes beginning at 18:53 UTC due to an interruption in network connectivity.

We rebooted the server approximately 10 minutes later.

The server is back online and we’re working with our data center team to determine the cause of the problem. (18:53 UTC — May 7)

[Resolved] opal10 not responding

Sun, 03 Apr 2022 10:23:00 +0000

Watching - Opal10 is back following an unexpected reboot. The cause is under investigation. (10:33 UTC — Apr 3)

Opal10 is not responding, we’re looking into it. (10:23 UTC — Apr 3)

[Resolved] Service disruption caused by firewall issue

Thu, 10 Feb 2022 19:43:00 +0000

About an hour ago we deployed a firewall update that had the unintended side effect of blocking our resolving DNS servers.

This caused various issues on customer sites on shared hosting, usually showing up as a “504 Gateway Timeout” error.

The issue has been resolved and all customer sites should be working now as they were prior to the change. (19:43 UTC — Feb 10)

[Resolved] DDOS on opal1

Sat, 22 Jan 2022 19:42:50 +0000

Fixed - We’ve seen no further issues on opal1. (16:31 UTC — Jan 23)

Watching - We’ve mitigated the attack and server conditions are back to normal. We’ll continue to monitor.

We’ll notify the targeted site’s operator and will work with them to prevent further issues. (20:11 UTC — Jan 22)

Opal1 (shared hosting in Washington DC) is currently experiencing elevated load due to a distributed denial of service attack.

We’re working to mitigate the attack at this time. (19:42 UTC — Jan 22)

[Resolved] opal1 not responding

Mon, 24 May 2021 16:56:18 +0000

Fixed - The problem was caused by a spike in system load due to a problem with a customer site. Conditions have returned to normal and we’ll work with that customer to prevent further issues. (17:13 UTC — May 24)

opal1.opalstack.com (WDC shared hosting) is currently responding to ping but not to SSH and HTTP requests. We’re investigating and will update this post when more info is available. (16:56 UTC — May 24)

[Resolved] Opal1 not responding

Sat, 17 Apr 2021 04:41:11 +0000

Fixed - Opal1 is back online. The outage appears to have been due to high load due to abnormally heavy memory usage by a customer app. We’re working with the customer to prevent this from happening again. (05:39 UTC — Apr 17)

Investigating - We’re rebooting Opal1 at this time. (04:53 UTC — Apr 17)

Opal1 is not responding to SSH and HTTP requests, we’re looking into it. (04:41 UTC — Apr 17)

[Resolved] Opal1 service disruption

Thu, 18 Feb 2021 11:00:00 +0000

At around 2021-02-18 11:00 UTC today the /tmp partition on opal1.opalstack.com ran out of available space. This disrupted service for applications that needed to write to temporary storage including PHP apps and the MariaDB database service.

At the time of the event our monitoring system was only checking the status of these system services and not their operational performance, so no alert was generated.

We corrected the problem at approximately 14:00 UTC and the server has been stable since that time.

We’re updating our monitoring system to perform the tests necessary to alert us to the conditions that caused this incident which will help us avoid this sort of extended disruption. (11:00 UTC — Feb 18)

[Resolved] Let's Encrypt rate limit for opalstacked.com subdomains

Wed, 21 Oct 2020 17:40:00 +0000

Fixed - We’ve made some changes to the way Let’s Encrypt works with the shared opalstacked.com domain and as a result the domain is no longer rate-limited. We’ll be posting updated documentation with the details later this week. (19:34 UTC — Nov 3)

Watching - Looks like we’ve hit our new limit, we’ll get this corrected ASAP. (20:56 UTC — Oct 27)

Fixed - We’ve confirmed that our increased rate limits are in effect, Calling this fixed, thanks for your patience :-) (16:36 UTC — Oct 25)

Watching - The LE service has increased our limit and at this time we are not seeing any rate-limiting for new LE certificates and renewals on any domains. We’ll continue to monitor. (16:03 UTC — Oct 24)

At this time the Let’s Encrypt service is rate-limiting certificate issue and renewal requests for the entire *.opalstacked.com domain. This is the domain that we use to provide free subdomains to our hosting customers.

We’re working with the LE team to get a higher limit. In the meantime if you are attempting to issue a LE cert for a site that uses your opalstacked.com subdomain then it may fail. If this happens you’ll receive a failure notice via email.

Until the rate limit is raised then the alternative is to use one of your own registered domains on your site instead of your opalstacked.com subdomain. If you’re not ready to point your domain at Opalstack, then you can create a testing subdomain and A record via your NS provider to point that subdomain at your Opalstack server.

We’ll update this issue as soon as the rate limit has been raised. (17:40 UTC — Oct 21)

[Resolved] Opal1 offline for scheduled maintenance

Sat, 01 Feb 2020 04:56:00 +0000

Fixed - The maintenance is complete. (15:47 UTC — Feb 1)

Watching - Opal1 is back online.

Customer applications installed via our control panel should come back online within the next 10 minutes. Other applications may need to be restarted manually.

We’ll close out this incident when we’ve been notified that the maintenance is complete. (08:16 UTC — Feb 1)

Opal1 is currently offline for scheduled maintenance as announced on 31 December 2019: https://status.opalstack.com/#scheduled-6

A 4 hour maintenance window has been scheduled but we’re hopeful that the actual downtime will be much less.

We’ll update this post when the server is back online. (04:56 UTC — Feb 1)