Shared: San Francisco (USA) on Opalstack Status

[Resolved] opal3.opalstack.com rebooted

Sat, 18 Oct 2025 19:20:03 +0000

Fixed - We’ve seen no further issues following the reboot. (19:46 UTC — Oct 18)

A short while ago opal3.opalstack.com (SFO shared hosting) was not responding due to heavy load. We’ve rebooted the machine to restore service. (19:20 UTC — Oct 18)

[Resolved] opal3 not responding

Sat, 01 Mar 2025 02:44:46 +0000

Fixed - We identified an abusive IP address that was flooding the network connection. The IP has been blocked and the server is now responding. (02:46 UTC — Mar 1)

opal3.opalstack.com (San Francisco shared hosting) is currently not responding. We’re investigating. (02:44 UTC — Mar 1)

[Resolved] Intermittent high load on opal14

Thu, 30 May 2024 19:47:00 +0000

Fixed - We believe we identified the customer site that has been causing these issues and have taken action to mitigate it. We will continue to monitor a couple more days. (15:59 UTC — Jun 2)

At this time we are tracking an high load condition on opal14.opalstack.com (San Francisco shared hosting) which has been causing the server to become unresponsive on an intermittent basis. The problem appears to be caused by high memory usage by the system Apache service.

We’re working to identify the customer site that is causing the high memory usage and will work with them to resolve the issue once we’ve identified it. (19:47 UTC — May 30)

[Resolved] Nginx offline on opal8

Mon, 08 Apr 2024 14:14:14 +0000

Fixed - All services on opal8 are back online. (14:36 UTC — Apr 8)

Identified - Nginx is back online, we’re now working to restore Apache for sites that use it. (14:22 UTC — Apr 8)

The front-end Nginx proxy on opal8 is currently down and as a result customer sites on opal8 are offline. We’re working to restore service at this time. (14:14 UTC — Apr 8)

[Resolved] Opal14 not responding

Sat, 30 Mar 2024 23:04:31 +0000

Fixed - There was a second incident of the same type of problem but on a different site operated by the same customer. We’ve mitigated the problem to prevent it from occurring again and the server is stable at this time. (14:53 UTC — Mar 31)

Watching - Opal14 is back online. The cause of the issue was high Apache RAM usage which resulted from a bad rewrite rule in a customer site. We’ve isolated the site and are working with the customer to correct the problem. We’ll continue to monitor. (23:20 UTC — Mar 30)

Opal14 (SFO shared hosting) is currently not responding. We’re investigating at this time. (23:04 UTC — Mar 30)

[Resolved] Opal14 is not responding

Sun, 07 Jan 2024 05:10:26 +0000

Fixed - Opal14 is back online following the reboot and is operating normally. (05:19 UTC — Jan 7)

opal14.opalstack.com is not responding. We suspect the cause is high load. We’re rebooting the server at this time and will update this post when the server is back online. (05:10 UTC — Jan 7)

[Resolved] Emergency reboot of opal3

Sat, 04 Nov 2023 21:12:25 +0000

Fixed - Opal3 is back online. (21:17 UTC — Nov 4)

opal3.opalstack.com is currently being rebooted to correct an issue with the system /tmp directory. Apologies for the unplanned maintenance. (21:12 UTC — Nov 4)

[Resolved] Opal8 service disruption

Thu, 20 Jul 2023 22:30:00 +0000

opal8.opalstack.com (San Francisco shared hosting) became non-responsive at 3:30 PM US/Pacific time. We suspected a temporary high load condition, but after several minutes the server remained unresponsive. At that point we decided to reboot the server.

The server is online and operating normally at this time. We’ll work to identify the site which caused the high load and will then work with that customer to guard against future incidents. (22:30 UTC — Jul 20)

[Resolved] Unplanned reboot of opal3

Sat, 10 Dec 2022 00:00:00 +0000

On Saturday 10 December at around 0800 UTC (midnight US/Pacific) opal3.opalstack.com in San Francisco became unresponsive. After several unsuccessful attempts to access the server for troubleshooting, we rebooted the server and it was back online around 0900 UTC (1am US/Pacific).

The outage appears to have been triggered by a very rapid spike in system memory usage. The high load unfortunately interrupted the system logging facilities so we weren’t able to precisely identify the cause, but we suspect that it was due to an buggy Apache rewrite rule in a customer application. (00:00 UTC — Dec 10)

[Resolved] Apache was down on opal3

Fri, 07 Oct 2022 15:31:09 +0000

An undetected fault on opal3.opalstack.com caused a temporary disruption in the Apache service. The problem has been corrected. (15:31 UTC — Oct 7)

[Resolved] opal8 not responding

Mon, 19 Sep 2022 21:26:00 +0000

Fixed - It appears that the earlier reboot was initiated by the system watchdog after the server became non-responsive under heavy load.

The server is operating normally at this time. (22:12 UTC — Sep 19)

Investigating - Opal8 is back online. The server rebooted unexpectedly so we’re investigating further. (21:29 UTC — Sep 19)

Opal8 is not responding at this time. We’re looking into it. (21:26 UTC — Sep 19)

[Resolved] MariaDB restarted on Opal8

Wed, 16 Feb 2022 03:00:00 +0000

At 2022-02-16 03:00 UTC, the MariaDB database service on opal8.opalstack.com was restarted due to high CPU utilization. This may have resulted in database connection errors or similar errors on websites that use the MariaDB service for their databases.

The service has remained stable following the restart. (03:00 UTC — Feb 16)

[Resolved] Service disruption caused by firewall issue

Thu, 10 Feb 2022 19:43:00 +0000

About an hour ago we deployed a firewall update that had the unintended side effect of blocking our resolving DNS servers.

This caused various issues on customer sites on shared hosting, usually showing up as a “504 Gateway Timeout” error.

The issue has been resolved and all customer sites should be working now as they were prior to the change. (19:43 UTC — Feb 10)

[Resolved] Intermittent connectivity to San Francisco servers including US webmail

Sat, 11 Dec 2021 15:09:34 +0000

Fixed - The maintenance was completed as scheduled and we’re seeing no further issues at this time. (19:58 UTC — Dec 11)

Network maintenance in one of our San Francisco facilities is causing intermittent connectivity issues for some customers on servers in that location.

This also affects our US webmail server which is located in San Francisco.

The maintenance is expected to be complete at 17:30 UTC Dec 11 2021 (9:30 AM US Pacific). (15:09 UTC — Dec 11)

[Resolved] opal3 not responding

Fri, 10 Sep 2021 12:02:00 +0000

Fixed - We’ve positively identified the cause of the memory spike and have mitigated it. (13:38 UTC — Sep 10)

Watching - The root cause was a massive spike in system load due to a runaway customer process using most of the available system RAM. We’ll work with the customer to ensure it doesn’t happen again. (13:17 UTC — Sep 10)

Watching - Opal3 is back online at this time. We’ll continue to monitor. (13:11 UTC — Sep 10)

Investigating - We’re rebooting opal3 at this time. (13:00 UTC — Sep 10)

Opal3 is currently not responding. We’re investigating. (12:02 UTC — Sep 10)

[Resolved] Apache was down on Opal8

Fri, 09 Jul 2021 21:08:36 +0000

The system Apache httpd service on opal8.opalstack.com was briefly offline due to a problem with its configured system resource limits.

During the incident customers visiting the Apache-powered sites may have seen a “502 bad gateway” error message.

The problem has been corrected and the service is working normally at this time. (21:08 UTC — Jul 9)

[Resolved] Opal8 not responding

Mon, 10 May 2021 21:23:34 +0000

Fixed - Repairs are complete and Opal8 is back online. (02:27 UTC — May 11)

Identified - Opal8 is now down for repair. The expected downtime is 30 minutes. (02:04 UTC — May 11)

Watching - The server has remained stable since it came back online after the initial incident.

We’re now waiting on the data center team to bring down the server for repair. (00:45 UTC — May 11)

Identified - Opal8 has a faulty memory module. We’re now scheduling a repair with the data center team. (21:39 UTC — May 10)

Investigating - Opal8 is back online after an unexpected reboot. We’re still investigating. (21:27 UTC — May 10)

Opal8 (shared server in San Francisco) is currently not responding. We’re looking into the issue and will follow up here when we have more info. (21:23 UTC — May 10)

[Resolved] Opal3 not responding

Sun, 17 Jan 2021 15:39:44 +0000

Fixed - The problem was caused by extremely high RAM usage from a process run by a hosting customer. We’ve notified the customer of the problem and have taken steps to prevent it from happening again. (16:59 UTC — Jan 17)

Fixed - Opal3 is stable after the reboot. We’ll update this post when we’ve determined the cause of the outage. (16:25 UTC — Jan 17)

Investigating - Opal3 is back online after a reboot. The cause of the outage is still under investigation. (15:59 UTC — Jan 17)

Investigating - We’re rebooting Opal3 at this time. (15:52 UTC — Jan 17)

opal3.opalstack.com is currently not responding. We’ll follow up here when more info is available. (15:39 UTC — Jan 17)

[Resolved] Apache httpd problems on opal3

Thu, 05 Nov 2020 01:31:00 +0000

Fixed - Calling this fixed! (21:40 UTC — Nov 6)

Watching - Apache httpd on Opal3 has been stable since our last update. We’ll continue to monitor. (04:32 UTC — Nov 5)

The Apache httpd service on Opal3 is bumping into some resource limits and some customers are reporting errors on their sites. We’ve made made some adjustments and are monitoring. (01:31 UTC — Nov 5)

[Resolved] Let's Encrypt rate limit for opalstacked.com subdomains

Wed, 21 Oct 2020 17:40:00 +0000

Fixed - We’ve made some changes to the way Let’s Encrypt works with the shared opalstacked.com domain and as a result the domain is no longer rate-limited. We’ll be posting updated documentation with the details later this week. (19:34 UTC — Nov 3)

Watching - Looks like we’ve hit our new limit, we’ll get this corrected ASAP. (20:56 UTC — Oct 27)

Fixed - We’ve confirmed that our increased rate limits are in effect, Calling this fixed, thanks for your patience :-) (16:36 UTC — Oct 25)

Watching - The LE service has increased our limit and at this time we are not seeing any rate-limiting for new LE certificates and renewals on any domains. We’ll continue to monitor. (16:03 UTC — Oct 24)

At this time the Let’s Encrypt service is rate-limiting certificate issue and renewal requests for the entire *.opalstacked.com domain. This is the domain that we use to provide free subdomains to our hosting customers.

We’re working with the LE team to get a higher limit. In the meantime if you are attempting to issue a LE cert for a site that uses your opalstacked.com subdomain then it may fail. If this happens you’ll receive a failure notice via email.

Until the rate limit is raised then the alternative is to use one of your own registered domains on your site instead of your opalstacked.com subdomain. If you’re not ready to point your domain at Opalstack, then you can create a testing subdomain and A record via your NS provider to point that subdomain at your Opalstack server.

We’ll update this issue as soon as the rate limit has been raised. (17:40 UTC — Oct 21)

[Resolved] Service disruption in San Francisco

Sun, 16 Aug 2020 13:00:00 +0000

Fixed - All SFO servers are now back online. (16:17 UTC — Aug 16)

Identified - vps15 and mx2.us are back online. (15:54 UTC — Aug 16)

Identified - The webmail server is back online and is operational with the exception of sending mail. We’re still waiting on the remaining servers to come back online. (15:26 UTC — Aug 16)

Identified - The data center team reports:

The issue has been identified as a utility power issue. Power is transferred to generators and is currently running stable and network connectivity is restored for the vast majority of the services.

We’re starting to see the remaining machines come online at this time. (15:08 UTC — Aug 16)

Investigating - opal3.opalstack.com is back online at this time. Other servers are still offline and we’re awaiting a report on the initial investigation from the data center team. (14:19 UTC — Aug 16)

Services in our San Francisco data center are currently disrupted. The following servers are affected:

community.opalstack.com
mx2.us.opalstack.com
opal3.opalstack.com
smtp2.us.opalstack.com
vps4.opalstack.com
vps5.opalstack.com
vps6.opalstack.com
vps13.opalstack.com
vps15.opalstack.com
vps16.opalstack.com
vps17.opalstack.com
vps19.opalstack.com
vps21.opalstack.com
vps24.opalstack.com
vps28.opalstack.com
webmail.us.opalstack.com

We’ll update this post as soon as more information is available. (13:00 UTC — Aug 16)

[Resolved] Opal3 offline

Sun, 12 Jan 2020 17:40:00 +0000

Fixed - There have been no further problems since the last update. (00:24 UTC — Jan 13)

Watching - As of 2020-01-12 22:20 UTC all Opal3 shell user home directories, PostgreSQL databases, and related control panel data have been restored.

All customer sites are back online and the server is operating normally at this time. We’ll continue to monitor.

We were not able to recover shell user passwords, so Opal3 customers will need to reset their shell user passwords via the control panel. Instructions are available in our documentation: Changing a Shell User’s Password (22:25 UTC — Jan 12)

Identified - At approximately 17:40 UTC (9:40 AM Pacific) on 12 January 2020 we ran a live test of our upcoming migration tool on opal3, our shared hosting server in San Francisco.

A serious bug in the migration tool resulted in the loss of all customer home directories and PostgreSQL databases on opal3.

We’re presently restoring opal3 from the most recent backup which was completed at 00:14 AM, about 7.5 hours prior to the incident.

We’ll update this incident with more information when the restoration is complete. (19:17 UTC — Jan 12)

Opal3 was offline. (17:40 UTC — Jan 12)