Opalstack Status

[Resolved] Opal17 server connectivity outage

Sun, 26 Jul 2026 23:00:00 +0000

July 27 01:27 UTC: The problem was caused by a failed network device in the server rack where opal17 is located. The failed component has been replaced, and connectivity to opal17 has been fully restored.

July 27 00:50 UTC: Connectivity to opal17 was recently disrupted by local network hardware issues, techs on site working to restore connectivity. (23:00 UTC — Jul 26)

[Resolved] US IMAP impacted by scheduled maintenance

Wed, 22 Jul 2026 15:02:00 +0000

Fixed - The backlog of incoming mail has been delivered. The maintenance is complete. (07:06 UTC — Jul 23)

Watching - imap1.us.opalstack.com is back online. The backlog of incoming emails is currently being processed. (05:55 UTC — Jul 23)

Investigating - The upstream management systems are back online and we’re now actively working to restore service. (03:47 UTC — Jul 23)

Investigating - The restoration of service is being delayed by an unfortunately-timed outage in our upstream provider’s management systems. We regret the extended downtime and are doing everything that we can to restore service as soon as possible. (02:28 UTC — Jul 23)

Investigating - The resolution of this issue is being delayed by an unfortunately-timed outage in our upstream provider’s management systems. We do regret the extended downtime and are doing everything we can to restore service as soon as possible. (02:24 UTC — Jul 23)

Investigating - The data center team is still working to bring the server back online. (00:30 UTC — Jul 23)

Investigating - The server still has not come back online, the data center team is investigating. (22:33 UTC — Jul 22)

Watching - The server is presently rebooting and should be back online soon. (22:13 UTC — Jul 22)

[July 23 05:38 UTC] The maintenance is complete - imap1.us.opalstack.com is back up and running.

[July 22 15:02 UTC] imap1.us.opalstack.com is currently offline for scheduled maintenance. We’ll update this post when the server is back online. (15:02 UTC — Jul 22)

[Resolved] DE webmail not responding

Sat, 11 Jul 2026 21:29:18 +0000

Fixed - Repairs to the host hardware are complete and the webmail server is back online. (22:49 UTC — Jul 11)

Identified - The outage is caused by a problem with the physical host hardware of the VPS which runs the webmail service. The data center team is investigating and estimated 30-60 minutes of downtime. (21:44 UTC — Jul 11)

Our DE webmail server at webmail.de.opalstack.com is currently not responding. We’re investigating at this time. (21:29 UTC — Jul 11)

[Resolved] Emergency security maintenance / reboots on all AlmaLinux (EL9) servers

Wed, 08 Jul 2026 09:14:21 +0000

Between 08:50 UTC and 09:10 UTC, we performed emergency security maintenance on all AlmaLinux (EL9) servers across all regions to apply critical security updates. This required rebooting the servers, which resulted in brief service interruptions during that window. The maintenance is complete and all affected systems are already back online. (09:14 UTC — Jul 8)

[Resolved] smtp.de.opalstack.com rebooted for emergency maintenance

Mon, 01 Jun 2026 19:17:08 +0000

Fixed - The SMTP server is back online. (19:24 UTC — Jun 1)

We’ve just rebooted smtp.de.opalstack.com for emergency maintenance. We expect the server to be back online within the next few minutes. (19:17 UTC — Jun 1)

[Resolved] Unplanned emergency maintenance on imap1.de.opalstack.com

Wed, 27 May 2026 00:37:00 +0000

May 27 00:37 UTC: imap1.de.opalstack.com encountered connectivity issues. We’re currently analyzing the server to get it back to full service.

May 27 01:34 UTC: The server was successfully rebooted and services are operational. We continue to monitor are check for any underlying issues.

May 27 13:30 UTC: We’ve seen no further issues and consider this to be resolved. (00:37 UTC — May 27)

[Resolved] Emergency security maintenance / reboots on all AlmaLinux (EL9) servers

Thu, 30 Apr 2026 07:31:21 +0000

Between 04:30 UTC and 06:15 UTC, we performed emergency security maintenance on all AlmaLinux (EL9) servers across all regions to apply critical security updates. This required rebooting the servers, which resulted in brief service interruptions during that window. The maintenance is complete and all affected systems are already back online. (07:31 UTC — Apr 30)

[Resolved] MariaDB disruption on opal9

Thu, 23 Apr 2026 04:27:00 +0000

Fixed - We’ve seen no further issues since our previous update. (06:53 UTC — Apr 23)

A short while ago the MariaDB service on opal9 (Frankfurt shared hosting) went offline due to a problem with its InnoDB storage. The problem has been corrected and the service is operating normally at this time. We’ll continue to monitor. (04:27 UTC — Apr 23)

[Resolved] MariaDB disruption on opal16

Wed, 22 Apr 2026 07:43:00 +0000

Fixed - We’ve seen no further issues since our previous update. (16:50 UTC — Apr 22)

A short while ago the MariaDB service on opal16 (Frankfurt shared hosting) went offline due to a problem with its InnoDB storage. The problem has been corrected and the service is operating normally at this time. We’ll continue to monitor. (07:43 UTC — Apr 22)

[Resolved] opal12 under attack

Fri, 03 Apr 2026 13:25:06 +0000

Fixed - The reboot is complete and server conditions are normal. (13:38 UTC — Apr 3)

opal12 (WDC shared hosting) is currently experiencing a DDOS attack against a customer site. We’ve identified the site and have taken measures to mitigate that attack, but we need to reboot the server in order to restore service.

We’re doing that now and will follow up when the server is back online. (13:25 UTC — Apr 3)

[Resolved] MariaDB disruption on opal15

Tue, 17 Mar 2026 15:54:17 +0000

Fixed - We’ve seen no further issues since our previous update. (14:21 UTC — Mar 20)

A short while ago the MariaDB service on opal15 (Frankfurt shared hosting) went offline due to a problem with its InnoDB storage. The problem has been corrected and the service is operating normally at this time. We’ll continue to monitor. (15:54 UTC — Mar 17)

[Resolved] MariaDB disruption on opal5

Sat, 21 Feb 2026 14:11:29 +0000

Fixed - We’ve seen no further issues. (13:30 UTC — Feb 23)

The MariaDB service on opal5 went down approximately 3 hours ago due to a problem with the InnoDB store. We’ve corrected that and the service is now operational. We’ll continue to monitor for problems. (14:11 UTC — Feb 21)

[Resolved] US SMTP was blocked by Microsoft

Mon, 26 Jan 2026 17:09:10 +0000

Our US SMTP service was temporarily blocked by Microsoft due to a spam outbreak from one of our customer accounts. The outbreak has been secured and Microsoft states that the block has been lifted, but that it may take up to 24 hours for the change to take full effect in their systems. (17:09 UTC — Jan 26)

[Resolved] Unplanned emergencey maintenance on smtp2.de.opalstack.com

Fri, 23 Jan 2026 01:24:00 +0000

Fixed - There have been no further issues since our previous update. (15:03 UTC — Jan 24)

Fri Jan 23 00:56 UTC: SMTP2 DE became unreachable and has entered an emergency hardware maintenance state. In the interim, you may temporarily reconfigure your mail client SMTP settings to use smtp1.de.opalstack.com instead. Undates to domains in the dashboard may experience delays.

Fri Jan 23 01:35 UTC: The server became operational but more maintenance is required. No further downtime is expected but we will continue to monitor. (01:24 UTC — Jan 23)

[Resolved] Unplanned Maintenance: webmail.de.opalstack.com

Thu, 18 Dec 2025 23:16:00 +0000

We’re looking into a recent unexpected outage affecting webmail.de.opalstack.com. The site is currently up, but our upstream provider will be doing emergency maintenance on the physical host, which will make the service unavailable during that time. We’ll share updates as soon as we have more details.

Fri Dec 19 02:06 - Server has been powered down for maintenance.

Fri Dec 19 02:11 - Server maintenance complete, operational (23:16 UTC — Dec 18)

[Resolved] unexpected reboot of opal13

Wed, 05 Nov 2025 17:57:32 +0000

Fixed - The firmware update is complete and the server is back online. (07:21 UTC — Nov 6)

Identified - opal13.opalstack.com is currently offline for a critical firmware update (06:33 UTC — Nov 6)

Identified - The data center team has determined that the reboot was caused by a firmware bug. We’ll be taking the server offline later this evening to update the firmware.

The server remains online and operating normally at this time. (21:18 UTC — Nov 5)

opal13.opalstack.com (Phoenix shared hosting) was offline for a few minutes following an unexpected reboot. The server is online and operating normally at this time. The cause of the reboot is under investigation. (17:57 UTC — Nov 5)

[Resolved] opal3.opalstack.com rebooted

Sat, 18 Oct 2025 19:20:03 +0000

Fixed - We’ve seen no further issues following the reboot. (19:46 UTC — Oct 18)

A short while ago opal3.opalstack.com (SFO shared hosting) was not responding due to heavy load. We’ve rebooted the machine to restore service. (19:20 UTC — Oct 18)

[Resolved] Unexpected reboot of opal9

Tue, 09 Sep 2025 07:55:00 +0000

Fixed - The cause of the reboot was caused by a deadlocked kernel process. We’ve seen no further issues following the reboot. (14:22 UTC — Sep 9)

A short while ago opal9.opalstack.com (Frankfurt shared hosting) rebooted for an unknown reason. The server is back online at this time and we’re investigating the cause of the reboot. (07:55 UTC — Sep 9)

[Resolved] Unexpected reboot of opal4

Mon, 01 Sep 2025 23:40:13 +0000

Fixed - The cause of the reboot was determined to be a deadlocked kernel process. We’ve seen no further issues following the reboot. (20:21 UTC — Sep 4)

A short while ago opal4.opalstack.com (Dallas shared hosting) rebooted for an unknown reason. The server is back online at this time and we’re now investigating the cause of the reboot. (23:40 UTC — Sep 1)

[Resolved] Control panel SSL certificate was expired

Fri, 29 Aug 2025 00:05:36 +0000

About 1 hour ago the SSL certificate for the control panel expired due to an administrative error on our end. While the certificate was expired, web traffic and API requests to https://my.opalstack.com/ were disrupted due to the invalid certificate.

The problem has been corrected as of a few moments ago and all services are back online. Apologies for the trouble. (00:05 UTC — Aug 29)

[Resolved] MariaDB offline on opal1

Mon, 18 Aug 2025 14:49:47 +0000

Fixed - We’ve seen no further issues with MariaDB on opal1. (18:22 UTC — Aug 19)

Watching - The restore is 100% complete and the server should be online and functioning. We will continue to monitor. (20:36 UTC — Aug 18)

Watching - The restore is 50% complete. (19:15 UTC — Aug 18)

Watching - Following the integrity checks the database service remains unstable, so we’re now rebuilding the system via a full dump/restore of all data. The database service will be unavailable until the restore is complete. (17:41 UTC — Aug 18)

Watching - The database service is back online. We’re now running integrity checks on customer databases. (14:58 UTC — Aug 18)

The MariaDB database service on opal1.opalstack.com (Washington DC shared hosting) is currently offline. We’re working to restore service at this time. (14:49 UTC — Aug 18)

[Resolved] Microsoft blocking mx1.us.opalstack.com and smtp1.de.opalstack.com

Sat, 16 Aug 2025 15:48:00 +0000

Fixed - The US MX block is now clear. (14:17 UTC — Aug 17)

Watching - The DE SMTP block has been cleared.

We’re still working with Microsoft to clear the US MX block. (17:37 UTC — Aug 16)

At this time mail sent from our DE SMTP server and mail forwarded from one of our US MX servers to outlook.com, hotmail.com, and live.com is being blocked by Microsoft. We’re working with Microsoft to correct the problem.

In the interim:

DE SMTP users can work around the SMTP block by configuring their clients to use smtp2.de.opalstack.com as their outgoing mail server.
US customers who use mail forwarding can work around the MX block by creating new MX records for mx2.us.opalstack.com with priority 5 and mx1.us.opalstack.com with priority 10. (15:48 UTC — Aug 16)

[Resolved] intermittent connectivity for Newark VPS

Sun, 27 Jul 2025 14:11:09 +0000

Fixed - Service to all Newark VPS has been restored. If you are a Newark VPS customer and notice any further issues on our VPS then please contact our support team for assistance. (15:31 UTC — Jul 28)

Identified - Please wait for further updates (00:22 UTC — Jul 28)

Identified - The data center team has made great progress; servers have been coming back online for the past hour or so, but they’re still working on stabilizing connectivity for Newark VPSs. We’ll update this post as soon as more information becomes available. (00:21 UTC — Jul 28)

Identified - The data center team is still working to restore connectivity to a handful of Newark VPS. We’ll update this post when more info is available. (17:40 UTC — Jul 27)

Our VPS data center in Newark is experiencing intermittent connectivity at this time. The cause is heating and cooling issues related to a power failure. The data center team is working to correct the problem at this time. (14:11 UTC — Jul 27)

[Resolved] Control panel and API offline

Tue, 22 Jul 2025 23:16:00 +0000

Fixed - The control panel updated was completed approximately 9 hours ago, apologies for the late update. (12:44 UTC — Jul 23)

The control panel at my.opalstack.com and related API are offline for scheduled maintenance. (23:16 UTC — Jul 22)

[Resolved] opal11 not responding

Tue, 15 Jul 2025 18:52:47 +0000

Fixed - We’re continuing to monitor the server to make sure there are no additional issues. (00:10 UTC — Jul 16)

Fixed - After some difficulty with the chassis swap and subsequent firmware updates, the data center team has completed repairs and opal11 is now back online. (23:52 UTC — Jul 15)

Identified - The data center team determined that a full chassis swap is required to repair the server. They are proceeding with that at this time. (20:12 UTC — Jul 15)

Identified - The cause of the outage appears to be a failed memory module. The data center team is working to repair it at this time. (19:02 UTC — Jul 15)

opal11 (Frankfurt DE shared hosting) is currently not responding. We’re investigating at this time. (18:52 UTC — Jul 15)

[Resolved] iCloud blocking US and DE SMTP

Thu, 26 Jun 2025 18:46:00 +0000

Fixed - The block was lifted as of 0600 UTC today. (21:19 UTC — Jun 27)

At this time Apple’s iCloud email service is blocking messages sent from our SMTP servers to Apple-hosted domains including icloud.com, me.com, and custom email domains that use the iCloud email service.

We’ve raised a ticket with Apple to determine the cause. (18:46 UTC — Jun 26)

[Resolved] unexpected reboot of opal4

Fri, 06 Jun 2025 12:46:27 +0000

Fixed - Opal4 has remained stable for the past several hours. We haven’t been able to determine the precise cause of the unexpected reboots earlier today but we suspect it was a possible fault in the server’s remote management interface. We’ve reset that system and are seeing further no problems at this time. (22:59 UTC — Jun 6)

Investigating - opal4 is back online at this time. Our investigation is still in progress. (16:46 UTC — Jun 6)

Investigating - opal4 is once again offline for troubleshooting and maintenance. During this time, the server may be placed into a rescue mode where it will respond to ping, but SSH connections will fail and other services will not be available. (16:01 UTC — Jun 6)

Investigating - opal4 is experiencing heavy load at this time. (15:21 UTC — Jun 6)

Investigating - Opal4 is back online at this time, the investigation is ongoing. (14:46 UTC — Jun 6)

Investigating - opal4 is now being taken offline for emergency maintenance. (13:59 UTC — Jun 6)

Investigating - The server was rebooted again unexpectedly, we’re investigating the cause. (13:29 UTC — Jun 6)

Investigating - We’ve lost connectivity to opal4 and suspect another reboot, we’re investigating further. (13:21 UTC — Jun 6)

opal4.opalstack.com (Dallas shared hosting) rebooted unexpectedly a few moments ago as around 12:25 UTC. The server is back online and operating normally at this time. (12:46 UTC — Jun 6)

[Resolved] heavy load on opal4

Wed, 04 Jun 2025 14:25:43 +0000

Fixed - The cause of the high load was a compromised customer website. We’ve locked down the affected site and have notified the site operator of the issue. (15:07 UTC — Jun 5)

Investigating - opal4 is back online. We’re still investigating the cause of the high load. (14:43 UTC — Jun 4)

Investigating - We’re rebooting opal4 at this time. (14:34 UTC — Jun 4)

opal4.opalstack.com (Dallas shared hosting) is currently experiencing heavy load resulting in very poor performance. We’re investigating at this time. (14:25 UTC — Jun 4)

[Resolved] Unexpected reboot of opal12

Sun, 01 Jun 2025 11:20:00 +0000

opal12.opalstack.com (Washington DC shared hosting) was rebooted unexpectedly on Sunday, 1 June 2025 at around 11:20 UTC. The reboot was caused by a power issue. The server was back online at 11:27 UTC and has been operating normally since (11:20 UTC — Jun 1)

[Resolved] opal5 not responding

Sat, 31 May 2025 16:24:02 +0000

Fixed - The reboot is complete and opal5 is back online. The cause of the outage was high load due to malicious traffic from an abusive network. The network has been blocked. (16:34 UTC — May 31)

Opal5 (Frankfurt shared hosting) is not responding, so we’re rebooting the machine to restore service. (16:24 UTC — May 31)

[Resolved] Inadvertent shutdown of opal1

Sat, 17 May 2025 13:04:47 +0000

At approximately 04:00 UTC today, opal1.opalstack.com (Washington DC shared hosting was inadvertently shut down by the data center. The server was rebooted shortly after the shutdown and is online and operating normally at this time. (13:04 UTC — May 17)

[Resolved] OPAL4 MariaDB Outage

Sun, 09 Mar 2025 08:12:00 +0000

Fixed - We’ve seen no further problems since our last update. (19:11 UTC — Mar 9)

Watching - The issue with database permissions is resolved and the MariaDB service is back online and operating normally. We will continue to monitor. (14:22 UTC — Mar 9)

Identified - We’ve discovered that there is still a problem with database permissions. We’re working to resolve that at this time. (13:29 UTC — Mar 9)

Fixed - The data restoration has completed and we are continuing to monitor the server as usage increases. (12:29 UTC — Mar 9)

Identified - The restoration is approximately 75% complete. (11:24 UTC — Mar 9)

Identified - The data restoration is approximately 50% complete. (10:32 UTC — Mar 9)

Identified - The MariaDB restore process is still running. (09:41 UTC — Mar 9)

Identified - The database corruption is unrecoverable so we will proceed with restoring from backups. The most recent valid backups were from 1.5 hours before the server crashed so we’ll proceed with the restore from there.

The restoration process varies in completion time. It may take up to 6 hours or more. We’ll keep this post updated. (08:37 UTC — Mar 9)

MariaDB on OPAL4 has crashed, we’re working to identify the cause and solution now. (08:12 UTC — Mar 9)

[Resolved] DE SMTP blocked by iCloud

Mon, 03 Mar 2025 20:21:00 +0000

Fixed - The block was cleared as of 14:22 UTC today. Mail sent to iCloud email addresses via our DE SMTP are no longer bouncing. (17:40 UTC — Mar 7)

Identified - Apple informed us that our SMTP is caught up in a block of a larger IP address range operated by our network provider. Our provider is working with Apple to resolve the issue.

If you are affected by this you can configure your mail client to use smtp2.de.opalstack.com as its SMTP server as a temporary workaround. (20:53 UTC — Mar 5)

Identified - Apple informed us that our SMTP is caught up in a block of a larger IP address range operated by our network provider. Our provider is working with Apple to resolve the issue.

If you are affected by this you can configure your mail client to use smtp2.de.opalstack.com as its SMTP server as a temporary workaround. (20:44 UTC — Mar 5)

At the present time email sent to @icloud.com addresses via our DE SMTP server is being either deferred or bounced. We’re working with Apple to resolve the issue and will update this post when more information is available.

In the meantime, DE mail customers can work around the issue by using our alternate DE SMTP server in their email client settings. The alternate server is: smtp2.de.opalstack.com (20:21 UTC — Mar 3)

[Resolved] opal3 not responding

Sat, 01 Mar 2025 02:44:46 +0000

Fixed - We identified an abusive IP address that was flooding the network connection. The IP has been blocked and the server is now responding. (02:46 UTC — Mar 1)

opal3.opalstack.com (San Francisco shared hosting) is currently not responding. We’re investigating. (02:44 UTC — Mar 1)

[Resolved] Inadvertent shutdown of opal12

Wed, 19 Feb 2025 06:00:00 +0000

At approximately 05:20 UTC today, opal12.opalstack.com (Washington DC shared hosting was inadvertently shut down by the data center. The server was rebooted shortly after the shutdown and is online and operating normally at this time. (06:00 UTC — Feb 19)

[Resolved] opal12 not responding

Tue, 21 Jan 2025 04:09:43 +0000

Fixed - The outage was caused by a very heavy spike in system load, itself caused by a brief (but intense) attack from an abusive IP address. The problem has been mitigated and the server has been stable since our previous update. (15:00 UTC — Jan 21)

Watching - opal12 is back online and stable at this time. The cause of the outage is still under investigation. (04:30 UTC — Jan 21)

opal12.opalstack.com (washington DC shared hosting) is currenty not responding. We’re investigating at this time. (04:09 UTC — Jan 21)

[Resolved] High load on opal15

Fri, 17 Jan 2025 17:14:19 +0000

Fixed - We’ve identified the cause of the problem and have taken measures to mitigate it as of about 3 hours ago. The server has been stable since that time. (21:11 UTC — Jan 17)

Watching - The high load was caused by heavy CPU usage from the MariaDB service. We’re still working to identify the exact cause but the server is stable at this time. (18:38 UTC — Jan 17)

opal15.opalstack.com (Frankfurt shared hosting) is currently experiencing heavy load. We’re working to resolve it at this time. (17:14 UTC — Jan 17)

[Resolved] MariaDB down on opal10

Thu, 16 Jan 2025 13:34:02 +0000

Fixed - The recovery is complete and the database service is back online. (19:00 UTC — Jan 16)

Identified - We’ve been unable to recover the MariaDB service in-place, so we’re restoring the service from the most recent backup which was completed about 13 hours ago at 04:21 UTC.

The estimated recovery time is 4 hours or less. We’ll update this post when more information is available. (17:25 UTC — Jan 16)

Identified - The MariaDB InnoDB store on opal10 is corrupted. We’re still working to recover but do not have an ETA at this time. (16:20 UTC — Jan 16)

At this time the MariaDB service on opal10 is offline. We’re working to restore service and will update this post when more information is available. (13:34 UTC — Jan 16)

[Resolved] OPAL7 MariaDB Server Offline

Sat, 07 Dec 2024 10:16:00 +0000

Fixed - The MariaDB recovery is complete and by all indications customer sites should be working normally at this time. If you have a site hosted on opal7 and notice any problems then please contact our support team for further assistance. (15:13 UTC — Dec 7)

Identified - At this time we are rebooting opal7. The recovery is nearly complete. (14:29 UTC — Dec 7)

Identified - The restoration process is 80% complete. (13:37 UTC — Dec 7)

Identified - Databases are still restoring. (13:15 UTC — Dec 7)

Identified - Databases are continuing to be restored. We do not have an ETA on when the full restore will be finished but we’ll keep this post updated. (12:11 UTC — Dec 7)

Identified - The recovery of MariaDB has been unsuccessful through all of the recovery methods. We will begin restoring the databases from latest backup.

The latest backup was taken ~4 hours before the MariaDB server went offline. (11:13 UTC — Dec 7)

Identified - The InnoDB transaction log has been corrupted. We’re working to restore service by running in recovery mode. (10:54 UTC — Dec 7)

The MariaDB server on OPAL7 has stopped responding to queries. We’re investigating this failure now. (10:16 UTC — Dec 7)

[Resolved] High load on opal13

Mon, 28 Oct 2024 20:02:00 +0000

Watching - The cause appears to have been an inbound attack from several IP addresses. We’ve blocked the IPs and server load has returned to normal levels. We will continue to monitor. (20:30 UTC — Oct 28)

At this time we’re tracking an intermittent high load condition on opal13.opalstack.com (San Francisco shared hosting). We’re working to identify the cause and will update this post when more information is available. (20:02 UTC — Oct 28)

[Resolved] US SMTP blocked by Microsoft

Tue, 01 Oct 2024 20:17:00 +0000

Fixed - Microsoft has lifted the block and their servers are now accepting mail from the US SMTP. (01:47 UTC — Oct 2)

At this time messages sent from the US SMTP server to Microsoft email domains like hotmail.com, live.com, outlook.com, and their international variants are being blocked by Microsoft.

The cause of the block has been identified and corrected and we are now waiting for Microsoft to remove the block.

We’ll update this post as soon as the block has been lifted or when there is other information to report. (20:17 UTC — Oct 1)

[Resolved] DE mail SSL certificates were briefly expired

Sun, 29 Sep 2024 09:30:00 +0000

Due to an unfortunate oversight on our end the SSL certificates for our DE mail servers expired earlier today at around 09:30 UTC, causing certificate warnings to appear in email client software using DE mailboxes and in our DE webmail system.

Renewed certificates were rolled out approximately 1 hour later and there have been no issues since that time (~10:30 UTC). If you continue to see a certificate warning when accessing DE email systems then please restart your software and/or your device. (09:30 UTC — Sep 29)

[Resolved] DE forwarding main server blocked by Microsoft

Sat, 21 Sep 2024 23:09:03 +0000

Fixed - We’ve seen no further issues since our previous update. (17:47 UTC — Sep 26)

One of our DE MX servers, mx2.de.opalstack.com on IP 178.162.221.165, was recently blocked by Microsoft. As a result, mail sent to addresses which forward to Microsoft services like hotmail.com, outlook.com, and their international variants would be bounced back to the original sender with an error message.

We’ve already resolved this issue with Microsoft but they say that it may take 24-48 hours for the block to be fully lifted.

We’ll update this post after we’ve seen no further blocks in a 24 hour period. (23:09 UTC — Sep 21)

[Resolved] MariaDB down on opal7

Thu, 19 Sep 2024 18:16:11 +0000

Fixed - The recovery and repair is complete and the MariaDB service is operating normally. (20:58 UTC — Sep 19)

Watching - The cause of the MariaDB outage was corruption in the service’s InnoDB store.

The service is now operational while running recovery and repair routines in the background. We expect the recovery and repair to take several hours but the service should remain operational in the interim. (18:45 UTC — Sep 19)

Investigating - The MariaDB service is back online at this time but service may be intermittent as we continue to troubleshoot. (18:36 UTC — Sep 19)

The MariaDB service on opal7.opalstack.com (Phoenix AZ shared hosting) is currently down. We’re attempting to restore service and will follow up as soon as possible. (18:16 UTC — Sep 19)

[Resolved] opal6 under heavy load

Thu, 19 Sep 2024 18:01:38 +0000

Fixed - The server is back online following the reboot. The cause was a runaway Apache httpd process consuming all of the server’s available RAM.

We’ll identify the site that caused the problem and will work with the site owner to prevent the problem from occurring again. (18:19 UTC — Sep 19)

Investigating - We’re unable to troubleshoot due to the extremely high load, so we are rebooting the server at this time. (18:06 UTC — Sep 19)

opal6.opalstack.com (Fraankfurt shared hosting) is currently under heavy load and is responding very slowly. We’re attempting to identify and correct the cause at this time and will follow up when more info is available. (18:01 UTC — Sep 19)

[Resolved] Intermittent network connectivity to opal12

Sat, 14 Sep 2024 18:41:20 +0000

Fixed - We’ve seen no further issues. (15:03 UTC — Sep 16)

Watching - We’ve identified and blocked an abusive IP address that was saturating the network. Connectivity is stable at this time. We will continue to monitor. (19:25 UTC — Sep 14)

Investigating - Connectivity is failing again, we’ll continue to troubleshoot. (18:56 UTC — Sep 14)

Watching - opal12 is back online at this time. We suspect that a network issue upstream in the data center was the cause. We’re awaiting confirmation and will continue to monitor. (18:50 UTC — Sep 14)

At this time network connectivity to opal12.opalstack.com is intermittent and customer sites may be inaccessible. We’re investigating at this time and will update when more information is available. (18:41 UTC — Sep 14)

[Resolved] opal12 blocked by Malwarebytes

Thu, 12 Sep 2024 15:46:00 +0000

Fixed - opal12.opalstack.com is no longer blocked by Malwarebytes. (20:31 UTC — Oct 1)

At this time opal12.opalstack.com (Washington DC shared hosting) is being blocked by Malwarebytes. The cause was an outbound SSH attack from a compromised customer shell account which caused the server to be listed in a major block list. As a result visitors who use Malwarebytes may see a warning when visiting sites hosted on opal12.

We’re working with the block list operator to have the server delisted and expect to have this resolved within the next couple of days.

In the meantime, you may be able to work around the problem by adding your site domain and/or server IP 207.244.121.250 to your Malwarebytes exception list. (15:46 UTC — Sep 12)

[Resolved] opal10 rebooted unexpectedly

Tue, 20 Aug 2024 18:02:51 +0000

Fixed - The cause of the reboot was a faulty power cable which has since been replaced. (20:47 UTC — Aug 28)

At approximately 17:36 UTC opal10.opalstack.com (Washington DC shared hosting) rebooted unexpectedly. The server was back online at approximately 17:42 UTC.

The server is operating normally at this time. We’re investigating the cause of the reboot and will update this post when more information is available. (18:02 UTC — Aug 20)

[Resolved] OPAL4 - MariaDB has crashed.

Wed, 31 Jul 2024 10:19:00 +0000

Fixed - We found and corrected a problem with database permissions and all database users should be able to connect now. (17:00 UTC — Jul 31)

Investigating - We’re getting reports that some users still cannot connect to their databases so we are investigating that now. (16:34 UTC — Jul 31)

Fixed - Databases have been restored and should be fully operational. (16:15 UTC — Jul 31)

Identified - The restore process is at 99% and the last databases are being restored. (15:25 UTC — Jul 31)

Identified - The database restores are 66% complete. (14:45 UTC — Jul 31)

Identified - The database restores are 51% complete. (13:58 UTC — Jul 31)

Identified - The database restores are in progress and are ~25% complete. (13:21 UTC — Jul 31)

Identified - The InnoDB data corruption is too bad to recover data from the database server. Tables are unreadable causing data dumps or repairs to the tables to be impossible.

We have started the database repair process from our backups that were 3-5 hours old when the server went offline. We will update this post with our progress. (12:26 UTC — Jul 31)

Identified - The data corruption on the server is in the innodb data which means there is potentially corrupted data across multiple databases due to how MySQL/MariaDB store data. We are still trying to recover and repair the data in the MariaDB server. If the data is too corrupt to repair we will restore the data from the latest backups. The latest backups were taken between 3 and 5 hours (depending on the dump time) before the database server crashed. (12:07 UTC — Jul 31)

MariaDB has crashed on OPAL4. We are investigating the cause of the crash and working to restore services. (10:19 UTC — Jul 31)

[Resolved] OPAL6 PostgreSQL upgrade issues.

Fri, 14 Jun 2024 09:34:00 +0000

Fixed - The database upgrade has been reverted and OPAL6 is back running on postgresql-11. (10:33 UTC — Jun 14)

After finishing the upgrade of PostgreSQL on Opal6 we discovered some inconsistencies and we’re investigating further. (09:34 UTC — Jun 14)

[Resolved] DNS outage (resolved)

Fri, 31 May 2024 11:45:00 +0000

About 1 hour ago a failed DNS sync with our alternate DNS provider caused an outage in our DNS servers. The outage lasted approximately 1 hour during which customers may have experience trouble accessing Opalstack systems.

The problem has been resolved and all DNS services are working normally at this time, however there may be lingering effects due to global DNS propagation. These issues will resolve themselves over the next few hours. You may be able to resolve them sooner for yourself by flushing your DNS and/or by power-cycling your devices. (11:45 UTC — May 31)

[Resolved] Intermittent high load on opal14

Thu, 30 May 2024 19:47:00 +0000

Fixed - We believe we identified the customer site that has been causing these issues and have taken action to mitigate it. We will continue to monitor a couple more days. (15:59 UTC — Jun 2)

At this time we are tracking an high load condition on opal14.opalstack.com (San Francisco shared hosting) which has been causing the server to become unresponsive on an intermittent basis. The problem appears to be caused by high memory usage by the system Apache service.

We’re working to identify the customer site that is causing the high memory usage and will work with them to resolve the issue once we’ve identified it. (19:47 UTC — May 30)

[Resolved] OPAL12 unresponsive.

Thu, 23 May 2024 19:39:00 +0000

Fixed - Everything looks normal and service has been restored. (21:16 UTC — May 23)

Watching - The server is now responding and we are carefully monitoring the server. (19:46 UTC — May 23)

OPAL12 became unresponsive, we’re investigating the cause and working with the datacenter to get the server back online. (19:39 UTC — May 23)

[Resolved] Temporary mail certificate error, now resolved

Fri, 19 Apr 2024 16:23:54 +0000

At approximately 0500 UTC today (19 April 2024) there was a problem with a routine certificate update which caused invalid certificates to be deployed to our mail servers.

The change was rolled back after about 90 minutes and the mail services are operating normally at this time. (16:23 UTC — Apr 19)

[Resolved] Mail forwarding issue (now resolved)

Wed, 17 Apr 2024 19:33:39 +0000

At around 16:15 UTC on 16 April 2024 a configuration update was deployed to our forwarding mail servers. The update inadvertently included some old configuration that we used when we initially rolled out SRS mail forwarding a few months ago. As a result of this error some mail sent to customer forwarding addresses may have been bounced back to the sender.

The configuration has been corrected today 17 April 2024 as of 19:05 UTC and the old configuration has been purged from our management systems to prevent this from occurring again. (19:33 UTC — Apr 17)

[Resolved] Nginx offline on opal8

Mon, 08 Apr 2024 14:14:14 +0000

Fixed - All services on opal8 are back online. (14:36 UTC — Apr 8)

Identified - Nginx is back online, we’re now working to restore Apache for sites that use it. (14:22 UTC — Apr 8)

The front-end Nginx proxy on opal8 is currently down and as a result customer sites on opal8 are offline. We’re working to restore service at this time. (14:14 UTC — Apr 8)

[Resolved] Control panel delays for US email

Wed, 03 Apr 2024 19:46:00 +0000

At the present time there is a backlog of mailbox and email address changes for the US email system that are waiting to be processed. We expect this backlog to be clear within the next 2-3 hours.

There is no delay in mail delivery, this is only affecting changes to email objects made via our customer control panel. (19:46 UTC — Apr 3)

[Resolved] Delays in US incoming mail

Tue, 02 Apr 2024 18:04:00 +0000

Fixed - All pending incoming US mail has been delivered. (20:30 UTC — Apr 2)

Watching - The high load condition is resolved and the backlog of pending mail is being delivered. (18:25 UTC — Apr 2)

At this time there is a high load condition on one of our US IMAP servers which is a variety of issues for some US mail customers including:

Delays in the receipt of incoming mail
Slow performance in IMAP mail clients
Slow performance in the Opalstack US webmail system

We’re working to mitigate the problem at this time and will update this post when more information is available. (18:04 UTC — Apr 2)

[Resolved] Opal14 not responding

Sat, 30 Mar 2024 23:04:31 +0000

Fixed - There was a second incident of the same type of problem but on a different site operated by the same customer. We’ve mitigated the problem to prevent it from occurring again and the server is stable at this time. (14:53 UTC — Mar 31)

Watching - Opal14 is back online. The cause of the issue was high Apache RAM usage which resulted from a bad rewrite rule in a customer site. We’ve isolated the site and are working with the customer to correct the problem. We’ll continue to monitor. (23:20 UTC — Mar 30)

Opal14 (SFO shared hosting) is currently not responding. We’re investigating at this time. (23:04 UTC — Mar 30)

[Resolved] Control panel delays for MariaDB items on opal16

Sun, 24 Mar 2024 21:01:08 +0000

Fixed - There have been no further delays. (14:01 UTC — Mar 25)

We are importing a large MariaDB database for a customer on opal16 at this time. The large import is putting heavy load on MariaDB which may result in a delay when creating/deleting new MariaDB databases/users and when changing MariaDB passwords.

We expect the import to be complete by 0300 UTC on Monday 25 March 2024 but hopefully sooner. (21:01 UTC — Mar 24)

[Resolved] Control panel processing delays

Mon, 18 Mar 2024 18:25:06 +0000

Fixed - The backlog of pending items is now clear. (22:04 UTC — Mar 18)

At this time there is a log backlog of pending operations in the control panel. We’re working to clear the backlog and will update this post when the issue is resolved. (18:25 UTC — Mar 18)

[Resolved] DNS issue for opalstack.com causing control panel delays and other issues

Tue, 12 Mar 2024 14:24:00 +0000

Fixed - DNS has propagated on our end and the backlog of pending control panel changes is now clear.

If you have DNS problems like “host not found” when accessing any Opalstack systems then please try flushing your local DNS cache: https://www.wikihow.com/Flush-DNS

For further assistance please email the Opalstack support team. (15:46 UTC — Mar 12)

Watching - We’ve identified the problem as 2 name servers which were failing to return records for opalstack.com hostnames. We’ve removed those name servers from our pool and are monitoring while the DNS change propagates. (15:00 UTC — Mar 12)

At this time there is a delay in processing new items and changes in the control panel at https://my.opalstack.com/. We’re looking into it and will follow up as soon as possible. (14:24 UTC — Mar 12)

[Resolved] Delays in mail forwarding to Google

Sat, 24 Feb 2024 01:31:36 +0000

Fixed - The bcklog of deferred mail is now clear. (11:55 UTC — Feb 24)

Google is once again rate-limiting messages forwarded from our MX servers. We’re working to resolve this and expect the backlog to be cleared over the next several hours.

As a temporary workaround, customers can route an Opalstack mailbox to their forwarding email address and then use that mailbox as a POP3 account in their Gmail settings.

We will update this post when the backlog of forwarded messages is cleared. (01:31 UTC — Feb 24)

[Resolved] Delays in mail forwarding to Google

Mon, 19 Feb 2024 16:44:01 +0000

Fixed - All of the pending forwarded mail has been delivered. (19:23 UTC — Feb 19)

Google had rate-limited 2 of our forwarding MX servers over the weekend resulting in forwarding delays for some users who forward mail to Gmail.

We’ve corrected the cause of the rate limit and expect all pending mail to be delivered over the next several hours. (16:44 UTC — Feb 19)

[Resolved] EU mail system blocked by Hotmail

Thu, 15 Feb 2024 15:21:00 +0000

Fixed - All blocks have been lifted. (03:30 UTC — Feb 16)

Watching - mx2.de.opalstack.com was added to the block list as well, we have submitted the delisting request and are awaiting action from Microsoft. (18:36 UTC — Feb 15)

Part of our EU mail infrastructure was blocked by Hotmail. The affected servers were:

smtp2.de.opalstack.com (outgoing mail)
mx1.de.opalstack.com (incoming mail and forwarding)

We’ve already dealt with the issue and Microsoft have confirmed that the servers have been unblocked, but it may take a few more hours to take full effect.

We’ll update this post when we’ve confirmed that mail is no longer being blocked. (15:21 UTC — Feb 15)

[Resolved] Recent disruptions to Apache service on opal16

Fri, 09 Feb 2024 17:32:55 +0000

Fixed - We’ve seen no further issues and are marking this as resolved. (16:39 UTC — Feb 10)

Watching - We believe we’ve mitigated this and will continue to monitor the server. (00:43 UTC — Feb 10)

We’ve seen some recent disruptions on the Apache httpd service on opal16 which were caused by high traffic to various customer sites.

We’re working to mitigate that and will update this post when it has been resolved. (17:32 UTC — Feb 9)

[Resolved] Change to default SPF and DMARC policies

Wed, 24 Jan 2024 20:03:12 +0000

On 23 January 2024 we began publishing default SPF and DMARC DNS records for domains that use Opalstack’s DNS and mail systems.

The default policies that we published were somewhat restrictive and were causing mail sent from some customer domains to go to spam so we’ve made the following changes to the policies:

The policy for SPF has been changed from hard fail “-all” to soft fail “~all”.
The policy for DMARC has been changed from “quarantine” to “none”.

The net result of these changes is that customer mail is still compliant with SPF and DMARC but with a less-restrictive policy to help legitimate mail arrive in your inbox. (20:03 UTC — Jan 24)

[Resolved] Opal14 is not responding

Sun, 07 Jan 2024 05:10:26 +0000

Fixed - Opal14 is back online following the reboot and is operating normally. (05:19 UTC — Jan 7)

opal14.opalstack.com is not responding. We suspect the cause is high load. We’re rebooting the server at this time and will update this post when the server is back online. (05:10 UTC — Jan 7)

[Resolved] German MX servers are on Hotmail block list

Sat, 06 Jan 2024 22:15:00 +0000

Fixed - The MX servers have been removed from the block list. (18:06 UTC — Jan 7)

mx1.de.opalstack.com and mx2.de.opalstack.com are listed on the Hotmail block list. As a result, mail forwarded to Hotmail domains like hotmail.com and outlook.com is being bounced back to the sender.

We’re working to have the servers removed from the block list and will update this post when that is done.

In the meantime, customers who have forwarding addresses which forward to Hotmail domains are advised to forward to an alternate address or direct mail to an Opalstack mailbox to ensure that they receive their mail. Instructions for updating email addresses are available in our user guide at: Editing Email Addresses (22:15 UTC — Jan 6)

[Resolved] Emergency MariaDB maintenance for opal4, opal7, and opal9

Sun, 03 Dec 2023 07:23:00 +0000

Fixed - The MariaDB recovery on opal9 is complete. Databases with names starting with tro through zzz are being restored from our 2 December 2023 backup.

Our plan now is to test the recovery process on a test server to determine why we were not able to import the dump files and then refine our maintenance procedures based on those findings. (22:21 UTC — Dec 3)

Watching - The MariaDB recovery on opal9 is still in progress. We expect the recovery to be complete in around 90 minutes. (21:21 UTC — Dec 3)

Watching - The MariaDB recovery on opal7 is complete.

The MariaDB recovery on opal9 is still in progress. We’ll update with an ETA as soon as possible. (21:12 UTC — Dec 3)

Watching - The MariaDB recovery on opal7 and opal9 is still in progress. We expect that it will take at least a couple of hours more to complete. (19:39 UTC — Dec 3)

Watching - The MariaDB recovery on opal7 and opal9 is still in progress. We expect that it will take at least a few more hours to complete.

The MariaDB recovery on opal4 is now complete, however we were not able to use the most recent backup for about 50% of the recovered databases. The remaining databases were recovered as follows:

Databases with names starting with lsa through shn were restored from our 2 December 2023 backup.
Databases with names starting with sho through zzz were restored from our 1 December 2023 backup. (16:56 UTC — Dec 3)

Watching - It seems some opal4 databases were not restored. We’re restoring those databases now.

The recovery on opal7 and opal9 is still in progress. (14:55 UTC — Dec 3)

Watching - The MariaDB recovery on opal4 is complete. Recovery on opal7 and opal9 is still in progress. (14:33 UTC — Dec 3)

Watching - The database restore is still in progress. We expect that it will take at least a few more hours. (13:47 UTC — Dec 3)

Watching - What happened?

We took the MariaDB database down to stop the ibdatafile from growing exponentially. The only way to do this is to dump all databases, delete the log files, update the server configuration, and restore the data.

The dump process worked without any errors. The clean up and server configuration completed without any errors. The restore process is where things failed.

Because of errors in 1 or more database(s) transactions failed to complete causing the restore to never get past a certain point. The amount of data in the mysql data directory would increase up to a certain point and then the data size would drop by half before repeating. After running the restore process twice and having errors in different spots we decided to extract each database from the monolithic backup we had taken previously. Why a monolithic backup? It’s usually faster to dump and restore, except in this case.

The extraction process is painfully slow compared to just running a working dump restore. That’s why this process is taking so long.

What didn’t we do?

We could have restored from the latest backup which would have been 24 hours or less old, however, that option comes with the significant risk of data loss. Rather than risk losing 24 hours of data we went with the slower, safer process.

The current restore process for each server is now past the previous failure point.

We’d sincerely apologize for the downtime this has caused for your apps that use MariaDB. In the tests we performed before the actual event we did not run into any of these errors. (12:15 UTC — Dec 3)

Watching - The data restoration process is still ongoing. There is currently no ETA we can provide. (10:29 UTC — Dec 3)

Watching - Opal4, Opal7, Opal9: We are still restoring data to the MariaDB databases. The exact amount of time left in the restoration process is unknown but we will update every hour until all of the restores have finished. (08:42 UTC — Dec 3)

On Sunday, 03 December 2023 at 0500 UTC we’ll be taking the managed MariaDB database service offline for maintenance on the following servers:

opal4.opalstack.com (Dallas)
opal7.opalstack.com (Phoenix)
opal9.opalstack.com (Frankfurt)

The maintenance window is 2 hours. During the maintenance sites and applications which use MariaDB (including WordPress sites) will not function.

We apologize for the short notice and any inconvenience. If you have any questions or concerns regarding the maintenance then please contact our support team. (07:23 UTC — Dec 3)

[Resolved] Emergency reboot of opal3

Sat, 04 Nov 2023 21:12:25 +0000

Fixed - Opal3 is back online. (21:17 UTC — Nov 4)

opal3.opalstack.com is currently being rebooted to correct an issue with the system /tmp directory. Apologies for the unplanned maintenance. (21:12 UTC — Nov 4)

[Resolved] US SMTP listed on SORBS block list

Fri, 27 Oct 2023 21:32:01 +0000

Fixed - The US SMTP server is no longer listed on the SORBS block list. (22:43 UTC — Oct 30)

Our US SMTP server is currently listed on the SORBS email block list. We’ve already gone through the de-listing process and expect the server to be de-listed in 48 hours or less.

We’ll update this post when we confirm the server is de-listed. (21:32 UTC — Oct 27)

[Resolved] US SMTP and community forum are offline

Wed, 18 Oct 2023 01:23:00 +0000

Fixed - The US SMTP server and community forum are back online. (01:35 UTC — Oct 18)

At this time our US SMTP server (outgoing mail) and our community forum are offline. We’re investigating and will update this post when more information is available. (01:23 UTC — Oct 18)

[Resolved] ns3.de.opalstack.com offline, causing control panel delays

Tue, 15 Aug 2023 21:58:00 +0000

At this time our German name server ns3.de.opalstack.com is offline for emergency maintenance.

The other 2 German regional NS are still online and domains that use those NS are working, but since we’re not able to push system updates to ns3.de.opalstack.com there are several pending control panel operations that cannot be completed until the server is back online.

These items will be finalized automatically once the server is back online.

Update: The server is now back online and updates are back to normal. (21:58 UTC — Aug 15)

[Resolved] Opal8 service disruption

Thu, 20 Jul 2023 22:30:00 +0000

opal8.opalstack.com (San Francisco shared hosting) became non-responsive at 3:30 PM US/Pacific time. We suspected a temporary high load condition, but after several minutes the server remained unresponsive. At that point we decided to reboot the server.

The server is online and operating normally at this time. We’ll work to identify the site which caused the high load and will then work with that customer to guard against future incidents. (22:30 UTC — Jul 20)

[Resolved] Partial DNS outage.

Fri, 30 Jun 2023 11:19:36 +0000

Fixed - The DNS issue is fully resolved now.

Note that while this issue affected only the Opalstack US name servers and only for the domain opalstack.com. Since all of our regional services use the opalstack.com domain, effects of this issue may have been seen by customers outside of the US.

If you’re still having trouble connecting to Opalstack services then please try flushing your local DNS cache (13:00 UTC — Jun 30)

During a routine DNS update our Secondary DNS provider captured an incomplete zone update and propagated it. This means that various DNS services for opalstack.com services (DNS, Mail, and Web) are inconsistently available.

We have removed the Secondary provider and we expect this to resolve over the next hour as the DNS TTL caches time out. (11:19 UTC — Jun 30)

[Resolved] Intermittent network connectivity in Washington DC

Wed, 14 Jun 2023 16:01:15 +0000

Fixed - The issue has been resolved. (17:50 UTC — Jun 15)

There have been intermittent network connectivity issues this morning in our Washington DC shared hosting facility, typically presenting as a network dropout lasting 1-2 seconds. The issues could potentially affect the opal1, opal10, and opal12 shared hosting servers.

The data center team believes they’ve identified the issue and are taking corrective action. We’ll update this post when more information is available. (16:01 UTC — Jun 14)

[Resolved] Intermittent VPS connectivity in Frankfurt

Wed, 31 May 2023 16:35:56 +0000

Fixed - There have been no further issues so we consider this to be resolved. (21:59 UTC — Jun 1)

Watching - The fix has been implemented. We’ll continue to monitor. (17:20 UTC — May 31)

The Frankfurt VPS data center is currently experiencing intermittent network connectivity issues. The data center team have identified the issue and are presently working on a solution.

We’ll update this post when more information is available. (16:35 UTC — May 31)

[Resolved] Opal9 offline for scheduled maintenance

Sun, 28 May 2023 04:04:39 +0000

Fixed - The maintenance is complete and opal9 is back online. (04:23 UTC — May 28)

opal9.opalstack.com is offline for scheduled maintenance. (04:04 UTC — May 28)

[Resolved] US SMTP was briefly disrupted

Sun, 07 May 2023 00:37:00 +0000

The US SMTP service was disrupted for approximately 5 minutes during system maintenance. (00:37 UTC — May 7)

[Resolved] US DNS under attack

Tue, 02 May 2023 04:48:43 +0000

Fixed - There have been no further issues since our last update. (13:44 UTC — May 2)

Watching - We’ve mitigated the worst of the attack and all services are online at this time. DNS lookups are still a bit slow so our troubleshooting is ongoing. (05:33 UTC — May 2)

Our US DNS infrastructure under attack, with impacts throughout our hosting services in all regions. We’re working to resolve the issue at this time. (04:48 UTC — May 2)

[Resolved] US webmail performance issues

Tue, 07 Mar 2023 18:20:00 +0000

Fixed - We’ve seen no further performance issues on the US webmail system since implementing the new proxy, so we consider this issue to be resolved. (19:18 UTC — Mar 17)

Watching - We’ve deployed a new IMAP proxy on the US webmail server and the performance seems much better at this time. We’ll continue to monitor. (01:20 UTC — Mar 17)

We’re aware that the US webmail server has been having intermittent timeout issues, particularly during US business hours.

We’re working to resolve this but we expect that it will take at least a couple of weeks. Our recommended workaround in the interim is to use a desktop mail client or to run your own webmail server separately from our managed webmail instance.

Desktop email client configuration info is available in our user guide at: Email Client Configuration

If you’d like to run your own webmail server setup instructions are available at: HOWTO run your own Roundcube webmail server on Opalstack (18:20 UTC — Mar 7)

[Resolved] Opal9 not responding

Wed, 01 Feb 2023 04:27:20 +0000

Fixed - The relocation is complete. (13:19 UTC — Feb 16)

Watching - The servers are now offline and being moved between datacenters. (09:27 UTC — Feb 16)

Watching - The servers are now offline and being moved between datacenters. (09:25 UTC — Feb 16)

Watching - The servers are now offline and being moved between datacenters. (09:21 UTC — Feb 16)

Fixed - OPAL9 unexpectedly lost network connection and was rebooted. After the reboot the boot process hung on a file system corruption. We’ve run the file system check on all file systems, updated the boot loader, system bios, and kernel versions and the server is now back online and functioning normally. (08:55 UTC — Feb 1)

Opal9 is currently not responding, we’re investigating. (04:27 UTC — Feb 1)

[Resolved] German IMAP/POP3 server offline for scheduled relocation

Tue, 24 Jan 2023 08:05:00 +0000

Fixed - The backlog of incoming mail has been delivered and the relocation is complete. (17:01 UTC — Jan 24)

Watching - The DE mail server is back online at this time and the backlog of incoming mail is now being processed. We’re awaiting the final word from the data center that the relocation is complete. (16:03 UTC — Jan 24)

As previously announced, our German IMAP/POP3 server will be offline for relocation to a new data center facility from 0800 to 1600 UTC on 24 January 2023: https://status.opalstack.com/schedules/38 (08:05 UTC — Jan 24)

[Resolved] DE SMTP was blocked by Abusix

Fri, 20 Jan 2023 15:06:02 +0000

Fixed - The DE SMTP is no longer blocked. (20:08 UTC — Jan 20)

Our DE SMTP server was listed in the Abusix Mail Intelligence block list. We’ve already initiated the delisting process and expect the issue to be resolved within the next 60 minutes. (15:06 UTC — Jan 20)

[Resolved] DE SMTP blocked by Proofpoint

Mon, 09 Jan 2023 10:38:08 +0000

Fixed - The DE SMTP server is no longer blocked by Proofpoint. (18:24 UTC — Jan 10)

At this time our German SMTP server is being blocked by the Proofpoint blocklist. We’ve dealt with the cause and have started the de-listing process.

We’ll update this post when the server has been removed from the list. (10:38 UTC — Jan 9)

[Resolved] Ongoing DNS DDOS Attack

Thu, 29 Dec 2022 12:00:00 +0000

Fixed - Post-mortem: this was a DNS flood attack which targeted all 9 of our regional name servers. The attack originated from several cloud provider subnets, including Google and Cloudflare.

To mitigate the attack we had to temporarily block the provider subnets. This had some unintended and undesirable side effects:

The subnets included public DNS resolver infrastructure run by Google and Cloudflare, so blocking them prevented those resolvers from looking up DNS records on Opalstack name servers. As a result, people who use those resolvers were temporarily unable to access opalstack.com (including this status page) and customer domains that use our name servers.
The block also prevented Google from looking up DNS records related to email such as SPF and DKIM. As a result, mail sent (or forwarded) to Gmail from Opalstack’s mail system was rejected by Google while the block was in place.

The block was lifted over 24 hours ago and there have been no further issues since that time.

Going forward, we’ll refine our mitigation techniques to avoid blocking major providers when possible. We’ll also move this status page to an externally-hosted domain to ensure that system status updates will be available regardless of the state of our infrastructure. (16:40 UTC — Dec 30)

Fixed - We’ve seen no further issues in the past several hours and consider this to be resolved. (21:18 UTC — Dec 29)

Watching - The attack has subsided, we’ll continue to monitor. (14:27 UTC — Dec 29)

Over the last few hours we’ve identified and mitigated an ongoing DDOS attack against our DNS infrastructure. This attack has also effected our Singapore and German DNS servers.

We’re continuing to monitor the attack. (12:00 UTC — Dec 29)

[Resolved] Delayed domain updates

Thu, 29 Dec 2022 02:21:36 +0000

Fixed - The backlog of domain updates is now clear. (04:48 UTC — Dec 29)

At this time (Thu Dec 29 02:17:55 UTC 2022) there is a backlog of domain updates which are waiting to be deployed to our name servers. Customers who are affected by this will see domains listed as “Pending” in their Opalstack dashboards.

The actual processing delay is on our Singapore name servers. If your domain is using the US or DE name servers then they’ll still appear as pending but the domains will actually be updated on the NS. Only domains that use the Singapore servers will actually be affected, and it’s only a small number at this time.

The cause of the issue appears to be network-related so we’re working with our upstream providers to resolve this.

We’ll update this post when more information is available. (02:21 UTC — Dec 29)

[Resolved] Unplanned reboot of opal3

Sat, 10 Dec 2022 00:00:00 +0000

On Saturday 10 December at around 0800 UTC (midnight US/Pacific) opal3.opalstack.com in San Francisco became unresponsive. After several unsuccessful attempts to access the server for troubleshooting, we rebooted the server and it was back online around 0900 UTC (1am US/Pacific).

The outage appears to have been triggered by a very rapid spike in system memory usage. The high load unfortunately interrupted the system logging facilities so we weren’t able to precisely identify the cause, but we suspect that it was due to an buggy Apache rewrite rule in a customer application. (00:00 UTC — Dec 10)

[Resolved] MariaDB disrupted on opal11

Fri, 18 Nov 2022 14:14:45 +0000

The MariaDB service on Opal11 (Frankfurt shared hosting) was briefly disrupted due to a service restart. The service is operating normally at this time. (14:14 UTC — Nov 18)

[Resolved] Slow performance on the US IMAP/POP3 service

Thu, 03 Nov 2022 17:57:23 +0000

Fixed - The backup is complete and the server load has returned to normal. (20:11 UTC — Nov 9)

At this time load on our US IMAP/POP3 server is intermittently very high because we’re running a full backup of the entire server.

We expect this to continue until the backup is complete, possible for another day or longer.

We’ll update this post when the backup is complete and load has returned to normal. (17:57 UTC — Nov 3)

[Resolved] opal13 not responding

Thu, 20 Oct 2022 15:41:15 +0000

Fixed - The unexpected reboot was caused by human error. The server has been stable since coming back online. (20:19 UTC — Oct 20)

Investigating - Opal13 is back online. The outage was due to an unexpected reboot. We are now investigating the cause. (15:48 UTC — Oct 20)

Opal13 (shared hosting in Phoenix AZ) is currently not responding. We’re investigating now and will follow up when more info is available. (15:41 UTC — Oct 20)

[Resolved] IPv6 email rejections

Wed, 19 Oct 2022 13:07:00 +0000

A misconfiguration in our IPv6 DNS settings was causing some mail providers (including Gmail) to reject some mail sent or forwarded from Opalstack’s mail system over the past 3-4 days. The problem has been corrected. (13:07 UTC — Oct 19)

[Resolved] Delays in US mail forwarding to @yahoo.com addresses

Fri, 14 Oct 2022 17:54:45 +0000

Fixed - Yahoo are no longer deferring mail forwarded from our US MX servers. (13:05 UTC — Oct 15)

Watching - Yahoo have begun to accept some mail forwarded from our US MX servers. We’ll continue to monitor. (20:34 UTC — Oct 14)

An abusive network was sending a very high volume of messages to a @yahoo.com forwarding address hosted on Opalstack’s US mail system. The high volume of mail has caused Yahoo to temporarily defer messages resulting in delayed delivery.

We’ve blocked the abusive network and expect that Yahoo will begin to accept the deferred messages over the next few hours as their systems register the overall message volume decrease. (17:54 UTC — Oct 14)

[Resolved] opal10 not responding

Thu, 13 Oct 2022 11:30:50 +0000

Fixed - The cause of the unexpected reboot was a power cable fault. The cable has been been replaced and the server is operating normally at this time. (16:01 UTC — Oct 13)

Investigating - Opal10 is back online after an unexpected reboot, we’ll continue to investigate. (11:33 UTC — Oct 13)

Opal10 (Washington DC shared hosting) is currently not responding, we’re investigating at this time. (11:30 UTC — Oct 13)

[Resolved] Apache was down on opal3

Fri, 07 Oct 2022 15:31:09 +0000

An undetected fault on opal3.opalstack.com caused a temporary disruption in the Apache service. The problem has been corrected. (15:31 UTC — Oct 7)

[Resolved] mx2.us.opalstack.com blocked by Microsoft

Thu, 06 Oct 2022 22:44:00 +0000

Fixed - mx2.us.opalstack.com is no longer blocked by Microsoft. (13:29 UTC — Oct 9)

Watching - Microsoft have stated mx2.us.opalstack.com has been de-listed and that it may take 24-48 hours for the change to fully propagate through their mail systems. We will continue to monitor. (22:26 UTC — Oct 7)

At this time Microsoft is blocking mail forwarded to hotmail.com and outlook.com addresses via our mx2.us.opalstack.com server. We believe we’ve identified the cause and are currently waiting on Microsoft to lift the block.

In the meantime, customers who forward mail from their Opalstack-hosted domains to a Hotmail or Outlook address may want to assign a Opalstack mailbox to their email address to assure that they do not miss any incoming messages.

We’ll update this post when we’ve either received a reply from Microsoft or when we can confirm that the block has been lifted. (22:44 UTC — Oct 6)

[Resolved] Slow connections or timeouts on US IMAP/POP3/Webmail

Thu, 06 Oct 2022 15:46:25 +0000

There was a ~30 minute period between 14:50 UTC and 15:20 UTC during which system load on our US IMAP server was high. This may have resulted in slow IMAP performance or connection timeouts, including connections from the US webmail system.

The condition has passed and we’re seeing no further issues at this time. (15:46 UTC — Oct 6)

[Resolved] opal8 not responding

Mon, 19 Sep 2022 21:26:00 +0000

Fixed - It appears that the earlier reboot was initiated by the system watchdog after the server became non-responsive under heavy load.

The server is operating normally at this time. (22:12 UTC — Sep 19)

Investigating - Opal8 is back online. The server rebooted unexpectedly so we’re investigating further. (21:29 UTC — Sep 19)

Opal8 is not responding at this time. We’re looking into it. (21:26 UTC — Sep 19)

[Resolved] Dashboard disk usage stats incorrect

Fri, 16 Sep 2022 15:52:03 +0000

Fixed - A fix has been deployed. Dashboard usage charts will return to normal in 8 hours when the old data has rotated out. (17:39 UTC — Sep 16)

There’s currently a problem with the display of disk usage statistics in the Opalstack dashboard which will cause customers to see reported usage much higher than their actual usage. We’re working on a fix at this time. (15:52 UTC — Sep 16)

[Resolved] Opal11 offline for scheduled relocation

Thu, 15 Sep 2022 07:00:40 +0000

Fixed - The relocation was completed ahead of schedule as of 10:30 UTC. (13:54 UTC — Sep 15)

As previously announced, opal11 has been shut down for physical relocation to a new data center facility. The expected downtime is approximately 8 hours.

We’ll update this post when the relocation is complete. (07:00 UTC — Sep 15)

[Resolved] Singapore VPS were briefly offline

Wed, 14 Sep 2022 18:18:24 +0000

Fixed - The cause of the incident was found to be a faulty power distribution unit. The faulty unit is being replaced and no further disruption of service is expected. (20:24 UTC — Sep 14)

Our Singapore VPS servers were briefly offline a few minutes ago. The network team is investigating and we’ll update this post as soon as more info is available. (18:18 UTC — Sep 14)

[Resolved] DE SMTP was briefly disrupted

Tue, 06 Sep 2022 14:52:06 +0000

The DE SMTP service was briefly disrupted due to a high load condition on the server. The problem is corrected and SMTP is working normally at this time. (14:52 UTC — Sep 6)

[Resolved] Opal10 not responding

Sat, 27 Aug 2022 04:46:09 +0000

Fixed - Opal10 is back online. (04:54 UTC — Aug 28)

Watching - Opal10 became non-responsive again so we’re rebooting it once more. (04:45 UTC — Aug 28)

Fixed - Opal10 is back online. The outage was caused by a runaway user process consuming too much memory. We’ll work with the related customer to resolve the issue. (05:08 UTC — Aug 27)

Investigating - We’re rebooting opal10 at this time. (04:54 UTC — Aug 27)

Opal10 (shared hosting in Washington DC) is currently not responding. We’re investigating. (04:46 UTC — Aug 27)

[Resolved] US SMTP blocked by Cloudmark

Sat, 27 Aug 2022 02:24:00 +0000

Fixed - We’ve confirmed that there are no further blocks by Cloudmark. (18:32 UTC — Aug 30)

Watching - Cloudmark have stated that they have de-listed the server IPs. We’ll continue to monitor until we see no further blocks in the mail logs. (23:19 UTC — Aug 29)

Watching - We’re still working with Cloudmark to have the US SMTP and MX servers delisted. (20:26 UTC — Aug 29)

Watching - We’ve discovered that the US MX servers are also blocked. This could cause mail forwarding to fail if the final recipient domain is using the Cloudmark block list, so we’ve requested delisting of those as well. (14:27 UTC — Aug 27)

At this time our US SMTP is being blocked by the Cloudmark Sender Intelligence block list. We’ve dealt with the cause and have started the de-listing process.

If you receive a bounce message, please do not attempt to delist the server yourself via the URL in the error message. We’re taking care of it.

We’ll update this post when the server has been removed from the list. (02:24 UTC — Aug 27)

[Resolved] Several London VPS are offline

Thu, 25 Aug 2022 22:38:00 +0000

Fixed - The issue is confirmed to be resolved. (04:17 UTC — Aug 26)

Watching - The data center team identified a switching issue and a fix has been implemented. All London VPS are back online at this time.

We’ll continue to monitor awaiting final confirmation from the data center that the issue is fully resolved. (23:40 UTC — Aug 25)

At this time several London VPS are offline due to a connectivity issue in the London data center.

The DC team are looking into the issue. We’ll update this item when more info is available. (22:38 UTC — Aug 25)

[Resolved] Slow logins on US SMTP

Thu, 18 Aug 2022 21:21:00 +0000

Fixed - The US SMTP server has remained stable following the upgrade and improved configuration. (04:43 UTC — Aug 19)

Watching - We’ve implemented a fix and authentication is working normally at this time. We’ll continue to monitor. (01:49 UTC — Aug 19)

Watching - The original US SMTP is back online. We’ll continue to troubleshoot and monitor. (00:36 UTC — Aug 19)

Identified - We’ve taken the original US SMTP offline for an emergency upgrade. (00:25 UTC — Aug 19)

Watching - The second server is online and is slowly picking up its share of the traffic.

We’ll continue to monitor until we’re satisfied that this issue is resolved. (23:21 UTC — Aug 18)

Identified - We believe the issue is higher-than-average mail traffic, so we’re adding a SMTP server to the pool to take up some of the load. (22:43 UTC — Aug 18)

At this time logins to Opalstack’s US SMTP server are running slowly. Customers may see longer message sending times as a result.

We’re investigating the issue and will update this post when more info is available. (21:21 UTC — Aug 18)

[Resolved] US SMTP was blocked by Microsoft

Fri, 12 Aug 2022 01:13:35 +0000

Our US SMTP service was briefly blocked by Microsoft today. We’ve already corrected the issues which caused the block and mail delivery to Microsoft addresses is working normally at this time. (01:13 UTC — Aug 12)

[Resolved] London VPS services offline

Mon, 01 Aug 2022 21:49:38 +0000

Fixed - The London VPS service has been stable since our last update. (14:07 UTC — Aug 3)

Watching - The data center reports that the power issue is resolved. All London VPS services (including the ns2.de.opalstack.com name server) are back online at this time. We will continue to monitor. (23:00 UTC — Aug 2)

Identified - The problem has been identified as a power issue; the data center team is still working to restore service. (22:21 UTC — Aug 2)

Identified - Note that our name server ns2.de.opalstack.com is located in the London data center. If your domain uses our DE name servers then you may possibly see intermittent problems accessing your site until this issue is resolved. (21:51 UTC — Aug 2)

Identified - The London VPS data center has lost connectivity again. The data center is working to restore service. (21:32 UTC — Aug 2)

Fixed - London VPS service is fully restored. (00:19 UTC — Aug 2)

Watching - Connectivity has been restored while the data center team continues to implement the fix. (23:30 UTC — Aug 1)

Identified - The issue has been identified and a fix is being implemented. (22:30 UTC — Aug 1)

Investigating - The London data center has confirmed a loss of connectivity. The investigation is ongoing. (22:09 UTC — Aug 1)

All Opalstack VPS located in London are currently offline. We’re investigating and will update this post when more info is available. (21:49 UTC — Aug 1)

[Resolved] Opal1 not responding

Fri, 08 Jul 2022 21:40:47 +0000

Fixed - Opal1 is back online and stable online following a reboot.

The issue was caused by high RAM usage by the server’s web-based database admin site. (21:58 UTC — Jul 8)

Opal1 (Washington DC shared hosting) is currently not responding to requests.

We suspect the issue is heavy server load, but we’re unable to log in to confirm so to restore service we’re now rebooting the server. (21:40 UTC — Jul 8)

[Resolved] MariaDB down on OPAL2.

Sun, 29 May 2022 04:51:00 +0000

Fixed - There have been no further issues. (18:13 UTC — May 29)

Watching - The MariaDB server is back online and fully functional. We are continuing to monitor the server. (14:32 UTC — May 29)

Identified - The restore is complete and we’re running a final check on the entire DB service. (13:38 UTC — May 29)

Identified - We are still restoring databases to the MariaDB server. (12:07 UTC — May 29)

Identified - The databases have been exported, the InnoDB data and log files have been rebuilt, and we are currently importing all databases. (09:04 UTC — May 29)

Identified - The MariaDB InnoDB data files and logs have become corrupt and our efforts to fix the corruption have not succeeded. In order to fix this we are exporting the databases, rebuilding the InnoDB data and log files, and then importing the databases. (07:01 UTC — May 29)

MariaDB is down on OPAL2. We are investigating the cause and working to restore service. (04:51 UTC — May 29)

[Resolved] Opal4 offline

Fri, 20 May 2022 15:45:00 +0000

Fixed - The issue is resolved and opal4 is back online. (16:31 UTC — May 20)

Opal4 (Dallas shared hosting) is currently offline due to a network issue. We’re working to restore service at this time. (15:45 UTC — May 20)

[Resolved] US SMTP blacklisted

Fri, 13 May 2022 15:56:36 +0000

Fixed - We’ve identified and shut down several websites with insecure contact forms which were sending spam. The site operators have been notified and the US SMTP is no longer blacklisted. (18:59 UTC — May 13)

Our US SMTP service is currently blacklisted. We’re working with the blacklist operators to identify the cause and will have it corrected as soon as possible. (15:56 UTC — May 13)

[Resolved] Opal12 was offline

Sat, 07 May 2022 18:53:00 +0000

Fixed - We’ve confirmed that the maintenance is complete and there are no further issues at this time. (02:35 UTC — May 14)

Watching - Opal12 is back online and we’re awaiting confirmation from the data center that the maintenance is complete. (02:20 UTC — May 14)

Watching - Opal12 is now offline for scheduled maintenance. (01:00 UTC — May 14)

Watching - Opal12 will be going offline for troubleshooting and repair in approximately 50 minutes at 9PM US/Eastern time. (00:12 UTC — May 14)

Watching - We’ve scheduled maintenance downtime on opal12 to correct the issue with the network controller. Details are at https://status.opalstack.com/schedules/36. (14:13 UTC — May 9)

Watching - We suspect that the server’s network controller is failing. We’re now working with the data center team to have the problem corrected. (19:54 UTC — May 7)

Opal12 (Washington DC shared hosting) was offline for approximately 25 minutes beginning at 18:53 UTC due to an interruption in network connectivity.

We rebooted the server approximately 10 minutes later.

The server is back online and we’re working with our data center team to determine the cause of the problem. (18:53 UTC — May 7)

[Resolved] opal5 not responding

Mon, 18 Apr 2022 17:31:00 +0000

Fixed - Opal5 is stable following the reboot. We’ll continue to monitor. (18:44 UTC — Apr 18)

Investigating - Opal5 is back online, we’re still investigating. (18:30 UTC — Apr 18)

Investigating - We’re rebooting opal5 at this time. (18:22 UTC — Apr 18)

Investigating - The high load condition has returned so we’re continuing our investigation. (18:11 UTC — Apr 18)

Fixed - Opal5 is back online. The problem was high load caused by heavy disk I/O activity due to high swap usage by a customer process. We’ve increased available swap and are working with the customer to resolve the issue. (17:51 UTC — Apr 18)

Opal5 (shared hosting in Frankfurt DE) is currently not responding to web and SSH requests. We’re investigating. (17:31 UTC — Apr 18)

[Resolved] opal10 not responding

Sun, 03 Apr 2022 10:23:00 +0000

Watching - Opal10 is back following an unexpected reboot. The cause is under investigation. (10:33 UTC — Apr 3)

Opal10 is not responding, we’re looking into it. (10:23 UTC — Apr 3)

[Resolved] Germany mail server offline.

Fri, 01 Apr 2022 08:30:00 +0000

Fixed - All repairs are complete and all DE mailboxes have been restored.

If you are a DE mail customer and are still seeing issues with your IMAP/POP3 email then please email Opalstack support for further assistance. (13:50 UTC — Apr 9)

Watching - The repairs for message dates and mailbox permissions are 75-80% complete. (20:37 UTC — Apr 7)

Watching - The repairs for dates and mailbox permissions are still in progress. We expect them to be completed today. (13:40 UTC — Apr 7)

Watching - The restore is complete.

We’re still working on lingering issues related to message dates and mailbox permissions which may take another day or two to fully resolve.

We’ll continue to post updates until the issues are resolved. (02:32 UTC — Apr 6)

Watching - The restore is 99% complete. (01:42 UTC — Apr 6)

Watching - The restore is 99% complete.

Once the restore has completed, there may be sporadic service disruptions for individual mailboxes as we identify lingering issues and correct them. (00:48 UTC — Apr 6)

Watching - The restore is 98-99% complete. (20:00 UTC — Apr 5)

Watching - The restore is 93-99% complete. (13:29 UTC — Apr 5)

Watching - The recovery is 85-95% complete. (01:22 UTC — Apr 5)

Watching - The recovery is 80-90% complete. (18:29 UTC — Apr 4)

Watching - The recovery is 75-85% complete. (13:28 UTC — Apr 4)

Watching - The recovery is 70-80% complete. (01:33 UTC — Apr 4)

Watching - The recovery is 65-75% complete. Note that there are several parallel restore processes so the overall recovery progress doesn’t reflect a percentage of the alphabet with regard to mailbox names. (19:41 UTC — Apr 3)

Watching - The recovery is 60-70% complete. (15:55 UTC — Apr 3)

Watching - The recovery is 45-60% complete. (02:22 UTC — Apr 3)

Watching - The recovery is 30-40% complete. We have to estimate with a range because of the parallel restore processes. (17:00 UTC — Apr 2)

Watching - The recovery is 24% complete. (13:30 UTC — Apr 2)

Watching - The recovery from backup is still in progress, albeit running slowly at only 12% complete.

We believe that we’ll be able to recover most of the data from the failed server, so we’re restoring that data in parallel with the backup recovery. This should speed up the overall recovery and provide the most recent data from prior to the failure. (04:19 UTC — Apr 2)

Watching - The recovery is in still progress.

We estimate that it is 11% complete and it is currently restoring mailboxes that being with “as”. (21:47 UTC — Apr 1)

Watching - DE webmail access is now working.

The mailbox restore is still in progress. (16:13 UTC — Apr 1)

Watching - We’re still noticing some login issues for webmail and are troubleshooting that now. (15:57 UTC — Apr 1)

Watching - We do not have high expectations for the outcome of the filesystem check, so we’ve deployed a new DE IMAP server and have re-created all DE mailboxes as new empty mailboxes on that server.

Customers should now be able to log in to our DE mail services to send and receive new mail. No changes to your email client settings should be required.

We’ll begin restoring the original mailbox contents ASAP from the most recent backup made on 30 March 2022 (the failure occurred while the 31 March backup was still in progress). The restore may take several days to complete, so we’ll keep this post updated until it is done.

We’ll attempt to recover mail from 31 March 2022 when the filesystem repairs on the failed server are complete. (15:30 UTC — Apr 1)

At around 08:30 UTC on 1 April 2022 our DE incoming mail server stopped responding, so we initiated an emergency reboot.

The server did not reboot correctly and further analysis identified a failing disk associated with outdated disk controller firmware and a failed disk controller battery.

As of 11:00 UTC the disc controller was repaired and a filesystem check was initiated.

Further updates follow in reverse-chronological order. (08:30 UTC — Apr 1)

[Resolved] MariaDB down on opal7

Fri, 04 Mar 2022 16:35:48 +0000

Fixed - The issue was due to a problem with the server’s /tmp filesystem. The problem has been corrected and all Opal7 services are now working normally. (16:49 UTC — Mar 4)

The MariaDB service on Opal7 is currently down, we’re working to restore service at this time. (16:35 UTC — Mar 4)

[Resolved] MariaDB restarted on Opal8

Wed, 16 Feb 2022 03:00:00 +0000

At 2022-02-16 03:00 UTC, the MariaDB database service on opal8.opalstack.com was restarted due to high CPU utilization. This may have resulted in database connection errors or similar errors on websites that use the MariaDB service for their databases.

The service has remained stable following the restart. (03:00 UTC — Feb 16)

[Resolved] Service disruption caused by firewall issue

Thu, 10 Feb 2022 19:43:00 +0000

About an hour ago we deployed a firewall update that had the unintended side effect of blocking our resolving DNS servers.

This caused various issues on customer sites on shared hosting, usually showing up as a “504 Gateway Timeout” error.

The issue has been resolved and all customer sites should be working now as they were prior to the change. (19:43 UTC — Feb 10)

[Resolved] opal6 offline

Tue, 08 Feb 2022 13:04:00 +0000

Fixed - The maintenance is complete and opal6 is back online. (22:56 UTC — Feb 8)

Watching - Opal6 is now offline for scheduled emergency maintenance. The expected downtime is 30 minutes. (22:40 UTC — Feb 8)

Watching - The data center team suspects that a fault in the server’s physical power switch caused the server to power down unexpectedly.

We’ve scheduled emergency maintenance for 2022-02-08 22:30 UTC (tonight) to investigate and replace the switch if necessary. (17:33 UTC — Feb 8)

Watching - Opal6 is back online. We’re still awaiting word from the data center to determine why the machine had been powered off. (14:09 UTC — Feb 8)

Identified - We discovered that opal6 had been powered down unexpectedly. The machine is rebooting at this time and we’re awaiting an update from the data center as to why the machine was powered down. (13:36 UTC — Feb 8)

Opal6 is currently offline, we’re looking into it. (13:04 UTC — Feb 8)

[Resolved] Several DE VPS offline due to emergency maintenance

Wed, 02 Feb 2022 21:18:38 +0000

Fixed - The emergency maintenance has been completed. (22:43 UTC — Feb 2)

Watching - All DE VPS are back online at this time. We’re awaiting confirmation from the data center that the maintenance is complete. (21:41 UTC — Feb 2)

At this time several customer VPS in Germany are offline due to emergency maintenance in the Frankfurt data center.

The estimated time of completion is 22:15 UTC (approximately 1 hour from the time of this post).

We’ll update this post when more information is available. (21:18 UTC — Feb 2)

[Resolved] SFO VPS service disruption

Wed, 26 Jan 2022 20:00:00 +0000

Fixed - We’ve received confirmation that the incident is resolved. (22:05 UTC — Jan 26)

Watching - All services are back online and the pending items in the dashboard control loop have been completed.

We’re awaiting final confirmation from our provider that the issue is fully resolved. (21:42 UTC — Jan 26)

Watching - The data center team has identified the issue and we’re starting to see services come back online. (21:36 UTC — Jan 26)

Identified - Our provider has notified us that this is a cloud storage connectivity issue in their SFO cloud service.

We’ll post another update when we have more information. (21:26 UTC — Jan 26)

Several San Francisco VPS are currently experiencing connectivity issues, including our community forum, US webmail, and US SMTP service.

A small number of customer web hosting VPS are also affected.

This same issue is causing delays in our dashboard control loop, so customer may see new/updated items in their dashboard stuck in a pending state until this is resolved.

We’re looking into this now and will follow up when more information is available. (20:00 UTC — Jan 26)

[Resolved] DDOS on opal1

Sat, 22 Jan 2022 19:42:50 +0000

Fixed - We’ve seen no further issues on opal1. (16:31 UTC — Jan 23)

Watching - We’ve mitigated the attack and server conditions are back to normal. We’ll continue to monitor.

We’ll notify the targeted site’s operator and will work with them to prevent further issues. (20:11 UTC — Jan 22)

Opal1 (shared hosting in Washington DC) is currently experiencing elevated load due to a distributed denial of service attack.

We’re working to mitigate the attack at this time. (19:42 UTC — Jan 22)

[Resolved] German SMTP blocked by Microsoft

Mon, 03 Jan 2022 21:40:00 +0000

Fixed - It’s been over a week and we’ve seen no blocks of the new server IP by MIcrosoft, so we consider this to be resolved. (15:04 UTC — Jan 17)

Watching - We informed Microsoft of our findings and the corrective action we’ve taken but we’ve still received no further communication from them.

To work around this we’ve moved smtp.de.opalstack.com to a new IP address. Customers do not need to change any settings - the DNS change should propagate automatically within the hour.

After the DNS change has propagated, customers should be able to send mail to email addresses that are hosted by Microsoft.

We’ll leave this incident open and will continue to monitor. (16:28 UTC — Jan 6)

Identified - We’ve still not heard from Microsoft but we believe we’ve identified the activity that triggered the block and are taking action to mitigate it at this time.

We’ll notify Microsoft of the actions we’re taking and continue to hold on for a response. (23:16 UTC — Jan 5)

Investigating - We’re still awaiting a response from Microsoft and have reached out to them as a reminder. (16:32 UTC — Jan 5)

On January 3 2022 at 20:47 UTC Microsoft began blocking mail sent from smtp.de.opalstack.com to domains that use the Microsoft 365 email system.

Microsoft’s own IP reputation and junk mail reporting systems indicate that there are no problems with smtp.de.opalstack.com so we’ve reached out to them for answers.

We’ll update this post as soon as more information is available.

In the interim as a possible workaround customers may create a new mailbox on our US mail server and use that temporarily for their outgoing email authentication. (21:40 UTC — Jan 3)

[Resolved] Opal4 not responding

Tue, 28 Dec 2021 18:50:52 +0000

Fixed - Opal4 is back online. We’ve reached out to our network provider for details regarding the outage. (19:10 UTC — Dec 28)

opal4.opalstack.com (Dallas shared hosting) is not responding. We’re looking into it. (18:50 UTC — Dec 28)

[Resolved] German SMTP blocked by Microsoft

Sat, 18 Dec 2021 19:30:00 +0000

Fixed - We’ve still not received confirmation from Microsoft that the block was lifted, but we’ve seen no further blocking activity since 23:50 UTC on 20 December 2021 so we consider this incident to be closed. (15:42 UTC — Dec 23)

Watching - Microsoft began accepting mail from smtp.de.opalstack.com as of 23:50 UTC on 20 December 2021.

We’re still awaiting official confirmation that the block has been lifted. (15:27 UTC — Dec 21)

On December 18 2021 at 19:30 UTC Microsoft began blocking mail sent from smtp.de.opalstack.com to domains that use the Microsoft 365 email system.

Microsoft’s own IP reputation and junk mail reporting systems indicate that there are no problems with smtp.de.opalstack.com so we’ve reached out to them for answers.

We’ll update this post as soon as more information is available.

In the interim as a possible workaround customers may create a new mailbox on our US mail server and use that temporarily for their outgoing email authentication. (19:30 UTC — Dec 18)

[Resolved] Intermittent connectivity to San Francisco servers including US webmail

Sat, 11 Dec 2021 15:09:34 +0000

Fixed - The maintenance was completed as scheduled and we’re seeing no further issues at this time. (19:58 UTC — Dec 11)

Network maintenance in one of our San Francisco facilities is causing intermittent connectivity issues for some customers on servers in that location.

This also affects our US webmail server which is located in San Francisco.

The maintenance is expected to be complete at 17:30 UTC Dec 11 2021 (9:30 AM US Pacific). (15:09 UTC — Dec 11)

[Resolved] German SMTP server blocked

Mon, 08 Nov 2021 12:00:00 +0000

Fixed - The DE SMTP server is no longer blocked. (15:39 UTC — Nov 10)

Our German SMTP server smtp.de.opalstack.com is currently on a block list.

We’ve identified and corrected the cause of the block, and expect the block to be lifted by November 11 2021. (12:00 UTC — Nov 8)

[Resolved] Mail server certificates expired

Thu, 30 Sep 2021 14:43:09 +0000

Fixed - The issue is now resolved. The root cause was an obsolete root certificate on our servers. The certificate has been updated. (15:10 UTC — Sep 30)

The SSL certificates used by Opalstack’s mail servers are expired. We’re working to correct that at this time and should have it resolved shortly. (14:43 UTC — Sep 30)

[Resolved] Intermittent high load on opal11

Thu, 16 Sep 2021 22:31:00 +0000

Fixed - There have been no further issues on opal11 in the past 24 hours so we consider this to be resolved. (14:00 UTC — Oct 2)

Investigating - We’ve mitigated most of the abusive traffic that’s been driving up high load on opal11 and performance should be much better at this time.

We are still seeing occasional spikes and will continue to investigate. (16:02 UTC — Oct 1)

Investigating - Our investigation into this issue is ongoing. We’ll provide more info when it is available. (15:35 UTC — Sep 30)

Our shared hosting server opal11.opalstack.com in Frankfurt Germany is experiencing an intermittent high load condition which is impacting system performance. We’re working to resolve this and will update this post when the issue is resolved. (22:31 UTC — Sep 16)

[Resolved] opal3 not responding

Fri, 10 Sep 2021 12:02:00 +0000

Fixed - We’ve positively identified the cause of the memory spike and have mitigated it. (13:38 UTC — Sep 10)

Watching - The root cause was a massive spike in system load due to a runaway customer process using most of the available system RAM. We’ll work with the customer to ensure it doesn’t happen again. (13:17 UTC — Sep 10)

Watching - Opal3 is back online at this time. We’ll continue to monitor. (13:11 UTC — Sep 10)

Investigating - We’re rebooting opal3 at this time. (13:00 UTC — Sep 10)

Opal3 is currently not responding. We’re investigating. (12:02 UTC — Sep 10)

[Resolved] Dashboard control loop issue

Sun, 08 Aug 2021 00:57:00 +0000

Fixed - The dashboard and API are back online.

The problem was caused by human error by one of our team while using our admin shell to make a mass update to a single customer’s websites during a migration. This kicked off a massive cascade of events which overloaded the control loop.

This was strictly an admin error - the WebFaction migration tool that many customers are using is safe to use.

We’ve rolled back the updates and have cleared the pending updates from the control loop and all systems are operating normally at this time.

This issue affected Apache/PHP-CGI applications in the dashboard. We believe there should not be any lingering issues but if you notice a problem with any of your Apache/PHP-CGI applications then please try an edit+save cycle on the application and wait a minute for the changes to take effect. If the problem persists then please contact Opalstack Support for further assistance. (02:12 UTC — Aug 8)

Identified - We’ve taken the dashboard offline while we work to resolve this. (01:05 UTC — Aug 8)

At this time there is an issue with our control loop which deploys changes from the dashboard to our servers. Dashboard changes will remain pending until the issue is resolved. We’ll update this post as soon as possible. (00:57 UTC — Aug 8)

[Resolved] Delay in dashboard operations

Sun, 01 Aug 2021 23:12:53 +0000

Fixed - The problem has been corrected and the dashboard operation backlog is cleared. (04:51 UTC — Aug 2)

At this time there is a significant delay in the time that it is taking for changes made in the Opalstack dashboard (ie creating or modifying domains, apps, sites, email, shell users, etc).

We’re investigating the issue and will update this post when more info is available. (23:12 UTC — Aug 1)

[Resolved] opal5 offline

Mon, 12 Jul 2021 09:32:04 +0000

Fixed - The null route has been removed and opal5 is back online. (09:53 UTC — Jul 12)

Identified - The problem seems to be a network null route due to an abuse issue that has been mis-handled by an upstream provider. We’re working on restoring service now. (09:40 UTC — Jul 12)

Opal5 (shared hosting in Frankfurt) is offline. We’re investigating at this time. (09:32 UTC — Jul 12)

[Resolved] Apache was down on Opal8

Fri, 09 Jul 2021 21:08:36 +0000

The system Apache httpd service on opal8.opalstack.com was briefly offline due to a problem with its configured system resource limits.

During the incident customers visiting the Apache-powered sites may have seen a “502 bad gateway” error message.

The problem has been corrected and the service is working normally at this time. (21:08 UTC — Jul 9)

[Resolved] German SMTP offline

Sat, 12 Jun 2021 10:45:00 +0000

Fixed - The certificate has been updated and SMTP is now working normally. We’re reviewing and revising our internal procedures to minimize the chance of this happening again in the future. (13:40 UTC — Jun 12)

German SMTP was offline due to a problem with the server certificate. (10:45 UTC — Jun 12)

[Resolved] Opal7 rebooted unexpectedly

Sat, 12 Jun 2021 02:50:00 +0000

Fixed - The maintenance was confirmed completed on Saturday shortly after our last update. (15:11 UTC — Jun 14)

Watching - Opal7 is back online and operating normally. We’re awaiting confirmation from the data center that the maintenance is complete. (17:04 UTC — Jun 12)

Identified - Opal7 is offline for emergency RAM replacement at this time. (16:51 UTC — Jun 12)

Identified - The cause of the reboot was a faulty memory module. We’re scheduling emergency maintenance at this time. (13:44 UTC — Jun 12)

Opal7 (shared hosting in Phoenix) was unexpectedly rebooted. We’re investigating the cause.

The server is online and operating normally at this time. (02:50 UTC — Jun 12)

[Resolved] Apache was briefly down on opal5

Thu, 10 Jun 2021 02:00:00 +0000

The Apache httpd service on opal5 was briefly down because the system /tmp directory was full.

The service was down for 2-3 minutes. We’ve identified the customer responsible for the problem and will work with them to ensure it does not happen again. (02:00 UTC — Jun 10)

[Resolved] German IMAP/POP3 service offline

Wed, 09 Jun 2021 09:28:00 +0000

Fixed - The Germain mail service is back online. The outage was caused by human error when deploying a new certificate. (09:39 UTC — Jun 9)

The German IMAP/POP3 service is currently offline. We’re currently investigating. (09:28 UTC — Jun 9)

[Resolved] Network outage affecting domains that use Opalstack DE name servers

Tue, 25 May 2021 00:20:00 +0000

Fixed - All DE services are back online. (01:42 UTC — May 26)

Watching - Service has been partially restored. The remaining affected servers are ns2.de.opalstack.com and 5 private customer VPS.

We’ll continue to monitor and update as the service is restored. (01:14 UTC — May 26)

Our DE name servers, SMTP, and several customer VPS located in Frankfurt, Germany are currently offline due to a connectivity issue with one of our regional providers there.

They’re working to restore service. In the interim, we are relocating our DE nameservers to new locations.

Since the DE name servers are affected, this is unfortunately affecting all domains that use Opalstack’s DE name servers. We’re hopeful that service will be restored quickly but if it is not then you may want to switch the NS for your domain to one of our other regional sets of name servers. Our regional NS are listed at: What are your name servers?

We’ll update this post as soon as more information is available. (00:20 UTC — May 25)

[Resolved] opal1 not responding

Mon, 24 May 2021 16:56:18 +0000

Fixed - The problem was caused by a spike in system load due to a problem with a customer site. Conditions have returned to normal and we’ll work with that customer to prevent further issues. (17:13 UTC — May 24)

opal1.opalstack.com (WDC shared hosting) is currently responding to ping but not to SSH and HTTP requests. We’re investigating and will update this post when more info is available. (16:56 UTC — May 24)

[Resolved] Temporary SSL problem in US mail system

Mon, 17 May 2021 06:00:00 +0000

At approximately 06:00 UTC on 17 May 2021 we inadvertently pushed the wrong set of SSL certificates to our US mail system.

Customers who attempted to access the mail system during that period may have seen certificate warning messages and would not have been able to access their mail.

The problem was fixed approximately 30 minutes later and the mail system has been working normally since that time. No incoming or outgoing mail was lost during the incident.

If you are still seeing a certificate error while attempting to use the US mail system then please restart your mail client and/or reboot your computer. If the problem persists then please contact Opalstack support for assistance. (06:00 UTC — May 17)

[Resolved] German SMTP IP range on MS block list

Wed, 12 May 2021 23:02:00 +0000

Fixed - Microsoft is now accepting mail from smtp.de.opalstack.com. (19:44 UTC — May 14)

Identified - We’ve identified and corrected the cause of the block, and have passed our findings along to Microsoft with a request to delist the German SMTP address.

We’ll update this post when Microsoft have confirmed that the server is no longer on the block list. (17:51 UTC — May 13)

We’ve received some reports that outlook.com and other Microsoft mail services are blocking smtp.de.opalstack.com.

We’re investigating now and will follow up when we have more info. (23:02 UTC — May 12)

[Resolved] Opal8 not responding

Mon, 10 May 2021 21:23:34 +0000

Fixed - Repairs are complete and Opal8 is back online. (02:27 UTC — May 11)

Identified - Opal8 is now down for repair. The expected downtime is 30 minutes. (02:04 UTC — May 11)

Watching - The server has remained stable since it came back online after the initial incident.

We’re now waiting on the data center team to bring down the server for repair. (00:45 UTC — May 11)

Identified - Opal8 has a faulty memory module. We’re now scheduling a repair with the data center team. (21:39 UTC — May 10)

Investigating - Opal8 is back online after an unexpected reboot. We’re still investigating. (21:27 UTC — May 10)

Opal8 (shared server in San Francisco) is currently not responding. We’re looking into the issue and will follow up here when we have more info. (21:23 UTC — May 10)

[Resolved] MariaDB down on opal9

Fri, 30 Apr 2021 15:48:02 +0000

Fixed - The MariaDB service on Opal9 is back online. The problem was due to an error in a monitoring check that has been corrected. (15:57 UTC — Apr 30)

The MariaDB (mysql) service on Opal9 is currently offline, we’re working to restore service at this time. (15:48 UTC — Apr 30)

[Resolved] Control panel and API down for scheduled maintenance.

Sat, 17 Apr 2021 04:45:00 +0000

Fixed - The control panel and API are back online. (05:06 UTC — Apr 17)

Watching - We expect the service will be back online in ~15 minutes. (04:48 UTC — Apr 17)

The control panel and API are down for scheduled maintenance. We’ll update this post when the service is back online. (04:45 UTC — Apr 17)

[Resolved] Opal1 not responding

Sat, 17 Apr 2021 04:41:11 +0000

Fixed - Opal1 is back online. The outage appears to have been due to high load due to abnormally heavy memory usage by a customer app. We’re working with the customer to prevent this from happening again. (05:39 UTC — Apr 17)

Investigating - We’re rebooting Opal1 at this time. (04:53 UTC — Apr 17)

Opal1 is not responding to SSH and HTTP requests, we’re looking into it. (04:41 UTC — Apr 17)

[Resolved] Opal7 server relocation

Wed, 14 Apr 2021 07:03:16 +0000

Fixed - The maintenance is now complete and opal7 is online. (13:39 UTC — Apr 14)

Watching - Opal7 is back online. We’ll continue to monitor. (13:09 UTC — Apr 14)

Investigating - Opal7 is offline again, the maintenance may be ongoing. Awaiting confirmation from the data center. (12:54 UTC — Apr 14)

Fixed - The migration is complete and Opal7 is back online. (12:33 UTC — Apr 14)

Watching - Opal7 is offline for the migration at this time. (08:00 UTC — Apr 14)

Our data center team will be moving opal7.opalstack.com to a new facility within the next 6 hours. During the move the server will be offline. (07:03 UTC — Apr 14)

[Resolved] Apache was down on Opal6

Mon, 29 Mar 2021 11:30:00 +0000

The Apache service on Opal6 was offline for approximately 2 hours after the server ran out of available semaphores. The problem has been corrected as of around 13:30 UTC (~1 hour ago) and we’ll be adjusting our monitoring system to detect and correct this condition when it occurs. (11:30 UTC — Mar 29)

[Resolved] IMAP1US Offline

Wed, 24 Mar 2021 09:50:00 +0000

Fixed - The US IMAP server has been back online since 2021-03-24 10:30 UTC. (15:00 UTC — Mar 24)

The IMAP1US server is currently down and we are working to restore access. (09:50 UTC — Mar 24)

[Resolved] DE SMTP on Proofpoint blacklist

Fri, 12 Mar 2021 21:19:00 +0000

Fixed - The DE SMTP IP is no longer on the Proofpoint blacklist. (21:35 UTC — Mar 15)

Watching - We’re still waiting on Proofpoint to delist the DE SMTP IP address. (17:17 UTC — Mar 15)

One of the Opalstack DE SMTP IP addresses is currently on the Proofpoint blacklist. We’re working to have the IP de-listed and expect this to be complete within the next 24-48 hours. (21:19 UTC — Mar 12)

[Resolved] Opal1 service disruption

Thu, 18 Feb 2021 11:00:00 +0000

At around 2021-02-18 11:00 UTC today the /tmp partition on opal1.opalstack.com ran out of available space. This disrupted service for applications that needed to write to temporary storage including PHP apps and the MariaDB database service.

At the time of the event our monitoring system was only checking the status of these system services and not their operational performance, so no alert was generated.

We corrected the problem at approximately 14:00 UTC and the server has been stable since that time.

We’re updating our monitoring system to perform the tests necessary to alert us to the conditions that caused this incident which will help us avoid this sort of extended disruption. (11:00 UTC — Feb 18)

[Resolved] DE IMAP service down

Fri, 12 Feb 2021 14:23:21 +0000

Fixed - The DE IMAP/POP3 service has been stable since our last update. (16:25 UTC — Feb 12)

Watching - The DE IMAP/POP3 service is back online with higher process and open files limits. We’ll continue to monitor. (14:55 UTC — Feb 12)

Watching - We’re rebooting time DE IMAP/POP3 server to pick up a configuration change. (14:46 UTC — Feb 12)

Watching - The DE IMAP/POP3 service is back online. We’ll continue to monitor as we look into the cause of the outage. (14:34 UTC — Feb 12)

The IMAP/POP3 service on our DE mail system is currently down. We’re working to restore service. (14:23 UTC — Feb 12)

[Resolved] Incoming mail problems in Germany

Tue, 09 Feb 2021 14:18:31 +0000

Fixed - The disk array rebuild is complete and the server has been stable since our last update. (15:50 UTC — Feb 10)

Watching - The DE mail service is back online at this time. The main issue was heavy load caused by the very high volume of deferred mail the resulted from the mail loop. This was exacerbated by the degraded disk array.

The networking issue was unrelated but has been resolved.

The disk array is still rebuilding but the server is operating normally. We’ll continue to monitor until the rebuild is complete. (18:54 UTC — Feb 9)

Identified - There seems to be multiple issues related to the DE IMAP server:

A forwarding loop was causing delivery delays for the past few hours.
A degraded disk array was impacting system performance
A failed network switch is currently blocking network connectivity

Our data center team is working on restoring connectivity at this time. (15:50 UTC — Feb 9)

Investigating - The DE IMAP server is now offline for a filesystem check. (15:16 UTC — Feb 9)

We’re receiving reports of slow mail delivery in our German mail system. We’re looking into it at this time. (14:18 UTC — Feb 9)

[Resolved] Intermittent errors on webmail

Tue, 02 Feb 2021 17:38:00 +0000

Fixed - Webmail has been stable since our last update. (20:59 UTC — Feb 3)

Watching - We’ve adjusted the webmail server configuration to accommodate higher traffic and the server is now stable. We’ll continue to monitor. (18:55 UTC — Feb 2)

At this time we’re seeing intermittent database errors and 50x responses on https://webmail.opalstacked.com/ - we’re investigating and will update this post when more info is available. (17:38 UTC — Feb 2)

[Resolved] US IMAP scheduled maintenance

Mon, 01 Feb 2021 03:02:00 +0000

Fixed - All mailboxes have been synced. (00:47 UTC — Feb 2)

Fixed - The US IMAP service is now up and running on the new hardware.

Mail received in the past 48-72 hours may not be immediately visible on the new hardware but will appear automatically as we perform a final sync over the next several hours.

We’ll post a final update here when the sync is complete. (05:30 UTC — Feb 1)

The US IMAP service is currently undergoing scheduled maintenance.

IMAP connectivity may be sporadic or unavailable for the next ~3 hours.

We’ll update this post when the maintenance is complete. (03:02 UTC — Feb 1)

[Resolved] US IMAP service not responding

Fri, 29 Jan 2021 20:30:23 +0000

Fixed - The US IMAP server is back online. (21:49 UTC — Jan 29)

Investigating - We’re rebooting the US IMAP server at this time. (20:38 UTC — Jan 29)

The US IMAP service is currently not responding, we’re looking into it. (20:30 UTC — Jan 29)

[Resolved] EU VPS and NS connectivity problems

Mon, 25 Jan 2021 18:56:00 +0000

Fixed - The upstream issue has been resolved. (22:05 UTC — Jan 25)

Watching - All EU services are back online. We’ll continue to monitor until resolution of the upstream issue is confirmed. (20:03 UTC — Jan 25)

Watching - At this time connectivity to most of our EU services has been restored. ns3.de.opalstack.com and one customer VPS remain affected. (19:29 UTC — Jan 25)

Identified - This issue is also affecting all of our DE name servers. We’re working to restore service ASAP but if you are using our DE NS and need an immediate workaround please change your NS over to our US name servers. (19:11 UTC — Jan 25)

Identified - We’ve confirmed that there is an outage in the Linode data center that hosts our EU VPS: https://status.linode.com/incidents/dm9mc14kv2t4

We’ll update this post when more info is available. (19:03 UTC — Jan 25)

There seems to be a loss of connectivity in our EU VPS datacenter. We’re investigating at this time. (18:56 UTC — Jan 25)

[Resolved] US SMTP on Cloudmark blacklist

Sat, 23 Jan 2021 18:43:28 +0000

Fixed - Our US SMTP is no longer on the Cloudmark blacklist. (16:06 UTC — Jan 25)

At this time Opalstack’s US SMTP is on the Cloudmark blacklist. We’re working to have the server delisted and expect that to be done within the next 24-48 hours. (18:43 UTC — Jan 23)

[Resolved] EU VPS network connectivity problems

Thu, 21 Jan 2021 15:23:34 +0000

Fixed - The network has been stable since our last update so we’re considering this to be resolved. (14:27 UTC — Jan 22)

Watching - EU VPS network connectivity is now stable. We’ll continue to monitor. (01:02 UTC — Jan 22)

Identified - EU VPS customers may see further connectivity issues while our upstream provider is performing emergency maintenance. (00:32 UTC — Jan 22)

Fixed - The maintenance is complete and connectivity to EU VPS remains stable. (16:12 UTC — Jan 21)

Watching - Connectivity to EU VPS is stable at this time. The maintenance is still in progress so we’ll continue to monitor. (15:56 UTC — Jan 21)

Connectivity to some EU VPS may be degraded due to active ongoing maintenance by our upstream provider. We’ll update this post as soon as the maintenance is complete. (15:23 UTC — Jan 21)

[Resolved] US IMAP service performance degraded

Tue, 19 Jan 2021 20:35:00 +0000

Fixed - The US IMAP server is back online and is operating normally. We’ll be upgrading the server in February to avoid future problems. (21:22 UTC — Jan 19)

Watching - We’ve rebooted the US IMAP server and are monitoring. (20:57 UTC — Jan 19)

The IMAP service on our US mail system is currently running very slowly. We’re working to correct that at this time.

This is also causing timeouts and errors for users accessing Opalstack webmail, including users of our German mail system. (20:35 UTC — Jan 19)

[Resolved] Opal3 not responding

Sun, 17 Jan 2021 15:39:44 +0000

Fixed - The problem was caused by extremely high RAM usage from a process run by a hosting customer. We’ve notified the customer of the problem and have taken steps to prevent it from happening again. (16:59 UTC — Jan 17)

Fixed - Opal3 is stable after the reboot. We’ll update this post when we’ve determined the cause of the outage. (16:25 UTC — Jan 17)

Investigating - Opal3 is back online after a reboot. The cause of the outage is still under investigation. (15:59 UTC — Jan 17)

Investigating - We’re rebooting Opal3 at this time. (15:52 UTC — Jan 17)

opal3.opalstack.com is currently not responding. We’ll follow up here when more info is available. (15:39 UTC — Jan 17)

[Resolved] rebooting US IMAP server

Mon, 11 Jan 2021 18:04:34 +0000

Fixed - The US IMAP server is back online. (19:02 UTC — Jan 11)

Investigating - The US IMAP server is not coming back from the reboot. We’re working with our provider to restore service ASAP. (18:27 UTC — Jan 11)

We’re doing an emergency reboot of the US IMAP server. Service should be restored in a moment. (18:04 UTC — Jan 11)

[Resolved] opal5 was offline

Thu, 07 Jan 2021 02:03:54 +0000

Opal5 was temporarily offline after running out of disk space. The server is online at this time and we’re working to make more storage available.

Additionally at the time of this incident our status page (this site) was failing save our updates. We’re working to correct that. (02:03 UTC — Jan 7)

[Resolved] US SMTP blacklisted

Tue, 05 Jan 2021 15:42:00 +0000

Fixed - The US SMTP is no longer blacklisted. (21:59 UTC — Jan 5)

smtp.us.opalstack.com is currently on the Cloudmark blacklist which is used by Comcast and other services. We’ve submitted a delisting request to Cloudmark and are waiting for a followup from them. (15:42 UTC — Jan 5)

[Resolved] Opal4 not responding

Mon, 28 Dec 2020 18:46:39 +0000

Fixed - Opal4 is back online and operating normally. (01:43 UTC — Dec 29)

Investigating - Opal4 is not responding at this time. (01:31 UTC — Dec 29)

Fixed - Opal4 is back online and is operating normally. We’ve confirmed another network attack was the cause. (19:07 UTC — Dec 28)

Opal4 is currently not responding to ping. We suspect this is due to another attack. (18:46 UTC — Dec 28)

[Resolved] Connectivity issues in Singapore

Sun, 20 Dec 2020 11:00:00 +0000

Fixed - Linode reports that the connectivity issues in Singapore have been resolved. (18:21 UTC — Dec 21)

Watching - All Singapore servers are back online at this time. We’ll continue to monitor. (17:31 UTC — Dec 21)

Watching - Linode are investigating a connectivity issue in our Singapore data center. During this time, users may experience connection timeouts and errors for all services deployed in this data center. We will share additional updates as we have more information: https://status.linode.com/incidents/wfghw8md5j8k (16:52 UTC — Dec 21)

Fixed - Linode will be performing an emergency network maintenance to upgrade an aggregate switch pair in the AP-South (Singapore) data center on Monday, December 21, 2020 from 21:00 UTC until 23:00 UTC. While we do not expect any downtime during this maintenance there may be a period of brief packet loss or latency. (source) (04:02 UTC — Dec 21)

Singapore VPS and NS may have experienced connectivity problems due to network connectivity issues in Linode’s Singapore data center: https://status.linode.com/incidents/zv7xb4plgzfm

Linode reports that the issue is now resolved. We’ve reached out to them to find out the cause of the recurring network problems in Singapore and the corrective action that they are taking to minimize future disruptions of service. (11:00 UTC — Dec 20)

[Resolved] Network outage in Singapore

Fri, 18 Dec 2020 17:30:00 +0000

Fixed - Linode reports the incident is resolved as of Dec 18, 2020 20:01 UTC. (21:17 UTC — Dec 18)

Watching - All Singapore services are back online, we’ll continue to monitor while Linode wraps this up. (18:24 UTC — Dec 18)

Watching - Linode has identified the cause of the outage and is implementing a fix, we’re seeing the Singapore servers coming back online at this time. (18:11 UTC — Dec 18)

Identified - If your domain is using our Singapore name servers then as an interim fix please switch your domain’s NS to one of our other regions: https://help.opalstack.com/article/48/what-are-your-name-servers (17:33 UTC — Dec 18)

An outage in the Linode Singapore data center is affecting connectivity to some Singapore NS and VPS: https://status.linode.com/incidents/k740qss44w58

We’ll update this post when we have more info. (17:30 UTC — Dec 18)

[Resolved] Singapore servers were offline

Wed, 16 Dec 2020 11:36:00 +0000

At 11:36 UTC on Dec 16, 2020 Linode reported a network connectivity issue in their Singapore datacenter: https://status.linode.com/incidents/bt9z0b25417j

Customer servers and NS in Singapore were affected by this issue and may have been inaccessible at that time. The NS issue may have also affected any domain that uses our Singapore NS.

Service was restored after approximately 2 hours and has been up since that time. (11:36 UTC — Dec 16)

[Resolved] Slow IMAP performance

Mon, 14 Dec 2020 21:14:00 +0000

Identified - The US IMAP service is currently under heavy load while importing mail for new customers. (02:15 UTC — Dec 23)

Fixed - All pending mail imports have been completed and the IMAP servers are operating normally. (22:17 UTC — Dec 19)

At this time we’re importing hundreds of gigabytes of mail for new customers migrating from WebFaction. This is causing heavy load on our IMAP servers so you may see slow performance in our webmail system, IMAP clients, and POP3 downloads over the next few days.

We’ll update this post when the imports are complete. (21:14 UTC — Dec 14)

[Resolved] Recurring attacks on opal4

Fri, 11 Dec 2020 19:17:00 +0000

Watching - Opal4 is stable at this time. (18:50 UTC — Dec 16)

Investigating - Opal4 is not responding, likely another attack. We’re investigating. (18:31 UTC — Dec 16)

Fixed - Opal4 is back online. The disruption was caused by another network attack. The replacement server will be set up soon and we’ll start scheduling the transition. (04:40 UTC — Dec 16)

Investigating - Opal4 is currently not responding to ping, investigating. (04:32 UTC — Dec 16)

Fixed - Opal4 is stable at this time. (21:30 UTC — Dec 11)

Watching - Opal4 is back online.

The earlier assessment of an external network problem was incorrect, this appears to have been another network attack on Opal4.

The attacks that have taking opal4 down haven’t seemed to be particularly massive - similar attacks happen on other shared hosts in our infrastructure and those machines are fine. Because of this we’re planning to replace opal4 in the coming weeks. We’ll contact all affected customers individually with the details as we finalize the plan. (21:08 UTC — Dec 11)

Identified - Opal4 is offline due to a network problem in the Leaseweb Dallas TX data center. We’re awaiting further updates from them at this time. (19:43 UTC — Dec 11)

opal4.opalstack.com in Dallas TX USA is currently not responding. We’re looking into it. (19:17 UTC — Dec 11)

[Resolved] Dashboard was not pushing changes

Wed, 09 Dec 2020 17:30:00 +0000

Four hours ago at approximately 17:30 UTC today on 09 Dec 2020 a bug was introduced into our admin dashboard that blocked the ability for new changes to be pushed out to servers.

This affected all dashboard operations: new apps, new sites, email, domains, etc were held back due to the bug.

It’s been corrected and most items should be caught up within the next 30 minutes. Let’s Encrypt certificates may take a while longer but will eventually go through. (17:30 UTC — Dec 9)

[Resolved] Apache down on opal6

Mon, 16 Nov 2020 11:30:00 +0000

Fixed - The Opal6 Apache httpd and PHP-FPM services were hitting an open files limit. We’ve raised the limit and the services are back online. (15:46 UTC — Nov 16)

The Apache httpd service on opal6 is currently down, we’re looking into it. (11:30 UTC — Nov 16)

[Resolved] Forwarding from US MX servers blocked by AT&T

Fri, 13 Nov 2020 19:01:00 +0000

Fixed - AT&T are no longer blocking mail forwarded from our US MX servers. (16:29 UTC — Nov 14)

Watching - AT&T says that our US MX servers will be unblocked in 24-48 hours. (22:01 UTC — Nov 13)

At this time AT&T is blocking mail forwarded from our US MX servers. We’re working with AT&T to resolve the problem. (19:01 UTC — Nov 13)

[Resolved] opal4 not responding

Wed, 11 Nov 2020 19:24:00 +0000

Fixed - Opal4 is operating normally. (19:44 UTC — Nov 28)

Investigating - Opal4 is currently under high load and not responding to most requests, we’re looking into it. (19:36 UTC — Nov 28)

Watching - Opal4 just experienced another load spike due to a network attack. We’ll continue to monitor. (04:49 UTC — Nov 15)

Fixed - Opal4 has been stable since our last update. (18:58 UTC — Nov 13)

Watching - The system was once again under heavy load due to a network attack. We’ll continue to monitor. (20:06 UTC — Nov 11)

Investigating - Opal4 is responding to ping but the system is under heavy load, we will continue to investigate. (20:00 UTC — Nov 11)

Fixed - Opal4 is back online. The cause was a brief network attack. (19:42 UTC — Nov 11)

Opal4 in Dallas TX USA is currently not responding to ping, we’re looking into it. (19:24 UTC — Nov 11)

[Resolved] Newark VPS hosting is offline

Tue, 10 Nov 2020 21:31:00 +0000

Fixed - Newark VPS services are back online. (22:15 UTC — Nov 10)

Identified - This is an upstream outage in our provider’s network. They are working to restore service at this time. (21:35 UTC — Nov 10)

Opalstack VPS in Newark NJ USA are currently offline. We’re looking into it. (21:31 UTC — Nov 10)

[Resolved] Apache httpd problems on opal3

Thu, 05 Nov 2020 01:31:00 +0000

Fixed - Calling this fixed! (21:40 UTC — Nov 6)

Watching - Apache httpd on Opal3 has been stable since our last update. We’ll continue to monitor. (04:32 UTC — Nov 5)

The Apache httpd service on Opal3 is bumping into some resource limits and some customers are reporting errors on their sites. We’ve made made some adjustments and are monitoring. (01:31 UTC — Nov 5)

[Resolved] US SMTP blocked by outlook.com

Sun, 25 Oct 2020 22:50:00 +0000

Fixed - Microsoft is no longer blocking our US SMTP. (06:08 UTC — Nov 2)

Watching - Microsoft has responded saying that our US SMTP will be removed from the blacklist within the next 24-48 hours. (04:13 UTC — Oct 31)

Identified - Microsoft have responded to our inquiry. It appears that the recent influx of WebFaction customers 🎉 has caused a sudden increase our outgoing mail volume and that this is what triggered the block.

We’ve verified via Microsoft’s own postmaster tools that there have been zero complaints regarding mail from our US SMTP and have provided this data to Microsoft for consideration.

We hope to follow up with a resolution to this issue as soon as possible. Thank you for your patience. (23:49 UTC — Oct 28)

Our US SMTP server smtp.us.opalstack.com is currently on a Microsoft block list used by the following domains and their non-US equivalents:

hotmail.com
live.com
msn.com
outlook.com

We’re working to get the server delisted at this time. (22:50 UTC — Oct 25)

[Resolved] Let's Encrypt rate limit for opalstacked.com subdomains

Wed, 21 Oct 2020 17:40:00 +0000

Fixed - We’ve made some changes to the way Let’s Encrypt works with the shared opalstacked.com domain and as a result the domain is no longer rate-limited. We’ll be posting updated documentation with the details later this week. (19:34 UTC — Nov 3)

Watching - Looks like we’ve hit our new limit, we’ll get this corrected ASAP. (20:56 UTC — Oct 27)

Fixed - We’ve confirmed that our increased rate limits are in effect, Calling this fixed, thanks for your patience :-) (16:36 UTC — Oct 25)

Watching - The LE service has increased our limit and at this time we are not seeing any rate-limiting for new LE certificates and renewals on any domains. We’ll continue to monitor. (16:03 UTC — Oct 24)

At this time the Let’s Encrypt service is rate-limiting certificate issue and renewal requests for the entire *.opalstacked.com domain. This is the domain that we use to provide free subdomains to our hosting customers.

We’re working with the LE team to get a higher limit. In the meantime if you are attempting to issue a LE cert for a site that uses your opalstacked.com subdomain then it may fail. If this happens you’ll receive a failure notice via email.

Until the rate limit is raised then the alternative is to use one of your own registered domains on your site instead of your opalstacked.com subdomain. If you’re not ready to point your domain at Opalstack, then you can create a testing subdomain and A record via your NS provider to point that subdomain at your Opalstack server.

We’ll update this issue as soon as the rate limit has been raised. (17:40 UTC — Oct 21)

[Resolved] Opal4 offline

Mon, 19 Oct 2020 20:40:41 +0000

Fixed - The issue was caused by a spike in system load due to temporary high network usage. The server is now stable. (20:44 UTC — Oct 19)

Opal4 (Dallas shared hosting) is currently not responding to ping. We’re looking into it at this time. (20:40 UTC — Oct 19)

[Resolved] Intermittent PHP-FPM failures on Opal4 and Opal5

Sun, 18 Oct 2020 15:34:00 +0000

Fixed - We’ve rebuilt most of our PHP stack and this issue is now resolved. Updated documentation and maybe a blog post are on the way. (02:54 UTC — Oct 26)

Watching - Over the last few weeks we’ve dealt with various PHP-FPM related problems. After trying various combinations of fixes to the server, apache, and PHP-FPM itself we’ve developed a new, more stable PHP-CGI stack to replace PHP-FPM. With PHP and WordPress powering large parts of the Internet (and businesses) having a stable, reliable stack for them is critical.

We rolled the new stack out to all servers yesterday afternoon and found some edge cases we hadn’t found in our previous testing.

We’ll be rolling those changes out again tonight. While we do there may be a brief downtime of less than 10 minutes across PHP based sites and the update happens. Once the sites are updated we’ll begin monitoring for any breakages that didn’t show up in our testing and fix them ASAP.

We’ll follow up with a blog post on exactly what we’ve changed and how things are working in the future. (04:21 UTC — Oct 23)

Watching - Our troubleshooting is going but we’ve improved the stability of PHP-FPM quite a bit over the past few days and are seeing no customer facing problems at this time. We’ll leave this incident open until we’re satisfied that we’ve got this issue nailed down. (17:32 UTC — Oct 21)

Watching - There have been intermittent PHP-FPM outages on Opal5 in the past several hours. Troubleshooting is ongoing. (20:45 UTC — Oct 19)

Watching - PHP-FPM for PHP 7.3 on Opal5 was briefly disrupted about 15 minutes ago. We’ll continue to troubleshoot and monitor. (23:24 UTC — Oct 18)

Watching - We’ve just deployed a potential fix and will continue to monitor. (19:01 UTC — Oct 18)

We’ve been investigating intermittent failures of PHP sites hosted on Opal4 (Dallas) and Opal5 (Frankfurt) which result in “503 Service Unavailable” responses on websites. We’ll post more details here as they become available. (15:34 UTC — Oct 18)

[Resolved] Opal4 (Dallas) offline

Sat, 17 Oct 2020 05:48:00 +0000

Fixed - The data center has reported that the issue is resolved. (08:38 UTC — Oct 17)

Watching - Opal4 is back online. We will continue to monitor. (07:37 UTC — Oct 17)

Identified - The data center reports “an outage related to a networking maintenance that was intended to be non-impacting in our DAL-12 datacenter”.

We’re awaiting further updates at this time. (06:52 UTC — Oct 17)

Investigating - The outage appears to be upstream in our data center. We’ve contacted them and are awaiting their response. (06:02 UTC — Oct 17)

opal4.opalstack.com located in Dallas TX is currently not responding to ping. We’re looking into it and will follow up on this post with more information as it becomes available. (05:48 UTC — Oct 17)

[Resolved] Service disruption in San Francisco

Sun, 16 Aug 2020 13:00:00 +0000

Fixed - All SFO servers are now back online. (16:17 UTC — Aug 16)

Identified - vps15 and mx2.us are back online. (15:54 UTC — Aug 16)

Identified - The webmail server is back online and is operational with the exception of sending mail. We’re still waiting on the remaining servers to come back online. (15:26 UTC — Aug 16)

Identified - The data center team reports:

The issue has been identified as a utility power issue. Power is transferred to generators and is currently running stable and network connectivity is restored for the vast majority of the services.

We’re starting to see the remaining machines come online at this time. (15:08 UTC — Aug 16)

Investigating - opal3.opalstack.com is back online at this time. Other servers are still offline and we’re awaiting a report on the initial investigation from the data center team. (14:19 UTC — Aug 16)

Services in our San Francisco data center are currently disrupted. The following servers are affected:

community.opalstack.com
mx2.us.opalstack.com
opal3.opalstack.com
smtp2.us.opalstack.com
vps4.opalstack.com
vps5.opalstack.com
vps6.opalstack.com
vps13.opalstack.com
vps15.opalstack.com
vps16.opalstack.com
vps17.opalstack.com
vps19.opalstack.com
vps21.opalstack.com
vps24.opalstack.com
vps28.opalstack.com
webmail.us.opalstack.com

We’ll update this post as soon as more information is available. (13:00 UTC — Aug 16)

[Resolved] intermittent connectivity problems on opal4

Thu, 23 Jul 2020 19:30:00 +0000

Fixed - Opal4 is stable. (20:29 UTC — Jul 23)

Watching - Opal4 is back online. The data center team believed that the connectivity issue was caused by a configuration problem which they corrected

We will continue to monitor. (19:50 UTC — Jul 23)

Investigating - Opal4 is being rebooted at this time. (19:40 UTC — Jul 23)

Opal4 is experiencing intermittent connectivity problems following this morning’s scheduled maintenance. The data center team is investigating. (19:30 UTC — Jul 23)

[Resolved] Opal4 network was disrupted

Mon, 13 Jul 2020 20:11:00 +0000

Network connectivity to opal4 was disrupted for 1 minute due to a short network attack.

The server is online and operating normally at this time. (20:11 UTC — Jul 13)

[Resolved] Opal4 under high load

Tue, 07 Jul 2020 22:54:08 +0000

Fixed - Opal4 is back online following a reboot. The high load was caused by a network attack. We’ve identified the site targeted by the attack and are working with the site owner to isolate the site from other customers on opal4. (23:13 UTC — Jul 7)

Opal4 is presently under high load and is mostly unresponsive. We’re rebooting it at this time. (22:54 UTC — Jul 7)

[Resolved] Cron mail not working

Sun, 14 Jun 2020 22:24:11 +0000

Fixed - Mail from cron jobs should be working normally now. (20:58 UTC — Jun 16)

At the present time mail is not being sent by cron jobs on our web servers. We’re looking into the cause. (22:24 UTC — Jun 14)

[Resolved] Opal4 not responding

Thu, 30 Apr 2020 21:48:36 +0000

Fixed - We believe this latest issue was caused by a DNS reflection attack.

We’re consulting with our upstream provider to see what they can do to protect the server and network from future attacks. (22:58 UTC — Apr 30)

Investigating - Opal4 is back online, investigation is ongoing. (21:56 UTC — Apr 30)

Opal4 is not responding. We’re looking into the cause. (21:48 UTC — Apr 30)

[Resolved] Apache was not responding on Opal4

Thu, 30 Apr 2020 15:10:00 +0000

Apache was once again not responding on Opal4 after hitting its configured ServerLimit. The underlying cause appears to have been a high number of PHP-FPM processes being re-spawned for one customer site. We’ve adjusted the PHP-FPM configuration for that site to avoid the problem in the future. (15:10 UTC — Apr 30)

[Resolved] Apache was not responding on Opal4

Fri, 24 Apr 2020 18:23:00 +0000

Fixed - The root cause appears to have been a misconfigured customer application. We’ve corrected the configuration and have updated our monitoring system to alert us when Apache approaches its configured limits. (17:22 UTC — Apr 28)

The Apache service on Opal4 was not responding for approximately 1 hour today beginning 18:23 UTC.

The problem was that the Apache service had hit its configured ServerLimit which meant that no further Apache processes could spawn to handle requests. That has been corrected and the service is online now.

We’re now investigating the root cause and also looking into why our own monitoring systems did not alert us to this condition. (18:23 UTC — Apr 24)

[Resolved] Connectivity issues in Frankfurt

Thu, 16 Apr 2020 19:00:00 +0000

Fixed - The data center team has reported that the issue has been resolved. (20:37 UTC — Apr 16)

Several servers in our Frankfurt DE datacenter have experienced network connectivity issues.

The data center team reports that the problem is due to high temperatures in the DC. Their investigation into the root cause is ongoing.

At this time all servers are online. We’ll update this post when more information is available. (19:00 UTC — Apr 16)

[Resolved] Opal4 load issues

Fri, 10 Apr 2020 23:40:00 +0000

Fixed - We’ve seen no further load issues since our last update. (20:00 UTC — Apr 24)

Watching - Opal4 has experienced two load spikes in the past 24 hours.

During the most recent spike we were able to positively identify one of the sites causing the problem and have taken steps to prevent it from happening again.

We’ll continue to monitor. (15:20 UTC — Apr 14)

Opal4 experienced a brief spike in system load between 23:40 and 23:50 UTC on Friday 10 April 2020.

The cause appears to have been high CPU usage due to a SYN flood attack.

The server is operating normally at this time. We’ll continue to monitor. (23:40 UTC — Apr 10)

[Resolved] Opal4 not responding

Wed, 26 Feb 2020 22:55:25 +0000

Fixed - We’ve successfully mitigated the attack and Opal4 has been stable since our last update. (16:02 UTC — Feb 28)

Watching - We’ve identified an ongoing attack pattern against sites on opal4 and have taken steps to mitigate it platform-wide. We’ll continue to monitor. (16:08 UTC — Feb 27)

Investigating - Opal4 is back online.

The problem was caused by high load following heavy RAM usage by the system `httpd` service. We’re still investigating the cause of the RAM usage. (23:22 UTC — Feb 26)

Opal4 is currently unresponsive. We’ll update this incident when more info is available. (22:55 UTC — Feb 26)

[Resolved] DE IMAP service offline

Wed, 12 Feb 2020 11:30:00 +0000

Fixed - The disk array rebuild is complete. (22:30 UTC — Feb 12)

Watching - The failed disk has been replaced and the DE IMAP service is operational at this time.

Disk performance may be slightly degraded while the disk array rebuilds.

We’ll post a final update when the array rebuild is complete. (17:27 UTC — Feb 12)

Watching - The DE IMAP service is back online. We’ll continue to monitor. (14:31 UTC — Feb 12)

Identified - The DE IMAP server has experienced a disk failure. The storage array is currently rebuilding. We’ll update with an ETA as soon as we have one. (14:20 UTC — Feb 12)

Our IMAP service in Germany has been offline since around 11:30 AM 12 Feb 2020. We’re working to identify the cause at this time. (11:30 UTC — Feb 12)

[Resolved] Opal1 offline for scheduled maintenance

Sat, 01 Feb 2020 04:56:00 +0000

Fixed - The maintenance is complete. (15:47 UTC — Feb 1)

Watching - Opal1 is back online.

Customer applications installed via our control panel should come back online within the next 10 minutes. Other applications may need to be restarted manually.

We’ll close out this incident when we’ve been notified that the maintenance is complete. (08:16 UTC — Feb 1)

Opal1 is currently offline for scheduled maintenance as announced on 31 December 2019: https://status.opalstack.com/#scheduled-6

A 4 hour maintenance window has been scheduled but we’re hopeful that the actual downtime will be much less.

We’ll update this post when the server is back online. (04:56 UTC — Feb 1)

[Resolved] US SMTP server blacklisted by Microsoft

Fri, 24 Jan 2020 14:15:00 +0000

Fixed - We’ve verified that Microsoft is now accepting mail from the new US SMTP IP. (16:42 UTC — Apr 7)

Watching - The new US SMTP IP has been live for the past 12+ hours and we are seeing no problems with it.

Additionally, Microsoft has finally responded to state that the original US SMTP IP has been delisted.

We’ll continue to monitor until we’re satisfied that this is resolved. (17:03 UTC — Apr 6)

Identified - Microsoft has continued to be unresponsive to our inquiries, leaving us no recourse except to move our US SMTP to a new IP address.

We will begin the transition on Monday April 6 at 3AM UTC (2020-04-06 03:00 UTC).

We anticipate that this transition will be minimally disruptive (if at all) and will not require any changes to customer mail client settings.

If you notice any problems sending mail during or after the transition, please contact our support team for assistance. (16:45 UTC — Apr 4)

Identified - Microsoft has been unresponsive to our inquiries for the past week so we’ve opened a new ticket with them.

According to Microsoft’s own postmaster tools the US SMTP is not blacklisted and no spam-like activity has been detected since 29 January 2020 - however, we’re still seeing bans for mail sent from the US SMTP to domains that use Office 365 email.

We’ll continue to work with MS to get this resolved. In the meantime, customers affected by this issue can work around it by creating a new mail user on our Frankfurt, Germany mail server and using that user for their SMTP authentication for outgoing mail. (21:28 UTC — Mar 16)

Watching - https://sender.office.com/ is reporting that our SMTP is no longer blocked but we’re still seeing intermittent bans in our mail logs. We’ve reached out to Microsoft for an update and are awaiting their response. (17:29 UTC — Mar 12)

Investigating - This issue is ongoing. We’ll continue to work towards resolution. (16:09 UTC — Feb 27)

Watching - Microsoft has once again assured us that our US SMTP has been removed from their blacklist. We’ll continue to monitor. (17:31 UTC — Feb 12)

Identified - We are still seeing ban activity In spite of the assurance from Microsoft that our US SMTP was removed from their blacklist. We’ll continue to work with them to get this resolved. (15:13 UTC — Feb 6)

Watching - Microsoft has delisted our US SMTP as of approximately 2:15 PM UTC 31 Jan 2020. It may take another couple of hours for the delisting to take full effect, so we will continue to monitor. (15:43 UTC — Jan 31)

Investigating - This issue is ongoing. We’ve taken all of the required actions as dictated by Microsoft so far and are waiting for them to delist the server.

In the meantime if you are a US customer affected by this block, feel free to use our German SMTP server to send mail: smtp.de.opalstack.com (18:32 UTC — Jan 28)

Our US SMTP service is currently on Microsoft’s blacklist. Messages sent to email addresses that use Microsoft’s email services (including hotmail.com and outlook.com) may be bounced back to their original sender.

We’re working with MS to have the server de-listed and will update this incident when it is done. (14:15 UTC — Jan 24)

[Resolved] Opal3 offline

Sun, 12 Jan 2020 17:40:00 +0000

Fixed - There have been no further problems since the last update. (00:24 UTC — Jan 13)

Watching - As of 2020-01-12 22:20 UTC all Opal3 shell user home directories, PostgreSQL databases, and related control panel data have been restored.

All customer sites are back online and the server is operating normally at this time. We’ll continue to monitor.

We were not able to recover shell user passwords, so Opal3 customers will need to reset their shell user passwords via the control panel. Instructions are available in our documentation: Changing a Shell User’s Password (22:25 UTC — Jan 12)

Identified - At approximately 17:40 UTC (9:40 AM Pacific) on 12 January 2020 we ran a live test of our upcoming migration tool on opal3, our shared hosting server in San Francisco.

A serious bug in the migration tool resulted in the loss of all customer home directories and PostgreSQL databases on opal3.

We’re presently restoring opal3 from the most recent backup which was completed at 00:14 AM, about 7.5 hours prior to the incident.

We’ll update this incident with more information when the restoration is complete. (19:17 UTC — Jan 12)

Opal3 was offline. (17:40 UTC — Jan 12)

[Resolved] Intermittent outages on opal4

Wed, 08 Jan 2020 00:36:00 +0000

Fixed - During last week’s maintenance our upstream provider updated the server firmware on opal4. We’ve seen no further issues since that time. (16:03 UTC — Apr 6)

Watching - The maintenance is complete and Opal4 is back online. We’ll continue to monitor. (02:58 UTC — Apr 2)

Identified - Opal4 is down for the emergency maintenance scheduled earlier today. The total expected downtime is less than one hour. (02:05 UTC — Apr 2)

Investigating - Opal4 will be going down this evening at 9PM US Central (2020-04-01 21:00 UTC-5) for emergency maintenance. The total downtime should be less than one hour.

The ongoing issue on opal4 has been high system load caused by two separate types of attacks:

High Apache RAM usage due to attacks against common targets like Wordpress sites
High CPU usage by the system firewall under a high volume of SYN packets

We’ve resolved the first problem by tuning our application firewall rules and by working with a couple of specific site owners that were receiving the brunt of the attacks.

The second problem is part of what we’re troubleshooting this evening.

At first glance the SYN packet issue would seem like a common SYN flood attack but other shared servers in our infrastructure receive similar amounts of traffic and don’t have any problems mitigating it.

In the past 24 hours we’ve discovered key differences in Opal4’s hardware compared to the other servers, so we’re working with our upstream provider to sort that out.

Tonight’s downtime is at their request to allow them to perform hardware diagnostics in support of that investigation. (23:24 UTC — Apr 1)

Watching - Opal4 just went through a brief spike in system load during which performance was degraded, but the system is back to normal at this time. (23:20 UTC — Mar 29)

Identified - We’ve identified two distinct attack patterns responsible for the intermittent outages on opal4 and are putting measures in place to mitigate them. (23:01 UTC — Mar 25)

Watching - This intermittent issue on Opal4 has started up again - we will continue to monitor as we work to resolve it. (19:05 UTC — Mar 25)

Fixed - We’ve seen no further issues in the past several hours. (16:38 UTC — Jan 8)

Watching - The problem appears to have been caused by temporary high load due to high CPU usage by the system firewall.

opal4 is stable at this time. We’ll continue to monitor. (01:21 UTC — Jan 8)

opal4.opalstack.com is experiencing intermittent outages. We’re looking into it and will update this item when we have more information. (00:36 UTC — Jan 8)

[Resolved] Control Panel CSS broken

Wed, 04 Dec 2019 16:15:00 +0000

The main control panel CSS was briefly broken this morning which may have rendered the panel unusable for customers. The error has been corrected. (16:15 UTC — Dec 4)

[Resolved] US MX server blacklisted by Cloudmark

Mon, 25 Nov 2019 19:19:00 +0000

Fixed - The MX server has been delisted. (17:30 UTC — Nov 26)

Watching - We believe the blacklisting happened due to an influx of spam that was subsequently forwarded to external forwarding targets for various customers. We’ve removed the spam from our mail queue and are standing by for the server to be de-listed. (20:10 UTC — Nov 25)

mx1.us.opalstack.com is presently blacklisted by Cloudmark. As a result, email forwarded to domains that use Cloudmark services may be delayed or bounced.

We’re presently working with Cloudmark to have the server delisted and will update this incident when more information is available. (19:19 UTC — Nov 25)

[Resolved] Control panel issue

Mon, 25 Nov 2019 07:00:00 +0000

At around 7AM UTC on 25 November 2019 a problem developed with the API tokens used by the control panel. As a result, mail users and other items added via the control panel after that time may not have been created correctly.

The token issue is resolved. If you notice any problems with mail users, shell users, applications, and other items created via the panel in the past 24-36 hours then please contact our support team for assistance. (07:00 UTC — Nov 25)