Opalstack - Notice history

We've successfully mitigated the attack and Opal4 has been stable since our last update.

We've identified an ongoing attack pattern against sites on opal4 and have taken steps to mitigate it platform-wide. We'll continue to monitor.

Opal4 is back online.

The problem was caused by high load following heavy RAM usage by the system `httpd` service. We're still investigating the cause of the RAM usage.

The disk array rebuild is complete.

The failed disk has been replaced and the DE IMAP service is operational at this time.

Disk performance may be slightly degraded while the disk array rebuilds.

We'll post a final update when the array rebuild is complete.

The DE IMAP service is back online. We'll continue to monitor.

The DE IMAP server has experienced a disk failure. The storage array is currently rebuilding. We'll update with an ETA as soon as we have one.

The maintenance is complete.

Opal1 is back online.

Customer applications installed via our control panel should come back online within the next 10 minutes. Other applications may need to be restarted manually.

We'll close out this incident when we've been notified that the maintenance is complete.

We've verified that Microsoft is now accepting mail from the new US SMTP IP.

The new US SMTP IP has been live for the past 12+ hours and we are seeing no problems with it.

Additionally, Microsoft has finally responded to state that the original US SMTP IP has been delisted.

We'll continue to monitor until we're satisfied that this is resolved.

Microsoft has continued to be unresponsive to our inquiries, leaving us no recourse except to move our US SMTP to a new IP address.

We will begin the transition on Monday April 6 at 3AM UTC (2020-04-06 03:00 UTC).

We anticipate that this transition will be minimally disruptive (if at all) and will not require any changes to customer mail client settings.

If you notice any problems sending mail during or after the transition, please contact our support team for assistance.

Microsoft has been unresponsive to our inquiries for the past week so we've opened a new ticket with them.

According to Microsoft's own postmaster tools the US SMTP is not blacklisted and no spam-like activity has been detected since 29 January 2020 - however, we're still seeing bans for mail sent from the US SMTP to domains that use Office 365 email.

We'll continue to work with MS to get this resolved. In the meantime, customers affected by this issue can work around it by creating a new mail user on our Frankfurt, Germany mail server and using that user for their SMTP authentication for outgoing mail.

https://sender.office.com/ is reporting that our SMTP is no longer blocked but we're still seeing intermittent bans in our mail logs. We've reached out to Microsoft for an update and are awaiting their response.

This issue is ongoing. We'll continue to work towards resolution.

Microsoft has once again assured us that our US SMTP has been removed from their blacklist. We'll continue to monitor.

We are still seeing ban activity In spite of the assurance from Microsoft that our US SMTP was removed from their blacklist. We'll continue to work with them to get this resolved.

Microsoft has delisted our US SMTP as of approximately 2:15 PM UTC 31 Jan 2020. It may take another couple of hours for the delisting to take full effect, so we will continue to monitor.

This issue is ongoing. We've taken all of the required actions as dictated by Microsoft so far and are waiting for them to delist the server.

In the meantime if you are a US customer affected by this block, feel free to use our German SMTP server to send mail: smtp.de.opalstack.com

During last week's maintenance our upstream provider updated the server firmware on opal4. We've seen no further issues since that time.

The maintenance is complete and Opal4 is back online. We'll continue to monitor.

Opal4 is down for the emergency maintenance scheduled earlier today. The total expected downtime is less than one hour.

Opal4 will be going down this evening at 9PM US Central (2020-04-01 21:00 UTC-5) for emergency maintenance. The total downtime should be less than one hour.

The ongoing issue on opal4 has been high system load caused by two separate types of attacks:

1. High Apache RAM usage due to attacks against common targets like Wordpress sites
2. High CPU usage by the system firewall under a high volume of SYN packets

We've resolved the first problem by tuning our application firewall rules and by working with a couple of specific site owners that were receiving the brunt of the attacks.

The second problem is part of what we're troubleshooting this evening.

At first glance the SYN packet issue would seem like a common SYN flood attack but other shared servers in our infrastructure receive similar amounts of traffic and don't have any problems mitigating it.

In the past 24 hours we've discovered key differences in Opal4's hardware compared to the other servers, so we're working with our upstream provider to sort that out.

Tonight's downtime is at their request to allow them to perform hardware diagnostics in support of that investigation.

Opal4 just went through a brief spike in system load during which performance was degraded, but the system is back to normal at this time.

We've identified two distinct attack patterns responsible for the intermittent outages on opal4 and are putting measures in place to mitigate them.

This intermittent issue on Opal4 has started up again - we will continue to monitor as we work to resolve it.

We've seen no further issues in the past several hours.

The problem appears to have been caused by temporary high load due to high CPU usage by the system firewall.

opal4 is stable at this time. We'll continue to monitor.