Opalstack - Notice history

All systems operational

Notice history

Jul 2021

opal5 offline
  • Resolved
    Resolved

    The null route has been removed and opal5 is back online.

  • Identified
    Identified

    The problem seems to be a network null route due to an abuse issue that has been mis-handled by an upstream provider. We're working on restoring service now.

  • Investigating
    Investigating

    Opal5 (shared hosting in Frankfurt) is offline. We're investigating at this time.

Apache was down on Opal8
  • Resolved
    Resolved

    The system Apache httpd service on opal8.opalstack.com was briefly offline due to a problem with its configured system resource limits.

    During the incident customers visiting the Apache-powered sites may have seen a "502 bad gateway" error message.

    The problem has been corrected and the service is working normally at this time.

Dallas network maintenance
  • Completed
    July 02, 2021 at 1:00 AM
    Completed
    July 02, 2021 at 1:00 AM
    The maintenance is complete.
  • In progress
    July 02, 2021 at 12:00 AM
    In progress
    July 02, 2021 at 12:00 AM

    Maintenance is now in progress

  • Planned
    July 02, 2021 at 12:00 AM
    Planned
    July 02, 2021 at 12:00 AM

    Our network provider in Dallas TX (USA) will be performing maintenance at 0600 UTC on 2 July 2021. The expected maintenance window will be up to 1 hour.

    During this maintenance customers may experience a few brief moments of higher latency/packet loss up to 15 minutes. Even though it's not expected, there is a chance of loss of connectivity during these moments.

    We have assurances from our provider that they will do their best to keep the impact as minimal as possible. Our support team will be available during and after the maintenance should any issues arise.

Jun 2021

Upcoming dashboard and API upgrade downtime
  • Completed
    June 24, 2021 at 8:30 AM
    Completed
    June 24, 2021 at 8:30 AM
    The maintenance is complete.
  • In progress
    June 24, 2021 at 3:00 AM
    In progress
    June 24, 2021 at 3:00 AM

    Maintenance is now in progress

  • Planned
    June 24, 2021 at 3:00 AM
    Planned
    June 24, 2021 at 3:00 AM

    On 24 June 2021 at 0300 UTC we will be taking the Opalstack dashboard and API offline for a major upgrade.

    The maintenance window is 4 hours, but we expect to have the job done in less time.

    We'll update this post as the maintenance progresses.

    2021-06-24 04:31 UTC: The dashboard and API are down for the upgrade at this time.

    2021-06-24 06:53 UTC: The upgrade is taking longer than expected and we've extended the window for another hour.

    2021-06-24 08:09 UTC: we hope to have this wrapped up within the next 20 minutes, by 0830 UTC.

    2021-06-24 08:30 UTC: the upgade is complete! 🎉

German SMTP offline
  • Resolved
    Resolved

    The certificate has been updated and SMTP is now working normally. We're reviewing and revising our internal procedures to minimize the chance of this happening again in the future.

  • Investigating
    Investigating

    German SMTP was offline due to a problem with the server certificate.

Opal7 rebooted unexpectedly
  • Resolved
    Resolved

    The maintenance was confirmed completed on Saturday shortly after our last update.

  • Monitoring
    Monitoring

    Opal7 is back online and operating normally. We're awaiting confirmation from the data center that the maintenance is complete.

  • Update
    Update

    Opal7 is offline for emergency RAM replacement at this time.

  • Identified
    Identified

    The cause of the reboot was a faulty memory module. We're scheduling emergency maintenance at this time.

  • Investigating
    Investigating

    Opal7 (shared hosting in Phoenix) was unexpectedly rebooted. We're investigating the cause.

    The server is online and operating normally at this time.

Apache was briefly down on opal5
  • Resolved
    Resolved

    The Apache httpd service on opal5 was briefly down because the system /tmp directory was full.

    The service was down for 2-3 minutes. We've identified the customer responsible for the problem and will work with them to ensure it does not happen again.

German IMAP/POP3 service offline
  • Resolved
    Resolved

    The Germain mail service is back online. The outage was caused by human error when deploying a new certificate.

  • Investigating
    Investigating

    The German IMAP/POP3 service is currently offline. We're currently investigating.

May 2021

Unsecured outgoing SMTP connections from web servers to be blocked beginning on 29 May 2021
  • Completed
    May 29, 2021 at 12:20 AM
    Completed
    May 29, 2021 at 12:20 AM
    The maintenance is complete.
  • In progress
    May 29, 2021 at 12:00 AM
    In progress
    May 29, 2021 at 12:00 AM

    Maintenance is now in progress

  • Planned
    May 29, 2021 at 12:00 AM
    Planned
    May 29, 2021 at 12:00 AM

    At 00:00 UTC on 29 May 2021 we will begin blocking unsecured outgoing SMTP connections from Opalstack web servers by blocking outgoing port 25 in our network firewalls.

    If you are running an application on an Opalstack web server that makes SMTP connections to external SMTP servers then please ensure that your application is configured to connect securely over port 465 or port 587.

    Applications that use the native PHP mail function and other apps that use sendmail directly on the server will not be affected by this change.

    2021-05-29 00:20 UTC The new firewall rules are now in effect on all servers.

Network outage affecting domains that use Opalstack DE name servers
  • Resolved
    Resolved

    All DE services are back online.

  • Monitoring
    Monitoring

    Service has been partially restored. The remaining affected servers are ns2.de.opalstack.com and 5 private customer VPS.

    We'll continue to monitor and update as the service is restored.

  • Investigating
    Investigating

    Our DE name servers, SMTP, and several customer VPS located in Frankfurt, Germany are currently offline due to a connectivity issue with one of our regional providers there.

    They're working to restore service. In the interim, we are relocating our DE nameservers to new locations.

    Since the DE name servers are affected, this is unfortunately affecting all domains that use Opalstack's DE name servers. We're hopeful that service will be restored quickly but if it is not then you may want to switch the NS for your domain to one of our other regional sets of name servers. Our regional NS are listed at: What are your name servers?

    We'll update this post as soon as more information is available.

opal1 not responding
  • Resolved
    Resolved

    The problem was caused by a spike in system load due to a problem with a customer site. Conditions have returned to normal and we'll work with that customer to prevent further issues.

  • Investigating
    Investigating

    opal1.opalstack.com (WDC shared hosting) is currently responding to ping but not to SSH and HTTP requests. We're investigating and will update this post when more info is available.

Temporary SSL problem in US mail system
  • Resolved
    Resolved

    At approximately 06:00 UTC on 17 May 2021 we inadvertently pushed the wrong set of SSL certificates to our US mail system.

    Customers who attempted to access the mail system during that period may have seen certificate warning messages and would not have been able to access their mail.

    The problem was fixed approximately 30 minutes later and the mail system has been working normally since that time. No incoming or outgoing mail was lost during the incident.

    If you are still seeing a certificate error while attempting to use the US mail system then please restart your mail client and/or reboot your computer. If the problem persists then please contact Opalstack support for assistance.

German SMTP IP range on MS block list
  • Resolved
    Resolved

    Microsoft is now accepting mail from smtp.de.opalstack.com.

  • Identified
    Identified

    We've identified and corrected the cause of the block, and have passed our findings along to Microsoft with a request to delist the German SMTP address.

    We'll update this post when Microsoft have confirmed that the server is no longer on the block list.

  • Investigating
    Investigating

    We've received some reports that outlook.com and other Microsoft mail services are blocking smtp.de.opalstack.com.

    We're investigating now and will follow up when we have more info.

May 2021 to Jul 2021

Next