Opalstack - Notice history

All systems operational

Notice history

Apr 2020

Opal4 not responding
  • Resolved
    Resolved

    We believe this latest issue was caused by a DNS reflection attack.

    We're consulting with our upstream provider to see what they can do to protect the server and network from future attacks.

  • Update
    Update

    Opal4 is back online, investigation is ongoing.

  • Investigating
    Investigating

    Opal4 is not responding. We're looking into the cause.

Apache was not responding on Opal4
  • Resolved
    Resolved

    Apache was once again not responding on Opal4 after hitting its configured ServerLimit. The underlying cause appears to have been a high number of PHP-FPM processes being re-spawned for one customer site. We've adjusted the PHP-FPM configuration for that site to avoid the problem in the future.

Apache was not responding on Opal4
  • Resolved
    Resolved

    The root cause appears to have been a misconfigured customer application. We've corrected the configuration and have updated our monitoring system to alert us when Apache approaches its configured limits.

  • Investigating
    Investigating

    The Apache service on Opal4 was not responding for approximately 1 hour today beginning 18:23 UTC.

    The problem was that the Apache service had hit its configured ServerLimit which meant that no further Apache processes could spawn to handle requests. That has been corrected and the service is online now.

    We're now investigating the root cause and also looking into why our own monitoring systems did not alert us to this condition.

Connectivity issues in Frankfurt
  • Resolved
    Resolved

    The data center team has reported that the issue has been resolved.

  • Investigating
    Investigating

    Several servers in our Frankfurt DE datacenter have experienced network connectivity issues.

    The data center team reports that the problem is due to high temperatures in the DC. Their investigation into the root cause is ongoing.

    At this time all servers are online. We'll update this post when more information is available.

Opal4 load issues
  • Resolved
    Resolved

    We've seen no further load issues since our last update.

  • Monitoring
    Monitoring

    Opal4 has experienced two load spikes in the past 24 hours.

    During the most recent spike we were able to positively identify one of the sites causing the problem and have taken steps to prevent it from happening again.

    We'll continue to monitor.

  • Investigating
    Investigating

    Opal4 experienced a brief spike in system load between 23:40 and 23:50 UTC on Friday 10 April 2020.

    The cause appears to have been high CPU usage due to a SYN flood attack.

    The server is operating normally at this time. We'll continue to monitor.

Mar 2020

No notices reported this month

Feb 2020

Opal4 not responding
  • Resolved
    Resolved

    We've successfully mitigated the attack and Opal4 has been stable since our last update.

  • Monitoring
    Monitoring

    We've identified an ongoing attack pattern against sites on opal4 and have taken steps to mitigate it platform-wide. We'll continue to monitor.

  • Update
    Update

    Opal4 is back online.

    The problem was caused by high load following heavy RAM usage by the system `httpd` service. We're still investigating the cause of the RAM usage.

  • Investigating
    Investigating

    Opal4 is currently unresponsive. We'll update this incident when more info is available.

DE IMAP service offline
  • Resolved
    Resolved

    The disk array rebuild is complete.

  • Update
    Update

    The failed disk has been replaced and the DE IMAP service is operational at this time.

    Disk performance may be slightly degraded while the disk array rebuilds.

    We'll post a final update when the array rebuild is complete.

  • Monitoring
    Monitoring

    The DE IMAP service is back online. We'll continue to monitor.

  • Identified
    Identified

    The DE IMAP server has experienced a disk failure. The storage array is currently rebuilding. We'll update with an ETA as soon as we have one.

  • Investigating
    Investigating

    Our IMAP service in Germany has been offline since around 11:30 AM 12 Feb 2020. We're working to identify the cause at this time.

Opal1 offline for scheduled maintenance
  • Resolved
    Resolved

    The maintenance is complete.

  • Monitoring
    Monitoring

    Opal1 is back online.

    Customer applications installed via our control panel should come back online within the next 10 minutes. Other applications may need to be restarted manually.

    We'll close out this incident when we've been notified that the maintenance is complete.

  • Investigating
    Investigating

    Opal1 is currently offline for scheduled maintenance as announced on 31 December 2019: https://status.opalstack.com/#scheduled-6

    A 4 hour maintenance window has been scheduled but we're hopeful that the actual downtime will be much less.

    We'll update this post when the server is back online.

WDC infrastructure maintenance
  • Completed
    February 01, 2020 at 8:10 AM
    Completed
    February 01, 2020 at 8:10 AM
    The maintenance is complete.
  • In progress
    February 01, 2020 at 4:55 AM
    In progress
    February 01, 2020 at 4:55 AM

    Maintenance is now in progress

  • Planned
    February 01, 2020 at 4:55 AM
    Planned
    February 01, 2020 at 4:55 AM

    Our Washington DC data center team will be performing infrastructure maintenance beginning at 11:55 PM EST on 31 January 2020 (04:55 UTC on 1 Feb 2020).

    The maintenance may last up to 4 hours during which the following servers will be offline:

    • opal1.opalstack.com (162.210.195.182)

    The server will be powered down gracefully at the beginning of the maintenance window and will be restarted when the maintenance is complete.

    We'll update https://status.opalstack.com/incidents/9 as the maintenance progresses.

Feb 2020 to Apr 2020

Next