Opalstack - Notice history

All systems operational

Notice history

Sep 2021

Mail server certificates expired
  • Resolved
    Resolved

    The issue is now resolved. The root cause was an obsolete root certificate on our servers. The certificate has been updated.

  • Investigating
    Investigating

    The SSL certificates used by Opalstack's mail servers are expired. We're working to correct that at this time and should have it resolved shortly.

Intermittent high load on opal11
  • Resolved
    Resolved

    There have been no further issues on opal11 in the past 24 hours so we consider this to be resolved.

  • Update
    Update

    We've mitigated most of the abusive traffic that's been driving up high load on opal11 and performance should be much better at this time.

    We are still seeing occasional spikes and will continue to investigate.

  • Update
    Update

    Our investigation into this issue is ongoing. We'll provide more info when it is available.

  • Investigating
    Investigating

    Our shared hosting server opal11.opalstack.com in Frankfurt Germany is experiencing an intermittent high load condition which is impacting system performance. We're working to resolve this and will update this post when the issue is resolved.

opal3 not responding
  • Resolved
    Resolved

    We've positively identified the cause of the memory spike and have mitigated it.

  • Update
    Update

    The root cause was a massive spike in system load due to a runaway customer process using most of the available system RAM. We'll work with the customer to ensure it doesn't happen again.

  • Monitoring
    Monitoring

    Opal3 is back online at this time. We'll continue to monitor.

  • Update
    Update

    We're rebooting opal3 at this time.

  • Investigating
    Investigating

    Opal3 is currently not responding. We're investigating.

Aug 2021

US IMAP/POP3 server relocation - downtime expected
  • Completed
    August 11, 2021 at 10:00 AM
    Completed
    August 11, 2021 at 10:00 AM
    The maintenance is complete.
  • In progress
    August 11, 2021 at 5:00 AM
    In progress
    August 11, 2021 at 5:00 AM

    Maintenance is now in progress

  • Planned
    August 11, 2021 at 5:00 AM
    Planned
    August 11, 2021 at 5:00 AM

    On 11 August 2021 beginning at 05:00 UTC (00:00 US Central) our data center team will be moving mail.us.opalstack.com to a new facility.

    During the move the server will be offline.

    Mail that arrives while the server is offline will be deferred and delivered after the maintenance is complete. No mail should be rejected or bounced during the maintenance.

    The mail server hostname and IP address will not change and no action is required on your part.

    The maintenance window is 6 hours but we expect the total downtime to be less than that.

    We will update our status page at http://status.opalstack.com/ as the maintenance progresses.

    If you have any questions or concerns regarding this migration then please email support@opalstack.com and someone from our team will get back to you as soon as possible.

opal4 relocation - downtime expected
  • Completed
    August 10, 2021 at 10:25 AM
    Completed
    August 10, 2021 at 10:25 AM
    The maintenance is complete.
  • In progress
    August 10, 2021 at 5:00 AM
    In progress
    August 10, 2021 at 5:00 AM

    Maintenance is now in progress

  • Planned
    August 10, 2021 at 5:00 AM
    Planned
    August 10, 2021 at 5:00 AM

    On 10 August 2021 beginning at 05:00 UTC (00:00 US Central) our data center team will be moving opal4.opalstack.com to a new facility.

    During the move the server will be offline.

    The maintenance window is 6 hours but we expect the total downtime to be less than that.

    We will update our status page at http://status.opalstack.com/ as the maintenance progresses.

    Email is hosted on separate hardware and is not affected by this maintenance - however a second maintenance period the following day will affect email.

    Your server IP address will not change.

    If you have any questions or concerns regarding this migration then please email support@opalstack.com and someone from our team will get back to you as soon as possible.

Dashboard control loop issue
  • Resolved
    Resolved

    The dashboard and API are back online.

    The problem was caused by human error by one of our team while using our admin shell to make a mass update to a single customer's websites during a migration. This kicked off a massive cascade of events which overloaded the control loop.

    This was strictly an admin error - the WebFaction migration tool that many customers are using is safe to use.

    We've rolled back the updates and have cleared the pending updates from the control loop and all systems are operating normally at this time.

    This issue affected Apache/PHP-CGI applications in the dashboard. We believe there should not be any lingering issues but if you notice a problem with any of your Apache/PHP-CGI applications then please try an edit+save cycle on the application and wait a minute for the changes to take effect. If the problem persists then please contact Opalstack Support for further assistance.

  • Identified
    Identified

    We've taken the dashboard offline while we work to resolve this.

  • Investigating
    Investigating

    At this time there is an issue with our control loop which deploys changes from the dashboard to our servers. Dashboard changes will remain pending until the issue is resolved. We'll update this post as soon as possible.

Delay in dashboard operations
  • Resolved
    Resolved

    The problem has been corrected and the dashboard operation backlog is cleared.

  • Investigating
    Investigating

    At this time there is a significant delay in the time that it is taking for changes made in the Opalstack dashboard (ie creating or modifying domains, apps, sites, email, shell users, etc).

    We're investigating the issue and will update this post when more info is available.

Jul 2021

opal5 offline
  • Resolved
    Resolved

    The null route has been removed and opal5 is back online.

  • Identified
    Identified

    The problem seems to be a network null route due to an abuse issue that has been mis-handled by an upstream provider. We're working on restoring service now.

  • Investigating
    Investigating

    Opal5 (shared hosting in Frankfurt) is offline. We're investigating at this time.

Apache was down on Opal8
  • Resolved
    Resolved

    The system Apache httpd service on opal8.opalstack.com was briefly offline due to a problem with its configured system resource limits.

    During the incident customers visiting the Apache-powered sites may have seen a "502 bad gateway" error message.

    The problem has been corrected and the service is working normally at this time.

Dallas network maintenance
  • Completed
    July 02, 2021 at 1:00 AM
    Completed
    July 02, 2021 at 1:00 AM
    The maintenance is complete.
  • In progress
    July 02, 2021 at 12:00 AM
    In progress
    July 02, 2021 at 12:00 AM

    Maintenance is now in progress

  • Planned
    July 02, 2021 at 12:00 AM
    Planned
    July 02, 2021 at 12:00 AM

    Our network provider in Dallas TX (USA) will be performing maintenance at 0600 UTC on 2 July 2021. The expected maintenance window will be up to 1 hour.

    During this maintenance customers may experience a few brief moments of higher latency/packet loss up to 15 minutes. Even though it's not expected, there is a chance of loss of connectivity during these moments.

    We have assurances from our provider that they will do their best to keep the impact as minimal as possible. Our support team will be available during and after the maintenance should any issues arise.

Jul 2021 to Sep 2021

Next