Opalstack - Notice history

All systems operational

Notice history

Oct 2021

End-of-life for Opalstack API v0 and the WebFaction migration tool
  • Completed
    October 09, 2021 at 12:00 PM
    Completed
    October 09, 2021 at 12:00 PM
    The maintenance is complete.
  • In progress
    October 09, 2021 at 12:00 AM
    In progress
    October 09, 2021 at 12:00 AM

    Maintenance is now in progress

  • Planned
    October 09, 2021 at 12:00 AM
    Planned
    October 09, 2021 at 12:00 AM

    On October 9 2021 we will be retiring the v0 API endpoint at https://my.opalstack.com/api/v0/.

    If you're using Opalstack's API then please update your code to use the v1 endpoint https://my.opalstack.com/api/v1/.

    The API v1 endpoint is documented at https://my.opalstack.com/api/v1/doc/.

    The WebFaction migration tool (wfmigrate) uses the old v0 endpoint, so that tool will be retired as well. WebFaction is officially shut down so the tool no longer has any use.

    If you have any questions or concerns then please email them to support @ opalstack.com and one of our team will get back to you.

Sep 2021

Mail server certificates expired
  • Resolved
    Resolved

    The issue is now resolved. The root cause was an obsolete root certificate on our servers. The certificate has been updated.

  • Investigating
    Investigating

    The SSL certificates used by Opalstack's mail servers are expired. We're working to correct that at this time and should have it resolved shortly.

Intermittent high load on opal11
  • Resolved
    Resolved

    There have been no further issues on opal11 in the past 24 hours so we consider this to be resolved.

  • Update
    Update

    We've mitigated most of the abusive traffic that's been driving up high load on opal11 and performance should be much better at this time.

    We are still seeing occasional spikes and will continue to investigate.

  • Update
    Update

    Our investigation into this issue is ongoing. We'll provide more info when it is available.

  • Investigating
    Investigating

    Our shared hosting server opal11.opalstack.com in Frankfurt Germany is experiencing an intermittent high load condition which is impacting system performance. We're working to resolve this and will update this post when the issue is resolved.

opal3 not responding
  • Resolved
    Resolved

    We've positively identified the cause of the memory spike and have mitigated it.

  • Update
    Update

    The root cause was a massive spike in system load due to a runaway customer process using most of the available system RAM. We'll work with the customer to ensure it doesn't happen again.

  • Monitoring
    Monitoring

    Opal3 is back online at this time. We'll continue to monitor.

  • Update
    Update

    We're rebooting opal3 at this time.

  • Investigating
    Investigating

    Opal3 is currently not responding. We're investigating.

Aug 2021

US IMAP/POP3 server relocation - downtime expected
  • Completed
    August 11, 2021 at 10:00 AM
    Completed
    August 11, 2021 at 10:00 AM
    The maintenance is complete.
  • In progress
    August 11, 2021 at 5:00 AM
    In progress
    August 11, 2021 at 5:00 AM

    Maintenance is now in progress

  • Planned
    August 11, 2021 at 5:00 AM
    Planned
    August 11, 2021 at 5:00 AM

    On 11 August 2021 beginning at 05:00 UTC (00:00 US Central) our data center team will be moving mail.us.opalstack.com to a new facility.

    During the move the server will be offline.

    Mail that arrives while the server is offline will be deferred and delivered after the maintenance is complete. No mail should be rejected or bounced during the maintenance.

    The mail server hostname and IP address will not change and no action is required on your part.

    The maintenance window is 6 hours but we expect the total downtime to be less than that.

    We will update our status page at http://status.opalstack.com/ as the maintenance progresses.

    If you have any questions or concerns regarding this migration then please email support@opalstack.com and someone from our team will get back to you as soon as possible.

opal4 relocation - downtime expected
  • Completed
    August 10, 2021 at 10:25 AM
    Completed
    August 10, 2021 at 10:25 AM
    The maintenance is complete.
  • In progress
    August 10, 2021 at 5:00 AM
    In progress
    August 10, 2021 at 5:00 AM

    Maintenance is now in progress

  • Planned
    August 10, 2021 at 5:00 AM
    Planned
    August 10, 2021 at 5:00 AM

    On 10 August 2021 beginning at 05:00 UTC (00:00 US Central) our data center team will be moving opal4.opalstack.com to a new facility.

    During the move the server will be offline.

    The maintenance window is 6 hours but we expect the total downtime to be less than that.

    We will update our status page at http://status.opalstack.com/ as the maintenance progresses.

    Email is hosted on separate hardware and is not affected by this maintenance - however a second maintenance period the following day will affect email.

    Your server IP address will not change.

    If you have any questions or concerns regarding this migration then please email support@opalstack.com and someone from our team will get back to you as soon as possible.

Dashboard control loop issue
  • Resolved
    Resolved

    The dashboard and API are back online.

    The problem was caused by human error by one of our team while using our admin shell to make a mass update to a single customer's websites during a migration. This kicked off a massive cascade of events which overloaded the control loop.

    This was strictly an admin error - the WebFaction migration tool that many customers are using is safe to use.

    We've rolled back the updates and have cleared the pending updates from the control loop and all systems are operating normally at this time.

    This issue affected Apache/PHP-CGI applications in the dashboard. We believe there should not be any lingering issues but if you notice a problem with any of your Apache/PHP-CGI applications then please try an edit+save cycle on the application and wait a minute for the changes to take effect. If the problem persists then please contact Opalstack Support for further assistance.

  • Identified
    Identified

    We've taken the dashboard offline while we work to resolve this.

  • Investigating
    Investigating

    At this time there is an issue with our control loop which deploys changes from the dashboard to our servers. Dashboard changes will remain pending until the issue is resolved. We'll update this post as soon as possible.

Delay in dashboard operations
  • Resolved
    Resolved

    The problem has been corrected and the dashboard operation backlog is cleared.

  • Investigating
    Investigating

    At this time there is a significant delay in the time that it is taking for changes made in the Opalstack dashboard (ie creating or modifying domains, apps, sites, email, shell users, etc).

    We're investigating the issue and will update this post when more info is available.

Aug 2021 to Oct 2021

Next