Opalstack - Dashboard control loop issue – Incident details

All systems operational

Dashboard control loop issue

Resolved
Operational
Started over 4 years agoLasted about 1 hour

Affected

General Infrastructure

Operational from 12:57 AM to 2:12 AM

Control Panel and API

Operational from 12:57 AM to 2:12 AM

Updates
  • Resolved
    Resolved

    The dashboard and API are back online.

    The problem was caused by human error by one of our team while using our admin shell to make a mass update to a single customer's websites during a migration. This kicked off a massive cascade of events which overloaded the control loop.

    This was strictly an admin error - the WebFaction migration tool that many customers are using is safe to use.

    We've rolled back the updates and have cleared the pending updates from the control loop and all systems are operating normally at this time.

    This issue affected Apache/PHP-CGI applications in the dashboard. We believe there should not be any lingering issues but if you notice a problem with any of your Apache/PHP-CGI applications then please try an edit+save cycle on the application and wait a minute for the changes to take effect. If the problem persists then please contact Opalstack Support for further assistance.

  • Identified
    Identified

    We've taken the dashboard offline while we work to resolve this.

  • Investigating
    Investigating

    At this time there is an issue with our control loop which deploys changes from the dashboard to our servers. Dashboard changes will remain pending until the issue is resolved. We'll update this post as soon as possible.