Opalstack - Notice history

All systems operational

Notice history

Aug 2021

US IMAP/POP3 server relocation - downtime expected
  • Completed
    August 11, 2021 at 10:00 AM
    Completed
    August 11, 2021 at 10:00 AM
    The maintenance is complete.
  • In progress
    August 11, 2021 at 5:00 AM
    In progress
    August 11, 2021 at 5:00 AM

    Maintenance is now in progress

  • Planned
    August 11, 2021 at 5:00 AM
    Planned
    August 11, 2021 at 5:00 AM

    On 11 August 2021 beginning at 05:00 UTC (00:00 US Central) our data center team will be moving mail.us.opalstack.com to a new facility.

    During the move the server will be offline.

    Mail that arrives while the server is offline will be deferred and delivered after the maintenance is complete. No mail should be rejected or bounced during the maintenance.

    The mail server hostname and IP address will not change and no action is required on your part.

    The maintenance window is 6 hours but we expect the total downtime to be less than that.

    We will update our status page at http://status.opalstack.com/ as the maintenance progresses.

    If you have any questions or concerns regarding this migration then please email support@opalstack.com and someone from our team will get back to you as soon as possible.

opal4 relocation - downtime expected
  • Completed
    August 10, 2021 at 10:25 AM
    Completed
    August 10, 2021 at 10:25 AM
    The maintenance is complete.
  • In progress
    August 10, 2021 at 5:00 AM
    In progress
    August 10, 2021 at 5:00 AM

    Maintenance is now in progress

  • Planned
    August 10, 2021 at 5:00 AM
    Planned
    August 10, 2021 at 5:00 AM

    On 10 August 2021 beginning at 05:00 UTC (00:00 US Central) our data center team will be moving opal4.opalstack.com to a new facility.

    During the move the server will be offline.

    The maintenance window is 6 hours but we expect the total downtime to be less than that.

    We will update our status page at http://status.opalstack.com/ as the maintenance progresses.

    Email is hosted on separate hardware and is not affected by this maintenance - however a second maintenance period the following day will affect email.

    Your server IP address will not change.

    If you have any questions or concerns regarding this migration then please email support@opalstack.com and someone from our team will get back to you as soon as possible.

Dashboard control loop issue
  • Resolved
    Resolved

    The dashboard and API are back online.

    The problem was caused by human error by one of our team while using our admin shell to make a mass update to a single customer's websites during a migration. This kicked off a massive cascade of events which overloaded the control loop.

    This was strictly an admin error - the WebFaction migration tool that many customers are using is safe to use.

    We've rolled back the updates and have cleared the pending updates from the control loop and all systems are operating normally at this time.

    This issue affected Apache/PHP-CGI applications in the dashboard. We believe there should not be any lingering issues but if you notice a problem with any of your Apache/PHP-CGI applications then please try an edit+save cycle on the application and wait a minute for the changes to take effect. If the problem persists then please contact Opalstack Support for further assistance.

  • Identified
    Identified

    We've taken the dashboard offline while we work to resolve this.

  • Investigating
    Investigating

    At this time there is an issue with our control loop which deploys changes from the dashboard to our servers. Dashboard changes will remain pending until the issue is resolved. We'll update this post as soon as possible.

Delay in dashboard operations
  • Resolved
    Resolved

    The problem has been corrected and the dashboard operation backlog is cleared.

  • Investigating
    Investigating

    At this time there is a significant delay in the time that it is taking for changes made in the Opalstack dashboard (ie creating or modifying domains, apps, sites, email, shell users, etc).

    We're investigating the issue and will update this post when more info is available.

Jul 2021

opal5 offline
  • Resolved
    Resolved

    The null route has been removed and opal5 is back online.

  • Identified
    Identified

    The problem seems to be a network null route due to an abuse issue that has been mis-handled by an upstream provider. We're working on restoring service now.

  • Investigating
    Investigating

    Opal5 (shared hosting in Frankfurt) is offline. We're investigating at this time.

Apache was down on Opal8
  • Resolved
    Resolved

    The system Apache httpd service on opal8.opalstack.com was briefly offline due to a problem with its configured system resource limits.

    During the incident customers visiting the Apache-powered sites may have seen a "502 bad gateway" error message.

    The problem has been corrected and the service is working normally at this time.

Dallas network maintenance
  • Completed
    July 02, 2021 at 1:00 AM
    Completed
    July 02, 2021 at 1:00 AM
    The maintenance is complete.
  • In progress
    July 02, 2021 at 12:00 AM
    In progress
    July 02, 2021 at 12:00 AM

    Maintenance is now in progress

  • Planned
    July 02, 2021 at 12:00 AM
    Planned
    July 02, 2021 at 12:00 AM

    Our network provider in Dallas TX (USA) will be performing maintenance at 0600 UTC on 2 July 2021. The expected maintenance window will be up to 1 hour.

    During this maintenance customers may experience a few brief moments of higher latency/packet loss up to 15 minutes. Even though it's not expected, there is a chance of loss of connectivity during these moments.

    We have assurances from our provider that they will do their best to keep the impact as minimal as possible. Our support team will be available during and after the maintenance should any issues arise.

Jun 2021

Upcoming dashboard and API upgrade downtime
  • Completed
    June 24, 2021 at 8:30 AM
    Completed
    June 24, 2021 at 8:30 AM
    The maintenance is complete.
  • In progress
    June 24, 2021 at 3:00 AM
    In progress
    June 24, 2021 at 3:00 AM

    Maintenance is now in progress

  • Planned
    June 24, 2021 at 3:00 AM
    Planned
    June 24, 2021 at 3:00 AM

    On 24 June 2021 at 0300 UTC we will be taking the Opalstack dashboard and API offline for a major upgrade.

    The maintenance window is 4 hours, but we expect to have the job done in less time.

    We'll update this post as the maintenance progresses.

    2021-06-24 04:31 UTC: The dashboard and API are down for the upgrade at this time.

    2021-06-24 06:53 UTC: The upgrade is taking longer than expected and we've extended the window for another hour.

    2021-06-24 08:09 UTC: we hope to have this wrapped up within the next 20 minutes, by 0830 UTC.

    2021-06-24 08:30 UTC: the upgade is complete! 🎉

German SMTP offline
  • Resolved
    Resolved

    The certificate has been updated and SMTP is now working normally. We're reviewing and revising our internal procedures to minimize the chance of this happening again in the future.

  • Investigating
    Investigating

    German SMTP was offline due to a problem with the server certificate.

Opal7 rebooted unexpectedly
  • Resolved
    Resolved

    The maintenance was confirmed completed on Saturday shortly after our last update.

  • Monitoring
    Monitoring

    Opal7 is back online and operating normally. We're awaiting confirmation from the data center that the maintenance is complete.

  • Update
    Update

    Opal7 is offline for emergency RAM replacement at this time.

  • Identified
    Identified

    The cause of the reboot was a faulty memory module. We're scheduling emergency maintenance at this time.

  • Investigating
    Investigating

    Opal7 (shared hosting in Phoenix) was unexpectedly rebooted. We're investigating the cause.

    The server is online and operating normally at this time.

Apache was briefly down on opal5
  • Resolved
    Resolved

    The Apache httpd service on opal5 was briefly down because the system /tmp directory was full.

    The service was down for 2-3 minutes. We've identified the customer responsible for the problem and will work with them to ensure it does not happen again.

German IMAP/POP3 service offline
  • Resolved
    Resolved

    The Germain mail service is back online. The outage was caused by human error when deploying a new certificate.

  • Investigating
    Investigating

    The German IMAP/POP3 service is currently offline. We're currently investigating.

Jun 2021 to Aug 2021

Next