Opalstack - Notice history

All systems operational

Notice history

Feb 2022

MariaDB restarted on Opal8
  • Resolved
    Resolved

    At 2022-02-16 03:00 UTC, the MariaDB database service on opal8.opalstack.com was restarted due to high CPU utilization. This may have resulted in database connection errors or similar errors on websites that use the MariaDB service for their databases.

    The service has remained stable following the restart.

Service disruption caused by firewall issue
  • Resolved
    Resolved

    About an hour ago we deployed a firewall update that had the unintended side effect of blocking our resolving DNS servers.

    This caused various issues on customer sites on shared hosting, usually showing up as a "504 Gateway Timeout" error.

    The issue has been resolved and all customer sites should be working now as they were prior to the change.

Emergency maintenance on opal6.opalstack.com
  • Completed
    February 08, 2022 at 10:56 PM
    Completed
    February 08, 2022 at 10:56 PM
    The maintenance is complete.
  • In progress
    February 08, 2022 at 10:30 PM
    In progress
    February 08, 2022 at 10:30 PM

    Maintenance is now in progress

  • Planned
    February 08, 2022 at 10:30 PM
    Planned
    February 08, 2022 at 10:30 PM

    On 8 February 2022 at 22:30 UTC opal6.opalstack.com will be taken offline for emergency maintenance related to the outage earlier today.

    The expected downtime is 30 minutes.

    We'll update this post when the maintenance begins and again when it is complete.

    • 2022-02-08 22:30: Opal6 is offline for maintenance.
    • 2022-02-08 22:56: The maintenance is complete and opal6 is back online.
opal6 offline
  • Resolved
    Resolved

    The maintenance is complete and opal6 is back online.

  • Update
    Update

    Opal6 is now offline for scheduled emergency maintenance. The expected downtime is 30 minutes.

  • Update
    Update

    The data center team suspects that a fault in the server's physical power switch caused the server to power down unexpectedly.

    We've scheduled emergency maintenance for 2022-02-08 22:30 UTC (tonight) to investigate and replace the switch if necessary.

  • Monitoring
    Monitoring

    Opal6 is back online. We're still awaiting word from the data center to determine why the machine had been powered off.

  • Identified
    Identified

    We discovered that opal6 had been powered down unexpectedly. The machine is rebooting at this time and we're awaiting an update from the data center as to why the machine was powered down.

  • Investigating
    Investigating

    Opal6 is currently offline, we're looking into it.

Several DE VPS offline due to emergency maintenance
  • Resolved
    Resolved

    The emergency maintenance has been completed.

  • Monitoring
    Monitoring

    All DE VPS are back online at this time. We're awaiting confirmation from the data center that the maintenance is complete.

  • Investigating
    Investigating

    At this time several customer VPS in Germany are offline due to emergency maintenance in the Frankfurt data center.

    The estimated time of completion is 22:15 UTC (approximately 1 hour from the time of this post).

    We'll update this post when more information is available.

Jan 2022

Changes to Apache/PHP-CGI apps
  • Completed
    September 10, 2022 at 12:00 PM
    Completed
    September 10, 2022 at 12:00 PM
    The maintenance is complete.
  • In progress
    January 31, 2022 at 12:00 PM
    In progress
    January 31, 2022 at 12:00 PM

    Maintenance is now in progress

  • Planned
    January 31, 2022 at 12:00 PM
    Planned
    January 31, 2022 at 12:00 PM

    On 31 January 2022 we'll begin a phased rollout of a change to our Apache/PHP-CGI setup to transition from PHP-CGI to PHP-FPM.

    This change will improve the performance of all Apache/PHP applications hosted at Opalstack. There are a high number of these apps installed throughout the platform, so this change should also improve the overall performance of our servers for all customers.

    We do not anticipate any disruptions related to this change.

    We'll update this maintenance item when the rollout begins and as it progresses.

    If you have any questions or concerns regarding this change then please email them to support@opalstack.com.

    • 2022-02-05 14:18 UTC - we've been rolling out these changes slowly across a small number of customer VPS and so far no problems have been noted. We'll be ramping up the rollout over the coming week.

    • 2022-05-09 14:00 UTC - all VPS have been updated to use the Apache/PHP-FPM stack. We've also deployed the changes on 1 shared server (opal14, SFO shared) as of 2022-05-09 0300 UTC and will monitor that server for a couple of days before moving on to the other shared servers.

    • 2022-05-22 11:00 UTC opal1 and opal12 have been updated to use the Apache/PHP-FPM stack.

    • 2022-09-03 08:00 UTC opal5 and opal6 have been updated to use the Apache/PHP-FPM stack.

    • 2022-09-09 08:00 UTC opal10 and opal11 have been updated to use the Apache/PHP-FPM stack.

    • 2022-09-10 12:00 UTC all remaining servers have been updated to use the Apache/PHP-FPM stack.

    This change is now complete.

SFO VPS service disruption
  • Resolved
    Resolved

    We've received confirmation that the incident is resolved.

  • Update
    Update

    All services are back online and the pending items in the dashboard control loop have been completed.

    We're awaiting final confirmation from our provider that the issue is fully resolved.

  • Monitoring
    Monitoring

    The data center team has identified the issue and we're starting to see services come back online.

  • Identified
    Identified

    Our provider has notified us that this is a cloud storage connectivity issue in their SFO cloud service.

    We'll post another update when we have more information.

  • Investigating
    Investigating

    Several San Francisco VPS are currently experiencing connectivity issues, including our community forum, US webmail, and US SMTP service.

    A small number of customer web hosting VPS are also affected.

    This same issue is causing delays in our dashboard control loop, so customer may see new/updated items in their dashboard stuck in a pending state until this is resolved.

    We're looking into this now and will follow up when more information is available.

DDOS on opal1
  • Resolved
    Resolved

    We've seen no further issues on opal1.

  • Monitoring
    Monitoring

    We've mitigated the attack and server conditions are back to normal. We'll continue to monitor.

    We'll notify the targeted site's operator and will work with them to prevent further issues.

  • Investigating
    Investigating

    Opal1 (shared hosting in Washington DC) is currently experiencing elevated load due to a distributed denial of service attack.

    We're working to mitigate the attack at this time.

Power cycle on German SMTP
  • Completed
    January 18, 2022 at 11:09 PM
    Completed
    January 18, 2022 at 11:09 PM
    The maintenance is complete.
  • In progress
    January 18, 2022 at 11:00 PM
    In progress
    January 18, 2022 at 11:00 PM

    Maintenance is now in progress

  • Planned
    January 18, 2022 at 11:00 PM
    Planned
    January 18, 2022 at 11:00 PM

    On 18 January 2022 at 23:00 UTC we'll be power-cycling the German SMTP server.

    While the server is restarting the German SMTP service will be unavailable.

    We expect the total downtime to be 15 minutes or less.

    • 2022-01-18 23:04 - we are shutting down the server at this time.
    • 2022-01-18 23:09 - the server is back online and the maintenance is complete.
Emergency network maintenance impacting Washington DC VPS service
  • Completed
    January 14, 2022 at 9:15 AM
    Completed
    January 14, 2022 at 9:15 AM
    The maintenance is complete.
  • In progress
    January 14, 2022 at 9:00 AM
    In progress
    January 14, 2022 at 9:00 AM

    Maintenance is now in progress

  • Planned
    January 14, 2022 at 9:00 AM
    Planned
    January 14, 2022 at 9:00 AM

    On Friday, 14 January 2022 at 4:00 AM US/Eastern (2022-01-14 0900 UTC) our Washington DC VPS data center will be performing network maintenance which may interrupt connectivity for up to 5 minutes for the following servers:

    • vps3.opalstack.com
    • vps7.opalstack.com
    • vps8.opalstack.com
    • vps10.opalstack.com
    • vps12.opalstack.com
    • vps18.opalstack.com
    • vps26.opalstack.com
    • vps27.opalstack.com
    • vps29.opalstack.com

    The affected servers will remain running during the loss of connectivity, and no customer action will be required following the maintenance.

Dec 2021

Opal4 not responding
  • Resolved
    Resolved

    Opal4 is back online. We've reached out to our network provider for details regarding the outage.

  • Investigating
    Investigating

    opal4.opalstack.com (Dallas shared hosting) is not responding. We're looking into it.

German SMTP blocked by Microsoft
  • Resolved
    Resolved

    We've still not received confirmation from Microsoft that the block was lifted, but we've seen no further blocking activity since 23:50 UTC on 20 December 2021 so we consider this incident to be closed.

  • Monitoring
    Monitoring

    Microsoft began accepting mail from smtp.de.opalstack.com as of 23:50 UTC on 20 December 2021.

    We're still awaiting official confirmation that the block has been lifted.

  • Investigating
    Investigating

    On December 18 2021 at 19:30 UTC Microsoft began blocking mail sent from smtp.de.opalstack.com to domains that use the Microsoft 365 email system.

    Microsoft's own IP reputation and junk mail reporting systems indicate that there are no problems with smtp.de.opalstack.com so we've reached out to them for answers.

    We'll update this post as soon as more information is available.

    In the interim as a possible workaround customers may create a new mailbox on our US mail server and use that temporarily for their outgoing email authentication.

Intermittent connectivity to San Francisco servers including US webmail
  • Resolved
    Resolved

    The maintenance was completed as scheduled and we're seeing no further issues at this time.

  • Investigating
    Investigating

    Network maintenance in one of our San Francisco facilities is causing intermittent connectivity issues for some customers on servers in that location.

    This also affects our US webmail server which is located in San Francisco.

    The maintenance is expected to be complete at 17:30 UTC Dec 11 2021 (9:30 AM US Pacific).

Dec 2021 to Feb 2022

Next