Opalstack - Notice history

All systems operational

Notice history

Jul 2023

Opal8 service disruption
  • Resolved
    Resolved

    opal8.opalstack.com (San Francisco shared hosting) became non-responsive at 3:30 PM US/Pacific time. We suspected a temporary high load condition, but after several minutes the server remained unresponsive. At that point we decided to reboot the server.

    The server is online and operating normally at this time. We'll work to identify the site which caused the high load and will then work with that customer to guard against future incidents.

Emergency MariaDB maintenance for opal3
  • Completed
    July 10, 2023 at 7:30 AM
    Completed
    July 10, 2023 at 7:30 AM
    The maintenance is complete.
  • In progress
    July 10, 2023 at 4:30 AM
    In progress
    July 10, 2023 at 4:30 AM

    Maintenance is now in progress

  • Planned
    July 10, 2023 at 4:30 AM
    Planned
    July 10, 2023 at 4:30 AM

    On Sunday, 09 July 2023 at 9:30 PM to 11:30 PM US/Pacific (04:30 to 06:30 UTC) we'll be taking the managed MariaDB database service offline for maintenance.

    During the maintenance sites and applications which use MariaDB (including WordPress sites) will not function until the maintenance is complete.

    We apologize for the short notice and any inconvenience. If you have any questions or concerns regarding the maintenance then please contact our support team.

    Update: During the maintenance an unrecoverable error occurred and the backup taken minutes before the maintenance window could not be restored. We are restoring databases from the newest backup which were taken about 18 hours before the maintenance started. That restore is in progress now and once the restoration is complete database access will resume as normal. Another update will follow in approximately 1 hour.

    Final Update: All databases have been restored and should be online now. We sincerely apologize for the error. In our tests of the maintenance process on other servers we did not see any of the errors we saw during the actual maintenance. We've updated the maintenance procedure and added additional redundant backups (in addition to the 2 that were already in place) so that this same failure is less likely to happen in the future.

Jun 2023

Partial DNS outage.
  • Resolved
    Resolved

    The DNS issue is fully resolved now.

    Note that while this issue affected only the Opalstack US name servers and only for the domain opalstack.com. Since all of our regional services use the opalstack.com domain, effects of this issue may have been seen by customers outside of the US.

    If you're still having trouble connecting to Opalstack services then please try flushing your local DNS cache

  • Investigating
    Investigating

    During a routine DNS update our Secondary DNS provider captured an incomplete zone update and propagated it. This means that various DNS services for opalstack.com services (DNS, Mail, and Web) are inconsistently available.

    We have removed the Secondary provider and we expect this to resolve over the next hour as the DNS TTL caches time out.

Scheduled downtime for opal5 relocation
  • Completed
    June 15, 2023 at 10:30 AM
    Completed
    June 15, 2023 at 10:30 AM
    The maintenance is complete.
  • In progress
    June 15, 2023 at 7:00 AM
    In progress
    June 15, 2023 at 7:00 AM

    Maintenance is now in progress

  • Planned
    June 15, 2023 at 7:00 AM
    Planned
    June 15, 2023 at 7:00 AM

    On 15 June 2023 at 0900 CET (UTC +2) our shared web server opal5.opalstack.com will be physically relocated to a new data center facility in Frankfurt. The server will be offline during the relocation and the expected downtime is 8 hours.

    During the downtime the server will be taken offline and relocated to a new data center facility in Frankfurt. Customer websites hosted on opal5 will be offline during the relocation.

    Email is hosted on separate hardware and is not affected by the relocation.

    We'll update our status page at http://status.opalstack.com/ at the beginning of the relocation window. If you would like to be informed of system status updates automatically then please subscribe at https://status.opalstack.com/subscribe.

    We realize that the timing of this relocation and the associated downtime are not ideal. If you would like to have less of an impact on your services and have your account migrated to a different server in advance of the scheduled relocation, or if you have any other questions or concerns, then please email our support team via support@opalstack.com and one of our team will assist you.

Intermittent network connectivity in Washington DC
  • Resolved
    Resolved

    The issue has been resolved.

  • Investigating
    Investigating

    There have been intermittent network connectivity issues this morning in our Washington DC shared hosting facility, typically presenting as a network dropout lasting 1-2 seconds. The issues could potentially affect the opal1, opal10, and opal12 shared hosting servers.

    The data center team believes they've identified the issue and are taking corrective action. We'll update this post when more information is available.

May 2023

Intermittent VPS connectivity in Frankfurt
  • Resolved
    Resolved

    There have been no further issues so we consider this to be resolved.

  • Monitoring
    Monitoring

    The fix has been implemented. We'll continue to monitor.

  • Investigating
    Investigating

    The Frankfurt VPS data center is currently experiencing intermittent network connectivity issues. The data center team have identified the issue and are presently working on a solution.

    We'll update this post when more information is available.

Opal9 offline for scheduled maintenance
  • Resolved
    Resolved

    The maintenance is complete and opal9 is back online.

  • Investigating
    Investigating

    opal9.opalstack.com is offline for scheduled maintenance.

MariaDB upgrades
  • Completed
    May 15, 2023 at 8:00 AM
    Completed
    May 15, 2023 at 8:00 AM
    The maintenance is complete.
  • In progress
    May 15, 2023 at 4:00 AM
    In progress
    May 15, 2023 at 4:00 AM

    Maintenance is now in progress

  • Planned
    May 15, 2023 at 4:00 AM
    Planned
    May 15, 2023 at 4:00 AM

    On Monday 15 May 2023 between 0400 UTC and 0800 UTC we'll be upgrading MariaDB to v10.6 on several servers. The database services will be down for approximately 5 minutes as each server is upgraded.

    The affected servers are:

    • Shared hosting: opal1.opalstack.com, opal3.opalstack.com, opal4.opalstack.com, opal5.opalstack.com, opal6.opalstack.com, opal7.opalstack.com, opal8.opalstack.com, opal9.opalstack.com, opal15.opalstack.com
    • VPS: vps6.opalstack.com, vps28.opalstack.com, vps51.opalstack.com, vps74.opalstack.com, vps95.opalstack.com, vps121.opalstack.com, vps130.opalstack.com, vps147.opalstack.com
US SMTP was briefly disrupted
  • Resolved
    Resolved

    The US SMTP service was disrupted for approximately 5 minutes during system maintenance.

US DNS under attack
  • Resolved
    Resolved

    There have been no further issues since our last update.

  • Monitoring
    Monitoring

    We've mitigated the worst of the attack and all services are online at this time. DNS lookups are still a bit slow so our troubleshooting is ongoing.

  • Investigating
    Investigating

    Our US DNS infrastructure under attack, with impacts throughout our hosting services in all regions. We're working to resolve the issue at this time.

May 2023 to Jul 2023

Next