Opalstack - Emergency MariaDB maintenance for opal3 – Maintenance details

Emergency MariaDB maintenance for opal3

Completed
Scheduled for July 10, 2023 at 4:30 AM – 7:30 AM

Affects

General Infrastructure

Updates
  • Completed
    July 10, 2023 at 7:30 AM
    Completed
    July 10, 2023 at 7:30 AM
    The maintenance is complete.
  • In progress
    July 10, 2023 at 4:30 AM
    In progress
    July 10, 2023 at 4:30 AM

    Maintenance is now in progress

  • Planned
    July 10, 2023 at 4:30 AM
    Planned
    July 10, 2023 at 4:30 AM

    On Sunday, 09 July 2023 at 9:30 PM to 11:30 PM US/Pacific (04:30 to 06:30 UTC) we'll be taking the managed MariaDB database service offline for maintenance.

    During the maintenance sites and applications which use MariaDB (including WordPress sites) will not function until the maintenance is complete.

    We apologize for the short notice and any inconvenience. If you have any questions or concerns regarding the maintenance then please contact our support team.

    Update: During the maintenance an unrecoverable error occurred and the backup taken minutes before the maintenance window could not be restored. We are restoring databases from the newest backup which were taken about 18 hours before the maintenance started. That restore is in progress now and once the restoration is complete database access will resume as normal. Another update will follow in approximately 1 hour.

    Final Update: All databases have been restored and should be online now. We sincerely apologize for the error. In our tests of the maintenance process on other servers we did not see any of the errors we saw during the actual maintenance. We've updated the maintenance procedure and added additional redundant backups (in addition to the 2 that were already in place) so that this same failure is less likely to happen in the future.