OPAL4 - MariaDB has crashed.

Fixed - We found and corrected a problem with database permissions and all database users should be able to connect now. (17:00 UTC — Jul 31)

Investigating - We’re getting reports that some users still cannot connect to their databases so we are investigating that now. (16:34 UTC — Jul 31)

Fixed - Databases have been restored and should be fully operational. (16:15 UTC — Jul 31)

Identified - The restore process is at 99% and the last databases are being restored. (15:25 UTC — Jul 31)

Identified - The database restores are 66% complete. (14:45 UTC — Jul 31)

Identified - The database restores are 51% complete. (13:58 UTC — Jul 31)

Identified - The database restores are in progress and are ~25% complete. (13:21 UTC — Jul 31)

Identified - The InnoDB data corruption is too bad to recover data from the database server. Tables are unreadable causing data dumps or repairs to the tables to be impossible.

We have started the database repair process from our backups that were 3-5 hours old when the server went offline. We will update this post with our progress. (12:26 UTC — Jul 31)

Identified - The data corruption on the server is in the innodb data which means there is potentially corrupted data across multiple databases due to how MySQL/MariaDB store data. We are still trying to recover and repair the data in the MariaDB server. If the data is too corrupt to repair we will restore the data from the latest backups. The latest backups were taken between 3 and 5 hours (depending on the dump time) before the database server crashed. (12:07 UTC — Jul 31)

MariaDB has crashed on OPAL4. We are investigating the cause of the crash and working to restore services. (10:19 UTC — Jul 31)