{
  "is": "issue",
  "title": "OPAL4 - MariaDB has crashed.",
  "body": "\u003cp\u003e\u003cem\u003eFixed\u003c/em\u003e - We found and corrected a problem with database permissions and all database users should be able to connect now. \n  \u003cspan class=\"faded\"\u003e(17:00 UTC — Jul 31)\u003c/span\u003e\n\n\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eInvestigating\u003c/em\u003e - We\u0026rsquo;re getting reports that some users still cannot connect to their databases so we are investigating that now. \n  \u003cspan class=\"faded\"\u003e(16:34 UTC — Jul 31)\u003c/span\u003e\n\n\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eFixed\u003c/em\u003e - Databases have been restored and should be fully operational. \n  \u003cspan class=\"faded\"\u003e(16:15 UTC — Jul 31)\u003c/span\u003e\n\n\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eIdentified\u003c/em\u003e - The restore process is at 99% and the last databases are being restored. \n  \u003cspan class=\"faded\"\u003e(15:25 UTC — Jul 31)\u003c/span\u003e\n\n\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eIdentified\u003c/em\u003e - The database restores are 66% complete. \n  \u003cspan class=\"faded\"\u003e(14:45 UTC — Jul 31)\u003c/span\u003e\n\n\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eIdentified\u003c/em\u003e - The database restores are 51% complete. \n  \u003cspan class=\"faded\"\u003e(13:58 UTC — Jul 31)\u003c/span\u003e\n\n\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eIdentified\u003c/em\u003e - The database restores are in progress and are ~25% complete. \n  \u003cspan class=\"faded\"\u003e(13:21 UTC — Jul 31)\u003c/span\u003e\n\n\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eIdentified\u003c/em\u003e - The InnoDB data corruption is too bad to recover data from the database server. Tables are unreadable causing data dumps or repairs to the tables to be impossible.\u003c/p\u003e\n\u003cp\u003eWe have started the database repair process from our backups that were 3-5 hours old when the server went offline. We will update this post with our progress. \n  \u003cspan class=\"faded\"\u003e(12:26 UTC — Jul 31)\u003c/span\u003e\n\n\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eIdentified\u003c/em\u003e - The data corruption on the server is in the innodb data which means there is potentially corrupted data across multiple databases due to how MySQL/MariaDB store data. We are still trying to recover and repair the data in the MariaDB server. If the data is too corrupt to repair we will restore the data from the latest backups. The latest backups were taken between 3 and 5 hours (depending on the dump time) before the database server crashed. \n  \u003cspan class=\"faded\"\u003e(12:07 UTC — Jul 31)\u003c/span\u003e\n\n\u003c/p\u003e\n\u003cp\u003eMariaDB has crashed on OPAL4. We are investigating the cause of the crash and working to restore services. \n  \u003cspan class=\"faded\"\u003e(10:19 UTC — Jul 31)\u003c/span\u003e\n\n\u003c/p\u003e\n",
  "createdAt": "2024-07-31 10:19:00 +0000 UTC",
  "lastMod": "2024-07-31 10:19:00 +0000 UTC",
  "permalink": "https://opalstackstatus.com/issues/2024-07-31-opal4-mariadb-has-crashed/",
  "severity": "down",
  "resolved": true,
  "informational": false,
  "resolvedAt": "2024-07-31 17:00:54",
  "affected": ["Shared: Dallas (USA)"],
  "filename": "2024-07-31-opal4-mariadb-has-crashed.md"
}