{
  "is": "issue",
  "title": "Opal3 offline",
  "body": "\u003cp\u003e\u003cem\u003eFixed\u003c/em\u003e - There have been no further problems since the last update. \n  \u003cspan class=\"faded\"\u003e(00:24 UTC — Jan 13)\u003c/span\u003e\n\n\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eWatching\u003c/em\u003e - As of 2020-01-12 22:20 UTC all Opal3 shell user home directories, PostgreSQL databases, and related control panel data have been restored.\u003c/p\u003e\n\u003cp\u003eAll customer sites are back online and the server is operating normally at this time. We\u0026rsquo;ll continue to monitor.\u003c/p\u003e\n\u003cp\u003eWe were not able to recover shell user passwords, so Opal3 customers will need to reset their shell user passwords via the control panel. Instructions are available in our documentation: \u003ca href=\"https://help.opalstack.com/article/45/managing-shell-users#changing-a-shell-users-password\"\u003eChanging a Shell User\u0026rsquo;s Password\u003c/a\u003e \n  \u003cspan class=\"faded\"\u003e(22:25 UTC — Jan 12)\u003c/span\u003e\n\n\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eIdentified\u003c/em\u003e - At approximately 17:40 UTC (9:40 AM Pacific) on 12 January 2020 we ran a live test of our upcoming migration tool on opal3, our shared hosting server in San Francisco.\u003c/p\u003e\n\u003cp\u003eA serious bug in the migration tool resulted in the loss of all customer home directories and PostgreSQL databases on opal3.\u003c/p\u003e\n\u003cp\u003eWe\u0026rsquo;re presently restoring opal3 from the most recent backup which was completed at 00:14 AM, about 7.5 hours prior to the incident.\u003c/p\u003e\n\u003cp\u003eWe\u0026rsquo;ll update this incident with more information when the restoration is complete. \n  \u003cspan class=\"faded\"\u003e(19:17 UTC — Jan 12)\u003c/span\u003e\n\n\u003c/p\u003e\n\u003cp\u003eOpal3 was offline. \n  \u003cspan class=\"faded\"\u003e(17:40 UTC — Jan 12)\u003c/span\u003e\n\n\u003c/p\u003e\n",
  "createdAt": "2020-01-12 17:40:00 +0000 UTC",
  "lastMod": "2020-01-12 17:40:00 +0000 UTC",
  "permalink": "https://opalstackstatus.com/issues/2020-01-12-opal3-offline/",
  "severity": "down",
  "resolved": true,
  "informational": false,
  "resolvedAt": "2020-01-13 00:24:07",
  "affected": ["Shared: San Francisco (USA)"],
  "filename": "2020-01-12-opal3-offline.md"
}