Servers: opal3.opalstack.com
On Saturday 10 December at around 0800 UTC (midnight US/Pacific) opal3.opalstack.com in San Francisco became unresponsive. After several unsuccessful attempts to access the server for troubleshooting, we rebooted the server and it was back online around 0900 UTC (1am US/Pacific).
The outage appears to have been triggered by a very rapid spike in system memory usage. The high load unfortunately interrupted the system logging facilities so we weren’t able to precisely identify the cause, but we suspect that it was due to an buggy Apache rewrite rule in a customer application. (00:00 UTC — Dec 10)