Servers: opal1.opalstack.com
At around 2021-02-18 11:00 UTC today the /tmp partition on opal1.opalstack.com ran out of available space. This disrupted service for applications that needed to write to temporary storage including PHP apps and the MariaDB database service.
At the time of the event our monitoring system was only checking the status of these system services and not their operational performance, so no alert was generated.
We corrected the problem at approximately 14:00 UTC and the server has been stable since that time.
We’re updating our monitoring system to perform the tests necessary to alert us to the conditions that caused this incident which will help us avoid this sort of extended disruption. (11:00 UTC — Feb 18)