Opalstack - Notice history

Opal5 is stable following the reboot. We'll continue to monitor.

Opal5 is back online, we're still investigating.

We're rebooting opal5 at this time.

The high load condition has returned so we're continuing our investigation.

Opal5 is back online. The problem was high load caused by heavy disk I/O activity due to high swap usage by a customer process. We've increased available swap and are working with the customer to resolve the issue.

Opal10 is back following an unexpected reboot. The cause is under investigation.

Opal10 is not responding, we're looking into it.

All repairs are complete and all DE mailboxes have been restored.

If you are a DE mail customer and are still seeing issues with your IMAP/POP3 email then please email Opalstack support for further assistance.

The repairs for message dates and mailbox permissions are 75-80% complete.

The repairs for dates and mailbox permissions are still in progress. We expect them to be completed today.

The restore is complete.

We're still working on lingering issues related to message dates and mailbox permissions which may take another day or two to fully resolve.

We'll continue to post updates until the issues are resolved.

The restore is 99% complete.

The restore is 99% complete.

Once the restore has completed, there may be sporadic service disruptions for individual mailboxes as we identify lingering issues and correct them.

The restore is 98-99% complete.

The restore is 93-99% complete.

The recovery is 85-95% complete.

The recovery is 80-90% complete.

The recovery is 75-85% complete.

The recovery is 70-80% complete.

The recovery is 65-75% complete. Note that there are several parallel restore processes so the overall recovery progress doesn't reflect a percentage of the alphabet with regard to mailbox names.

The recovery is 60-70% complete.

The recovery is 45-60% complete.

The recovery is 30-40% complete. We have to estimate with a range because of the parallel restore processes.

The recovery is 24% complete.

The recovery from backup is still in progress, albeit running slowly at only 12% complete.

We believe that we'll be able to recover most of the data from the failed server, so we're restoring that data in parallel with the backup recovery. This should speed up the overall recovery and provide the most recent data from prior to the failure.

The recovery is in still progress.

We estimate that it is 11% complete and it is currently restoring mailboxes that being with "as".

DE webmail access is now working.

The mailbox restore is still in progress.

We're still noticing some login issues for webmail and are troubleshooting that now.

We do not have high expectations for the outcome of the filesystem check, so we've deployed a new DE IMAP server and have re-created all DE mailboxes as new empty mailboxes on that server.

Customers should now be able to log in to our DE mail services to send and receive new mail. No changes to your email client settings should be required.

We'll begin restoring the original mailbox contents ASAP from the most recent backup made on 30 March 2022 (the failure occurred while the 31 March backup was still in progress). The restore may take several days to complete, so we'll keep this post updated until it is done.

We'll attempt to recover mail from 31 March 2022 when the filesystem repairs on the failed server are complete.

The issue was due to a problem with the server's /tmp filesystem. The problem has been corrected and all Opal7 services are now working normally.

The emergency maintenance has been completed.

All DE VPS are back online at this time. We're awaiting confirmation from the data center that the maintenance is complete.