Opalstack - Notice history

There have been no further issues.

The MariaDB server is back online and fully functional. We are continuing to monitor the server.

The restore is complete and we're running a final check on the entire DB service.

We are still restoring databases to the MariaDB server.

The databases have been exported, the InnoDB data and log files have been rebuilt, and we are currently importing all databases.

The MariaDB InnoDB data files and logs have become corrupt and our efforts to fix the corruption have not succeeded. In order to fix this we are exporting the databases, rebuilding the InnoDB data and log files, and then importing the databases.

The issue is resolved and opal4 is back online.

We've identified and shut down several websites with insecure contact forms which were sending spam. The site operators have been notified and the US SMTP is no longer blacklisted.

We've confirmed that the maintenance is complete and there are no further issues at this time.

Opal12 is back online and we're awaiting confirmation from the data center that the maintenance is complete.

Opal12 is now offline for scheduled maintenance.

Opal12 will be going offline for troubleshooting and repair in approximately 50 minutes at 9PM US/Eastern time.

We've scheduled maintenance downtime on opal12 to correct the issue with the network controller. Details are at https://status.opalstack.com/schedules/36.

We suspect that the server's network controller is failing. We're now working with the data center team to have the problem corrected.

Opal5 is stable following the reboot. We'll continue to monitor.

Opal5 is back online, we're still investigating.

We're rebooting opal5 at this time.

The high load condition has returned so we're continuing our investigation.

Opal5 is back online. The problem was high load caused by heavy disk I/O activity due to high swap usage by a customer process. We've increased available swap and are working with the customer to resolve the issue.

Opal10 is back following an unexpected reboot. The cause is under investigation.

Opal10 is not responding, we're looking into it.

All repairs are complete and all DE mailboxes have been restored.

If you are a DE mail customer and are still seeing issues with your IMAP/POP3 email then please email Opalstack support for further assistance.

The repairs for message dates and mailbox permissions are 75-80% complete.

The repairs for dates and mailbox permissions are still in progress. We expect them to be completed today.

The restore is complete.

We're still working on lingering issues related to message dates and mailbox permissions which may take another day or two to fully resolve.

We'll continue to post updates until the issues are resolved.

The restore is 99% complete.

The restore is 99% complete.

Once the restore has completed, there may be sporadic service disruptions for individual mailboxes as we identify lingering issues and correct them.

The restore is 98-99% complete.

The restore is 93-99% complete.

The recovery is 85-95% complete.

The recovery is 80-90% complete.

The recovery is 75-85% complete.

The recovery is 70-80% complete.

The recovery is 65-75% complete. Note that there are several parallel restore processes so the overall recovery progress doesn't reflect a percentage of the alphabet with regard to mailbox names.

The recovery is 60-70% complete.

The recovery is 45-60% complete.

The recovery is 30-40% complete. We have to estimate with a range because of the parallel restore processes.

The recovery is 24% complete.

The recovery from backup is still in progress, albeit running slowly at only 12% complete.

We believe that we'll be able to recover most of the data from the failed server, so we're restoring that data in parallel with the backup recovery. This should speed up the overall recovery and provide the most recent data from prior to the failure.

The recovery is in still progress.

We estimate that it is 11% complete and it is currently restoring mailboxes that being with "as".

DE webmail access is now working.

The mailbox restore is still in progress.

We're still noticing some login issues for webmail and are troubleshooting that now.

We do not have high expectations for the outcome of the filesystem check, so we've deployed a new DE IMAP server and have re-created all DE mailboxes as new empty mailboxes on that server.

Customers should now be able to log in to our DE mail services to send and receive new mail. No changes to your email client settings should be required.

We'll begin restoring the original mailbox contents ASAP from the most recent backup made on 30 March 2022 (the failure occurred while the 31 March backup was still in progress). The restore may take several days to complete, so we'll keep this post updated until it is done.

We'll attempt to recover mail from 31 March 2022 when the filesystem repairs on the failed server are complete.