MX (DE) on Opalstack Status

[Resolved] DE forwarding main server blocked by Microsoft

Sat, 21 Sep 2024 23:09:03 +0000

Fixed - We’ve seen no further issues since our previous update. (17:47 UTC — Sep 26)

One of our DE MX servers, mx2.de.opalstack.com on IP 178.162.221.165, was recently blocked by Microsoft. As a result, mail sent to addresses which forward to Microsoft services like hotmail.com, outlook.com, and their international variants would be bounced back to the original sender with an error message.

We’ve already resolved this issue with Microsoft but they say that it may take 24-48 hours for the block to be fully lifted.

We’ll update this post after we’ve seen no further blocks in a 24 hour period. (23:09 UTC — Sep 21)

[Resolved] EU mail system blocked by Hotmail

Thu, 15 Feb 2024 15:21:00 +0000

Fixed - All blocks have been lifted. (03:30 UTC — Feb 16)

Watching - mx2.de.opalstack.com was added to the block list as well, we have submitted the delisting request and are awaiting action from Microsoft. (18:36 UTC — Feb 15)

Part of our EU mail infrastructure was blocked by Hotmail. The affected servers were:

smtp2.de.opalstack.com (outgoing mail)
mx1.de.opalstack.com (incoming mail and forwarding)

We’ve already dealt with the issue and Microsoft have confirmed that the servers have been unblocked, but it may take a few more hours to take full effect.

We’ll update this post when we’ve confirmed that mail is no longer being blocked. (15:21 UTC — Feb 15)

[Resolved] German MX servers are on Hotmail block list

Sat, 06 Jan 2024 22:15:00 +0000

Fixed - The MX servers have been removed from the block list. (18:06 UTC — Jan 7)

mx1.de.opalstack.com and mx2.de.opalstack.com are listed on the Hotmail block list. As a result, mail forwarded to Hotmail domains like hotmail.com and outlook.com is being bounced back to the sender.

We’re working to have the servers removed from the block list and will update this post when that is done.

In the meantime, customers who have forwarding addresses which forward to Hotmail domains are advised to forward to an alternate address or direct mail to an Opalstack mailbox to ensure that they receive their mail. Instructions for updating email addresses are available in our user guide at: Editing Email Addresses (22:15 UTC — Jan 6)

[Resolved] German IMAP/POP3 server offline for scheduled relocation

Tue, 24 Jan 2023 08:05:00 +0000

Fixed - The backlog of incoming mail has been delivered and the relocation is complete. (17:01 UTC — Jan 24)

Watching - The DE mail server is back online at this time and the backlog of incoming mail is now being processed. We’re awaiting the final word from the data center that the relocation is complete. (16:03 UTC — Jan 24)

As previously announced, our German IMAP/POP3 server will be offline for relocation to a new data center facility from 0800 to 1600 UTC on 24 January 2023: https://status.opalstack.com/schedules/38 (08:05 UTC — Jan 24)

[Resolved] Germany mail server offline.

Fri, 01 Apr 2022 08:30:00 +0000

Fixed - All repairs are complete and all DE mailboxes have been restored.

If you are a DE mail customer and are still seeing issues with your IMAP/POP3 email then please email Opalstack support for further assistance. (13:50 UTC — Apr 9)

Watching - The repairs for message dates and mailbox permissions are 75-80% complete. (20:37 UTC — Apr 7)

Watching - The repairs for dates and mailbox permissions are still in progress. We expect them to be completed today. (13:40 UTC — Apr 7)

Watching - The restore is complete.

We’re still working on lingering issues related to message dates and mailbox permissions which may take another day or two to fully resolve.

We’ll continue to post updates until the issues are resolved. (02:32 UTC — Apr 6)

Watching - The restore is 99% complete. (01:42 UTC — Apr 6)

Watching - The restore is 99% complete.

Once the restore has completed, there may be sporadic service disruptions for individual mailboxes as we identify lingering issues and correct them. (00:48 UTC — Apr 6)

Watching - The restore is 98-99% complete. (20:00 UTC — Apr 5)

Watching - The restore is 93-99% complete. (13:29 UTC — Apr 5)

Watching - The recovery is 85-95% complete. (01:22 UTC — Apr 5)

Watching - The recovery is 80-90% complete. (18:29 UTC — Apr 4)

Watching - The recovery is 75-85% complete. (13:28 UTC — Apr 4)

Watching - The recovery is 70-80% complete. (01:33 UTC — Apr 4)

Watching - The recovery is 65-75% complete. Note that there are several parallel restore processes so the overall recovery progress doesn’t reflect a percentage of the alphabet with regard to mailbox names. (19:41 UTC — Apr 3)

Watching - The recovery is 60-70% complete. (15:55 UTC — Apr 3)

Watching - The recovery is 45-60% complete. (02:22 UTC — Apr 3)

Watching - The recovery is 30-40% complete. We have to estimate with a range because of the parallel restore processes. (17:00 UTC — Apr 2)

Watching - The recovery is 24% complete. (13:30 UTC — Apr 2)

Watching - The recovery from backup is still in progress, albeit running slowly at only 12% complete.

We believe that we’ll be able to recover most of the data from the failed server, so we’re restoring that data in parallel with the backup recovery. This should speed up the overall recovery and provide the most recent data from prior to the failure. (04:19 UTC — Apr 2)

Watching - The recovery is in still progress.

We estimate that it is 11% complete and it is currently restoring mailboxes that being with “as”. (21:47 UTC — Apr 1)

Watching - DE webmail access is now working.

The mailbox restore is still in progress. (16:13 UTC — Apr 1)

Watching - We’re still noticing some login issues for webmail and are troubleshooting that now. (15:57 UTC — Apr 1)

Watching - We do not have high expectations for the outcome of the filesystem check, so we’ve deployed a new DE IMAP server and have re-created all DE mailboxes as new empty mailboxes on that server.

Customers should now be able to log in to our DE mail services to send and receive new mail. No changes to your email client settings should be required.

We’ll begin restoring the original mailbox contents ASAP from the most recent backup made on 30 March 2022 (the failure occurred while the 31 March backup was still in progress). The restore may take several days to complete, so we’ll keep this post updated until it is done.

We’ll attempt to recover mail from 31 March 2022 when the filesystem repairs on the failed server are complete. (15:30 UTC — Apr 1)

At around 08:30 UTC on 1 April 2022 our DE incoming mail server stopped responding, so we initiated an emergency reboot.

The server did not reboot correctly and further analysis identified a failing disk associated with outdated disk controller firmware and a failed disk controller battery.

As of 11:00 UTC the disc controller was repaired and a filesystem check was initiated.

Further updates follow in reverse-chronological order. (08:30 UTC — Apr 1)

[Resolved] Incoming mail problems in Germany

Tue, 09 Feb 2021 14:18:31 +0000

Fixed - The disk array rebuild is complete and the server has been stable since our last update. (15:50 UTC — Feb 10)

Watching - The DE mail service is back online at this time. The main issue was heavy load caused by the very high volume of deferred mail the resulted from the mail loop. This was exacerbated by the degraded disk array.

The networking issue was unrelated but has been resolved.

The disk array is still rebuilding but the server is operating normally. We’ll continue to monitor until the rebuild is complete. (18:54 UTC — Feb 9)

Identified - There seems to be multiple issues related to the DE IMAP server:

A forwarding loop was causing delivery delays for the past few hours.
A degraded disk array was impacting system performance
A failed network switch is currently blocking network connectivity

Our data center team is working on restoring connectivity at this time. (15:50 UTC — Feb 9)

Investigating - The DE IMAP server is now offline for a filesystem check. (15:16 UTC — Feb 9)

We’re receiving reports of slow mail delivery in our German mail system. We’re looking into it at this time. (14:18 UTC — Feb 9)