Opalstack - Notice history

All systems operational

Notice history

Jan 2023

German IMAP/POP3 server offline for scheduled relocation
  • Resolved
    Resolved

    The backlog of incoming mail has been delivered and the relocation is complete.

  • Monitoring
    Monitoring

    The DE mail server is back online at this time and the backlog of incoming mail is now being processed. We're awaiting the final word from the data center that the relocation is complete.

  • Investigating
    Investigating

    As previously announced, our German IMAP/POP3 server will be offline for relocation to a new data center facility from 0800 to 1600 UTC on 24 January 2023: https://status.opalstack.com/schedules/38

German IMAP email server relocation downtime
  • Completed
    January 24, 2023 at 4:00 PM
    Completed
    January 24, 2023 at 4:00 PM
    The maintenance is complete.
  • In progress
    January 24, 2023 at 8:00 AM
    In progress
    January 24, 2023 at 8:00 AM

    Maintenance is now in progress

  • Planned
    January 24, 2023 at 8:00 AM
    Planned
    January 24, 2023 at 8:00 AM

    On 24 January 2023 at 0900 CET (UTC +1) our German IMAP/POP3 mail server will be physically relocated to a new data center facility in Frankfurt. The server will be offline during the relocation and the expected downtime is 8 hours.

    Customer mailboxes hosted on the server will not be available during the relocation. Mail that arrives while the server is offline will be deferred and delivered once the mail server is back online (no incoming mail will be lost).

    Mail forwarding addresses configured at https://my.opalstack.com/mail/ will continue to forward mail normally.

    Customers will still be able to send mail during the downtime under the following circumstances:

    • Web applications will be able to send mail via SMTP and sendmail
    • POP3 desktop email clients will be able to send mail via SMTP.
    • IMAP desktop email clients should be able to send mail via SMTP but will not be able to store a copy of the sent message because storing messages is a function of the IMAP server.
    • Webmail will not be available because it uses the IMAP server for authentication.

    We realize that the timing of this relocation is not ideal. If you have business-critical mailboxes and would like them to be available during the relocation then please email support@opalstack.com to make arrangements to migrate your mailbox in advance.

    We'll update the status page at the beginning of the relocation window. If you would like to be informed of system status updates automatically then please subscribe at https://status.opalstack.com/subscribe.

DE SMTP was blocked by Abusix
  • Resolved
    Resolved

    The DE SMTP is no longer blocked.

  • Investigating
    Investigating

    Our DE SMTP server was listed in the Abusix Mail Intelligence block list. We've already initiated the delisting process and expect the issue to be resolved within the next 60 minutes.

Mail service maintenance
  • Completed
    January 19, 2023 at 4:00 AM
    Completed
    January 19, 2023 at 4:00 AM
    The maintenance is complete.
  • In progress
    January 19, 2023 at 3:00 AM
    In progress
    January 19, 2023 at 3:00 AM

    Maintenance is now in progress

  • Planned
    January 19, 2023 at 3:00 AM
    Planned
    January 19, 2023 at 3:00 AM

    We'll be restarting the various mail services on our MX, SMTP, and IMAP servers at beginning at 0300 UTC on 19 Jan 2023 (10:00PM US/Eastern). The maintenance window is 1 hour.

    Mail clients, including the Opalstack webmail system, may experience intermittent connection errors during the maintenance window.

DE SMTP blocked by Proofpoint
  • Resolved
    Resolved

    The DE SMTP server is no longer blocked by Proofpoint.

  • Investigating
    Investigating

    At this time our German SMTP server is being blocked by the Proofpoint blocklist. We've dealt with the cause and have started the de-listing process.

    We'll update this post when the server has been removed from the list.

Dec 2022

Ongoing DNS DDOS Attack
  • Update
    Update

    Post-mortem: this was a DNS flood attack which targeted all 9 of our regional name servers. The attack originated from several cloud provider subnets, including Google and Cloudflare.

    To mitigate the attack we had to temporarily block the provider subnets. This had some unintended and undesirable side effects:

    • The subnets included public DNS resolver infrastructure run by Google and Cloudflare, so blocking them prevented those resolvers from looking up DNS records on Opalstack name servers. As a result, people who use those resolvers were temporarily unable to access opalstack.com (including this status page) and customer domains that use our name servers.
    • The block also prevented Google from looking up DNS records related to email such as SPF and DKIM. As a result, mail sent (or forwarded) to Gmail from Opalstack's mail system was rejected by Google while the block was in place.

    The block was lifted over 24 hours ago and there have been no further issues since that time.

    Going forward, we'll refine our mitigation techniques to avoid blocking major providers when possible. We'll also move this status page to an externally-hosted domain to ensure that system status updates will be available regardless of the state of our infrastructure.

  • Resolved
    Resolved

    We've seen no further issues in the past several hours and consider this to be resolved.

  • Monitoring
    Monitoring

    The attack has subsided, we'll continue to monitor.

  • Investigating
    Investigating

    Over the last few hours we've identified and mitigated an ongoing DDOS attack against our DNS infrastructure. This attack has also effected our Singapore and German DNS servers.

    We're continuing to monitor the attack.

Delayed domain updates
  • Resolved
    Resolved

    The backlog of domain updates is now clear.

  • Investigating
    Investigating

    At this time (Thu Dec 29 02:17:55 UTC 2022) there is a backlog of domain updates which are waiting to be deployed to our name servers. Customers who are affected by this will see domains listed as "Pending" in their Opalstack dashboards.

    The actual processing delay is on our Singapore name servers. If your domain is using the US or DE name servers then they'll still appear as pending but the domains will actually be updated on the NS. Only domains that use the Singapore servers will actually be affected, and it's only a small number at this time.

    The cause of the issue appears to be network-related so we're working with our upstream providers to resolve this.

    We'll update this post when more information is available.

Unplanned reboot of opal3
  • Resolved
    Resolved

    On Saturday 10 December at around 0800 UTC (midnight US/Pacific) opal3.opalstack.com in San Francisco became unresponsive. After several unsuccessful attempts to access the server for troubleshooting, we rebooted the server and it was back online around 0900 UTC (1am US/Pacific).

    The outage appears to have been triggered by a very rapid spike in system memory usage. The high load unfortunately interrupted the system logging facilities so we weren't able to precisely identify the cause, but we suspect that it was due to an buggy Apache rewrite rule in a customer application.

Nov 2022

MariaDB disrupted on opal11
  • Resolved
    Resolved

    The MariaDB service on Opal11 (Frankfurt shared hosting) was briefly disrupted due to a service restart. The service is operating normally at this time.

Slow performance on the US IMAP/POP3 service
  • Resolved
    Resolved

    The backup is complete and the server load has returned to normal.

  • Investigating
    Investigating

    At this time load on our US IMAP/POP3 server is intermittently very high because we're running a full backup of the entire server.

    We expect this to continue until the backup is complete, possible for another day or longer.

    We'll update this post when the backup is complete and load has returned to normal.

Nov 2022 to Jan 2023

Next