Opalstack - Notice history

All systems operational

Notice history

Mar 2021

Apache was down on Opal6
  • Resolved
    Resolved

    The Apache service on Opal6 was offline for approximately 2 hours after the server ran out of available semaphores. The problem has been corrected as of around 13:30 UTC (~1 hour ago) and we'll be adjusting our monitoring system to detect and correct this condition when it occurs.

IMAP1US Offline
  • Resolved
    Resolved

    The US IMAP server has been back online since 2021-03-24 10:30 UTC.

  • Investigating
    Investigating

    The IMAP1US server is currently down and we are working to restore access.

DE SMTP on Proofpoint blacklist
  • Resolved
    Resolved

    The DE SMTP IP is no longer on the Proofpoint blacklist.

  • Monitoring
    Monitoring

    We're still waiting on Proofpoint to delist the DE SMTP IP address.

  • Investigating
    Investigating

    One of the Opalstack DE SMTP IP addresses is currently on the Proofpoint blacklist. We're working to have the IP de-listed and expect this to be complete within the next 24-48 hours.

Feb 2021

Webmail system maintenance
  • Completed
    February 19, 2021 at 5:26 AM
    Completed
    February 19, 2021 at 5:26 AM
    The maintenance is complete.
  • In progress
    February 19, 2021 at 5:03 AM
    In progress
    February 19, 2021 at 5:03 AM

    Maintenance is now in progress

  • Planned
    February 19, 2021 at 5:03 AM
    Planned
    February 19, 2021 at 5:03 AM

    The Opalstack webmail system will be down for maintenance beginning at 05:00 UTC on 19 February 2021.

    The maintenance window is 1 hour but the downtime will probably be much less than that.

    2021-02-19 05:03 UTC: Webmail is down for maintenance at this time.

    2021-02-19 05:26 UTC: Webmail maintenance is complete.

Opal1 service disruption
  • Resolved
    Resolved

    At around 2021-02-18 11:00 UTC today the /tmp partition on opal1.opalstack.com ran out of available space. This disrupted service for applications that needed to write to temporary storage including PHP apps and the MariaDB database service.

    At the time of the event our monitoring system was only checking the status of these system services and not their operational performance, so no alert was generated.

    We corrected the problem at approximately 14:00 UTC and the server has been stable since that time.

    We're updating our monitoring system to perform the tests necessary to alert us to the conditions that caused this incident which will help us avoid this sort of extended disruption.

DE IMAP service down
  • Resolved
    Resolved

    The DE IMAP/POP3 service has been stable since our last update.

  • Update
    Update

    The DE IMAP/POP3 service is back online with higher process and open files limits. We'll continue to monitor.

  • Update
    Update

    We're rebooting time DE IMAP/POP3 server to pick up a configuration change.

  • Monitoring
    Monitoring

    The DE IMAP/POP3 service is back online. We'll continue to monitor as we look into the cause of the outage.

  • Investigating
    Investigating

    The IMAP/POP3 service on our DE mail system is currently down. We're working to restore service.

Incoming mail problems in Germany
  • Resolved
    Resolved

    The disk array rebuild is complete and the server has been stable since our last update.

  • Monitoring
    Monitoring

    The DE mail service is back online at this time. The main issue was heavy load caused by the very high volume of deferred mail the resulted from the mail loop. This was exacerbated by the degraded disk array.

    The networking issue was unrelated but has been resolved.

    The disk array is still rebuilding but the server is operating normally. We'll continue to monitor until the rebuild is complete.

  • Identified
    Identified

    There seems to be multiple issues related to the DE IMAP server:

    1. A forwarding loop was causing delivery delays for the past few hours.
    2. A degraded disk array was impacting system performance
    3. A failed network switch is currently blocking network connectivity

    Our data center team is working on restoring connectivity at this time.

  • Update
    Update

    The DE IMAP server is now offline for a filesystem check.

  • Investigating
    Investigating

    We're receiving reports of slow mail delivery in our German mail system. We're looking into it at this time.

Opal2 (Singapore) maintenance downtime
  • Completed
    February 06, 2021 at 11:08 PM
    Completed
    February 06, 2021 at 11:08 PM
    The maintenance is complete.
  • In progress
    February 06, 2021 at 10:00 PM
    In progress
    February 06, 2021 at 10:00 PM

    Maintenance is now in progress

  • Planned
    February 06, 2021 at 10:00 PM
    Planned
    February 06, 2021 at 10:00 PM

    On Sunday February 7 2021 at 06:00 AM Singapore Standard time (2021-02-06 22:00 UTC) we will migrate opal2.opalstack.com to new hardware.

    The expected downtime for the migration will be 2 hours during which your sites on opal2.opalstack.com will be unavailable.

    The new server IP address will be: 209.58.165.79

    If you are using external name servers for domains that are pointing at opal2.opalstack.com then you will need to update the A records for those domains to point at the new server IP address no later than the maintenance start time. DNS records for domains that use Opalstack's name servers will be updated automatically.

    Email is hosted on separate hardware and is not affected by this maintenance.

Jan 2021

US IMAP service not responding
  • Resolved
    Resolved

    The US IMAP server is back online.

  • Update
    Update

    We're rebooting the US IMAP server at this time.

  • Investigating
    Investigating

    The US IMAP service is currently not responding, we're looking into it.

EU VPS and NS connectivity problems
  • Resolved
    Resolved

    The upstream issue has been resolved.

  • Update
    Update

    All EU services are back online. We'll continue to monitor until resolution of the upstream issue is confirmed.

  • Monitoring
    Monitoring

    At this time connectivity to most of our EU services has been restored. ns3.de.opalstack.com and one customer VPS remain affected.

  • Update
    Update

    This issue is also affecting all of our DE name servers. We're working to restore service ASAP but if you are using our DE NS and need an immediate workaround please change your NS over to our US name servers.

  • Identified
    Identified

    We've confirmed that there is an outage in the Linode data center that hosts our EU VPS: https://status.linode.com/incidents/dm9mc14kv2t4

    We'll update this post when more info is available.

  • Investigating
    Investigating

    There seems to be a loss of connectivity in our EU VPS datacenter. We're investigating at this time.

US SMTP on Cloudmark blacklist
  • Resolved
    Resolved

    Our US SMTP is no longer on the Cloudmark blacklist.

  • Investigating
    Investigating

    At this time Opalstack's US SMTP is on the Cloudmark blacklist. We're working to have the server delisted and expect that to be done within the next 24-48 hours.

EU VPS network connectivity problems
  • Resolved
    Resolved

    The network has been stable since our last update so we're considering this to be resolved.

  • Monitoring
    Monitoring

    EU VPS network connectivity is now stable. We'll continue to monitor.

  • Identified
    Identified

    EU VPS customers may see further connectivity issues while our upstream provider is performing emergency maintenance.

  • Resolved
    Resolved

    The maintenance is complete and connectivity to EU VPS remains stable.

  • Monitoring
    Monitoring

    Connectivity to EU VPS is stable at this time. The maintenance is still in progress so we'll continue to monitor.

  • Investigating
    Investigating

    Connectivity to some EU VPS may be degraded due to active ongoing maintenance by our upstream provider. We'll update this post as soon as the maintenance is complete.

US IMAP service performance degraded
  • Resolved
    Resolved

    The US IMAP server is back online and is operating normally. We'll be upgrading the server in February to avoid future problems.

  • Monitoring
    Monitoring

    We've rebooted the US IMAP server and are monitoring.

  • Investigating
    Investigating

    The IMAP service on our US mail system is currently running very slowly. We're working to correct that at this time.

    This is also causing timeouts and errors for users accessing Opalstack webmail, including users of our German mail system.

Jan 2021 to Mar 2021

Next