Opalstack - Notice history

All systems operational

Notice history

Feb 2021

Webmail system maintenance
  • Completed
    February 19, 2021 at 5:26 AM
    Completed
    February 19, 2021 at 5:26 AM
    The maintenance is complete.
  • In progress
    February 19, 2021 at 5:03 AM
    In progress
    February 19, 2021 at 5:03 AM

    Maintenance is now in progress

  • Planned
    February 19, 2021 at 5:03 AM
    Planned
    February 19, 2021 at 5:03 AM

    The Opalstack webmail system will be down for maintenance beginning at 05:00 UTC on 19 February 2021.

    The maintenance window is 1 hour but the downtime will probably be much less than that.

    2021-02-19 05:03 UTC: Webmail is down for maintenance at this time.

    2021-02-19 05:26 UTC: Webmail maintenance is complete.

Opal1 service disruption
  • Resolved
    Resolved

    At around 2021-02-18 11:00 UTC today the /tmp partition on opal1.opalstack.com ran out of available space. This disrupted service for applications that needed to write to temporary storage including PHP apps and the MariaDB database service.

    At the time of the event our monitoring system was only checking the status of these system services and not their operational performance, so no alert was generated.

    We corrected the problem at approximately 14:00 UTC and the server has been stable since that time.

    We're updating our monitoring system to perform the tests necessary to alert us to the conditions that caused this incident which will help us avoid this sort of extended disruption.

DE IMAP service down
  • Resolved
    Resolved

    The DE IMAP/POP3 service has been stable since our last update.

  • Update
    Update

    The DE IMAP/POP3 service is back online with higher process and open files limits. We'll continue to monitor.

  • Update
    Update

    We're rebooting time DE IMAP/POP3 server to pick up a configuration change.

  • Monitoring
    Monitoring

    The DE IMAP/POP3 service is back online. We'll continue to monitor as we look into the cause of the outage.

  • Investigating
    Investigating

    The IMAP/POP3 service on our DE mail system is currently down. We're working to restore service.

Incoming mail problems in Germany
  • Resolved
    Resolved

    The disk array rebuild is complete and the server has been stable since our last update.

  • Monitoring
    Monitoring

    The DE mail service is back online at this time. The main issue was heavy load caused by the very high volume of deferred mail the resulted from the mail loop. This was exacerbated by the degraded disk array.

    The networking issue was unrelated but has been resolved.

    The disk array is still rebuilding but the server is operating normally. We'll continue to monitor until the rebuild is complete.

  • Identified
    Identified

    There seems to be multiple issues related to the DE IMAP server:

    1. A forwarding loop was causing delivery delays for the past few hours.
    2. A degraded disk array was impacting system performance
    3. A failed network switch is currently blocking network connectivity

    Our data center team is working on restoring connectivity at this time.

  • Update
    Update

    The DE IMAP server is now offline for a filesystem check.

  • Investigating
    Investigating

    We're receiving reports of slow mail delivery in our German mail system. We're looking into it at this time.

Opal2 (Singapore) maintenance downtime
  • Completed
    February 06, 2021 at 11:08 PM
    Completed
    February 06, 2021 at 11:08 PM
    The maintenance is complete.
  • In progress
    February 06, 2021 at 10:00 PM
    In progress
    February 06, 2021 at 10:00 PM

    Maintenance is now in progress

  • Planned
    February 06, 2021 at 10:00 PM
    Planned
    February 06, 2021 at 10:00 PM

    On Sunday February 7 2021 at 06:00 AM Singapore Standard time (2021-02-06 22:00 UTC) we will migrate opal2.opalstack.com to new hardware.

    The expected downtime for the migration will be 2 hours during which your sites on opal2.opalstack.com will be unavailable.

    The new server IP address will be: 209.58.165.79

    If you are using external name servers for domains that are pointing at opal2.opalstack.com then you will need to update the A records for those domains to point at the new server IP address no later than the maintenance start time. DNS records for domains that use Opalstack's name servers will be updated automatically.

    Email is hosted on separate hardware and is not affected by this maintenance.

Jan 2021

US IMAP service not responding
  • Resolved
    Resolved

    The US IMAP server is back online.

  • Update
    Update

    We're rebooting the US IMAP server at this time.

  • Investigating
    Investigating

    The US IMAP service is currently not responding, we're looking into it.

EU VPS and NS connectivity problems
  • Resolved
    Resolved

    The upstream issue has been resolved.

  • Update
    Update

    All EU services are back online. We'll continue to monitor until resolution of the upstream issue is confirmed.

  • Monitoring
    Monitoring

    At this time connectivity to most of our EU services has been restored. ns3.de.opalstack.com and one customer VPS remain affected.

  • Update
    Update

    This issue is also affecting all of our DE name servers. We're working to restore service ASAP but if you are using our DE NS and need an immediate workaround please change your NS over to our US name servers.

  • Identified
    Identified

    We've confirmed that there is an outage in the Linode data center that hosts our EU VPS: https://status.linode.com/incidents/dm9mc14kv2t4

    We'll update this post when more info is available.

  • Investigating
    Investigating

    There seems to be a loss of connectivity in our EU VPS datacenter. We're investigating at this time.

US SMTP on Cloudmark blacklist
  • Resolved
    Resolved

    Our US SMTP is no longer on the Cloudmark blacklist.

  • Investigating
    Investigating

    At this time Opalstack's US SMTP is on the Cloudmark blacklist. We're working to have the server delisted and expect that to be done within the next 24-48 hours.

EU VPS network connectivity problems
  • Resolved
    Resolved

    The network has been stable since our last update so we're considering this to be resolved.

  • Monitoring
    Monitoring

    EU VPS network connectivity is now stable. We'll continue to monitor.

  • Identified
    Identified

    EU VPS customers may see further connectivity issues while our upstream provider is performing emergency maintenance.

  • Resolved
    Resolved

    The maintenance is complete and connectivity to EU VPS remains stable.

  • Monitoring
    Monitoring

    Connectivity to EU VPS is stable at this time. The maintenance is still in progress so we'll continue to monitor.

  • Investigating
    Investigating

    Connectivity to some EU VPS may be degraded due to active ongoing maintenance by our upstream provider. We'll update this post as soon as the maintenance is complete.

US IMAP service performance degraded
  • Resolved
    Resolved

    The US IMAP server is back online and is operating normally. We'll be upgrading the server in February to avoid future problems.

  • Monitoring
    Monitoring

    We've rebooted the US IMAP server and are monitoring.

  • Investigating
    Investigating

    The IMAP service on our US mail system is currently running very slowly. We're working to correct that at this time.

    This is also causing timeouts and errors for users accessing Opalstack webmail, including users of our German mail system.

Dec 2020

Opal4 not responding
  • Resolved
    Resolved

    Opal4 is back online and operating normally.

  • Investigating
    Investigating

    Opal4 is not responding at this time.

  • Resolved
    Resolved

    Opal4 is back online and is operating normally. We've confirmed another network attack was the cause.

  • Investigating
    Investigating

    Opal4 is currently not responding to ping. We suspect this is due to another attack.

Connectivity issues in Singapore
  • Resolved
    Resolved

    Linode reports that the connectivity issues in Singapore have been resolved.

  • Update
    Update

    All Singapore servers are back online at this time. We'll continue to monitor.

  • Monitoring
    Monitoring

    Linode are investigating a connectivity issue in our Singapore data center. During this time, users may experience connection timeouts and errors for all services deployed in this data center. We will share additional updates as we have more information: https://status.linode.com/incidents/wfghw8md5j8k

  • Resolved
    Resolved

    Linode will be performing an emergency network maintenance to upgrade an aggregate switch pair in the AP-South (Singapore) data center on Monday, December 21, 2020 from 21:00 UTC until 23:00 UTC. While we do not expect any downtime during this maintenance there may be a period of brief packet loss or latency. (source)

  • Investigating
    Investigating

    Singapore VPS and NS may have experienced connectivity problems due to network connectivity issues in Linode's Singapore data center: https://status.linode.com/incidents/zv7xb4plgzfm

    Linode reports that the issue is now resolved. We've reached out to them to find out the cause of the recurring network problems in Singapore and the corrective action that they are taking to minimize future disruptions of service.

Network outage in Singapore
  • Resolved
    Resolved

    Linode reports the incident is resolved as of Dec 18, 2020 20:01 UTC.

  • Update
    Update

    All Singapore services are back online, we'll continue to monitor while Linode wraps this up.

  • Monitoring
    Monitoring

    Linode has identified the cause of the outage and is implementing a fix, we're seeing the Singapore servers coming back online at this time.

  • Identified
    Identified

    If your domain is using our Singapore name servers then as an interim fix please switch your domain's NS to one of our other regions: https://help.opalstack.com/article/48/what-are-your-name-servers

  • Investigating
    Investigating

    An outage in the Linode Singapore data center is affecting connectivity to some Singapore NS and VPS: https://status.linode.com/incidents/k740qss44w58

    We'll update this post when we have more info.

Singapore servers were offline
  • Resolved
    Resolved

    At 11:36 UTC on Dec 16, 2020 Linode reported a network connectivity issue in their Singapore datacenter: https://status.linode.com/incidents/bt9z0b25417j

    Customer servers and NS in Singapore were affected by this issue and may have been inaccessible at that time. The NS issue may have also affected any domain that uses our Singapore NS.

    Service was restored after approximately 2 hours and has been up since that time.

Slow IMAP performance
  • Resolved
    Resolved

    All pending mail imports have been completed and the IMAP servers are operating normally.

  • Monitoring
    Monitoring

    The US IMAP service is currently under heavy load while importing mail for new customers.

  • Investigating
    Investigating

    At this time we're importing hundreds of gigabytes of mail for new customers migrating from WebFaction. This is causing heavy load on our IMAP servers so you may see slow performance in our webmail system, IMAP clients, and POP3 downloads over the next few days.

    We'll update this post when the imports are complete.

Dec 2020 to Feb 2021

Next