Opalstack - Notice history

All systems operational

Notice history

Apr 2021

MariaDB down on opal9
  • Resolved
    Resolved

    The MariaDB service on Opal9 is back online. The problem was due to an error in a monitoring check that has been corrected.

  • Investigating
    Investigating

    The MariaDB (mysql) service on Opal9 is currently offline, we're working to restore service at this time.

Control panel and API down for scheduled maintenance.
  • Resolved
    Resolved

    The control panel and API are back online.

  • Monitoring
    Monitoring

    We expect the service will be back online in ~15 minutes.

  • Investigating
    Investigating

    The control panel and API are down for scheduled maintenance. We'll update this post when the service is back online.

Opal1 not responding
  • Resolved
    Resolved

    Opal1 is back online. The outage appears to have been due to high load due to abnormally heavy memory usage by a customer app. We're working with the customer to prevent this from happening again.

  • Update
    Update

    We're rebooting Opal1 at this time.

  • Investigating
    Investigating

    Opal1 is not responding to SSH and HTTP requests, we're looking into it.

Dashboard & API maintenance downtime
  • Completed
    April 17, 2021 at 5:07 AM
    Completed
    April 17, 2021 at 5:07 AM
    The maintenance is complete.
  • In progress
    April 17, 2021 at 4:00 AM
    In progress
    April 17, 2021 at 4:00 AM

    Maintenance is now in progress

  • Planned
    April 17, 2021 at 4:00 AM
    Planned
    April 17, 2021 at 4:00 AM

    The Opalstack dashboard at https://my.opalstack.com/ and the related API will be taken offline for maintenance at 04:00 UTC on 17 April 2021. The expected downtime is 1 hour or less.

Opal7 server relocation
  • Resolved
    Resolved

    The maintenance is now complete and opal7 is online.

  • Monitoring
    Monitoring

    Opal7 is back online. We'll continue to monitor.

  • Investigating
    Investigating

    Opal7 is offline again, the maintenance may be ongoing. Awaiting confirmation from the data center.

  • Resolved
    Resolved

    The migration is complete and Opal7 is back online.

  • Monitoring
    Monitoring

    Opal7 is offline for the migration at this time.

  • Investigating
    Investigating

    Our data center team will be moving opal7.opalstack.com to a new facility within the next 6 hours. During the move the server will be offline.

Mar 2021

Apache was down on Opal6
  • Resolved
    Resolved

    The Apache service on Opal6 was offline for approximately 2 hours after the server ran out of available semaphores. The problem has been corrected as of around 13:30 UTC (~1 hour ago) and we'll be adjusting our monitoring system to detect and correct this condition when it occurs.

IMAP1US Offline
  • Resolved
    Resolved

    The US IMAP server has been back online since 2021-03-24 10:30 UTC.

  • Investigating
    Investigating

    The IMAP1US server is currently down and we are working to restore access.

DE SMTP on Proofpoint blacklist
  • Resolved
    Resolved

    The DE SMTP IP is no longer on the Proofpoint blacklist.

  • Monitoring
    Monitoring

    We're still waiting on Proofpoint to delist the DE SMTP IP address.

  • Investigating
    Investigating

    One of the Opalstack DE SMTP IP addresses is currently on the Proofpoint blacklist. We're working to have the IP de-listed and expect this to be complete within the next 24-48 hours.

Feb 2021

Webmail system maintenance
  • Completed
    February 19, 2021 at 5:26 AM
    Completed
    February 19, 2021 at 5:26 AM
    The maintenance is complete.
  • In progress
    February 19, 2021 at 5:03 AM
    In progress
    February 19, 2021 at 5:03 AM

    Maintenance is now in progress

  • Planned
    February 19, 2021 at 5:03 AM
    Planned
    February 19, 2021 at 5:03 AM

    The Opalstack webmail system will be down for maintenance beginning at 05:00 UTC on 19 February 2021.

    The maintenance window is 1 hour but the downtime will probably be much less than that.

    2021-02-19 05:03 UTC: Webmail is down for maintenance at this time.

    2021-02-19 05:26 UTC: Webmail maintenance is complete.

Opal1 service disruption
  • Resolved
    Resolved

    At around 2021-02-18 11:00 UTC today the /tmp partition on opal1.opalstack.com ran out of available space. This disrupted service for applications that needed to write to temporary storage including PHP apps and the MariaDB database service.

    At the time of the event our monitoring system was only checking the status of these system services and not their operational performance, so no alert was generated.

    We corrected the problem at approximately 14:00 UTC and the server has been stable since that time.

    We're updating our monitoring system to perform the tests necessary to alert us to the conditions that caused this incident which will help us avoid this sort of extended disruption.

DE IMAP service down
  • Resolved
    Resolved

    The DE IMAP/POP3 service has been stable since our last update.

  • Update
    Update

    The DE IMAP/POP3 service is back online with higher process and open files limits. We'll continue to monitor.

  • Update
    Update

    We're rebooting time DE IMAP/POP3 server to pick up a configuration change.

  • Monitoring
    Monitoring

    The DE IMAP/POP3 service is back online. We'll continue to monitor as we look into the cause of the outage.

  • Investigating
    Investigating

    The IMAP/POP3 service on our DE mail system is currently down. We're working to restore service.

Incoming mail problems in Germany
  • Resolved
    Resolved

    The disk array rebuild is complete and the server has been stable since our last update.

  • Monitoring
    Monitoring

    The DE mail service is back online at this time. The main issue was heavy load caused by the very high volume of deferred mail the resulted from the mail loop. This was exacerbated by the degraded disk array.

    The networking issue was unrelated but has been resolved.

    The disk array is still rebuilding but the server is operating normally. We'll continue to monitor until the rebuild is complete.

  • Identified
    Identified

    There seems to be multiple issues related to the DE IMAP server:

    1. A forwarding loop was causing delivery delays for the past few hours.
    2. A degraded disk array was impacting system performance
    3. A failed network switch is currently blocking network connectivity

    Our data center team is working on restoring connectivity at this time.

  • Update
    Update

    The DE IMAP server is now offline for a filesystem check.

  • Investigating
    Investigating

    We're receiving reports of slow mail delivery in our German mail system. We're looking into it at this time.

Opal2 (Singapore) maintenance downtime
  • Completed
    February 06, 2021 at 11:08 PM
    Completed
    February 06, 2021 at 11:08 PM
    The maintenance is complete.
  • In progress
    February 06, 2021 at 10:00 PM
    In progress
    February 06, 2021 at 10:00 PM

    Maintenance is now in progress

  • Planned
    February 06, 2021 at 10:00 PM
    Planned
    February 06, 2021 at 10:00 PM

    On Sunday February 7 2021 at 06:00 AM Singapore Standard time (2021-02-06 22:00 UTC) we will migrate opal2.opalstack.com to new hardware.

    The expected downtime for the migration will be 2 hours during which your sites on opal2.opalstack.com will be unavailable.

    The new server IP address will be: 209.58.165.79

    If you are using external name servers for domains that are pointing at opal2.opalstack.com then you will need to update the A records for those domains to point at the new server IP address no later than the maintenance start time. DNS records for domains that use Opalstack's name servers will be updated automatically.

    Email is hosted on separate hardware and is not affected by this maintenance.

Feb 2021 to Apr 2021

Next