Opalstack - Notice history

All systems operational

Notice history

Oct 2022

opal13 not responding
  • Resolved
    Resolved

    The unexpected reboot was caused by human error. The server has been stable since coming back online.

  • Update
    Update

    Opal13 is back online. The outage was due to an unexpected reboot. We are now investigating the cause.

  • Investigating
    Investigating

    Opal13 (shared hosting in Phoenix AZ) is currently not responding. We're investigating now and will follow up when more info is available.

IPv6 email rejections
  • Resolved
    Resolved

    A misconfiguration in our IPv6 DNS settings was causing some mail providers (including Gmail) to reject some mail sent or forwarded from Opalstack's mail system over the past 3-4 days. The problem has been corrected.

Delays in US mail forwarding to @yahoo.com addresses
  • Resolved
    Resolved

    Yahoo are no longer deferring mail forwarded from our US MX servers.

  • Monitoring
    Monitoring

    Yahoo have begun to accept some mail forwarded from our US MX servers. We'll continue to monitor.

  • Investigating
    Investigating

    An abusive network was sending a very high volume of messages to a @yahoo.com forwarding address hosted on Opalstack's US mail system. The high volume of mail has caused Yahoo to temporarily defer messages resulting in delayed delivery.

    We've blocked the abusive network and expect that Yahoo will begin to accept the deferred messages over the next few hours as their systems register the overall message volume decrease.

opal10 not responding
  • Resolved
    Resolved

    The cause of the unexpected reboot was a power cable fault. The cable has been been replaced and the server is operating normally at this time.

  • Update
    Update

    Opal10 is back online after an unexpected reboot, we'll continue to investigate.

  • Investigating
    Investigating

    Opal10 (Washington DC shared hosting) is currently not responding, we're investigating at this time.

Apache was down on opal3
  • Resolved
    Resolved

    An undetected fault on opal3.opalstack.com caused a temporary disruption in the Apache service. The problem has been corrected.

Sep 2022

opal8 not responding
  • Resolved
    Resolved

    It appears that the earlier reboot was initiated by the system watchdog after the server became non-responsive under heavy load.

    The server is operating normally at this time.

  • Update
    Update

    Opal8 is back online. The server rebooted unexpectedly so we're investigating further.

  • Investigating
    Investigating

    Opal8 is not responding at this time. We're looking into it.

Dashboard disk usage stats incorrect
  • Resolved
    Resolved

    A fix has been deployed. Dashboard usage charts will return to normal in 8 hours when the old data has rotated out.

  • Investigating
    Investigating

    There's currently a problem with the display of disk usage statistics in the Opalstack dashboard which will cause customers to see reported usage much higher than their actual usage. We're working on a fix at this time.

Opal11 offline for scheduled relocation
  • Resolved
    Resolved

    The relocation was completed ahead of schedule as of 10:30 UTC.

  • Investigating
    Investigating

    As previously announced, opal11 has been shut down for physical relocation to a new data center facility. The expected downtime is approximately 8 hours.

    We'll update this post when the relocation is complete.

Scheduled downtime for opal11.opalstack.com
  • Completed
    September 15, 2022 at 10:30 AM
    Completed
    September 15, 2022 at 10:30 AM
    The maintenance is complete.
  • In progress
    September 15, 2022 at 7:00 AM
    In progress
    September 15, 2022 at 7:00 AM

    Maintenance is now in progress

  • Planned
    September 15, 2022 at 7:00 AM
    Planned
    September 15, 2022 at 7:00 AM

    opal11.opalstack.com will be physically disconnected and relocated to a new data center facility in Frankfurt on 15 September 2022.

    The maintenance window for the relocation is:

    • Time: 0700 UTC to 1500 UTC
    • Duration: 8 hours

    Customer websites and other services hosted on opal11 will be offline and unavailable during the relocation.

    Email is hosted on separate hardware and is not affected by the relocation.

    If you have any questions or concerns regarding the relocation then please email support@opalstack.com and someone from our team will get back to you as soon as possible.

Singapore VPS were briefly offline
  • Resolved
    Resolved

    The cause of the incident was found to be a faulty power distribution unit. The faulty unit is being replaced and no further disruption of service is expected.

  • Investigating
    Investigating

    Our Singapore VPS servers were briefly offline a few minutes ago. The network team is investigating and we'll update this post as soon as more info is available.

Aug 2022

Opal10 not responding
  • Resolved
    Resolved

    Opal10 is back online.

  • Monitoring
    Monitoring

    Opal10 became non-responsive again so we're rebooting it once more.

  • Resolved
    Resolved

    Opal10 is back online. The outage was caused by a runaway user process consuming too much memory. We'll work with the related customer to resolve the issue.

  • Update
    Update

    We're rebooting opal10 at this time.

  • Investigating
    Investigating

    Opal10 (shared hosting in Washington DC) is currently not responding. We're investigating.

US SMTP blocked by Cloudmark
  • Resolved
    Resolved

    We've confirmed that there are no further blocks by Cloudmark.

  • Update
    Update

    Cloudmark have stated that they have de-listed the server IPs. We'll continue to monitor until we see no further blocks in the mail logs.

  • Update
    Update

    We're still working with Cloudmark to have the US SMTP and MX servers delisted.

  • Monitoring
    Monitoring

    We've discovered that the US MX servers are also blocked. This could cause mail forwarding to fail if the final recipient domain is using the Cloudmark block list, so we've requested delisting of those as well.

  • Investigating
    Investigating

    At this time our US SMTP is being blocked by the Cloudmark Sender Intelligence block list. We've dealt with the cause and have started the de-listing process.

    If you receive a bounce message, please do not attempt to delist the server yourself via the URL in the error message. We're taking care of it.

    We'll update this post when the server has been removed from the list.

Several London VPS are offline
  • Resolved
    Resolved

    The issue is confirmed to be resolved.

  • Monitoring
    Monitoring

    The data center team identified a switching issue and a fix has been implemented. All London VPS are back online at this time.

    We'll continue to monitor awaiting final confirmation from the data center that the issue is fully resolved.

  • Investigating
    Investigating

    At this time several London VPS are offline due to a connectivity issue in the London data center.

    The DC team are looking into the issue. We'll update this item when more info is available.

Slow logins on US SMTP
  • Resolved
    Resolved

    The US SMTP server has remained stable following the upgrade and improved configuration.

  • Update
    Update

    We've implemented a fix and authentication is working normally at this time. We'll continue to monitor.

  • Monitoring
    Monitoring

    The original US SMTP is back online. We'll continue to troubleshoot and monitor.

  • Identified
    Identified

    We've taken the original US SMTP offline for an emergency upgrade.

  • Monitoring
    Monitoring

    The second server is online and is slowly picking up its share of the traffic.

    We'll continue to monitor until we're satisfied that this issue is resolved.

  • Identified
    Identified

    We believe the issue is higher-than-average mail traffic, so we're adding a SMTP server to the pool to take up some of the load.

  • Investigating
    Investigating

    At this time logins to Opalstack's US SMTP server are running slowly. Customers may see longer message sending times as a result.

    We're investigating the issue and will update this post when more info is available.

US SMTP was blocked by Microsoft
  • Resolved
    Resolved

    Our US SMTP service was briefly blocked by Microsoft today. We've already corrected the issues which caused the block and mail delivery to Microsoft addresses is working normally at this time.

Aug 2022 to Oct 2022

Next