Opalstack - Notice history

All systems operational

Notice history

Aug 2020

Service disruption in San Francisco
  • Resolved
    Resolved

    All SFO servers are now back online.

  • Update
    Update

    vps15 and mx2.us are back online.

  • Update
    Update

    The webmail server is back online and is operational with the exception of sending mail. We're still waiting on the remaining servers to come back online.

  • Identified
    Identified

    The data center team reports:

    The issue has been identified as a utility power issue. Power is transferred to generators and is currently running stable and network connectivity is restored for the vast majority of the services.

    We're starting to see the remaining machines come online at this time.

  • Update
    Update

    opal3.opalstack.com is back online at this time. Other servers are still offline and we're awaiting a report on the initial investigation from the data center team.

  • Investigating
    Investigating

    Services in our San Francisco data center are currently disrupted. The following servers are affected:

    • community.opalstack.com
    • mx2.us.opalstack.com
    • opal3.opalstack.com
    • smtp2.us.opalstack.com
    • vps4.opalstack.com
    • vps5.opalstack.com
    • vps6.opalstack.com
    • vps13.opalstack.com
    • vps15.opalstack.com
    • vps16.opalstack.com
    • vps17.opalstack.com
    • vps19.opalstack.com
    • vps21.opalstack.com
    • vps24.opalstack.com
    • vps28.opalstack.com
    • webmail.us.opalstack.com

    We'll update this post as soon as more information is available.

Scheduled maintenance on ns2.de.opalstack.com
  • Completed
    August 07, 2020 at 3:00 AM
    Completed
    August 07, 2020 at 3:00 AM
    The maintenance is complete.
  • In progress
    August 07, 2020 at 2:00 AM
    In progress
    August 07, 2020 at 2:00 AM

    Maintenance is now in progress

  • Planned
    August 07, 2020 at 2:00 AM
    Planned
    August 07, 2020 at 2:00 AM

    ns2.de.opalstack.com will be taken offline at 2020-08-07 02:00:00 AM UTC for scheduled maintenance. A 2-hour window is allocated, however, the actual downtime should be around 45 - 60 minutes.

    No impact is expected as the other two regional NS will still be online during the maintenance.

Scheduled maintenance on ns3.de.opalstack.com
  • Completed
    August 07, 2020 at 1:00 AM
    Completed
    August 07, 2020 at 1:00 AM
    The maintenance is complete.
  • In progress
    August 07, 2020 at 12:00 AM
    In progress
    August 07, 2020 at 12:00 AM

    Maintenance is now in progress

  • Planned
    August 07, 2020 at 12:00 AM
    Planned
    August 07, 2020 at 12:00 AM

    ns3.de.opalstack.com will be taken offline at 2020-08-07 00:00:00 AM UTC for scheduled maintenance. A 2-hour window is allocated, however, the actual downtime should be around 45 - 60 minutes.

    No impact is expected as the other two regional NS will still be online during the maintenance.

Scheduled maintenance on ns2.sg.opalstack.com
  • Completed
    August 03, 2020 at 5:00 PM
    Completed
    August 03, 2020 at 5:00 PM
    The maintenance is complete.
  • In progress
    August 03, 2020 at 4:00 PM
    In progress
    August 03, 2020 at 4:00 PM

    Maintenance is now in progress

  • Planned
    August 03, 2020 at 4:00 PM
    Planned
    August 03, 2020 at 4:00 PM

    ns2.sg.opalstack.com will be taken offline at 2020-08-03 04:00:00 PM UTC for scheduled maintenance. A 2-hour window is allocated, however, the actual downtime should be around 45 - 60 minutes.

    No impact is expected as the other two regional NS will still be online during the maintenance.

Jul 2020

intermittent connectivity problems on opal4
  • Resolved
    Resolved

    Opal4 is stable.

  • Monitoring
    Monitoring

    Opal4 is back online. The data center team believed that the connectivity issue was caused by a configuration problem which they corrected

    We will continue to monitor.

  • Update
    Update

    Opal4 is being rebooted at this time.

  • Investigating
    Investigating

    Opal4 is experiencing intermittent connectivity problems following this morning's scheduled maintenance. The data center team is investigating.

Scheduled maintenance on opal4.opalstack.com
  • Completed
    July 23, 2020 at 5:30 PM
    Completed
    July 23, 2020 at 5:30 PM
    The maintenance is complete.
  • In progress
    July 23, 2020 at 1:00 PM
    In progress
    July 23, 2020 at 1:00 PM

    Maintenance is now in progress

  • Planned
    July 23, 2020 at 1:00 PM
    Planned
    July 23, 2020 at 1:00 PM

    opal4.opalstack.com (Dallas TX USA shared hosting) will taken offline on 23 July 2020 beginning at 13:00 UTC (8:00 AM US Central) to swap out its network interface. The scheduled maintenance window is 1 hour. We'll update this post as the maintenance progresses.

    2020-07-23 12:26 UTC: due to a scheduling error our data center has started the maintenance 1 hour earlier than planned. We'll update this post when the maintenance is complete.

    2020-07-23 15:27 UTC: opal4 has been back online for 2.5 hours but we're waiting on confirmation from the data center that the work is complete.

    2020-07-23 17:11 UTC: network connectivity to opal4 has been interrupted temporarily while the data center completes some final tasks.

Opal4 network was disrupted
  • Resolved
    Resolved

    Network connectivity to opal4 was disrupted for 1 minute due to a short network attack.

    The server is online and operating normally at this time.

Service-wide reboots
  • Completed
    July 12, 2020 at 7:00 AM
    Completed
    July 12, 2020 at 7:00 AM
    The maintenance is complete.
  • In progress
    July 12, 2020 at 2:00 AM
    In progress
    July 12, 2020 at 2:00 AM

    Maintenance is now in progress

  • Planned
    July 12, 2020 at 2:00 AM
    Planned
    July 12, 2020 at 2:00 AM

    We will be rebooting our entire fleet of servers on Saturday July 11 2020 beginning at 8:00 PM US Mountain (2020-07-12 02:00 GMT) to apply operating system updates.

    2020-07-12 02:27 UTC: reboots are now in progress.

    2020-07-12 07:00 UTC: all reboots are complete.

Opal4 under high load
  • Resolved
    Resolved

    Opal4 is back online following a reboot. The high load was caused by a network attack. We've identified the site targeted by the attack and are working with the site owner to isolate the site from other customers on opal4.

  • Investigating
    Investigating

    Opal4 is presently under high load and is mostly unresponsive. We're rebooting it at this time.

Jun 2020

Cron mail not working
  • Resolved
    Resolved

    Mail from cron jobs should be working normally now.

  • Investigating
    Investigating

    At the present time mail is not being sent by cron jobs on our web servers. We're looking into the cause.

Jun 2020 to Aug 2020

Next