{
  "is": "issue",
  "title": "Intermittent outages on opal4",
  "body": "\u003cp\u003e\u003cem\u003eFixed\u003c/em\u003e - During last week\u0026rsquo;s maintenance our upstream provider updated the server firmware on opal4. We\u0026rsquo;ve seen no further issues since that time. \n  \u003cspan class=\"faded\"\u003e(16:03 UTC — Apr 6)\u003c/span\u003e\n\n\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eWatching\u003c/em\u003e - The maintenance is complete and Opal4 is back online. We\u0026rsquo;ll continue to monitor. \n  \u003cspan class=\"faded\"\u003e(02:58 UTC — Apr 2)\u003c/span\u003e\n\n\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eIdentified\u003c/em\u003e - Opal4 is down for the emergency maintenance scheduled earlier today. The total expected downtime is less than one hour. \n  \u003cspan class=\"faded\"\u003e(02:05 UTC — Apr 2)\u003c/span\u003e\n\n\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eInvestigating\u003c/em\u003e - Opal4 will be going down this evening at 9PM US Central (2020-04-01 21:00 UTC-5) for emergency maintenance. The total downtime should be less than one hour.\u003c/p\u003e\n\u003cp\u003eThe ongoing issue on opal4 has been high system load caused by two separate types of attacks:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eHigh Apache RAM usage due to attacks against common targets like Wordpress sites\u003c/li\u003e\n\u003cli\u003eHigh CPU usage by the system firewall under a high volume of SYN packets\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eWe\u0026rsquo;ve resolved the first problem by tuning our application firewall rules and by working with a couple of specific site owners that were receiving the brunt of the attacks.\u003c/p\u003e\n\u003cp\u003eThe second problem is part of what we\u0026rsquo;re troubleshooting this evening.\u003c/p\u003e\n\u003cp\u003eAt first glance the SYN packet issue would seem like a common SYN flood attack but other shared servers in our infrastructure receive similar amounts of traffic and don\u0026rsquo;t have any problems mitigating it.\u003c/p\u003e\n\u003cp\u003eIn the past 24 hours we\u0026rsquo;ve discovered key differences in Opal4\u0026rsquo;s hardware compared to the other servers, so we\u0026rsquo;re working with our upstream provider to sort that out.\u003c/p\u003e\n\u003cp\u003eTonight\u0026rsquo;s downtime is at their request to allow them to perform hardware diagnostics in support of that investigation. \n  \u003cspan class=\"faded\"\u003e(23:24 UTC — Apr 1)\u003c/span\u003e\n\n\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eWatching\u003c/em\u003e - Opal4 just went through a brief spike in system load during which performance was degraded, but the system is back to normal at this time. \n  \u003cspan class=\"faded\"\u003e(23:20 UTC — Mar 29)\u003c/span\u003e\n\n\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eIdentified\u003c/em\u003e - We\u0026rsquo;ve identified two distinct attack patterns responsible for the intermittent outages on opal4 and are putting measures in place to mitigate them. \n  \u003cspan class=\"faded\"\u003e(23:01 UTC — Mar 25)\u003c/span\u003e\n\n\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eWatching\u003c/em\u003e - This intermittent issue on Opal4 has started up again - we will continue to monitor as we work to resolve it. \n  \u003cspan class=\"faded\"\u003e(19:05 UTC — Mar 25)\u003c/span\u003e\n\n\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eFixed\u003c/em\u003e - We\u0026rsquo;ve seen no further issues in the past several hours. \n  \u003cspan class=\"faded\"\u003e(16:38 UTC — Jan 8)\u003c/span\u003e\n\n\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eWatching\u003c/em\u003e - The problem appears to have been caused by temporary high load due to high CPU usage by the system firewall.\u003c/p\u003e\n\u003cp\u003eopal4 is stable at this time. We\u0026rsquo;ll continue to monitor. \n  \u003cspan class=\"faded\"\u003e(01:21 UTC — Jan 8)\u003c/span\u003e\n\n\u003c/p\u003e\n\u003cp\u003eopal4.opalstack.com is experiencing intermittent outages. We\u0026rsquo;re looking into it and will update this item when we have more information. \n  \u003cspan class=\"faded\"\u003e(00:36 UTC — Jan 8)\u003c/span\u003e\n\n\u003c/p\u003e\n",
  "createdAt": "2020-01-08 00:36:00 +0000 UTC",
  "lastMod": "2020-01-08 00:36:00 +0000 UTC",
  "permalink": "https://opalstackstatus.com/issues/2020-01-08-intermittent-outages-on-opal4/",
  "severity": "down",
  "resolved": true,
  "informational": false,
  "resolvedAt": "2020-04-06 16:03:39",
  "affected": ["Shared: Dallas (USA)"],
  "filename": "2020-01-08-intermittent-outages-on-opal4.md"
}