<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <link rel="alternate" type="text/html" href="https://opalstackstatus.com/"/>
    <title>Shared: Singapore (SG) on Opalstack Status</title>
    <link>https://opalstackstatus.com/affected/shared-singapore-sg/</link>
    <description>Incident history</description>
    <generator>github.com/cstate</generator>
    <language>en</language>
    
    <lastBuildDate>2022-05-29T04:51:00+00:00</lastBuildDate>
    <updated>2022-05-29T04:51:00+00:00</updated>
    
    
    
      <atom:link href="https://opalstackstatus.com/affected/shared-singapore-sg/index.xml" rel="self" type="application/rss+xml" />
    
    
      <item>
        <title>[Resolved] MariaDB down on OPAL2.</title>
        <link>https://opalstackstatus.com/issues/2022-05-29-mariadb-down-on-opal2/</link>
        <pubDate>Sun, 29 May 2022 04:51:00 +0000</pubDate>
        <guid>https://opalstackstatus.com/issues/2022-05-29-mariadb-down-on-opal2/</guid>
        <category>2022-05-29 18:13:07</category>
        <description>&lt;p&gt;&lt;em&gt;Fixed&lt;/em&gt; - There have been no further issues. 
  &lt;span class=&#34;faded&#34;&gt;(18:13 UTC — May 29)&lt;/span&gt;

&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Watching&lt;/em&gt; - The MariaDB server is back online and fully functional. We are continuing to monitor the server. 
  &lt;span class=&#34;faded&#34;&gt;(14:32 UTC — May 29)&lt;/span&gt;

&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Identified&lt;/em&gt; - The restore is complete and we&amp;rsquo;re running a final check on the entire DB service. 
  &lt;span class=&#34;faded&#34;&gt;(13:38 UTC — May 29)&lt;/span&gt;

&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Identified&lt;/em&gt; - We are still restoring databases to the MariaDB server. 
  &lt;span class=&#34;faded&#34;&gt;(12:07 UTC — May 29)&lt;/span&gt;

&lt;/p&gt;</description>
        <content type="html">&lt;p&gt;&lt;em&gt;Fixed&lt;/em&gt; - There have been no further issues. 
  &lt;span class=&#34;faded&#34;&gt;(18:13 UTC — May 29)&lt;/span&gt;

&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Watching&lt;/em&gt; - The MariaDB server is back online and fully functional. We are continuing to monitor the server. 
  &lt;span class=&#34;faded&#34;&gt;(14:32 UTC — May 29)&lt;/span&gt;

&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Identified&lt;/em&gt; - The restore is complete and we&amp;rsquo;re running a final check on the entire DB service. 
  &lt;span class=&#34;faded&#34;&gt;(13:38 UTC — May 29)&lt;/span&gt;

&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Identified&lt;/em&gt; - We are still restoring databases to the MariaDB server. 
  &lt;span class=&#34;faded&#34;&gt;(12:07 UTC — May 29)&lt;/span&gt;

&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Identified&lt;/em&gt; - The databases have been exported, the InnoDB data and log files have been rebuilt, and we are currently importing all databases. 
  &lt;span class=&#34;faded&#34;&gt;(09:04 UTC — May 29)&lt;/span&gt;

&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Identified&lt;/em&gt; - The MariaDB InnoDB data files and logs have become corrupt and our efforts to fix the corruption have not succeeded. In order to fix this we are exporting the databases, rebuilding the InnoDB data and log files, and then importing the databases. 
  &lt;span class=&#34;faded&#34;&gt;(07:01 UTC — May 29)&lt;/span&gt;

&lt;/p&gt;
&lt;p&gt;MariaDB is down on OPAL2. We are investigating the cause and working to restore service. 
  &lt;span class=&#34;faded&#34;&gt;(04:51 UTC — May 29)&lt;/span&gt;

&lt;/p&gt;
</content>
      </item>
    
      <item>
        <title>[Resolved] Service disruption caused by firewall issue</title>
        <link>https://opalstackstatus.com/issues/2022-02-10-service-disruption-caused-by-firewall-issue/</link>
        <pubDate>Thu, 10 Feb 2022 19:43:00 +0000</pubDate>
        <guid>https://opalstackstatus.com/issues/2022-02-10-service-disruption-caused-by-firewall-issue/</guid>
        <category>2022-02-10 19:48:19</category>
        <description>&lt;p&gt;About an hour ago we deployed a firewall update that had the unintended side effect of blocking our resolving DNS servers.&lt;/p&gt;
&lt;p&gt;This caused various issues on customer sites on shared hosting, usually showing up as a &amp;ldquo;504 Gateway Timeout&amp;rdquo; error.&lt;/p&gt;
&lt;p&gt;The issue has been resolved and all customer sites should be working now as they were prior to the change. 
  &lt;span class=&#34;faded&#34;&gt;(19:43 UTC — Feb 10)&lt;/span&gt;

&lt;/p&gt;</description>
        <content type="html">&lt;p&gt;About an hour ago we deployed a firewall update that had the unintended side effect of blocking our resolving DNS servers.&lt;/p&gt;
&lt;p&gt;This caused various issues on customer sites on shared hosting, usually showing up as a &amp;ldquo;504 Gateway Timeout&amp;rdquo; error.&lt;/p&gt;
&lt;p&gt;The issue has been resolved and all customer sites should be working now as they were prior to the change. 
  &lt;span class=&#34;faded&#34;&gt;(19:43 UTC — Feb 10)&lt;/span&gt;

&lt;/p&gt;
</content>
      </item>
    
      <item>
        <title>[Resolved] Let&#39;s Encrypt rate limit for opalstacked.com subdomains</title>
        <link>https://opalstackstatus.com/issues/2020-10-21-lets-encrypt-rate-limit-for-opalstackedcom-subdomains/</link>
        <pubDate>Wed, 21 Oct 2020 17:40:00 +0000</pubDate>
        <guid>https://opalstackstatus.com/issues/2020-10-21-lets-encrypt-rate-limit-for-opalstackedcom-subdomains/</guid>
        <category>2020-11-03 19:34:59</category>
        <description>&lt;p&gt;&lt;em&gt;Fixed&lt;/em&gt; - We&amp;rsquo;ve made some changes to the way Let&amp;rsquo;s Encrypt works with the shared opalstacked.com domain and as a result the domain is no longer rate-limited. We&amp;rsquo;ll be posting updated documentation with the details later this week. 
  &lt;span class=&#34;faded&#34;&gt;(19:34 UTC — Nov 3)&lt;/span&gt;

&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Watching&lt;/em&gt; - Looks like we&amp;rsquo;ve hit our new limit, we&amp;rsquo;ll get this corrected ASAP. 
  &lt;span class=&#34;faded&#34;&gt;(20:56 UTC — Oct 27)&lt;/span&gt;

&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Fixed&lt;/em&gt; - We&amp;rsquo;ve confirmed that our increased rate limits are in effect, Calling this fixed, thanks for your patience :-) 
  &lt;span class=&#34;faded&#34;&gt;(16:36 UTC — Oct 25)&lt;/span&gt;

&lt;/p&gt;</description>
        <content type="html">&lt;p&gt;&lt;em&gt;Fixed&lt;/em&gt; - We&amp;rsquo;ve made some changes to the way Let&amp;rsquo;s Encrypt works with the shared opalstacked.com domain and as a result the domain is no longer rate-limited. We&amp;rsquo;ll be posting updated documentation with the details later this week. 
  &lt;span class=&#34;faded&#34;&gt;(19:34 UTC — Nov 3)&lt;/span&gt;

&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Watching&lt;/em&gt; - Looks like we&amp;rsquo;ve hit our new limit, we&amp;rsquo;ll get this corrected ASAP. 
  &lt;span class=&#34;faded&#34;&gt;(20:56 UTC — Oct 27)&lt;/span&gt;

&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Fixed&lt;/em&gt; - We&amp;rsquo;ve confirmed that our increased rate limits are in effect, Calling this fixed, thanks for your patience :-) 
  &lt;span class=&#34;faded&#34;&gt;(16:36 UTC — Oct 25)&lt;/span&gt;

&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Watching&lt;/em&gt; - The LE service has increased our limit and at this time we are not seeing any rate-limiting for new LE certificates and renewals on any domains. We&amp;rsquo;ll continue to monitor. 
  &lt;span class=&#34;faded&#34;&gt;(16:03 UTC — Oct 24)&lt;/span&gt;

&lt;/p&gt;
&lt;p&gt;At this time the Let&amp;rsquo;s Encrypt service is rate-limiting certificate issue and renewal requests for the entire *.opalstacked.com domain. This is the domain that we use to provide free subdomains to our hosting customers.&lt;/p&gt;
&lt;p&gt;We&amp;rsquo;re working with the LE team to get a higher limit. In the meantime if you are attempting to issue a LE cert for a site that uses your opalstacked.com subdomain then it may fail. If this happens you&amp;rsquo;ll receive a failure notice via email.&lt;/p&gt;
&lt;p&gt;Until the rate limit is raised then the alternative is to use one of your own registered domains on your site instead of your opalstacked.com subdomain. If you&amp;rsquo;re not ready to point your domain at Opalstack, then you can create a testing subdomain and A record via your NS provider to point that subdomain at your Opalstack server.&lt;/p&gt;
&lt;p&gt;We&amp;rsquo;ll update this issue as soon as the rate limit has been raised. 
  &lt;span class=&#34;faded&#34;&gt;(17:40 UTC — Oct 21)&lt;/span&gt;

&lt;/p&gt;
</content>
      </item>
    
  </channel>
</rss>
