The Grid Status

February 2017

Pages not being built
Processing times are back to normal
Feb 4, 13:32-14:21 UTC

January 2017

No incidents reported for this month.

December 2016

Images not processing
This incident has been resolved.
Dec 2, 00:19-07:25 UTC

November 2016

New images not processing
A communication problem was found and fixed in our AMQP service.
Nov 23, 15:41-15:59 UTC
Increased error rates on image processing
Second fix was successful
Nov 2, 10:28-12:47 UTC

October 2016

.io domain resolution issues
The anomaly appears to be over.
Oct 28, 09:23-11:24 UTC
Occational 503
This incident has been resolved.
Oct 26, 20:37 - Oct 27, 01:04 UTC
Issues with DNS resolution
Services appear to have resumed for our US users.
Oct 21, 18:28-22:29 UTC

September 2016

Partial Outage for Web App
This incident has been resolved.
Sep 11, 22:32-23:07 UTC
New images not being processed
This incident has been resolved.
Sep 11, 01:56-02:47 UTC
[Scheduled] testing.thegrid.io -> app.thegrid.io move
app.thegrid.io is now the new (V2) webapp. Login and activation links with testing.thegrid.io still works via a redirect. The old webapp, is available on v1.thegrid.io In the future, it will be completely removed.
Sep 5, 09:00-10:38 UTC
Partial unavailability on testing.thegrid.io [503 ERROR]
This incident has been resolved.
Sep 4, 21:00-22:15 UTC
v2 webapp (testing.thegrid.io) down
This incident has been resolved.
Sep 4, 18:18-19:21 UTC
Websites give 503
This incident has been resolved.
Sep 1, 22:25-22:46 UTC

August 2016

Uploaded images not showing
Issue only affects a small subset of images
Aug 23, 06:39-07:02 UTC
Sites updates not being published
Backlog has been cleared and we are back to normal.
Aug 22, 04:51-05:23 UTC
Site serving outage
New load balancer setup seems to be operating nominally.
Aug 16, 15:01-16:12 UTC

July 2016

Websites giving 503 error
This incident has been resolved.
Jul 30, 11:57-13:18 UTC
Temporary site solving blockage
The blockage has been cleared and we're back in normal operation.
Jul 27, 07:51-08:23 UTC
Webpages served slower than normally
This incident has been resolved.
Jul 20, 21:30-22:08 UTC

June 2016

Webpages not shown, gives 504 instead
This incident has been resolved.
Jun 22, 20:34-20:38 UTC
Images loading slowly or timing out
We are back to normal, no issues seen the last 2 hours.
Jun 13, 18:47-21:14 UTC
Some images timing out
We've unclogged the processing pipeline and should be back to normal. Degraded performance with timeouts may have been experienced from 12.00 UTC to 22.30 UTC.
Jun 6, 22:17-22:34 UTC
[Scheduled] Database maintenance
The scheduled maintenance has been completed.
Jun 6, 10:25-12:50 UTC

May 2016

Unable to load posts or publish
A fix for the issue was deployed at 19.45 UTC
May 22, 18:36-21:45 UTC
Pages gives 503 error
Been running OK for last 12 hours, albeit with reduced redundancy.
May 10, 14:50 - May 11, 10:19 UTC

April 2016

Images not processing
The backlog has now been cleared.
Apr 27, 22:46-22:57 UTC
Changes not getting published
This only affected 2 new sites, not every site as previously assumed. The two sites were also fixed at around 12.00 UTC.
Apr 27, 09:57-15:38 UTC

March 2016

[Scheduled] Database maintenance
Database upgrades completed. We're monitoring the performance of the new setup.
Mar 21, 09:00-11:51 UTC
Issue processing solved pages
This incident has been resolved.
Mar 15, 08:30-09:18 UTC
Images often not showing for content cards
This incident has been resolved.
Mar 14, 10:13-12:21 UTC
Page builds and image processing slower than normal
Back to normal processing times.
Mar 9, 22:14-22:27 UTC
Site updates not coming through
Site redesigns are now processing as normal again.
Mar 6, 21:41-21:48 UTC
[Scheduled] Scheduled database maintenance
The scheduled maintenance has been completed.
Mar 4, 20:00-20:08 UTC
Slow page serving
We have updated the IPFS caching network to increase performance.
Mar 4, 14:35-17:42 UTC
Occational 504 errors for sites
We've increased the number of backend servers, to be more robust against intermittent glitches.
Mar 4, 11:17-12:28 UTC

February 2016

504 or timeout for pages
This incident has been resolved.
Feb 15, 17:55-19:04 UTC
Image analytics stuck
This incident has been resolved.
Feb 12, 07:31-11:09 UTC
Site updates not coming through
This incident has been resolved.
Feb 10, 01:26-01:42 UTC
Changes not getting published
All changes have now been published, and we're operating as normally.
Feb 1, 21:53-22:13 UTC

January 2016

[Scheduled] [scheduled] Site serving proxy update
The scheduled maintenance has been completed.
Jan 31, 20:52-21:11 UTC
Disruption with site serving network
Services should be fully recovered now.
Jan 15, 19:52-20:38 UTC

December 2015

Sites may not update after configuration change
This incident has been resolved.
Dec 12, 16:53-19:56 UTC
Some images not showing up
Measurement backlog has been cleared and system has been in normal operation for some time.
Dec 7, 14:11 - Dec 10, 09:59 UTC

November 2015

Database availability problem
According to our database provider, our database servers had a hardware or networking failure that has now been resolved.
Nov 12, 20:14-20:26 UTC
503 for api.thegrid.io
Fix has been deployed and verified.
Nov 6, 17:31-17:54 UTC

October 2015

Platform downtime
Heroku routing is back up
Oct 21, 10:43-12:18 UTC
Some images not showing (500)
A Redis DB cache had exceeded configured memory, and prevented retrieval of image URLs We've flushed the cache and are now operating without issues. Measures for preventing issue from occuring again has been planned.
Oct 20, 22:48-23:34 UTC
[Scheduled] Database upgrade
The scheduled maintenance has been completed.
Oct 7, 15:30-16:22 UTC
New images not processed
A bug in new deploy caused images to not process between Friday 20.50 UTC and Saturday 12.30 UTC. The deploy has now been reverted and all queued jobs have been completed.
Oct 3, 12:47 UTC
Long image load times
A particular image processing request was causing workers to get killed off continiously, causing degraded load times and occational 503 responses for images on newly built pages. The offending image processing request has been processed&cached now, and we are adding measures to prevent this from happening again.
Oct 2, 15:57 UTC

September 2015

No incidents reported for this month.