Recent Site Outage

A few days ago I pushed an update to the site. Although everything tested fine locally, apparently one of the included changes introduced a serious bug impacting performance. I haven't tracked down the cause yet, but I've rolled back the code which seems to have helped:

The black line spiking up is CPU usage. Normally it's around 2 minutes per hour (i.e. the host machine is actually sitting idle most of the time). For the last few days it's been around 50 minutes per hour - meaning it's been spinning its wheels like mad. And when it's like that, every request is extremely slow - minutes instead of milliseconds.

Sorry about that. :(

It may take some time to track down, and the odds are high that I'll push a bad release or two out while trying to bisect down to the exact bug - and it takes a few minutes of monitoring to even notice that it's occurring. So please be patient.



Unknown said...

Haven't noticed anything obvious, performance-wise, but the work is much appreciated. This kind of thing can be difficult to pin down. Thanks (for this, and just for having this site at all)

Unknown said...

its wonderful and awesome articles. its very useful information for people