2017-03-17

Recent Site Outage

A few days ago I pushed an update to the site. Although everything tested fine locally, apparently one of the included changes introduced a serious bug impacting performance. I haven't tracked down the cause yet, but I've rolled back the code which seems to have helped:

The black line spiking up is CPU usage. Normally it's around 2 minutes per hour (i.e. the host machine is actually sitting idle most of the time). For the last few days it's been around 50 minutes per hour - meaning it's been spinning its wheels like mad. And when it's like that, every request is extremely slow - minutes instead of milliseconds.

Sorry about that. :(

It may take some time to track down, and the odds are high that I'll push a bad release or two out while trying to bisect down to the exact bug - and it takes a few minutes of monitoring to even notice that it's occurring. So please be patient.

Ω

1 comment:

Bill Lowrey said...

Haven't noticed anything obvious, performance-wise, but the work is much appreciated. This kind of thing can be difficult to pin down. Thanks (for this, and just for having this site at all)