2017-03-17

Recent Site Outage

A few days ago I pushed an update to the site. Although everything tested fine locally, apparently one of the included changes introduced a serious bug impacting performance. I haven't tracked down the cause yet, but I've rolled back the code which seems to have helped:

The black line spiking up is CPU usage. Normally it's around 2 minutes per hour (i.e. the host machine is actually sitting idle most of the time). For the last few days it's been around 50 minutes per hour - meaning it's been spinning its wheels like mad. And when it's like that, every request is extremely slow - minutes instead of milliseconds.

Sorry about that. :(

It may take some time to track down, and the odds are high that I'll push a bad release or two out while trying to bisect down to the exact bug - and it takes a few minutes of monitoring to even notice that it's occurring. So please be patient.

Ω

No comments: