Back in 1992 Queen Elizabeth II talked about annus horribilis. Now, since the romans didn’t had the notion of seven-day weeks it is perhaps a bit far-fetched to talk about a septimana horribilis but this week has been a week of surprises, none of which very pleasant.
Last Sunday a new season of the reality show Big Brother premiered on TV11, which spawned a large interest for the web cast of the events unfolding in a house in the outskirts of Stockholm. The ratings for the TV broadcast has subsided somewhat since but the interest on the web has been and is astonishing. Since we modularized the code we wrote for our main video service, TV4 Play, it was possible to release a video site on a tight deadline. We had however almost no time for load testing so the actual premiere was the first real load testing. On this first day we saw that the controller fetching updates from Twitter (via Apigee) was in the millions and there also was other parts of the code that could have been optimized. Luckily one of the benefits of running a service in the cloud is that we can compensate by adding more processing capacity with the flick of a button, by just increasing the number of Heroku dynos we could buy time. Despite this we decided on releasing a patch a few minutes after midnight that avoids unnecessary checks on the session and increases the interval at which Twitter updates is fetched.
At Tuesday night we are pretty sure that we have found the most critical parts that can be optimized. There are of course things that you cannot control, and on Tuesday night on of the large backbones in Europe run into difficulties. And for a couple of hours a few of our sites are so slow that they take forever to load.
On Wednesday afternoon our content management system, built upon Polopoly, fills up its logs with stack overflow errors. We are in luck and can find out the operation that got us there in an hour or so but it still takes four hours before we have fixed the corrupt database again. Our public service competitor SVT also have difficulties with their Polopoly installation this afternoon.
On Thursday the network communication consultants Cygate set up firewall rules that isolate our video management system from the rest of the world. Despite Cygate’s payoff “Alltid där” (Always there) the servers aren’t there and none of the videos on our sites can be watched.
On Friday a test on the platform used for video subscriptions goes wrong and for a while no-one can log in or see videos.
Somewhere between Friday and Saturday Vizrt, the company that encodes videos for mobile devices have a hardware failure and the problem isn’t fixed until Saturday night which means that no new content comes ut to our iPhone apps.
Since then we have experienced another errors on this Sunday and Monday but we certainly hope we get more time to actually do something productive this week.