This Blog Has Moved!

Right, so yes, five years ago I moved to github pages, and never bothered to redirect any of these pages there. Now I've moved on from there, and... Finally I am using my real domain, trishagee.com . My blog is now at trishagee.com/blog .  See you there!
Post a Comment

Christmas decorations teach me a lesson about troubleshooting

And now, after an absence of several weeks, you get to see how long it takes me to write some of these posts.


I was putting up the Christmas decorations one Saturday when my worst fear was realised1 - one of my three strings of lights was not working.

The first two went up fine.  The third lit up when I plugged it in, and in less than a second went out.  Curses.  This is not what I wanted, this was supposed to be a short exercise in making my tiny little flat look festive.

So I set about the tedious task of starting from the end closest to the plug and replacing every bulb, one by one, with a spare one to see if it magically lit up again.  When it doesn't, you take the spare back out and replace it with the original bulb.  I remember my parents going through this ritual every Christmas, the tediousness of this activity is more memorable than the fleeting joy of shinies.

While I was doing this, my mind was back on the job I'd been doing at work the previous week - battling an Internet Explorer 7 performance problem.  We have automated performance tests which give us an indication of the load time for our application in Chrome and IE, and some time in the previous couple of weeks our IE performance had significantly degraded in the development code.  Due to a number of too-boring-to-explain-here circumstances, the last known good revision was four days and nearly 250 revisions earlier than the first revision that showed the performance problem.

Since we couldn't see anything to indicate it was an environmental problem, the logical next step was to pinpoint the revision which caused the problem, so we could either fix it or get performance gains from somewhere else in the system.


The most obvious way to do this, given there were no obvious suspects, is with a binary search of the revisions.  Our last known good revision was 081, our first poor performing one was 240.  So the thing to do is to check revision 160, see if it falls on the poor or good performance side.


If 160 proves to be a poor performer, check revision 120....


...if 160 is fine, test revision 200...

...and keep splitting the revisions by half until you find the suspect.

So of course that's what I want to do with my stupid Christmas lights.  I do not want to sequentially check each light bulb, that has a worst-case number-of-bulbs-tried = n, where n is the number of bulbs (probably a couple of hundred, although it felt like several thousand).  So, in computer speak, O(n).  The binary search algorithm is O(log n).  At university, this Big Oh had no context for me.  But when you've taken 10 minutes to get a quarter of the way through your Christmas lights, and you diagnosed your IE performance problems... well, actually it took days.  But the point is, a binary search for the missing bulb would definitely have been a Good Thing.

I know you're dying to know if I tracked down the problem in Internet Explorer - I did.  What's the worst case when you're doing a binary search?  It's when the thing you're looking for is veeeery close to either your start point or your end point.

The revision number I was after was number 237.  Sigh.


And my Christmas tree lights?  Well, through the boredom I remembered that modern lights are in sections, so they have a sort of built-in binary search - well, limited segments will go dark if a single bulb is out - which allows you to narrow down your problem area.  Since the whole string was out, I figured something else was probably wrong.

Turned out the plug had come out of the socket.

So:
Lesson 1: Theoretical computer science does have a place when you care about how long something takes.  When it's you feeling the pain, you'll do anything to make it stop.

Lesson 2: When diagnosing a problem you will always biased towards what you think it is, in the face of actual evidence.  I was afraid I would have to search the whole set of lights for a blown bulb, so that's the problem I was looking for when the lights failed.  In actual fact it was a problem with a much simpler solution.



1 - OK, "worst fear" in this very limited context only - it's not like I lie awake at night in July afraid that one of the bulbs on my Christmas lights has blown.

Comments

Post a Comment

Comments have been disabled since this blog is no longer active.

Popular posts from this blog

Dissecting the Disruptor: What's so special about a ring buffer?

Image
Recently we open sourced the LMAX Disruptor , the key to what makes our exchange so fast.  Why did we open source it?  Well, we've realised that conventional wisdom around high performance programming is... a bit wrong. We've come up with a better, faster way to share data between threads, and it would be selfish not to share it with the world.  Plus it makes us look dead clever. On the site you can download a technical article explaining what the Disruptor is and why it's so clever and fast.  I even get a writing credit on it, which is gratifying when all I really did is insert commas and re-phrase sentences I didn't understand. However I find the whole thing a bit much to digest all at once, so I'm going to explain it in smaller pieces, as suits my  NADD audience. First up - the ring buffer.  Initially I was under the impression the Disruptor was just the ring buffer.  But I've come to realise that while this data structure is at the heart of the patte
Read more

Dissecting the Disruptor: Writing to the ring buffer

Image
This is the missing piece in the end-to-end view of the Disruptor.  Brace yourselves, it's quite long.  But I decided to keep it in a single blog so you could have the context in one place. The important areas are: not wrapping the ring; informing the consumers; batching for producers; and how multiple producers work. ProducerBarriers The Disruptor code has interfaces and helper classes for the Consumer s, but there's no interface for your producer, the thing that writes to the ring buffer.  That's because nothing else needs to access your producer, only you need to know about it.  However, like the consuming side, a ProducerBarrier is created by the ring buffer and your producer will use this to write to it. Writing to the ring buffer involves a two-phase commit.  First, your producer has to claim the next slot on the buffer.  Then, when the producer has finished writing to the slot, it will call commit on the ProducerBarrier . So let's look at the first bi
Read more

Dissecting the Disruptor: How do I read from the ring buffer?

Image
The next in the series of understanding the Disruptor pattern developed at LMAX . After the last post we all understand ring buffers and how awesome they are.  Unfortunately for you, I have not said anything about how to actually populate them or read from them when you're using the Disruptor. ConsumerBarriers and Consumers I'm going to approach this slightly backwards, because it's probably easier to understand in the long run.  Assuming that some magic has populated it: how do you read something from the ring buffer? (OK, I'm starting to regret using Paint/ Gimp .  Although it's an excellent excuse to purchase a graphics tablet if I do continue down this road.  Also UML gurus are probably cursing my name right now.) Your Consumer is the thread that wants to get something off the buffer.  It has access to a ConsumerBarrier , which is created by the RingBuffer and interacts with it on behalf of the Consumer .  While the ring buffer obviously needs a se
Read more