After a long hiatus from blogging I have decided to get back online. I feel like there is plenty to comment on regardless of whether anyone is listening or not.
On a professional note; I just had to cancel a vacation in order to debug a performance problem at work. It is one of those particularly gnarley problems that does not occur regularly. It is always interesting to get back into the mindset of solving performance problems. They require an extreme amount of rigor and analysis. It's a different mindset than pure development.
I find then when developing I will often change several things at once and then re-run my code testing each of my changes. Typically the changes I make are independent of one another; that is I don't try and fix the same thing more than one way. It reminds me of a couple of core tenets of debugging these types of problems…
1. Jim McCarthy (formally of the VC++ team) has a principle from his book that states – get to known state and stay there. This is paramount to trying to solve these types of issues. You can’t introduce change into something that is not stable.
2. Change one variable at a time. Measure. Correlate. Confirm. Consider changing another variable. We had someone change two things in the environment and the problem disappeared. It’s bad enough it happens sporadically, but now we cannot attribute the disappearance to any one thing.
3. Get to a simple reproducible case as soon as possible. In our case it takes us three hours to recreate the problem – IF (big if) it’s going to happen again. That means we only get 2-3 attempts per working day. We are pretty close to getting that down to an 18 minute case; which means we spend much more time working (not waiting).
Time to get back to running tests rather than writing about them.
No comments:
Post a Comment