It would be so easy to dog pile on Microsoft’s poor, beleaguered media player. Take it from me, daylight savings and leap year handling is no fun and easy enough to get wrong or fail to test properly. These sorts of bugs make it past quality assurance from time to time, more often than we like to admit really. You remember those routers a few years back that all checked their clocks against an external source in lock step, bogging down large networks?
Edge cases are hard to test by definition. In retrospect, a leap year seems easy enough to check but think about all of the core cases QA has to cover with a media device. Playback, media synchronization, all the UI bits, battery handling, and many more besides. In thinking about the full test suite for a media player, would it occur to you to permute the system clock through each day of a standard year, let alone a leap year?
It turns out that in the Zune case, the bug apparently originates in a lower level chip driver. So even if Microsoft had tested their own software stack fully, that doesn’t guarantee they would have flushed out this integration issue.
The difficulty of testing the unanticipated is why techniques like fuzzing were developed. This is a permutation of a security axiom, though. An engineer can easily build a system that they cannot manage to break, this doesn’t mean the system lacks faults. It says more about the biases and perceptions of the engineer. At least fuzzing illustrates a bit of creativity in trying to get past an engineering team’s built-in limitations.
The open source development model, in contrast to Microsoft’s secretive methods, would be easy to advance as an anodyne. It has its advantages in terms of transparency, that many more engineering eyes would be combing through the code increasing the odds of spotting a problem like this. There is also such a rich tradition of re-use that allows any given project to build on the momentum of core libraries benefit from external achievements in quality and functionality. For popular libraries and tools, many other users have tackled the integration scenarios a new project is likely to encounter. Using an open source library means higher level projects can feed fixes back into lower level components, fixing their own issues as they encounter them.
Unfortunately, testing is one of the areas where open source projects are constantly short handed. Everyone wants to write new features, finding bugs and submitting patches just seem less glamorous. Many projects require a certain level of patches submitted by a potential contributor before granting them full commit rights to the code repository. I am sure that helps, to a degree, but I think it may as often chill interest in contributing code.
My only real takeaway from this little debacle is to be reminded that hacking on code is only one element of a successful project. I read many excellent thinkers on the subject of incorporating security and usability alongside core development on projects. Quality testing may not even have the appeal of these other not strictly coding aspects of projects but it needs its own top notch advocates who bring the same creativity and zeal to reduce the odds of an embarrassing but easy to commit defect like this own escaping into the wild.