The Gold CD

I try to be pretty conscience of what I install on a PC whether it’s a test PC or my own laptop. So I was surprised when I was working with Mike Kelly this weekend and he said well just drop Ruby onto the test machine. My reflex was to safeguard the laptop. Nothing against Ruby, I just like to control my environment and wanted to think about what the install might do. A little investigation shows that a Ruby install adds 3,000 folders and 21,000 files. Yeah twenty-one thousand files.

Ruby also added 48 keys to the registry and 109 values. I should mention that I spent several years testing installs so my reflex comes from experience. One product I tested for a number of years, and at the end of the dev process, I’d test the install as we worked towards the gold CD. (The gold CD is the CD that went to manufacturing. I’ve also done sample tests of the manufactured CD to ensure the production quality of manufacturing. I could digress on the topic of sample size but not today.) I’ve never been able to just install an application since then. In fact, I installed Ruby on a scrap laptop first to check out the install. Old habit.

This also led me to find once again that a couple of old favorite utilities don’t work well with Vista. This isn’t a gripe against Vista, every now and then I go through a pruning process of my test utilities for different reasons. In fact, I blogged about the tool I was hunting for this past weekend. I needed a better tool to checkout dlls loaded in the environment.

So a couple of points. One, if Mike Kelly was willing to drop software onto a laptop without knowing what (if anything) it might add or alter to the environment, then ouch, no doubt testers could tamper with an environment unknowingly. I joked with Mike about this for the balance of the weekend. It was pretty funny. So you might want to think about how you may alter an environment as a factor in the old “it used to work on my machine.”

Did you know that even the order in which you startup applications can matter? How? If multiple applications use the same dll and the same dll is loaded on a workstation in different directories – then the first application started loads the dll. It’s possible (I’ve witnessed this) to have the next application use the loaded dll instead of calling the version it may need from a different directory. See shared in-memory on the following reference and you can see why I want/need a detailed view of the dlls loaded. DLL hell can get a little crazy and take some meticulous tracking to figure it out.

If you want more details on install testing, I wrote an article on installs for Better Software recently, you can find it here. It’s called Navigating the Install.

A second point – when you’re trying to isolate factors, consider the environment. An investigation in this direction has often resolved the “it works on my machine” syndrome. I’ve used tracking tools to see what a developer has loaded on their workstation versus my test PC. On my cleanest personal laptop, I keep a running notepad file with the date and what I’ve installed so at least I can backtrack and possibly determine if a recent install is a culprit. It’s a minor chore and I think worth it. I track toolbar and browser add-in’s too because anything dropped on a PC counts.

21,000 files huh? Mmm.

Comments

It never ceases to amaze me how similar experiences can lead different people to near opposite "reflexes" (my brother and I are great examples of this that I won't get into).

Installing software screws stuff up *all the time*. Which is why I'm constantly downloading utilities, opensource software, crap that's likely to be a virus but looks interesting, why I don't frequently leave running virus scanners, malware detectors, spyware killers, etc. My thought process being "There is exactly one state I'm sure that no user's machine will be in (at least for long)... Clean. If I treat the test machine like a user is likely to treat their machine and the software *still* works, it's probably pretty stable."

This also comes from "Since I can't prove that the software will ever actually work, I might as well spend my time demonstrating complex cases where it does (or doesn't) work".

Your reflex makes *at least* as much sense as mine. I can almost never figure out *what* the problem is on my "anti-clean" test machines, but sure do find some cool bugs, high priority bugs that wouldn't have been found otherwise!

This is probably not a question of either/or, it's probably a good case for a team of testers who are not carbon copies of one another and having multiple test environments!

--
Scott Barber
Chief Technologist, PerfTestPlus
Vice President & Executive Director, Association for Software Testing
sbarber@perftestplus.com

Scott,

I should point out that I don't just work on pristine environments. When I'm testing an install, then I test with clean so I can be sure when we ship a product what happens with the install on a specific environment. But I also test with what I refer to as "dirty" environments because dirty is more realistic. More and more people have no idea what's on their PCs between plug-ins, downloads, etc. And this is where the other utilities become so valuable - tools like dependency walker are great ... see Adam's comment on my post re: seeking a tool.

As for virus scannning - I keep a fairly tight eye on my machines. It's not uncommon for me to kick off full scans before I sleep. I also run automatic detections on my network that kick off every hour. I regularly scan all flash drives and my external drives too. I think paranoid is a good thing.