A while back, I started learning and using Systemtap to help in understanding code flow, and in debugging. It's a really neat tool, but looking at it from the perspective of a tester, I immediately realized that it was going to be useful for fault injection. After some searching, the only thing I could find was a reference to the SCSI fault injection test harness. This was a good start, but obviously the same concept would be useful to apply to other areas of the kernel as well. Today, I noticed that someone else has finally started along that path.
It's fairly new, but the authors claim to have ported the SCSI fault injection test harness to their new fault injection framework. It is small, easy to understand, and they have provided some examples already. It also looks like it will have a home in the Systemtap tree fairly soon as well. This is one of those things I wish I had about 7 years ago!