Ahh I hadn't tested transitional, just strict and quirks. Great test cases by the way. In the end, it was the same problem. Since it was quirks AND transitional I did some more digging. As near as I can tell, when using fx.show() and fx.hide() the units can only be "px" so instead of patching fx.custom() it looks like patching fx.show() and fx.hide() is the way to go. It seemed to work fine starting and ending at 1 instead of 0 for all browsers I tested (IE6/7, FF2, Opera 9) so it doesn't have to be special cased.
The only issue I can see here is if someone uses fx.custom() directly then it might flash if they start it at zero. It might be something that should be covered in documentation instead of code though.
I attached a patch with the changes. Any help testing the millions of possible situations to make sure the bug is really fixed and it didn't mess up anything else would be great.