I'm having a similar problem (infinite loop, same line), but it has nothing to do with the :not selector, so this may help in chasing down the bug. Given the following HTML structure:
This query causes the infinite loop:
Notably, these queries do NOT cause loops:
It seems that the trouble comes when you have an initial selector (the A at the end) that does match at least one element, so checkSet isn't empty, then you step back to a selector that fails for ALL elements in checkSet (the span selector), so checkSet is now filled with false, then you step back one more level to a selector that has both a tag and a class specifier. preFilter.TAG() goes into an infinite loop trying to find an element to run isXML() on. I'm not sure where the fix for this should go, but that should be enough to track it down, hopefully.
Incidentally, I'm running Fx3.0.4 on Linux with Firebug enabled.