When writing a custom selector and giving it a function as the rule, returning truthy from the function passes the check, and returning falsy fails. That is the case unless the function returns undefined. Although undefined is falsy, the check is considered to have passed.

While I think any such function should return a strict BOOL, there is no official documentation (that I can find, anyway) on how a custom selector should be written and what these functions should be returning.

This stems from a trouble ticket opened to TinyMCE ( http://www.tinymce.com/develop/bugtracker_view.php?id=4975 ). I believe they need to correct their function to return BOOL, but I felt this should be brought to the attention of the jQuery core team as it could be happening in many other places.

See this fiddle: http://jsfiddle.net/Pw43h/