As I figured out, XUL documents are identified as XML documents (via isXMLDoc()) and that creates a problem when you want to operate with certain elements with jQuery in XUL (radio button, checkboxes, various lists etc).

The problem is with these lines:

var fix = jQuery.isXMLDoc(elem) ? {} : {
	"for": "htmlFor",
....
	selected: "selected"
};

The quick fix was to do this:

var fix = jQuery.isXMLDoc(elem) ? {value: "value"} : {
....
};

The quick overview of the code following the fix shows no big problems, but without any test/build environment for jQuery that is all I can do about it.