once and memory are used by $.Deferred.
unique, stopOnFalse and addAfterFire are useful for event-like lists (unique ensures the same handler will only be called once, stopOnFalse ensures you can "interrupt" firing the list, addAfterFire makes it so when you add a new callback to the list while it is firing then said callback won't be fired as part of current run).
relocate I added after a discussion with Yehuda at jqcon when he told me he had a need for "Deferreds that would have some kind of callback re-ordering facility" (a reverse option would be nice to have callbacks called from the last to the first one in the same vein btw).
queue is more of an experimental flag I put hoping, at the time, that we may be able to replace existing queues with $.Callbacks-based queues.
Apart from queue, these flags are pretty generic in what they do. Just tried to cover as much ground as possible. Not sure about the flag names to be honest, as usual, any help with that is more than welcome.
I'm also not sure the complexity added with the filter parameter is really needed. Just let me know what your review brings and I'll gladly cut (this was the sense of my request for a code review).