OK, I tried with that, and it still happened.
But then I played around a bit more, and it seems the problem was I was using an alert to check if it fired twice, and I kept getting two alerts. I switched to using console.log and only one message was logged. I combined both, so had an alert and a log in the handler, and got two alerts and two logs to the console.
So, it seems it is the alert that is causing the problem for me. I guess because it holds up execution of the handler, so the unbind doesn't happen until the alert is called.
I'm new to jQuery, so don't know enough of the inner workings to know if there is a clever workaround for this. Although, I've also only just discovered console.log, and so will probably be using that instead of alert from now on anyway, so may not encounter the problem again.