This happens to me all the time as I tend to log in and out a lot because I play so many different games during the day.
I was pretty sure that what Windypuff says about the "ghost" queue was happening, but personally I like to just quit the game and start it up again. That way I have more control over what's happening.