In my experience, it's a variable. Drunks are always a drag; children can be a real pain. Then again, so can anyone who is disrespectful of the performer: I just can't stand rudeness in any context. And speaking of that, magicians can be a real pain too, in that they can think their way is the ONLY way, instead of considering that other people can do something differently. . . an d just as well. Had a guy in a magic shop in Montreal hold forth on the superiority of ALL his presentations--twas enough to make me barf.
Fortunately, in every crowd, even if there is someone who's a hassle, I've found almost everyone else is really fun and easy to deal with.
Ã…lso, even though it seems to be a personal affront to have the rotten spectator in the audience, the rest of the audience can't stand him/her either, so the energy works against him/ her and for you (though it sure can be hard to accept that at the time).