I don't think it's a matter of whether or not some kind of assault would occur. If you were (uninvited) in a room full of naked women, most, if not all of the women would feel violated to some degree that you were getting some enjoyment out of seeing them naked. As straight men, we understand that, and somehow that understanding translates to straight men feeling violated that gay men might be getting some enjoyment out of seeing us naked. I think it's dumb, but I understand where that connection comes from.