There are many reasons to be against gay rights in general and gay marriage in particular:
Democrats against gay marriage
but as the secular left continues to define the mainstream on this issue, the arguments in favor continue to evolve. The "how does it hurt you" is a powerful one for most people who pretty much don't think too deeply on any issue.
One way it hurts all of us is that we have broken down our marriage culture in this society for decades. Beginning with no fault divorce, and a whole legal structure that supports the notion of individual rights within a marriage. We have a mainstream media that promotes a conflicted notion of gender equality where women basically get to choose when they are equal and when they are "partners". How many times have you seen a women hit a man on tv and there be no reprecussions? Of course "he deserved it" but if women and men are equal... you know the rest.
Allowing and institutionalizing gay relationships as equal to normal ones further solidifies the individualistic views of relationships. Marriage is about your personal happiness. Whatever you may want at the time. So you can be married for 10 years with two kids. Then "reveal" you were gay all along and we as society must not judge you harshly. Also as a young man, if you're single, married, gay, single father, whatever. Doesn't matter. By allowing the gay to be celebrated, it ensures the rest of the circumstances must be accepted.
Today nearly 7 out of 10 black children were born out of wedlock to parents not married. For whites it is up to 4 out of 10. It was thought to be a crisis in the black community in the 1960's when it was 4 out of 10. And it was. The lack of stability. The lack of moral structure. The lack of economic vitality. All serve to hurt all of us. More of the bad: crime, disease, disfunction. That we as a society have to deal with one way or the other.
That's one way gay marriage hurts you and me. It is setting up circumstances that promote bad situations for society. The promotion of the idea that "it's none of your business" when we are all in each others' business. We pay for each other's healthcare, education, and the results of bad choices.
We need to re-establish higher standards in society rather than lower them to satisfy people's instincts, urges, or fetishes. Our society needs strong marriages for the promotion of what's right and good. It's helps men and it helps women.