That depends on what you're complimenting.
If you're telling a guy he's hot, people are going to think you're gay no matter what. If you tell him he has a nice ***... same deal.
If you're complimenting him on something like a t-shirt or a golf swing, it's all about how you say it.
Ways to compliment a t-shirt:
"Nice shirt, it new?"
"Hey, when did you get that shirt?"
...then followed up with the where is it from, how much, etc. You don't compliment a guy on his appearance, you compliment him on something he owns, and may like to own yourself.
Ways NOT to compliment a t-shirt:
"That color looks good on you."
"That's a nice shirt, it fits you well."
...These are examples of complimenting a guy's appearance, and not his possessions.
Ways to do it in golf:
"Geez, that ball is gone."
"Do that again and we're giving you a handicap."
...These refer to his sporting prowess, and the competition at hand.
Ways NOT to do it in golf:
"I love watching you swing that club"
"You move your hips perfectly during those swings"
...These refer to the his appearance, or how he moves his body. Guys should not notice these things, or atleast not verbalize that they do. They are unrelated to the competition and speak of an admiration of him as a man.
Hopefully that helps :P