You used to envy this man. Sitting there with his hand on a 23-year-old's thigh while he sips his latte, he makes your banana-pancake domestic life feel lame. But lately that guy's beginning to seem—to you, your friends, and your wife—well, kind of creepy. His brazen rejection of the life stage that most of his peers have gotten to is starting to make it look like there's something wrong with him.

Joe (who asked that only his first name be used), a 39-year-old union organizer in New York who's never been married, has been getting disapproving looks from his friends ever since he turned 30. "There is nothing like a group of married people—especially with kids—when you come into their circle with a younger, thin woman," he says. "It's a terrible reaction."

"These guys get labeled playboy, loser, commitment-phobe," says Carl Weisman, author of So Why Have You Never Been Married? According to U.S. Census Bureau statistics, in 1980 only 6 percent of men between 40 and 44 had never been married; in 2008 it was 16 percent. But even though there are more of them around, men with long-term single status still have a hard time explaining their situation to potential dates, who see a guy entering middle age without ever having been married as damaged goods. In fact, a man whose marriage failed spectacularly tends to arouse less suspicion than a straight, still-single 41-year-old. "If he's over 40, you would hope that he's divorced," says Janis Spindel, a high-end matchmaker in New York who gets calls from hundreds of single women asking for setups. Evidence that even unmarried men in their mid-thirties are suspect is in her fee structure: The up-front charge for guys under 35 is $25,000; for those 35-plus it's $50,000.

If you ask a guy in his late thirties or early forties why he isn't married, he'll have his answer—you could call it his defense—ready. For some, the rationale is a self-fulfilling prophecy. Joe tends to date women younger than he, who are less likely to want to settle down than those his own age. "I would still like to have kids," he says. "But if I date someone who's 40, it's going to be chaos, a lot of pressure if we want kids—because we have to start that immediately, and even then you're not guaranteed."

Travis, 37, is a vow-averse management consultant in Baltimore whose singlehood is regarded with bemusement by his friends' wives because he has a 24-year-old girlfriend and takes trips to Vegas twice a year with married buddies. "Travis likes being spontaneous, doing new things," his friend Toni, a 33-year-old stay-at-home mom, says. "He has the misconception that if you're married it's a ball and chain." Besides, Toni adds, Travis "is the most socially skilled person I've ever met. He has no problem picking up the best-looking girl in the room." For anyone with that much game, being bonded to one person for life would be like losing a huge part of his identity. "I'm just not ready," Travis says. "A lot of people want to get married and are looking—I'm not." He holds that his approach to life reflects the way most men feel, and that the pent-up frustration of his married friends only confirms this. "They're the bad influences, and they're the ones who are married. Going out with me is an excuse to get crazy again," he says. "Some of them take off their wedding rings at bars just to talk to girls and flirt."

In fact, seeing friends' marriages fail can make never-wed guys more gun-shy. Which brings us to another stereotype: that these guys are afraid of being saddled with a less-than-perfect match. Eric Mark, 41, a partner at a national consulting firm in Los Angeles who last year was the best man at a friend's second nuptials (after having been a guest at the first), estimates that 70 to 75 percent of the weddings he's attended have resulted in broken marriages. Mark, like a lot of unmarried guys past 40, says that anecdotal evidence like that should be a satisfactory explanation for why he's still single. Why hustle down the aisle if you'll just end up single again at 45? But Mark's real problem may be his hustling. A peek at his BlackBerry calendar reveals he barely has enough free time to wink at a hot woman—let alone go on a date. He travels 30 to 50 times a year for his job, does pro bono work, and performs stand-up comedy at L.A. clubs. "I've spread myself so thin it makes me question how I can have a social life at all, work on existing relationships, let alone start a new one," Mark says. His friend Louisa, 37, says it's not time that's the issue; it's pickiness. "Eric has a very specific idea of what he likes," she says. His ideal partner is smart, compassionate, and well-educated and has a sense of humor. "And looks like Kate Beckinsale in The Last Days of Disco," Mark says.

This kind of overreaching is typical of the unmarried guy in his early forties, who tends to be convinced that nothing is good enough for him—that the perfect partner isn't right around the corner.

If the fortysomething unmarried isn't written off as being overly fussy or just plain weird, there is another label that he can find himself tagged with: gay. Travis, who is often teasingly called a diva by those close to him, recounts how five or six years ago a female friend said, "I really want to talk to you. Are you gay?" He laughs. "I was like, 'No,' and she's probably apologized for it 10 times since." Being an unmarried straight guy isn't exactly a career enhancer, either. Travis left a job where his lifestyle didn't fit the company culture. "I used to work for a commercial bank that was very conservative," he says. "They expect the country club, and taking customers and their wives out. That would be very awkward, because I certainly wouldn't be doing that."

Maybe some men know themselves too well to ever commit to a banana-pancake lifestyle. And maybe they don't deserve to be subjected to judgmental looks, because the kind of honesty that prevents them from entering a relationship just for the sake of it is the unmarried fortysomething's saving grace. Weisman believes that instead of looking askance at the perennially unmarried man, society should applaud him—even the cad. "The only thing worse than a playboy who refuses to commit is a playboy who gets married," he says.

Feel ostracized for staying single? Tell us about it below in the comments section.

Check out these top stories from Details:

MEET THE NEW BACHELOR
He's got money to burn and a line of women out the door—and the ink on the divorce papers is barely dry.

THE BIRTH-CONTROL EXTREMISTS
Young, single men terrified of unwanted pregnancies, and sick of condoms, are turning to vasectomies for liberation.

DOES EVERYONE THINK YOU'RE GAY?
For some guys, no amount of evidence to the contrary can kill a certain rumor.