The gay actor, who plays Jeremy Piven's character's assistant Lloyd on the hit show, endured a tough upbringing with his strict mother, a born-again Christian, and his "distant" dad.
He tells dating blog MySinglePeeps.com, "What I figured out about my family is my mother just judged me all the time and made me feel defective and unimportant. My father was neglectful and distant. My sister's message to me was you're not very interesting."
Lee's already-complicated relationship with his parents was strained even more when he 'came out' to them at the age of 22.
His mum kicked him out of the family home, marking the start of an 11-and-a-half year estrangement, which would continue after a brief reconciliation.
And Lee, 42, admits his troubled home life has deeply affected the way he behaves in relationships.
He says, "I need someone to make me feel important because my parents didn't...
"I've dated men who were kind of mean to me. The last guy, we were smoking a lot of pot and he said, 'This isn't working for me.' I said, 'Sex or the whole thing?' He said, 'Haven't you noticed how much pot I've been smoking? I've needed to do drugs to be with you.'"
But the actor insists being famous has done little to bring him luck in the dating game: "Being on TV has been a strange situation because now I meet people who are like, 'You're on TV. Let's f**k,' and I'm not into that. I would love to couple up with someone. At this point I'm not a nester but I'd love to find someone to nest with."