The comedian was just 10 years old when his father James and older siblings Paul, 18, and Peter, 15, were killed in 1974 after their jet attempted to land in North Carolina.
In a candid interview with talk show host Oprah Winfrey, he reveals the loss became a part of his identity, and he didn't even begin to grieve until he was 18 years old and left home for university.
He says, "For years, I sort of thought that that was my secret name. That the loss was my name, if you know what I mean... (The experience) is who you are. I liked the idea that you have a secret name. And that's a name that no one can ever really pronounce, you know, because it's who you are. There's a magic to your secret name...
"I didn't really feel the loss until I was in college. Then, I was in bad shape... I was just so sad about it. I was 10 when they died. It's only eight years later when you go to college... It seemed like a long time at the time, but now, at age 48, it seems like the blink of an eye."
Colbert's full interview airs Sunday (07Oct12) on Oprah's Next Chapter.