Olson played romantic leads in several other films, but neither the mild "The Boy from Oklahoma" (1954) nor the dreadful and paranoid actioner "Big Jim McLain" (1952) got much attention. By then married to songwriter Alan Jay Lerner, Olson, after playing several second leads ("So Big" 1953; a TV version of "The Women" 1955, with Olson as the gentle Peggy), retired from acting. After divorcing Lerner, she returned with the first of five films for Disney over the next dozen years, "Pollyanna" (1960), and was back to helpful homemaker parts for "The Absent-Minded Professor" (1960), "Son of Flubber" (1963) and "Smith!" (1969).
Olson has subsequently played a handful of film roles in the timid, gay-themed love triangle, "Making Love" (1982) and the all-star "Airport 1975" (1975, as Linda Blair's mother). Occasional TV work came along too: "Kingston: Confidential" (1977), starring Raymond Burr, found Olson atypically cast as the head of a company, while "Paper Dolls" (1984) saw her back in form as the wife of a ruthless CEO (Lloyd Bridges).