The director has withdrawn from the remake of the film about a man and his six-foot tall invisible rabbit.
Variety reports that Speilberg gave Fox the news this week. The studio had agreed to have DreamWorks finance 50% of production through its new funding relationship with Reliance, and either distribute domestically or internationally through its arrangement with Disney.
The film has faced challenges since Spielberg committed last August including finding a star to play Elwood P. Dowd, the character immortalized by James Stewart in the 1950 original film.
Fox 2000 will continue working on Harvey and could re-approach Spielberg and Downey, Variety says.
It wasn't immediately clear which picture Spielberg will direct instead, but possibilities include the Abraham Lincoln Civil War film scripted by Tony Kushner, and the Jeff Nathanson-written The 39 Clues.