The Buffy the Vampire Slayer creator admits he seriously considered ditching the superhero blockbuster to make his childhood dreams come true, but he couldn't bear the thought of someone else taking the reigns on his Avengers follow-up.
He tells CNN, "I'm as angry as you should be. When I heard that (he was in the running to direct the Star Wars film) I was like, 'I wonder... no, I really can't do that. Oh, I already have a job.' I wouldn't clear (be able to finish making) The Avengers. I'm having so much fun with that sequel right now, just with the script, that I couldn't imagine not doing it."
Both The Avengers follow-up and Star Wars: Episode VII are tentatively scheduled for release in the summer of 2015.
Whedon is the latest to take his name out of the running to make Disney's reboot of the Star Wars franchise - Guillermo Del Toro, J.J. Abrams, Brad Bird and Zack Snyder have all distanced themselves from the sci-fi project.
However, Whedon knows what he wants to see in Star Wars: Episode VII: "I'd like to see a little edge... It got very portentous, which was justified by the story, but I think we'd all like to see a little anarchy in the mix."
According to reports, Disney bosses are now considering X-Men: First Class director Matthew Vaughn for the top job.
Star Wars creator George Lucas sold his production company LucasFilm to bosses at the family film giant last year (12).