The music mogul recently unveiled Spears and singer/actress Demi Lovato as the new additions to the U.S. show's judging panel and the audition stages got underway with tryouts in Austin, Texas last week (24May12).
Reports suggested the Toxic hitmaker's first day as a judge ended in confusion when she abruptly left the room after a contestant butchered her song Hold It Against Me, but Spears has since laughed off the claims, insisting she was "just taking a little break".
Now Cowell and Lovato have poked fun at the allegations in a new video interview for The X Factor website, joking that it was the former Disney star who actually stormed off unexpectedly.
In the footage, Lovato adopts an English accent and says, "I was like, you know, tired of Simon, so I had to just, like, go."
Cowell then got serious to dismiss the speculation altogether, telling the interviewer: "There was no walking off. We have breaks every five or six songs. You can go have a cigarette, you can go to the loo (toilet). So as soon as anyone left their chairs, it was reported... that someone had left the show - and they hadn't!"
Spears and Lovato replaced Nicole Scherzinger and Paula Abdul for the second season of the American version of the talent show.