The music mogul has made millions through his string of reality shows, which also include American Idol and America's Got Talent in the U.S.
But bosses at performers’ union Equity aren't happy about the lucrative profits - insisting the contestants who provide the entertainment deserve to be paid for their efforts.
Union executives have tabled a motion about the issue and if the idea pulls in enough support it could lead to a court battle against TV executives.
The motion reads, "The contestants in such programmes are often compelled to enter into restrictive contracts and because of a loophole for competitions they generally do not get paid. These programmes may be very popular with the public but are based on exploitation and humiliation of vulnerable people, which cannot be acceptable.
"The public's demand for high-quality entertainment should be met by professional drama and light entertainment which has been replaced by this cheap exploitation.".
Bosses at Talkback Thames, the company behind Britain's Got Talent and The X Factor, have hit back insisting the shows give ordinary people the chance to transform their lives with a music career.
They say in a statement, "Britain's Got Talent and The X Factor are talent competitions. They are not employment in their own right and therefore Equity rates do not apply. Contestants chose to enter to compete for a substantial prize - a cash prize of £100,000 and a performance on the Royal Variety Show for Britain's Got Talent and a recording contract worth £1,000,000 on The X Factor.".
The news comes just two months after a legal ruling in France which means production companies in the country must now pay contestants to appear. Three contestants on the French version of reality show Temptation Island won compensation after a judge decided they were entitled to overtime pay for working 24 hours a day.