The retired New York Giants star, 51, was arrested on 6 May (10) at a hotel in Montebello, New York amid allegations he paid a 16-year-old girl $300 (£200) for sex.
Last month (Jul10), he entered a not guilty plea to charges of third-degree rape, patronising a prostitute, sexual abuse and endangering a child.
But as the case approaches its pre-trial stage on Wednesday (25Aug10), Rockland County Court Judge William Nelson stepped down.
He did not provide a reason for his decision, and will be replaced by Supreme Court Justice William Kelly, according to the Rockland District Attorney's office.
If convicted, Taylor faces up to four years in prison.