BOSTON — A man accused of raping and stalking a mentally disabled man is being held without bail in Bristol County after he was allegedly seen peeping in the window of the alleged victim’s group home in Foster.

Buddy Smith, 22, of Fall River, is accused of terrorizing the 25-year-old man, even as his family moved him from place to place to get him away from Smith.

Smith is charged with rape, indecent assault and battery on a retarded person and witness intimidation after allegedly sexually assaulting the man many times between 2004 and 2006, said Gregg Miliote, a spokesman for Bristol County District Attorney C. Samuel Sutter.

Smith was out on $1,000 bail while awaiting a July 21 trial. He was indicted in June 2006.

On Tuesday, Smith was ordered held without bail until a May 21 bail revocation hearing after he allegedly tracked the man to a group home in Foster, on the Rhode Island-Connecticut border. Witnesses told the police they saw Smith peering into the window of the group home on March 20 and driving by the home on April 9, Miliote said.

The man’s family told the police they have moved him to various group homes in Rhode Island to try to protect him from Smith.

“They have moved him around several times,” said Miliote. “Mr. Smith allegedly found out where this victim had been moved to, went to this group home and was peering in, looking for him.”

Smith’s attorney, Kenneth van Colen, did not immediately return calls seeking comment yesterday.

The alleged victim’s family said Smith, his uncle, William Senay, 52, of Fall River, and several other men gang raped him multiple times, beginning in 2004.

Senay has also been charged with rape and indecent assault and battery on a retarded person, and is scheduled to go to trial July 21. No one else has been charged, Miliote said.

Senay’s attorney, Kenneth Littman, did not immediately return a call seeking comment.

It was not immediately clear whether Smith could face charges in Rhode Island. Phone messages left for Tiverton, Middletown and Foster police and an e-mail message left with a spokesman for the state attorney general’s office were not immediately returned.

Colleen Lutkevich, executive director of The Massachusetts Coalition of Families and Advocates for the Retarded, said the alleged victim’s mother contacted her more than a year ago, saying Smith had repeatedly stalked and threatened her and her son. The alleged victim has mild to moderate mental retardation, Lutkevich said.

“He’s been following this boy and his family on an almost daily basis,” said Lutkevich. “Every time they move him, this guy would find him again.”

The alleged victim’s family said Smith went to the man’s house in Tiverton, in January 2006, and stalked him repeatedly over the next two years, including at a library in Middletown, in December, Lutkevich said.

The assaults allegedly began in 2004 shortly after the victim met Smith through a friend in Fall River. Smith is accused of taking the victim to a church parking lot in Fall River, where he allegedly raped him. Miliote said prosecutors believe Smith raped the man multiple times.