The trouble began in April when his ex-wife, Sheila, published an unflattering book suggesting he bullied her into an annulment. Then came allegations that Joe’s brother and political adviser, Michael, had been having an affair with a teenage babysitter. Joe seethed as the press badgered. His polls plummeted: once considered the leading political light of the younger generation of Kennedys, Joe saw his favorable ratings drop from 60 percent last January to 34 percent in July.

After cousin John F. Kennedy Jr. piled on with an editorial in his magazine, George, that dubbed Joe and Michael ““poster boys for bad behavior,’’ Joe proclaimed that the political had become too personal. On vacation in Hyannis Port with his wife, Beth, he decided to skip the governor’s race–a move that surprised aides on the verge of renting campaign-office space. ““People are very angry at members of my family, me in particular,’’ Kennedy told supporters. He plans to seek re-election to Congress next year and might run for his uncle Ted’s Senate seat in 2000. But last week’s announcement suggests the Camelot luster, already faded, is all but gone.