Trial, scandal and decline
In 1768, Calvert was accused of abduction and rape by Sarah Woodcock, a noted beauty who kept a milliner's shop at Tower Hill. He was indicted at Kingston Assizes, and put on trial, pleading not guilty by reason of consent. After deliberating for an hour and twenty minutes the jury acquitted Calvert, believing that Woodcock did not make adequate attempts to escape.
Much salacious gossip accompanied the trial, and in the same year, one of Calvert's willing sexual partners, Sophia Watson, found it opportune to write a salacious autobiography entitled Memoirs of the Seraglio of the Bashaw of Merryland, by a Discarded Sultana (London, 1768) Her readers were left in no doubt as to whom she was referring to, which further harmed Baltimore's reputation. Sultana Watson offered many intimate details of life in the seraglio, including the predictable, unkind suggestion that Baltimore himself was barely able to satisfy one, let alone eight, mistresses.
Following his acquittal Frederick left England, presumably hoping that his notoriety did not extend to Europe. In this he seems to have been at least partly correct, as in July 1769 the British Ambassador to Russia reported that "Lord Baltimore arrived here last week from Sweden; I had the honour to present him to the Empress, who was pleased to receive his Ld extremely graciously." Nevertheless, Calvert's brush with the law does not appear to have affected his unconventional living arrangements. Count Maximilian von Lamberg wrote of his travels:
In 1769, my Lord was travelling with eight women, a physician, and two negroes, which he called his corregidores, who were entrusted with the discipline of his little seraglio. With the aid of his physician, he conducted odd experiments on his houris: he fed the plump ones only acid foods and the thin ones milk and broth. He arrived at Vienna with the train I have described; when the chief of police requested him to declare which of the eight ladies was his wife, he replied that he was an Englishman, and that when he was called upon to give an account of his sexual arrangements, if he could not settle the matter with his fists, it was his practice to set out instantly on his travels again.
By this time it is evident that he was suffering from financial difficulties, and in 1768 he sold the family's great estate at Woodcote Park, apparently to a wealthy Soho upholsterer.