Personal life
On 2 January 1766, he married Elizabeth Boscawen, daughter of Hon. Edward Boscawen, Admiral of the Blue, and sister to George Boscawen, 3rd Viscount Falmouth. Elizabeth died 15 June 1828. Together they had four daughters and nine sons:
Henry Charles Somerset, 6th Duke of Beaufort (1766–1835), who married Lady Charlotte Leveson-Gower, a daughter of Granville Leveson-Gower, 1st Marquess of Stafford and Lady Susanna Stewart (a daughter of the 6th Earl of Galloway), in 1791.
Lord Charles Henry Somerset (1767–1831), who married Hon. Elizabeth Courtenay, daughter of William Courtenay, 2nd Viscount Courtenay, in 1788. After her death in 1815, he married Lady Mary Poulett, daughter of John Poulett, 4th Earl Poulett, in 1821.
Lord Edward Somerset (1768–1769), who died young.
Lord Norborne Berkeley Henry Somerset (4 May 1771 – 19 October 1838)
Lady Elizabeth Somerset (1773–1836), who married Very Rev Charles Talbot, Dean of Salisbury, son of Rev. Hon. George Talbot (a son of the 1st Baron Talbot) and Hon. Anne Bouverie (a daughter of the 1st Viscount Folkestone), in 1796.
Lady Frances Elizabeth Somerset (1774–1841)
Lady Harriet Isabella Elizabeth Somerset (1775–1855), who married Col. Hugh Henry Mitchell, a son of Hugh Henry Mitchell, MP for Ballyshannon and Enniskillen, in 1804.
Gen. Lord Robert Edward Henry Somerset (1776–1842), who married Hon. Louisa Augusta Courtenay, daughter of William Courtenay, 2nd Viscount Courtenay, in 1805.
Lord Arthur John Henry Somerset (1780–1816), who married Hon. Elizabeth Boscawen, daughter of George Boscawen, 3rd Viscount Falmouth, in 1808.
Rev. Lord William Somerset (1784–1861), who married Elizabeth Molyneux, daughter of Lt.-Gen. Sir Thomas Molyneux, 5th Baronet, in 1813. After her death in 1843, the following year he married Frances Westby Brady, daughter of Henry Brady of Raheen Manor, County Clare, and widow of Cornelius O'Callaghan of Ballynahinch (died 1829).
Lady Anne Elizabeth Somerset (1786–1803), who died unmarried.
Col. Lord John Thomas Henry Somerset (1787–1846), who married Lady Catherine Annesley, daughter of Arthur Annesley, 1st Earl of Mountnorris and Hon. Sarah Cavendish (a daughter of Sir Henry Cavendish, 2nd Baronet and the 1st Baroness Waterpark), in 1814.
FitzRoy James Henry Somerset, 1st Baron Raglan (1788–1855), who married Lady Emily Wellesley-Pole, daughter of William Wellesley-Pole, 3rd Earl of Mornington, in 1814.
Lord Beaufort died on 11 October 1803 and was buried at St Michael and All Angels Church, Badminton. His last will was dated from 21 June 1789 to 11 September 1800. He was succeeded in his titles by his eldest son, Henry
