Personal life
On 10 July 1858, he married Caroline Theresa Towneley (1838–1873) at the Royal Bavarian Chapel in London. She was the daughter and co-heiress of Colonel Charles Towneley, a Member of Parliament for Sligo, and Lady Caroline Molyneux (daughter of William Molyneux, 2nd Earl of Sefton). They lived at Wytham Abbey in Berkshire (now Oxfordshire) and had four surviving children:
Lady Mary Caroline Bertie (1859–1938), who married Edmund FitzAlan-Howard, 1st Viscount FitzAlan of Derwent (the last Lord Lieutenant of Ireland) on 5 August 1879.
Capt. Montagu Charles Francis Bertie, Lord Norreys (1860–1919), who predeceased his father. He married Rose Riversdale Glynn, granddaughter of George Glyn, 1st Baron Wolverton, with issue:
Montagu Towneley-Bertie, 13th Earl of Lindsey and 8th Earl of Abingdon
Hon. Arthur John Bertie (26 December 1861 – 10 January 1862), died aged 15 days
Lady Alice Josephine Bertie (1865–1950), who married Sir Gerald Portal on 1 February 1890. After his death, she married Maj. Robert Reyntiens on 5 October 1897.
Lady Cecil Josephine Bertie (1873–1895), who married Brig-Gen. Paul Aloysius Kenna on 18 July 1895.
After the death of his first wife in 1873, he remarried to Gwendoline Mary Dormer (1865–1942) on 16 October 1883. Gwendoline was the daughter of James Charlemagne Dormer, a British Army officer who was only two years Bertie's senior. Together, they were the parents of four more children:
Lady Gwendoline Theresa Mary "Goonie" Bertie (1885–1941), who married John, son of Conservative politician Lord Randolph Churchill and the American heiress Jennie Jerome, and brother of Winston Churchill.
Maj. Hon. Arthur Michael Cosmo Bertie (1886–1957), who married Aline Rose Ramsay, daughter of George Arbuthnot-Leslie, on 15 May 1929. After her death, he married Lilian Isabel Crackanthorpe, daughter of Charles Edward Cary-Elwes, on 7 May 1949.
Lt-Cdr. Hon. James Willoughby Bertie (1901–1966), who married Lady Jean Crichton-Stuart, daughter of John Crichton-Stuart, 4th Marquess of Bute, on 12 June 1928.
Lady Elizabeth Constance Mary Bertie (1895–1987), married first Major Sigismund Trafford on 21 April 1914. After his death, she married Col. Henry Cartwright on 5 September 1956.
In 1911, he broke up and sold the family estates at Albury and Great Haseley.
The Earl of Abingdon died on 10 March 1928, aged 91, at Oaken Holt in Oxfordshire, in South East England. He was buried at Abingdon Abbey in Abingdon. His widow lived to age 77 and died on 16 September 1942.