Marriage and children
He married Grace Louisa Staples, daughter of Rt. Hon. John Staples and Hon. Henrietta Molesworth (daughter of Richard Molesworth, 3rd Viscount Molesworth) on 12 October 1807. They had five sons and five daughters:
John Butler, 2nd Marquess of Ormonde (1808–1854), who married Frances Jane (died 26 August 1903), daughter of Gen. Hon. Sir Edward Paget
Lord Walter Wandesford Butler (1814–1861), Army officer who was a Captain in the Kilkenny Militia.
Captain Lord James Wandesford Butler (1815–1893), who married Lady Rachel Evelyn Russell, daughter of the 6th Duke of Bedford
James Francis Butler (1857 - 1933)
Grace Louisa Butler
Georgia Maud Sita Butler (1862 - 1863)
Julian George Butler (1864 - 1939)
Louisa Grace Butler (1816–1896), married Thomas Fortescue, 1st Baron Clermont
Lord Richard Molesworth Butler (1818–1838)
Lieut Lord Charles Wandesford Butler (1820–1857)
Lady Harriet Eleanor Butler (died 28 September 1885), married Robert Fowler, 1st son of Rt Rev Robert Fowler, Bishop of Ossory, Ferns and Leighlin
Anne Butler (died 1849), married John Wynne, of Hazlewood House, Sligo
Lady Elizabeth Butler (died 1892)
Lady Mary Charlotte Butler (died 1840)
In 1827, the London residence of Lord and Lady Ormonde was recorded as being 14 Weymouth Street, Westminster. Several of their children suffered from Epilepsy, which was attributed to the sudden deaths of their second son Lord Walter Butler, who died suddenly at his home, No. 16 South Frederick Street, Dublin, in 1861, and their fourth son Lord Charles Butler, who died at his mother's residence, Marlay House, Dublin, after a series of seizures in 1857.