Sir Edward's only son was a Lieutenant Colonel in the 6th Dragoons. When he died in 1771 at Calais, Walpole wrote to Mann, "My brother has lost his son, and it is no misfortune, though he was but three and thirty, and had very good parts; but he was sunk into such a habit of drinking and gaming, that the first ruined his constitution, and the latter would have ruined his father."