Charles Carroll of Carrolton’s oldest daughter Mary (1770-1846) married Richard Caton in 1787, and they settled at Brooklandwood, an estate in Baltimore County (currently St. Paul’s school). They subsequently had 4 daughters Marianne (1788 -1853), Elizabeth (1790 -1862),Louisa (1793-1874), Emily (1794/5 -1867). They are collectively referred to as the Caton Sisters. They were close to their grandfather and spent much time at the Carroll House and at Doughoregan Manor, as well as their parents home. When Charles Carroll of Carrollton died in 1832, the Carroll house passed to Mary Caton, and when she died in 1846,ownership passed on to the Caton sisters.
Marianne, Elizabeth (Bess) and Louisa spent much of there early adulthood in England, and all three would go on to marry members of the British nobility. Emily married John MacTavish, a Canadian businessman, and stayed in Maryland. She eventually cared for the aging signer at her home in downtown Baltimore until he died.
When her mother died in 1846, it fell to Emily to manage the Carroll house, which had been rented to a series of tenants. Emily had become close to a Redemptorist priest, Father Gabriel Rumpler in Baltimore, and through that connection, she proposed to the sisters that the Annapolis property be transferred to the congregation for $6,000, a sum which covered needed repairs, back fees and taxes.