The Carroll House and the surrounding site has been expanded, altered and re-developed over the past 350 years, but the property remains one of the very few essentially intact 18th-century sites in the Annapolis Historical District. The story of the House spans several periods: Pre 1700, the original Carroll purchases, the original brick house and expansions, the Redemptorist Novitiate and the current use.

The land around the Charles Carroll House was first settled in the late 1600s as part of Anne Arundel Towne. Dwelling houses, wharves, warehouses and Proctor’s Tavern (a public inn and meeting place for the provincial legislature) were located along the shore line during those early years.

A surviving land deed focusing on this lot dates to 1687. It states that there was a 24-foot-wide dwelling on the site – the approximate width of the existing one-story eastern structure and its adjacent footprint. Research indicates that this “Frame House” was actually standing on the site by 1687 almost 20 years before Carroll ownership.

Charles Carroll the Settler (CCS) bought various pieces of the property in Annapolis between 1701 and 1716, soon after the city Annapolis replaced St. Mary’s City as Maryland’s capital. In 1701, CCS bought the first property from postman John Perry at corner of Market and Southeast (now Duke of Gloucester) Streets. In 1702, Charles Carroll of Annapolis (CCA) was born at his parents’ home in Annapolis. The family eventually purchased almost one-quarter of the lots in Annapolis, the land around Charles Carroll House among them. In 1706, CCS bought the frame house and nearby lots from Henry Ridgely.

Soon after CCS died in 1720, CCA started construction on his brick house 10 feet to the west of the Frame House, where his widowed mother and sisters continued to live. The initial brick house was two-and-one-half stories on the street side, with a gambrel roof, in the style of many of the earliest Annapolis brick dwellings. CCA also probably improved some of the surrounding landscape at the same time. The homes stood side by side until approximately 1728-1735 when they were joined by a two-story passage way creating one house with as many as 29 rooms. In 1737, Charles Carroll of Carrollton (CCC) was born in this house. During this period, CCA began selling off adjacent lots but retained the property which encompass the St. Mary’s church and schools today.

The next addition to the House was made in the early 1770s. Charles Carroll of Carrollton, also known as “Charley,” had returned home from Europe and married Mary Darnall at his father’s house in Annapolis in 1768,  CCA gave the young couple the house as a wedding gift and he began to spend most of his time at Doughhoregan, the large family estate near Ellicott City. Charley and his new bride began to enlarge the brick house. One of the first additions was made to the connecting passage way by expanding it to the north to create a library, one of the most elegant rooms in the house.

By the 1770s, as the family grew, CCC raised the gambrel roof replacing it with the gable roof and massive chimneys. At the same time he was terracing his formal gardens. The one story connecting passage way grew to match the height of the central brick structure. By 1790s, it became the four-story house seen today.

In 1821, at the age of 83, CCC moved to his daughter’s home in Baltimore and the family rented the Annapolis property. The family stayed involved in Annapolis and the site, though. In 1822, the first Catholic church (St. Mary’s) was built on Carroll property under the auspices of Mary Caton, daughter of Charley’s only son, Charles Carroll of Homewood (CCH).  In 1825 CCH died, possibly at Carroll House, and was buried outside Annapolis at the family burial ground. He was re-interred in the Carroll House garden and is the only family member with a grave on the site. CCC died in Batimore in 1832, and left the property to his daughter, Mary. When she died in 1846, the property passed on to the 4 Caton sisters.  For 20 years, the property was leased to a variety of tenants. Attempts were made to sell the property, but no offers were forthcoming.