Harriet Cowles Swift was born in Hartford, Connecticut, and in 1855 married William Judah Keyser, born in nearby Norwalk. The Keyser family was among the founding members of the community of Milton in Santa Rosa County, Florida, where William became a lumber baron, owning many mills and exporting as far as Baghdad. They subsequently lived in Pensacola where the business, Keyser, Judah & Co. was based and where they became prominent members of the community. When her husband died in 1877, her son, William Swift Keyser, having graduated from Yale in 1878, returned to Pensacola to head the flourishing business. In 1910, his mother, one of the town’s most prolific builders, bought up an area on South Palafox Street that had been the site for the Sisters of Mercy Convent School; she built most of the buildings put up there shortly before her death. One of the buildings is known as the Keyser building, and upstairs houses Keyser Hall.