The house was built in 1869 as a summer home for Joseph Peterson, a prominent tobacco trader. He died in 1889, leaving the property to his granddaughters, Eliza and Carrie Lindenberger. Eliza married Harry Dumesnil, Carrie married Edward Rowland and they lived in the house with their families. Mrs. Dumesnil lived there until she died in 1948.

After Mrs. Dumesnil's death, the house and grounds were sold to the Louisville Board of Education. In the mid 50's, it became a teachers club, the only one of its kind in the country. In 1976, the house achieved local landmark status, which protects it from exterior change without approval of the Louisville Landmark Commission. In 1977, the Crescent Hill Community Council leased the house and made it available to groups on a rental basis. In 1982, when the Board of Education declared the property as surplus, the house, carriage house and 1.3 acres were sold to the newly formed, non-profit Peterson-Dumesnil House Foundation. All rental proceeds are used to steadily improve the house and grounds.