Hazard's Castle was constructed by Joseph Peace Hazard, the son of Rowland Hazard, (the founder of the Peacedale Mills) and Mary Peace. Joseph was born on February 17, 1807 in Burlington, New Jersey. The family moved to Peacedale, Rhode Island in 1820. During 1821, Joseph began working in the Peacedale mill run by his older brothers Roland and Isaac.

In 1835, Joseph constructed a woolen mill at Peacedale, which he ran for several years and then leased out. In 1856 Joseph traveled to England and Italy and thereafter spent most of the years until 1879 traveling abroad. During the last two years of his travels he also visited Japan and Iceland after which he returned to Peacedale.

Joseph was the owner of  several large tracts of land in the Narragansett Pier area and was instrumental in the early development of the town as a major tourist destination. In 1846, he began construction of what is now known as Hazard's Castle. The castle derives it's name from the two towers on either side of the building.