Stone & Clay can be mined on site for just time & effort.
This is the only other cost that is unavoidable. You'll need 17 for the Small House/Entryway (included in the price of the plot), 47 for the Main Hall and 14/18/24 (depending on wing choice) for each of the 3 wings. As part of the purchase price the plot will already have an anvil, lumber pile and a chest with enough material to build an unfurnished Small House. You'll need to do some questing to earn the rights to purchase the plots.