As I noted previously, New England farmers raised more rocks than crops over the past few centuries. The last ice age scraped all the dirt off the granite and formed Long Island. However, after the farmers gave up on New England, and more efficient fuel sources replaced wood, New England's forests have returned in abundance. My previous proposal for a New England gabion house uses the abundant rocks, but needs spray foam, which is not a low impact product, to provide insulation. Lately, I've been reading about slip-chip walls. Basically, wood chips are coated in clay and lightly stuffed into a wall form then allowed to dry out. As in many low impact homes, the thicker the walls, the better. Some builders use reed mats as permanent forms. Other builders have used a double wall of (free) pallets for a straw-clay infill on the Pine Ridge reservation. Straw and wood chips trap air which increases the insulation properties of the in-fill. Processed hemp will work as well, but, foolishly, it's not legal to grow in the United States. That may be coming to end though. Colorado farmers harvested their first crop recently.
Anyway, there is plenty of chatter on American websites about the potential of wood chip- clay slip infill. Apparently, the ideas have more history in Germany, but I have not found the German sites yet, except for the German wiki article on Lehmbau. This German site has some info and an english version as well.
Natural Builders Northeast - my part of the country.
Pot Kettle Black has pictures of a house in Groton Mass. under construction with chip-slip
Natural Design Build
Green Space Collaborative
Network Earth has a good long article on clay and home construction
FoxMaple , where this image comes from
Of course, a passive solar orientation along with proper summer shading will go a long way towards making this a home that is easy to keep comfortable with minimal costs.