Do you need to drive far from Pawling to find a bigger town ?

Yes, if far means at least half an hour. Eastern Dutchess County is kind of on its own, and though Pawling isn't quite as isolated as some other towns here, it's not near any bigger population centers. Carmel in Putnam County has a decent-sized population, but the two nearest cities to Pawling of any consequence are Poughkeepsie (northwest) and Danbury (southeast).

Both cities are about half an hour away.