Beautiful, Quaint Town, Not far from New York City?

I am looking for a nice village-type town to stay while visiting Manhattan. My actual NY stay is 5 days (2 days in the city, 3 days away from it). I am looking for small boutique and antique shops, flea market finds, etc.. that won't necessarily empty my wallet. My stay this year is December 6-10. Any recommendations? Thanks! M