One NYC shop to avoid is Second Hand Rose near Union Square. When I first started going there, I got loads of bargains but the vinyl is greatly overpriced nowadays.
I'd love to go up the stairs in the store though - the owner once told me there were 000's of 45's up there and they had never got round to sorting though them.