But for many years, architecture buffs have made a pilgrimage to this Connecticut coastal town for its modernist buildings.