Clarence House, St. James’s Palace and Spencer House