I.e., The People had better vacate the symbolic "Custom House" if they don't wish to be a part of society. Without them, the spot belongs to the men of "destructive rapacity," who will come "looking for me," i.e., for the Devil.