The building is far from literally empty, but this line is symbolic: the "Custom House" is established society, and it is "empty" because an organized society has yet to be established. In its absence, this is what remains: the screams of brutal torture wrought by naked greed.