If you refuse a companion joining, they disappear. Rest at an inn, and return to their spawn spot (for Hirge by Brother Gab at first, then inside the temple.
David, is it possible to code it so they just say no, and then stay there. Would make things easier for the player.