Unfortunately, you cannot complete the four Wizard Tower quests as long as your reputation with the Wizards Guild is low (you are not let into the towers in this case). Yet you need to complete these quests to gain access to Icemist. So David implemented (in theory) the possibilty of giving the Orb of Shadows to the Conjurer so you can restore both, your reputation with Thuram and the Wizards Guild, allowing you to continue the tower quests, and finally be granted access to Icemist.
I just checked the dialogue with the Conjurer and the problem is that in order to reach the part where he tells you to return the Orb, you need to have a low reputation with Thuram. The game doesn't check for your reputation with the Wizards Guild at this point. So the problem here seems to be that you raised your Thuram reputation by paying a fine and now can't trigger the Orb returning option.
This might be an oversight from David and probably be fixed in a future update, but for now your only option seems to be to somehow lower your Thuram reputation to -10 or lower and talk to the Conjurer again (this also requires a PER 3 check).