Phantasma knows better than to wander off on his own, which is why I suspect others to be involved. And if you don't do all your assignments you're not getting your allowance for the next Hogsmead trip. You'd probably just waste it anyway, it can be put away in the bank to be saved.