I've jungled Fizz before
He's got a relatively slow clear and is prone to getting counterjungled by Lee Sin and other strong early game junglers.
If you're ahead on him though, their squishies are basically free kills for you. Also, you can't pick him into just any team comp. If they've got a beefy team, you're completely screwed.