is it possible to remove the scarf part, it's annoying....

Nope.

I don't even see the scarf.

The scarf is on my back so you cannot see it.
Its annoying during wushu

I think its not possible because the scarf is one with the hat as one item.

No its not possible
The item itself called "Daimyo hat and scarf " man
You can't remove the scarf

Well it might just be possible my friend got a gentlemen essential without the eye thingy. It only had that hat but it's rare or some shit like that. Just give it a try and you might just get the hat without the scarf for a high price or whatever, pm someone from the item forging team.

Nope.