Are there any? Last time I got kicked from #Toribash for advertising my duel server in 2 lines. Is it even prohibited? Are there any rules that could make it clear and me able to complain about ircops if they kick / ban me not regarding me breaking the rules?

IRC is a completely separate entity from the forums, and as such doesn't really have any rules. Certain channels like #support will be moderated in a slightly fairer manner, but #toribash is pretty much gloves off. Just try not to piss off operators, or be annoying, which I guess for some people is quite a challenge.

RULES OF IRC:

Don't get on ops nerves.
Don't be annoying.
Don't advertise ingame shit in #toribash.
Don't advertise ingame shit in #toribash.
Don't advertise ingame shit in #toribash.
Don't advertise ingame shit in #toribash.
Don't advertise ingame shit in #toribash.
Don't advertise ingame shit in #toribash.

#support is better, but it's for support.

If you want to promote your duel server, globals/broadcasts/whatever they're called exist.

e: Pre-emptive response to 'but those cost 5k/10k!': They cost money because ads are fucking annoying. No one wants a continuous stream of ads in their face.

Adding

Don't advertise ANYTHING in #support
Don't try to use #support for general conversation

I also want to ask about these signs: @%~- What are those?

~ is owner - access to all chanserv functions and is the Q (Star Trek reference) of the channel.
& is protected - access to some chanserv functions
@ is operator - can kick/ban and promote people to @ and lower
% is half operator - can kick/ban and promote people to +
+ is voice - can talk when the channel is moderated (+m)

That's basically it, there are some more privileges etc but if you want to know those you should go ahead and find out yourself. ^^

~ and & do not necessarily imply ChanServ likes you. They can be set by either ChanServ or another ~, or an IRC operator. But, in the latter two cases, ChanServ won't consider them to be the same as the ~s and &s it hands out.