I know that no two players should ever have similar-enough if not identical textures, but what exactly is the limit? Does circumstance count, when there's consent from both players to use or sell the same texture? How different does an edited texture have to be compared to the original in order to use, if edited versions of pre-existing textures can be used at all?

It is hard to explain the "limit".

First of all, you are only allowed to take the idea of the texture, not the texture itself and secondly, direct permission is required from the original artist/owner of the texture to perform edits on his textures.

Also, the original owner/maker of the texture decides the "limits" of how much designs you can take from his textures.

Allow myself to help.
Plagiarism is when a work is taken a sold without permission, whether edited or not. The limits are variable, as stated above, by the original artist's wishes.
However, even with permission, toribash rules state that one texture and recolours/variations of said texture are not allowed to be resold to others, the original must swap owner with or before edits. This was to try and increase the amount of creativity amongst the community and to stop some of the smaller scammers.
The idea is free to take, but remember its a concept. If you make a head too similar, they can say that the specifics of the head were mimicked and call a plagiarism claim - resulting in a ban.
It is safest to either attempt to purchase the texture you have your eye on, or make something of a similar genre that is irrelevant to the specifics of the head - like, the shapes, colours, concept, etc.

Anything more than that and I'd be nit-picking.

Addendum to Monobi's post:

If it's a very generic head, like a smiley face or something, we'll usually ignore plagiarism claims.