What would key frames be used for? I accidentally hit it all the time and it's just annoying. I honestly see no use for it.

They're used so people can make the watcher of the replay see what the creator wants.
and shift+K removes iirc