Some pets only show the "Press R to tame" thing for a very brief fraction of a second - Wisents, Wolves and Black Bears for example. Impossible to start taming on those. Apparently some animals have to be beaten into mercy mode in order to be tamed.
If you hit your own pet, you go grey. This is...