Sounds like you've had some horrible experiences, I'm sorry to read that.
A little rough and tumble in the right setting has always been a, predominantly male, form of teasing and play. It's never affected me in the way it has you though so it's hard for me to agree that its never the right thing because I've not had an issue with it (but know bullying all to well).
As you said, context and relationships. This is all about clicks though.