As you grow larger your going to pick up haters no matter what. Some people will get upset because your growing faster then them, some people are going to decide that YouTube is here for them and EVERYTHING should cater to their tastes and still some people are just doing it to literally because they think their funny. I get through it simply by doing the math, if I have 3 dislikers, and 50 likes I think Im doing okay. Also as you build a following your viewers will fight for you and argue for you.
But regardless, if your going to do anything in the entertainment industry then you have to be ready for problem viewers. If you want to see a good view on this from comedians and movie stars watch the movie Hecklers. Its a movie Jamie Kennedy put out after filming Son of the Mask and how the hecklers from that movie almost made him quit making movies. It gave me a very good insight that everyone will have to deal with problem people sooner or later, so you better come up with a way to do so.