I'm going to have to disagree with people saying the first 100. As I reached 100 subs back in August 2013 and I am still growing slowly, maybe even slower.
So I have to say the first 1000 subscribers would be the hardest, I think once you reach that, you're clearly doing something right.