We've been with YouTube for 3 years now. We've seen a lot of Subscribers come and go. Just keep in mind that some people may "fall out of love" with your channel. This doesn't mean you are a failure, it means that people change. Also, as mentioned YouTube also removes inactive accounts. There was actually one moment about a year ago that YouTube did a purge of inactive accounts. Our channel lost between 500-600 subs. That was a bit of a hit, but if you think about it, it's a blessing. Those people were never really watching in the first place. Don't worry about it. Just make sure you are producing quality content.

Just our 0.02. -Randy (gettingtheregreen)