How is your upload schedule?
You should make sure to upload videos on a regular basis, or else people might just lose interest in you.
Remember to have catchy titles, and a topic of your video.
And maybe most important of all: Make sure your subscribers feel befriended by you. Don't treat them as "your fans", but your friends whom you couldn't have made it without.
I'm sure this will help you out a lot.
Other than that: Just keep doing your thing, don't give up and I'm sure that your traffic will be more balanced out in the future.