Fast internet speeds will help with this, however, you need to look at the upload process too. Copyright/duplication check, sensitive content checks, language checks, speech detection/recognition checks - and I may be wrong, but I think the technology of the YouTube video upload breaks the video up into pieces as part of the process, so it's more than a simple file transfer over the internet.
Add all these factors to a slow internet speed, and I can see why you may have problems. I tend to do other work during the upload, so I use my time well. I create a word document plan with tags, description, title, keyword research, first comment etc. I also create my thumbnail in this time too. so by the time it's uploaded I have everything ready to add.
Hope that helps!