Is it worth just upgrading the front and rear doors? Will it improve the sound quality?
Yes it will improve over stock if you get a component system that replaces front speakers and tweeters. However replacing my SR5 stock head unit dramatically improved the sound with the new (Polk) speakers. Got the rear door speakers replaced too, I got cheap Sony's and it was so bad, I had to fade it out. You can't even hear them so well, it's located too low IMO. Then added a powered subwoofer (Infinity) and again, huge improvement. Amplified the speakers and now it's like sitting in a concert every day.. So I would start with the front speakers. Crutchfield will give you all the brakets, installation stuff you need for "free".