I happen to think the 4 Runner has a lot of noise. I have the 04 SE also. I notice the noise at modest to highway speeds.
It is partly a function of the shape of the truck and the mirrors, rack, etc. that catch the wind. There also probably is not enough soundproofing in some areas.
I also have a problem with wind noise with the sunroof open (opening the rear window a bit helps some). When I reach up and depress the spring-loaded plastic piece at the front of the sunroof, the wind noise drops dramatically.
The 4 Runner is a nice vehicle, but it leaves a lot to be desired in these areas.