Yeah they call that the 'deflector' I think, and I have that installed too. I'm sure it would be even worse without that. I'm talking about the fairing you can buy and install separately. Should have made my link more obvious:
Wind Fairing for Rack / 1165mm/1255mm(W) - by Front Runner
Just read your post today, but I have installed that wind-fairing and it has helped a lot with wind noise and, in my opinion, it looks better.