I don't think I'm getting my point across clearly. Ok, say you pull off on the side of the road during a wind storm. As the wind pushes against your car, how much does that affect the suspension? Very minimal amount. This is true whether you're standing still or going 70mph. It's aerodynamics that affects this wandering, not suspension. Anyway, this is how it made sense in my head.
I understand what you are saying, but I disagree very strongly. Without considering wind, simply changing toe-in can dramatically change how well a car tracks. Different suspension geometries can have a significant difference as well. It doesn't take much load on some suspensions to make what may seem to be very small angle changes on the wheels, but those very small angle changes can cause the car to wander at speed.