Chiming in from Canada (I'm in AB) with some feedback.
I have a 2019 TRD OffRoad. I'm not an off-roader. The truck is stock and is my daily driver. I drive 95% in RWD mode and it works fine. I have only engaged 4WD in winter, and usually when engaged, it is in 4H (instead of 4L) because I am driving on the highway, or going somewhere on city streets. You're right in that 4H and 4L should only be engaged when tires can slip (eg. driving on snow, ice, slush, rain, loose gravel, etc). I have needed to use 4H for example when going skiing - for my situation the TransCanada was all snow and ice covered, and I needed the front wheels to help pull the truck through snow, ice, slush. Driving in 4H in these conditions helps a lot and I can feel the additional traction provided to the vehicle. In the winter, if I'm in 4H, I find I hardly ever need to shift from 4H to 2L and back and forth. This is because even on plowed roads, there is still snow, ice, slush, etc. left on the road. So on the days I need to use 4H, it usually stays in 4H until I get to my destination. You just need to note the road condition and adjust accordingly. Oh, I should mention I have 2 sets of rims/tires. One set of rim has winter tires and I put them on usually by November. They come off sometime in May. The other set of rims has the factory all season tires.
I've driven the Spray Lakes trail in my other front-wheel drive car (don't have it any more). I haven't driven it in my 4runner yet. I would think the 4runner will be fine and soak up the bumps because it has larger profile tires and is also heavier (compared with your Q5). The road is basically loose gravel and dirt, so you can drive it in 4H for additional traction.
I'm not sure if you're aware, but those other terrain modes on the 4runner require the truck to be in 4L in order to be activated. When you're in 4L, you need to keep the speedometer under a certain speed (I can't recall what it is but probably under 20 km/h). The terrain modes would come in handy if you're stuck, or need the truck's power to climb over things.
The 4runner will be a nice adventure vehicle. It has comfortable room for passengers, and had adequate cargo space. With the rear seat folded, it has lots of cargo space. You can take it off the beaten path
