What state are we talking about here OP?
I live in NY, and the most extreme, legal roads/areas you can drive would be doable with a SR5.
SR5 will suffice for for 95% of the situations you will encounter, even out west. And considering that your end goal is to get to a suitable parking spot to then get out and hunt, I don't see the SR5 as a compromise or step down from the Trail in this situation.
Trail does give you a locking diff. Someone else mentioned wider wheels too, though I'm not sure on that myself. Locking diff is great and definitely gives you some added confidence for some of the more technical driving you might encounter. But then again, picking the right approach and using a little more gas can probably get the SR5 over similar terrain.
To sum it up, I've seen Crazy subaru owners take their low-clearance, AWD, no low-range Outbacks up some ridiculous jeep trails....granted they're crazy and likely see some serious repair bills, but the SR5 is more than capable of getting through most of the terrain you will or might even potentially see.