There's four problems I see with a dedicated trail rig. One, you're GOING to break it, so you gotta have a trailer and a tow rig. Two, once you've got a trailer and a tow rig you aren't going to bother keeping it street legal, so point to points are out of the question, you have to end where you started and there can't be too much pavement in between. Three, because it isn't your daily, you're not in a hurry to finish projects and the damned thing spends more time in your garage in pieces and you spend less time on the trails. Four, if you have any money at all to spend on it, it gets to be SERIOUSLY competent really quick, and it'll get a lot harder to find a trail that challenges it - all your regular trails will be far too tame for a properly build trail rig.
I'd love to have a truck dedicated to competition and the really really hard trails, but it won't ever replace my regular truck. I think staying street legal is the best way to enjoy the sport.