In my mind, the answer really depends on how long you plan to keep the vehicle. If you are the type of person that gets a new car every 2 years, then I would go with the TRD Pro as it will have much higher resale value. A heavily modified vehicle isn't worth any more than a bone stock vehicle in the used car market so you can lose a lot of money if you decide you don't like 4-wheeling.
But if your primary goal is to build a great performing truck that you plan to keep for a long time, by all means go with the TRD Off-Road and modify it with vastly superior aftermarket parts for a fraction of the cost of getting a stock TRD Pro. As you mentioned, even a stock TRD Pro will require some mods if you intend on actually taking it off road...at a minimum, you will soon want larger tires, a higher lift, better skids, and proper sliders.
The only thing you may want to solicit some more advice on is whether KDSS is worth the money. I have it on my truck and absolutely love how it handles on and offroad, but honestly I have zero frame of reference as I've never driven one without KDSS. At around $2k for that option though, you could easily fund a very nice suspension upgrade just by foregoing KDSS.
I bought a 2018 TRD Offroad about 3 months ago with KDSS for around $38k new and probably spent around $2k upgrading tires, suspension and upper control arms, and sliders, but I'm confident my $40k Offroad will beat the pants off of a $55k TRD Pro!
Good luck!