Where it becomes an issue depends on the load out of your vehicle. For an example an E-rated tire only has enough carrying capacity to equal an OEM 4Runner SL P-spec tire if it is run at 45 PSI or higher (thanks to that site for doing the math). If you run any lower than that you are actually reducing the capacity of the tires lower than what Toyota engineers designed the vehicle for and reducing your overall GVWR. So, things like your payload rating, tow rating, etc in the manual all go in the trash can.
Running that way at highway speeds with any kind of load generates more heat and can increase chances of failure.
I am not an engineer that designs vehicles and tires so I won’t run lower than what the manufacturers of vehicles and tires say I should. I’d rather spill coffee on myself than risk a blowout by going against what someone way smarter than me decided is required for a vehicle.