Adammsu,
I have the exact same issue. This apparently is a common issue since the Toyota parts guy said they have them in stock. (They have to have a request for a part 3 times in a month in order to meet the qualification of stocking the part.)
Heat definitely is the cause.
The part number is 69120-35050 (Driver's Side) Actuator Assembly. My dealer wants $117. I found them on the net for $87. I asked about installation, and was immediately told $145 - proving it is a fairly common problem. He said it takes about 1 1/2 hours - so I guessing it cannot be that difficult. I have not tried to do it myself, yet.
Also, another solenoid that is subject to heat lock-up is the Automatic Transmission Shift Lock. I prevents you from taking it out of Park (use the manual override push button if needed.) It is part number 33058-35040 Bracket Assembly. $45 at the dealer, $31 on-line.