I have an 03 and mine broke last year. It is more than likely the weld where that bar connects to the other part on the side. I am surprised the dealer is saying it would not be covered by warranty. I think I would try another dealer if you have another one close.
When mine broke, I called Toyota and complained and showed them links to the issue here on this forum. My complaint was that this is a known issue and has never been fixed by Toyota. I used the logic that Toyota never fixed anything even when they know they have a problem with something. The 2010 4runner has had this issue too. Mine broke when the gas pedal issue was hot. Plus, I have two Toyota's now and have owned several over the years including Lexus. They had on file all those records of previous Toyotas I have owned. That may have helped in them deciding to cover the part cost.
Toyota agreed to pay for the part if I would pay the labor to have it replaced. The part was over 500.00. I ordered it from a local dealer, paid for it, and Toyota reimbursed me for it. I used a local upholstery shop to actually install it because the seat cover has to be removed of course and put onto the new frame. The labor was 200.00.
Good luck, but this should be a warranty fix in your case in my opinion.