I vote keep it too. My number one priority when shopping for toyotas (and I have owned many) is rust trumps all other factors. I'd rather have a 300k truck with mechanical issues and not a speck of rust than a 100k truck with frame rust or rusty rockers.
BTW, I have seen some pretty impressive restorations of faded red paint just by claybar, polish and a crap-load of elbow grease.
:edit: