I know what you mean. The paint came right off the ones on my '96 and it's just milky white plastic under the paint.
My '96 is black so I painted the center caps metallic orange for baseball season (Giants) and got another set off ebay and painted them candy apple red for football season (49ers).
The only problem is now my son has the car away at college in Southern California.
But I think the caps were all silver originally if you just wanted to re-paint them stock.
The centers on my 97' are silver. I just paint them every other year or when the silver color comes off. Lot cheaper than buying new one's because they will eventually get crappy too.
On my '97 I did a light sanding with very fine sandpaper just to scuff the surface a little then laid down 4 coats of arctic silver Duplicolor,filled in the "T" with Toyota red touch up paint and then covered everything with 5 coats of clearcoat and that should hold up for more than a few years.
Just remember to use automotive paint and a good quality clearcoat to withstand being whipped with road dust and the UV rays and you should get very good results.