I'd say you're probably fine driving it with this code, but if it's a case of your catalytic converter being partially clogged, your engine performance will suffer. I think the main thing is having to deal with getting it to pass smog. If I eliminated other possible causes, like replacing the O2 sensor and needed it to pass smog, I might try a product called Cataclean.
https://amzn.to/38we1hd
It gets mixed reviews but some people have claimed it fixed their issue. So, before I'd drop big money on a catalytic converter replacement, I'd try the Cataclean and hope it worked to clear the CEL code.
I've gotten the P0420 code many times and it's always on a long road trip. Back at home, it never pops up.
This article discusses the possible causes of a P0420 code:
Toyota Parts | P0420 Diagnostic Code - Trouble Code Diagnosis Guide
This article discusses the side affects of having a clogged cat:
What Are The Bad Catalytic Converter Symptoms? | CAR FROM JAPAN