Mine is black and I went through the same thing you did. I ended up using Meguiar's NXT Generation Tech Wax 2.0 Paste (
Meguiar's Direct NXT Generation® Tech Wax® 2.0 Paste).
The ultimate key to getting this done right is to use a polishing cloth to finish. Remove the paste from the paint first using a terry cloth. You'll still see swirls in the paint. Finish with a nice polishing cloth and it does wonders. If you use a terry cloth on the final polishing step, you'll see swirls.
As for the scratches, the NXT Gen paste does a good job of removing them. The fine scratches are typically only seen when my 4R is in the garage. Outside, you can't see them as much.