Put your hand in a plastic sandwich bag and slide it over the paint surface after washing. If it feels perfectly smooth, clay isn't needed. If you are waxing, and do not clay, you are sealing in, and potentially damaging the paint, if you do not clay. While waxing, you may pick up something in the applicator pad or buffer that will scratch the paint. So, its best to be safe and clean the paint before waxing/sealing.
As soon as you are done washing, your paint is getting recontaminated. Wash mitts are design to be gentle and not scratch the paint. They do not have enough (or any, really) scrubbing power to take off bugs and tar, etc. Clay lifts the contaminates off safely, and holds them away from the paint.
Twelve car cleaning myths - CNN.com
Number 1 on this list.