No reason to be afraid. Just be smart when driving and don't be reckless.
Trust me I can't do anything hardcore...just your average trails...
Some of the trails I've done in So Cal are Anza Borrego, Ocotillo Wells, Fish Creek, Big Bear (service roads), and Mojave Road.