I probably caught you too late, but I live up here.
You do need an Adventure Pass if you intend to park anywhere on any of the trails. Your campground host can usually sell you one. I think it's $30, but you can get a second one for only $5 more, so split it with your buddy.
The best place to start is anywhere north of the lake between Fawnskin and Big Bear City.
There are a ton of trails. You should be able to find a free 4x4 tail map at 7-11 or a few of the local gas stations. For future use, you can download it here:
http://backcountry4x4.com/files/Big_...l_Map_PDF2.pdf
This give you a great idea of what is up here and the difficulty of the trails.