Off-roading/adventuring is new to me and I just spent the weekend camping and driving along FSRs, came home and used my condo's hose to spray down the car and it looks 90% cleaner - but it seems there is also a bit of a film/layer of dirt stuck on - both the car body and windows. I don't want to RUB it because I assume rubbing all that dirt+grit will just create scratches.

Short of going to a touchless car wash, how do I do a better cleanup job for the car? I want to be able to do it more regularly when returning from adventures.

And is there anything I should take note of/be doing sort of getting the dirt off? I've never waxed/etc. a car before in my life

This is also obviously not a show car I'm aiming to keep in pristine condition haha but still want to take some level of care of it!

Once you get what you can with a pressure washer, you can wash it using the two bucket method, some automotive soap, and a microfiber mit. Tons of videos on Youtube about this.
Another method is to spray with a foam/soap gun and let the soap sit for a while before rinsing. Never tried this myself.

If you require even more of the dirt off of it, then a clay bar would be next, but probably not needed in your case.

Next time you go out, spray the whole underside/rocker panels with Turtle Wax spray wax. It keeps the dirt from really even sticking to the panel. I used to put 3-5 coats of hand wax on my old rig, and sprayed down the undercarriage with spray wax. I went wheeling all the time, and never had issues with mud not coming off even after wheeling multiple days/weeks in a row and not really washing it more than 2x a month at most.