Sounds like worn tie rods. Easy way to check: jack up the front end and grasp the tire at the 3 and 9 o'clock positions. If the tire wiggles from side to side, it's the tie rods. From top to bottom, ball joints (lower).
So I jacked up the front end and at the 3/9 o'clock positions there was play on the drivers side, and none on the passenger side. At the 12/6 positions there was 0 play on both tires.
So I had my dad wiggle the tire while I searched for what was loose and this is exactly where the problem is. Also I had him wiggle the tire when I held on to the steering wheel and it felt like the same knocking sensation I described originally.
The red arrow is exactly where there is play when I pinch thru the boot. That same boot on the passenger side has no wiggle room.