I'd remove the assembly and shake it out like you mentioned. You could also try vacuuming it out with the hose from the vacuum. Orrrrrrr just leave it in there, what's it gonna hurt?
__________________
2017 T4R TRD Pro
Cement - Dobinsons 2/1" (300/599), IMS 700/701, Air Lift 1000 air bags (60804), Pelfreybilt hidden winch mount, Warn VR8 w/ Amsteel synth, 2021 ORP LED headlights, DD SS3 Sport yellow fogs, CBI steel skid plates, Rough Country 20" LED bar, Gobi stealth no rise w/ ladder, ARTEC sliders, ARB 63qt fridge, Canvasback liners, homemade spare tire hitch carrier, ARB single air compressor w/ Slee mount, bronze TRD Pro wheels