I spent some time on this over the weekend, because I was never very happy with the first install attempt. So I ripped the box back out and tried your "soldering a switch into the blue and yellow wires" trick... and broke the blue wire free from the board in the process. Go me!
So I bought another remote box ($20), some 22 gauge wire and some phone line crimp-on wire taps from Radio Shack, which worked far better for tapping into those blue and yellow wires. I used heat shrink as wire protector and routed them out through the same hole as the mic cable. Then, I just soldered the two new wires to a stereo miniplug with a right angle that I scavenged from my box-o-crap, and it works marvelously.
When I want to listen to the CB through the AUX plug, I just select it as the source and we're good to go. When I don't I can pull the plug or just switch to a different source and the handset speaker works as usual. Yes, the handset speaker is always on, but this solution works as well as Antman's and I didn't have to drill another hole for a switch.
Antman, you might not be as prolific as some other poster, but you rocked this one. Thanks!