Obviously, the wiring is wrong. I don't know where that little chart you got came from but something is not right.
Here's how you fix it:
1. Go to your local salvage yard or junk yard and get a new Toyota plug (which seems to be missing from the last install). Give yourself about 3-4 inches of wire with the plug.
2. Buy the correct plug to fit from the Toyota plug to your new stereo (Crutchfield is a good place to start they are SUPER helpful for anything car audio!).
3. Un-do all the butt connectors you've got on there and then attach the bare Toyota wires to the used harness you got in step 1 (butt connectors work here but make sure you're using the right ones for the right gauge of wire or they could come loose, soldering is better if you can). Then just plug everything together!
This way if you ever change the head unit again it's all cleaned up and you may not even need to get a new adapter. Trying to match the old wiring to a new harness is extremely hard.
P.S. it's common to have wires from the factory wiring that don't connect as well as from the new harness. Most new harnesses also have a solid blue wire and orange wire (dimmer and something else) that aren't used, yours doesn't seem to though.