I have the same thing, it does not vibrate but when I push the brake pedal I can feel a little click. Sometimes I associate it with the gear shifting, sometimes not. Really annoying, but Toyota said its normal.
However, I also notice that when coming to a near stop, I get a subtle vibration...