I constantly upshift and downshift between 3-4-D(5), I like the feeling of holding a gear a little longer than the ECU likes me to, and I feel like I can't get that kind of control with my foot. So usually when slowing down I'll pull over to 4 then down to 3 (not over-revving the engine, obviously), then I'll upshift from a stop waiting for the engine to shift into 3rd automatically and I'll take it from there. The position of the lever is really easy to make this a comfortable option (don't have to push a button to move the lever, only have to move the shift lever one "slot").
On the highway, I don't like to have the engine try to downshift for me when passing because there are times when I'll push the pedal a little too far and get that annoyingly rapid 5-4-3 down shift (why doesn't it just go to 3?), so I'll pull it into 4 and match the revs with the gas pedal then, just like magic I'm in the prime power band and around I go without shifting into 3. At speeds from about 60-75 mph, I can still grab 3rd gear if I really need the go, but I'll do that with my foot and let the computer handle that shift because it occurs pretty close to redline.
I'm not sure what this does to the drivetrain. I'm sure it introduces more wear and tear because you are using the drivetrain to help remove energy from the car and not as much the brakes. Upshifting manually I don't think would cause any extra wear on the tranny. I'll be sure to have a full tranny service when the owner's manual suggests, but until then I've been doing this for 20k+ miles and it doesn't seem to have any ill effects...