Well, you accelerate a little faster due to less rotational mass, but let's say when you approach a hill at a constant speed, the hill should theoretically slow you down faster...but it won't.
So, downsides are that it is easier to stall the car and that the gearbox becomes a tad noisier, due to less mass = less damping. So, you'd have to up the idle speed to prevent continuous stalls.
The transmission gets more wear and tear because the peak torque is higher as the damping action of the mass is lost.
The notable downside would be a reduction in fuel economy, also.
So, you don't necessarily want to go too light unless your application is for drag.