you have shitty bearings in your tranny synchronizers...some of us here on the boards with 2003 - 2004 models have had problems similar to this with bad transmission bearings...my dealer was going to give me a new transmission too, but I only had 25,000 miles on my car, and they decided to rip it apart and overhaul it under warranty instead...they found no real wear, just faulty bearings from the factory...some of the other boys here will probably chime in too...if its making a gearing noise its not the DAMN manifold!!! does it sound like its in reverse all of the sudden????...post more detail and I'l try to help from there
I had my synchronizers, 2nd,3rd,4th,5th,6th gears and inner-outer bearings on the differental all replaced, due to this noise. Toyota even had the clutch, release fork, bearing and pressure plate replaced too. I still have this noise during the same 3rd gear around the same RPM's, but the other gears it's very slight. My mechanic stated that it's the position the 3rd gear sits in and while spinning at that RPM will make the noise. When my RPM's are higher at like 4000 it's not there. My mechanic said that there is really no way to explain it in words that's just how it's designed. He also drove my car and told me, "what are you worried about, your car shifts fine and every gear has power, your car is fine Nate don't worry, you worry to much". He also told me "You bought a performance car not a luxury car" "if you expect this engine and tranny to be quiet get a luxury car". I don't know if you ever heard rally car gears, but they sure chirp and wine like crazy, like they're going to fall apart, but keep pulling strong all the way through the race. I went through the same thing as you bro, seven times my car was at Toyota and all in all I basically got a new tranny and clutch. For my bearings in the differential, they needed to be changed anyway from driving to fast all the time on the freeway...well maybe they were just defective too.
Still today, two years later my car still makes the noise, but still drives and shifts like a champ. The noise isn't getting any louder then it was before. You can't notice the noise if you have an exhaust...lol