there's a lot of personal opinion here, so here's mine.
The 335i can't compete with the S4 because the price difference is nearly $8k (335xi M Sport vs. S4).
With that in mind, I think a 335xi M Sport does remarkably well against an S4.
The M3 and RS4 are the competitors for sure, but it's difficult to compare because their power delivery strategies are entirely different (RWD vs. AWD, respectively).
I ended up getting a RWD 335i because I wanted to own a RWD car once in my life, wanted to save nearly $14k, and figured I could add the tweaks with the M Performance parts I felt were what I wanted (Suspension, PPK, Exhaust; brake upgrades are relatively pointless for me since I don't track my cars as much as I used to with the Evo).
I like that BMW offers all the choices though...Audi just doesn't, and the price for an A4 S line came within thousands of my 335i RWD, and THAT performance/handling gap is significant (stock for stock).
2012 335i Mineral Gray
M Performance Whore: PPK|Exhaust|Brakes|Performance Suspension Struts & Springs|LSD|Black Kidney Grilles|CF Spoiler and Mirror Caps|Bav Stage 1|400Ms|BMWF30.com