I know many people disagree, but I just don't see the S4 and 335i as real competitors. Granted, they are comparable in many respects and you can spec out the cars for a similar price so they fit in the same category for the so-called enthusiast.
My objections are twofold:
1. I've found it tough to find S4/5s to test drive, never had a problem going to my dealer and driving a 335i prior to purchase. The greater likelihood of having to order a car vs. being able to buy off the lot means that we're looking at two potentially very dissimilar customer pools.
2. The starting MSRP difference between the S4 and 335i is roughly 5,000. Not that most people are buying base models, but a difference of over 10 percent in starting price is not insignificant.
Obviously, I'm just speculating, but the above factors make me think that, given the non-enthusiast nature of the average consumer, there are a decent amount of people who would consider a 335i and not want to look at an S4/5.