There's a lot more to go wrong on xDrive cars. For example, on the E46, the front control arms were notorious for wearing out far more quickly than on the RWD cars, and they cost much more to replace as well. You'll pay a performance and fuel economy penalty, of course, no matter what bench racers try to tell you.
It's the buyer's decision as to whether the many tradeoffs are worth it. I don't think AWD belongs on a sports car (it's an opinion -- please, can we leave the Ur-Quattro and GTR out of this), but for a big sedan, well, it's not like you're getting mad RWD performance anyway.
I've got my eye on the F31, and the xDrive is a selling point because I never want to put on chains in the mountains. If it weren't for ski trips, I'd never even consider getting xDrive.