This version of the 320 is arguably the best:
http://jalopnik.com/let-chris-harris...bmw-1514877066
It's cheaper than the 328, but gives you just enough fun to make it stand out on its own merits. I've driven one (a friend's), and can confirm it's a blast for the price. If it were much more, it might not be worth it, imho.
Otherwise, the 320, as others have said, is a 328 without the option choices. You can make up the power difference (and then some), but the same options aren't available last I checked, and once you've spent the extra $, you've defeated the whole point of getting the 320 over the 328 in the first place.
In my humble opinion, if you know you'll just mod the 320 to make it more like a 328, just get he 328. No brainer for my money.