I have a 318d Sport and whilst the 320d will be faster, I'm not sure you will really notice day to day, and the 318d really does have quite decent performance. Plus you can drive it like you stole it in sport mode and it will still do 50mpg.
Of course, so will the 320d, but in the end you'll be fine wither way, so I would go for the spec you most like the look of.
Current: Skoda Octavia Scout 4x4 DSG
Previous: F30 318d Sport, R56 MINI Dooper 2010, E93 325i M-Sport auto 2007, E91 320d SE 2005, E46 318Ci Coupe 2002