Make sure you drive an F30 with M adaptive suspension.
I say go for it. The E60 is not a very good looking car, IMO. And it has not aged very well. Going to the F30 (especially if you get Tech package, Driver's Assistance package, etc) will feel like a leap forward 10 years into the future.
2016 M3 - current
2013 ActiveHybrid 3 - sold
2012 M3 coupe - sold