You don't have to worry about being too "ginger" with the door, I've personally never had issue with it, nor have my passengers. Haven't found the position of the handle awkward either (shrug).
Quite interesting you're having an issue. I guess if you go with "half" brute force you should be fine.
