I've been driving red leather BMW's for 5yrs, never had any issues - not with blue jeans, wear or anything really.
The only thing I do, is make sure I almost lift myself out of the car using the sill, rather than brushing against the bolster.
Go for it
Mat