I get this too and I've not bought my M4 yet but every other time I bought a new car. Bottom line is you shouldn't care what other people say as their comments are driven by myriad of differnent reasons from jealously to not understanding others passion for cars. Personally, my main drive for making money was for a car. Obviously this is now second to family but, like you, I'm more than providing there.
My advice is, having spent so much on your car, DO NOT let these people spoil it for you. You deserve it. Enjoy