If you are playing loud music late at night, you might make my life less enjoyable, but that doesn't make you the worst sort of person because you're not doing me any actual harm. And if I ask you to turn it down or call the police and complain about the noise of your party, then I'm making your life less enjoyable too.

Also, tax collectors and the people who charge me money for goods and services are definitely making my life less enjoyable. They should stop doing that. :p