First of all, smoking has limitations placed on it so it is not a right, it is a privilege (like driving, there are age limitations, taxes, restrictions as to where it is and is not legal, etc.). Breathing is a right. If a privilege interferes with a right, the right takes precedence. For years this was turned on its head with smoking, as non-smokers had to suffer this addiction in public locations. Are some people still upset by this? You betcha!
By your own admission, it is difficult to come up with another habit that is legal and yet so harmful to both the user and those around them. Driving creates pollutants, but the benefits to society are numerous, drinking is more confined (driving drunk is illegal, public drunkenness is illegal, etc.) and restrictions are harsher including jail time for some infractions, drug abuse is illegal, sexual deviation is illegal, and so on. I would say you smokers are just lucky your addictive behavior is tolerated by society to the extent that it is. Now you want us to be nicer about it?