You say it's the only difference as if that's a small thing; it's an enormous thing. First off, there is the ability, in some instances, to ask to not receive certain unsolicited “real space” mail as well, although admittedly, lots of it still seems to make it through the door. There is a huge difference, because in the case of direct mail, physical marketing, the costs are being borne by the sender, and those are sizeable costs. They are not going to send out billions of pieces of paper, because the cost is too great, for a very low return. They really require some kind of legitimate return.
In the case of electronic messages like spam, it actually flips: the cost is almost costless from the perspective of the sender. Indeed, as they use things such as botnets and the like to send out their messages, in many instances it truly is costless. We bear the cost.
We bear the cost sometimes just in delayed time, but it's more than that. Our Internet service providers almost uniformly subscribe to various filtering services, costs that are ultimately borne by the consumer. Almost all network providers have to maintain additional equipment just to deal with the excess spam that isn't the legitimate stuff. We ultimately pay the price.
And it's not just network providers. Think of the schools, think of the hospitals that set up their own networking infrastructures, which at the end of the day are being forced to pay for this. I'd argue it's not an “only” here; it's a pretty significant cost that's being borne, and it's a clear cost shift from the person doing the marketing to the person who's on the receiving end, even though they never ask for it and in many instances don't want to have anything to do with it.