The problem with plastic is that when it ends up in nature and begins to degrade, it disintegrates into particles—microplastics—and various organisms ingest those particles. There's still a lot of research being done on that, but we know that is a problem specific to plastic.
In addition, it's important to consider the entire plastic life cycle, from production to use. Extracting the raw materials used to make plastics, as well as many other materials, is very energy-intensive, so that is a major consideration. Chemicals are used to extract those raw materials. Just as ore is extracted to manufacture metals, petroleum is extracted to produce plastic. All the forest equipment used to cut the trees is another factor. The chain of production involves a range of energy inputs or chemicals.
No matter which consumer good we look at, we quantify the greenhouse gas emissions as well as the other pollutants, which cause other problems, so we can do our comparisons. Such pollutants are found in the plastic production chain, on top of the microplastics, which can cause problems if ingested at the end of their life. Metals involve other kinds of problems. Obviously, there's a risk of shifting those problems if we aren't careful. The type of problem and extent of the problem depend on which products or systems are being compared.