I'm not a biologist. I don't actively study the approximate ecological impacts of net pen systems. The other challenge is, to be frank, that when you ask which is the most important or has the greatest impact, that is always, to a certain extent, in the eye of the beholder. I will provide you an answer, but my answer is going to be informed by my understanding of other people's work and my own values.
For net pen related work, from my perspective, the greatest chronic ecological effects would be those with benthic impacts, as well as in-shore impacts of waste accumulation in poorly sited settings. If we're taking impacts on salmonids off the table, in terms of sea lice and other negative interactions through disease transmission, I would probably say some of the more interesting and challenging approximate impacts associated with poorly sited farms and the benthic impacts associated with those. Again, those are impacts that can be remediated with time. If you fallow a site, my understanding is that in many instances within five years you can have a great deal of recovery in a lot of cases. Of course, depending on the locale and the setting, there can be big issues of negative predator interactions—seals and sea lions drowning in nets—but from my understanding, those tend to be relatively episodic. No one wants seals and sea lions drowning in nets. These situations are potentially avoidable.
I'm not sure if this helps.