I think it's very difficult. You can play around with the thresholds, and make thresholds higher or lower in order to ensure that certain smaller parties don't get elected. However, if you're opening a system that's more proportional, which I don't think is necessarily a bad thing, it also means that small parties of all sorts of persuasion can get representation in Parliament. I don't think you can avoid that when you're making a system more proportional.
Again, I'm not saying that I'm necessarily against more proportionality, but you have to accept that this is one of the things that might come with it, a greater chance for parties, such as radical rights or other types of radical parties, to gain a foothold.