Again, my experience with this certainly comes directly from living the dream and having worked for different governments and seen what worked and what didn't. You're right. It's a difficult question. I know that Public Services and Procurement Canada is working on it and looking at different models. I wouldn't want to prejudge its efforts.
The features you need to get right are that, if you do it right, and when I have seen it work, the performance management system becomes an incentive not only for the companies but for the project managers inside government to improve their performance. You end up with procurement officers who want to have vendors that are scoring the highest grades. It becomes a mark of pride to them to have good suppliers. They proactively work with suppliers to improve their grades before they get rated.
When we were subject to systems that worked, that's why it worked, because the person on the other side who was buying was as interested in our performance as we were and was constructive in helping us become a better company. When you do that, procurement will get better for everybody. Companies will deliver, they will enjoy that, the government customer will enjoy it, and we'll end up with better companies and a better procurement system. The trick is to use it as a way to encourage everyone to do better, not as some kind of filter where you weed the bad from the good.