Having worked in the Swedish government for five years, I think the disadvantage of only having an office in the government is that when the political majority of the government changes, the policies change quite radically and very quickly. If you have an independent function outside of the government, you create some stability in the work and in the investigation of how to continue to do the work.
As we have here in Sweden, it is preferable to have both an office or people responsible within the government, and an independent national rapporteur.