It may have been the minister's office that contacted you, but it was the PM's office that was setting them up, and then it was communicated to you by the minister's office that this meeting was to occur.
You say that it's unusual, and I'm wondering, did you not ask for the rationale and was none provided? This, as you say, isn't an everyday occurrence, and the Prime Minister's Office is showing a specific interest in this particular file. They gave you no rationale as to why this file and why these meetings?