It doesn't seem like you were actually very involved in anything in the department. When you have conflicts going on 82% of the time, it seems convenient to say, “Oh, it wasn't me. My ADM, who reported to me, must have been getting orders from somebody else.”
Who else would he get orders from to do this, if not from you or the minister?