I wouldn't do that on the first challenge, but I might if it was becoming obvious that confidence in the chair had been lost. The chair would obviously need to then remove himself.
I don't think challenging the chair Mr. Speaker means loss of confidence in the chair. There may be a legitimate reason. The chair may legitimately err as a human and may get bad advice.