Yes, the Arab-Berber community is at the top and all the others are below.
There is also a caste system within the black Fulani community. For example, when we talk about the Fulani and Soninke, there are Soninke nobles, Soninke slaves, Soninke blacksmiths, and Soninke griots; there are many categories.
The problem is that, among the Moors, the Arab-Berbers, that is accompanied by actual oppression. For example, an Arab-Berber may take a black, reduce him to slavery, deprive him of freedom, and condemn him to forced labour. However, a Fulani or Soninke noble, for example, cannot do that. He can only exercise matrimonial discrimination against the slave, but he may not deprive him of freedom or exploit him physically or economically.