You're right, that could technically apply in situations that are not necessarily perceived as being elder abuse. However, it has to be the combination of age and other personal circumstances. Therefore, logic would apply in cases where the victim is an elderly person. To that extent, as long as the combination of age and other personal circumstances do apply, it would favour an application for older victims, if you want, rather than a 30-year-old with some health problems.
In addition, other aggravating factors would cover those situations. If we read the list of aggravating factors, one would expect that elder abuse situations would be covered by that aggravating factor.