Certainly. I've heard of cases where a caregiver works in what otherwise seems to be a very good home. This is the employer they need to carry them through to the end, and then something will happen or the family has a change of plans. It turns out they don't necessarily care as much about the caregiver as they might have let on, and they end the employment and the caregiver is left in the lurch.
If the caregiver is sick for an extended period and can't necessarily do the tasks associated with the job, they're going to suffer when the employer says this can't work out because it's just not working for the family. But the caregiver, in the long run, is again the one who would suffer the most in that case.