Understanding Ethereum's proof of stake: Can cartels (at least probabilistically) punish the minority by ignoring their votes?

From my understanding, every staker must vote in ethereum for the chains they choose. What I don't understand is, what if 51% of stakers ignored the votes of someone with low balance (like 32 ETH). Can they lead to them being slashed and lose money?

Is there a good explanation of this problem from a technical point of view, and how it's mitigated, and what trade-offs are accepted?