if it is newest kernels(where nat has special process to cleanup conntrack) and you have some sort of unidirectional flood from one of users - it might happen
most probably then just issue of X520, they can't load balance pppoe you need to check, maybe you have too many rules in firewall or conntrack has too many connections