Only remove users which are not SHAPER_NOTLIVE

It could be we have multiple remove events queued, this is mostly
the case if we do a full reload from file and we populate very
many events with IP conflicts at once.