I don't see why there has to be created a helper object for each entry. Collections with custom hashing strategies exist, in Trove4j or (IIRC) Guava.
Helper objects still would have linear complexity instead of quadratic, though indeed much slower.
Your idea with a quick class check might also work to prevent the helper object creation in case it's not necessary. Although it would still be much slower than just an equivalent call of HashSet.removeAll.
And as for me, I don't need that number-aware comparison. I don't need for arrays to be equal either, although this feature seems to be more useful to me. If I needed those features, I'd think about it anyway by using a special data structure, like a map/set with a custom hashing strategy, and reserve "-" just as a handy syntactic sugar for copy+removeAll.
But that's only my preference. Maybe others think differently.