With caching, 512 intset entries, and 64 bit intset values, that's under 1 microsecond to intersect using the naive binary search algorithm for intersection.
I don't believe that small sets are an issue for performance.
For me, the real question is whether there are ways of getting good performance and lower memory overhead across the entire range of object sizes in Redis.