An implementation that lets the size overflow and returns a pointer to a block that isn't large enough for "an array of nmemb objects, each of whose size is size" is not conforming with that specification.

That specification gives the implementation exactly two options: return NULL, or return a pointer to a block of sufficient size.