Home Original page

Remove NotImplemented handling from the ufunc machinery (almost) by njsmith · Pull Request #5864 · numpy/numpy

added 2 commits

May 11, 2015 14:50
… case

The ndarray richcompare function has a special case for handling
string dtypes (which currently cannot be handled by
ufuncs). Traditionally this was handled by the ufuncs returning
NotImplemented, and then falling through to a special case. By moving
the special case to the top of the richcompare function, it becomes
unnecessary for the ufuncs to return NotImplemented in this case.
The ndarray richcompare function has special case code for handling
void dtypes (esp. structured dtypes), since there are no ufuncs for
this. Previously, we would attempt to call the relevant
ufunc (e.g. np.equal), and then when this failed (as signaled by the
ufunc returning NotImplemented), we would fall back on the special
case code. This commit moves the special case code to before the
regular code, so that it no longer requires ufuncs to return
NotImplemented.

Technically, it is possible to define ufunc loops for void dtypes
using PyUFunc_RegisterLoopForDescr, so technically I think this commit
changes behaviour: if someone had registered a ufunc loop for one of
these operations, then previously it might have been found and
pre-empted the special case fallback code; now, we use the
special-case code without even checking for any ufunc. But the only
possible use of this functionality would have been if someone wanted
to redefine what == or != meant for a particular structured dtype --
like, they decided that equality for 2-tuples of float32's should be
different from the obvious thing. This does not seem like an important
capability to preserve.

There were also several cases here where on error, an array comparison
would return a scalar instead of raising. This is supposedly
deprecated, but there were call paths that did this that had no
deprecation warning. I added those warnings.
ndarray special methods like __add__ have a special case where if the
right argument is not an ndarray or subclass, and it has higher
__array_priority__ than the left argument, then we return
NotImplemented and let the right argument handle the operation.

ufuncs have traditionally had a similar but different special case,
where if it's a 2 input - 1 output ufunc, and the right argument is
not an ndarray (exactly, subclasses don't count), and when converted
to an ndarray ends up as an object array (presumably b/c it doesn't
have a meaningful coercion route, though who knows), and it has a
higher __array_priority__ than the left argument AND it has a
__r<operation>__ attribute, then they return NotImplemented.

In practice this latter special case is not used by regular ndarrays,
b/c anytime it would need to be triggered, the former special case
triggers first and the ufunc is never called. However, numpy.ma did
not have the former special case, and was thus relying on the ufunc
special case. This commit adds the special case to the numpy.ma
special methods directly, so that they no longer depend on the quirky
ufunc behaviour.

It also cleans up the relevant test to things that actually should be
true in general, instead of just testing some implementation details.
This was redundant/broken anyway.

See numpygh-5844 for discussion. See the massive comment added to
ufunc_object.c:get_ufunc_arguments for a discussion of the deprecation
strategy here -- it turns out that array_richcompare is such a
disaster zone that we can't quite wholly eliminate NotImplemented
quite yet. But this removes most NotImplementeds, and lays the
groundwork for eliminating the rest in a release or two.

@charris charris added this to the 1.10.0 release milestone

May 14, 2015

@mhvk mhvk mentioned this pull request

May 14, 2015

charris added a commit to charris/numpy that referenced this pull request

Jun 13, 2015 
Also added back some extended error messages that were in original
PR numpy#5864.

@mhvk mhvk mentioned this pull request

Jun 16, 2015

mhvk added a commit to mhvk/astropy that referenced this pull request

Jun 17, 2015 
In numpy-dev, the __index__ can no longer be used to multiply lists;
for discussion about why, really, this was always broken & wrong, see
numpy/numpy#5864.

This was referenced

Jun 17, 2015

mhvk added a commit to mhvk/astropy that referenced this pull request

Jun 17, 2015 
In numpy-dev, the __index__ can no longer be used to multiply lists;
for discussion about why, really, this was always broken & wrong, see
numpy/numpy#5864.

mhvk added a commit to mhvk/astropy that referenced this pull request

Jun 24, 2015 
In numpy-dev, the __index__ can no longer be used to multiply lists;
for discussion about why, really, this was always broken & wrong, see
numpy/numpy#5864.

patti pushed a commit to patti/astropy that referenced this pull request

Jun 30, 2015 
In numpy-dev, the __index__ can no longer be used to multiply lists;
for discussion about why, really, this was always broken & wrong, see
numpy/numpy#5864.

dhomeier pushed a commit to dhomeier/astropy that referenced this pull request

Aug 11, 2015 
In numpy-dev, the __index__ can no longer be used to multiply lists;
for discussion about why, really, this was always broken & wrong, see
numpy/numpy#5864.

@mhvk mhvk mentioned this pull request

Jan 26, 2016