Changes proposed in PEP3121 and PEP384 have now been applied to the decimal module!

Thank you for the patch and the big amount of work that you are doing on so
many modules!

I'm afraid though that the patch is not acceptable in its current state:

    1) The unit tests do not pass. This could be fixed.

    2) The patch slows down _decimal by 25% (!).


If 2) cannot be fixed due to the increased amount of state lookups, I'm
firmly -1 on applying pep-3121 changes to _decimal (and any other speed
sensitive module).

> 2) The patch slows down _decimal by 25% (!).

Ouch. That's an interesting observation, because I guess other modules could be affected as well.

Robin, can you please take care that other modules don't have significant performance regressions after your work? I'm not sure what kind of changes it involves, or if it's possible at all, but we don't want to slow down Python for what is mostly (IMO) a code cleanup.

I have removed some redundant modulestate lookups and the testsuite now executes the decimal tests as fast as before the patch is applied. (at least for me).
May I ask how you tested the decimal performance?

Regarding the failing test:
It appears that the hackcheck() method in typeobject.c is responsible for this failure: 

static int
hackcheck(PyObject *self, setattrofunc func, char *what)
{
    PyTypeObject *type = Py_TYPE(self);
    while (type && type->tp_flags & Py_TPFLAGS_HEAPTYPE)
        type = type->tp_base;
    /* If type is NULL now, this is a really weird type.
       In the spirit of backwards compatibility (?), just shut up. */
    if (type && type->tp_setattro != func) {
        PyErr_Format(PyExc_TypeError,
                     "can't apply this %s to %s object",
                     what,
                     type->tp_name);
        return 0;
    }
    return 1;
}

As the context-type is now a heaptype the while-loop will continue to jump to the basetype of the conext-type, which is the object-type. There it ultimately fails when it compares func (context_setattr) to the tp_setattr of object.
Anyway, I do not understand the purpose of the hackcheck method, so I can not propose any kind of solution for this problem.

The test suite is not a good benchmark: it also tests decimal.py. For
numerical performance I'm running:

  cd Modules/_decimal/tests
  ../../../python bench.py

You can hit Ctrl-C after the first cdecimal result, since that's usually
already a pretty good indicator of overall performance. On 64-bit, for
9 digits of precision cdecimal is currently only around 1.5 times slower
than float. I want to keep that.

Running an unpatched _decimal.c three times gives (Linux, 64-bit, Core2 Duo):

0.162576s  0.165146s  0.163242s

With your second patch:

0.204383s  0.204383s  0.206919s

> Regarding the failing test:
> It appears that the hackcheck() method in typeobject.c is responsible for this failure: 

Thanks for the analysis. Perhaps Martin can comment on that.

hackcheck fixes the "Carlo Verry hack", which goes like this:

py> object.__setattr__(str, 'lower', str.upper)
py> 'dammit Carlo!'.lower()
'DAMMIT CARLO!'
(from http://bugs.jython.org/issue1058)

It shouldn't be possible to monkey-patch a builtin type; I believe this is to prevent crashes when self has the incorrect layout. Other than that, I find that an arbitrary restriction, except that setattr/attribute assignment prevent an assignment from occurring, it shouldn't be possible to bypass this by calling __setattr__. So if the restriction could be lifted, it should be lifted in both cases.

What specific decimal test is failing?

> What specific decimal test is failing?

# Attributes cannot be deleted
    for attr in ['prec', 'Emax', 'Emin', 'rounding', 'capitals', 'clamp',
                 'flags', 'traps']:
        self.assertRaises(AttributeError, c.__delattr__, attr)

test test_decimal failed -- Traceback (most recent call last):
  File "/home/stefan/pydev/pep-3121-cpython/Lib/test/test_decimal.py", line 3683, in test_invalid_context
    self.assertRaises(AttributeError, c.__delattr__, attr)
  File "/home/stefan/pydev/pep-3121-cpython/Lib/unittest/case.py", line 571, in assertRaises
    return context.handle('assertRaises', callableObj, args, kwargs)
  File "/home/stefan/pydev/pep-3121-cpython/Lib/unittest/case.py", line 135, in handle
    callable_obj(*args, **kwargs)
TypeError: can't apply this __delattr__ to object object

1 test failed:
    test_decimal

In order to avoid the significant slowdown:  Could we create a new
kind of method (METH_STATE) and change ceval to pass a state struct
that contains the thread and the module state as the first parameter
if the METH_STATE flag is present?

This sounds like a question for python-dev (or perhaps python-ideas).

Yes, python-ideas is probably better. -- I just noticed that the
total slowdown is even 40% if the global variable "cached_context"
is also placed into the module state (as it should).

Sorry Robin, I was wrong about the context -- it should be fine
since it's thread-local.  So the slowdown is back to 25%.

I would like to reject this until either the performance problems
are solved or someone actually uses _decimal in multiple interpreters.

If you do, you get tons of warnings about libmpdec being reinitialized,
so I suspect no one has ever done that in Python 3.3+ (I can't imagine
that such scary warnings would not be reported).

stage: resolved

messages: + msg168536