ENH: use more fine-grained critical sections in array coercion internals (#30514) by charris · Pull Request #30620 · numpy/numpy

Backport of #30514.

Towards addressing #30494. Right now the critical section I remove here in PyArray_FromAny_int shows up in the profile for the script in that issue.

I added the critical section in 5a031d9 and this partially reverts that change. On reflection, it's not a good idea to introduce a scaling bottleneck here in service of a sort of wonky thing to do: mutating the operand of np.array() while the array is being created.

Instead, we should error in those cases. Like we already do without the critical section!

I also needed to add new, more fine-grained critical sections, in PyArray_DiscoverDTypeAndShape_Recursive and PyArray_AssignFromCache_Recursive to avoid data races due to use of the PySequence_Fast API.

I also updated the tests for this to allow the cases affected by this to raise errors instead of succeeding, since that success relies on introducing scaling bottlenecks for valid read-only uses.

Here's a Samply profile output run using this PR on the script from #30494 - I no longer see a scaling bottleneck inside the array coercion routines: https://share.firefox.dev/44KLZxs