This is either a "won't fix" or an "impossible to fix" depending on your point of view.

PATIENT:
     It hurts whenever I press ctrl-C
DOCTOR:
     Then don't press ctrl-C

The problem is that ctrl-C can provoke an interrupt at any point in the program, and thus break presumed invariants.
try-finally has the same problem.

From the observable side effects it is indistinguishable whether an interrupt occurs after the last instruction in an __enter__ function or after the first (side-effect-less) instruction after the __enter__.
Likewise the last pre-__exit__ and first in-__exit__ instructions are effectively the same from the point of view from observable side-effects, but are different from the point of view of the handling of exceptions.