Messages (8) msg228809 - (view) Author: Kevin Dyer (Kevin.Dyer) Date: 2014-10-08 18:33

The following can be used to generate a file called mega_concat.py:

N = 2**17
my_array = []
for i in range(N):
  my_array.append("\"\"")
print '+'.join(my_array)

Then, on Ubuntu 14.04, 32-bit:

$ python mega_concat.py 
Segmentation fault (core dumped)
$ python3 mega_concat.py 
RuntimeError: maximum recursion depth exceeded during compilation

Seems like this is a simple out-of-memory issue.

msg228823 - (view) Author: Josh Rosenberg (josh.r) * Date: 2014-10-08 23:53

It's not out of memory at all. It's (probably) a blown stack during compilation optimization. Modern Py3 has "fixed" this by simply preventing silly levels of literal concatenation like this causing indefinite call stack expansion; the older ones just allowed the call stack to grow out of control (and the stack can blow long before memory is exhausted).

I doubt a fix would be backported, and a fix for Py3 seems unnecessary unless you can come up with a scenario where the recursion depth error is unacceptable (eval-ing arbitrary user input doesn't count, for obvious reasons).

msg228824 - (view) Author: Kevin Dyer (Kevin.Dyer) Date: 2014-10-09 00:04

Gotcha. Thanks for the explanation.

Two questions:

1) Given that it's probably not an out-of-memory issue, is it possible that this could turn into something more malicious? (e.g., an arbitrary code execution vulnerability?)
2) It's not obvious to me why eval-ing arbitrary user input doesn't count. Isn't an interpreter segfault a bad thing, even if it's a somewhat degenerate use case?

msg228852 - (view) Author: STINNER Victor (vstinner) * Date: 2014-10-09 10:02

> Then, on Ubuntu 14.04, 32-bit:
> $ python mega_concat.py 
> Segmentation fault (core dumped)

What is your Python version? It looks like Ubuntu Trusty provides Python 2.7.6. I'm unable to reproduce the issue on ArchLinux 32-bit with Python 2.7.8. The issue was maybe fixed between 2.7.6 and 2.7.8.

[haypo@arch32 ~]$ cat megaconcat.py 

N = 2**17
my_array = []
for i in range(N):
  my_array.append("\"\"")
print('+'.join(my_array))

[haypo@arch32 ~]$ python2 -V
Python 2.7.8

[haypo@arch32 ~]$ ulimit -v 8000
[haypo@arch32 ~]$ python2 megaconcat.py 
Traceback (most recent call last):
  File "megaconcat.py", line 5, in <module>
    my_array.append("\"\"")
MemoryError

msg228856 - (view) Author: Josh Rosenberg (josh.r) * Date: 2014-10-09 10:59

You're supposed to use that code to create a file (the output file is just ""+""+""+""+""+""+""+""+""+...+"").

If you want something that won't MemoryError generating the file, this is a "memory free" version of the code to generate the file:

import itertools, sys
sys.stdout.writelines(itertools.repeat('""+', 2**17))
print('""')

Run that code (saved as whatever file name you like) and pipe the output to mega_concat.py, then run mega_concat.py to repro.

msg228859 - (view) Author: STINNER Victor (vstinner) * Date: 2014-10-09 11:46

> You're supposed to use that code to create a file (the output file is just ""+""+""+""+""+""+""+""+""+...+"").

Oh ok, the bug is in Python compiler. It's probably an overflow of the C stack.

The bug was fixed in Python 3.3 by the issue #5765: see changeset ab02cd145f56. See also the issue #19098 ("sys.setrecursionlimit(2**30) breaks interactive shell") and the changeset c3df31cbcd0a.

Nick Coghlan wrote in the issue #5765: "However, agreed on the won't fix for 3.2 and 2.7, although I'd consider it at least for 2.7 if someone went through and worked out a patch that applies cleanly."


$ python2
Python 2.7.5 (default, Sep 25 2014, 13:57:38) 
[GCC 4.8.3 20140911 (Red Hat 4.8.3-7)] on linux2>>> compile('1' + '+1'*10**5, '<string>', 'exec')
<code object <module> at 0x7f8c5ba2c1b0, file "<string>", line 1>
>>> compile('1' + '+1'*10**6, '<string>', 'exec')
Erreur de segmentation (core dumped)

$ python3
Python 3.3.2 (default, Jun 30 2014, 17:20:03) 
[GCC 4.8.3 20140624 (Red Hat 4.8.3-1)] on linux
>>> compile('1' + '+1'*10**3, '<string>', 'exec')
<code object <module> at 0x7f7298d479c0, file "<string>", line 1>
>>> compile('1' + '+1'*10**4, '<string>', 'exec')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: maximum recursion depth exceeded during compilation

msg241305 - (view) Author: Aleksandr Yeganov (ayeganov) * Date: 2015-04-17 03:01

This is a backport of https://bugs.python.org/issue5765. I've pretty much taken the patch directly and applied it. I did have to modify the test code to use a different method.

msg367344 - (view) Author: Zachary Ware (zach.ware) * Date: 2020-04-27 01:35

As 2.7 has reached EOL, I'm closing this issue.

History Date User Action Args 2022-04-11 14:58:08adminsetgithub: 66773 2020-04-27 01:35:40zach.waresetstatus: open -> closed

nosy: + zach.ware
messages: + msg367344

resolution: out of date
stage: resolved

2015-04-17 03:01:21ayeganovsetfiles: + issue_22583.patch

nosy: + ayeganov
messages: + msg241305

keywords: + patch

2014-10-09 11:46:26vstinnersetnosy: + ncoghlan

messages: + msg228859
title: Segmentation fault with string concatenation -> C stack overflow in the Python 2.7 compiler

2014-10-09 10:59:11josh.rsetmessages: + msg228856 2014-10-09 10:02:04vstinnersetnosy: + vstinner
messages: + msg228852
2014-10-09 00:04:23Kevin.Dyersetmessages: + msg228824 2014-10-08 23:53:34josh.rsetnosy: + josh.r
messages: + msg228823
2014-10-08 18:33:29Kevin.Dyercreate