Issue 36981: asyncio transport.write() memory out

Issue36981

Created on 2019-05-21 02:23 by viocal, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (14)
msg342973 - (view) Author: (viocal) Date: 2019-05-21 02:23
in asyncio
when  filedata than free memory(hardware) 
will be memory out Or killed by OS

for buf in filedata:
   transport.write(buf)
#to client


I  try it todo:
abort  transporting to protect application be killed by OS
modified selector_events.py

    def _write_ready(self):
        assert self._buffer, 'Data should not be empty'

        if self._conn_lost:
            return
        try:
            n = self._sock.send(self._buffer)
        except (BlockingIOError, InterruptedError):
            pass
        except Exception as exc:
            self._loop._remove_writer(self._sock_fd)
            self._buffer.clear()
            self._fatal_error(exc, 'Fatal write error on socket transport')
            if self._empty_waiter is not None:
                self._empty_waiter.set_exception(exc)
            return
        else:
            try:
               if n:
                   del self._buffer[:n]
               self._maybe_resume_protocol()  # May append to buffer.
               if not self._buffer:
                   self._loop._remove_writer(self._sock_fd)
                   if self._empty_waiter is not None:
                       self._empty_waiter.set_result(None)
                   if self._closing:
                       self._call_connection_lost(None)
                   elif self._eof:
                       self._sock.shutdown(socket.SHUT_WR)
            except Exception as exc: #(MemoryError)
                 self._buffer.clear()
                 self._loop._remove_writer(self._sock_fd)
                 self._fatal_error(exc, 'Fatal write error on Selector SocketTransport write ready')
                 if self._empty_waiter is not None:
                     self._empty_waiter.set_exception(exc)
                 return
msg342982 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-05-21 06:37
The correct approach is using Protocol.pause_writing() / Protocol.resume_writing() callbacks.
No trivial, though.
See `StreamWriter.drain()` for example of implementation.
msg343010 - (view) Author: (viocal) Date: 2019-05-21 11:07
I use rotocol.pause_writing() / Protocol.resume_writing()
but results is no change(memory out Or killed by OS)
msg343014 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-05-21 11:18
I'm sorry but I cannot tell why you are using `pause_writing` incorrectly without looking on the code.
msg343015 - (view) Author: (viocal) Date: 2019-05-21 11:33
for buf in filedata:
    asc.resume_writing()
    asc.transport.write(buf)
    asc.pause_writing()
msg343017 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-05-21 11:42
No. It doesn't work this way.
pause_writing is a protocol callback called from transport when the internal buffer is full.
In reaction to this callback the producer should stop calling transport.write() and resume writing after getting `resume_writing()`.

Flow control is hard. As I wrote you can take a look at asyncio streams for example how to do it.
msg343024 - (view) Author: (viocal) Date: 2019-05-21 12:07
thanks you 
but I think protocol.resume_writing() / protocol.pause_writing() is auto called   by Protocol
because set transport.set_write_buffer_limits(high=65536*2, low=16384*2) #default (high=65536, low=16384)
msg343032 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-05-21 12:19
No. pause_writing/resume_writing are protocol callbacks called by transport.
User code should respond to these callbacks by stopping sending data to transport (transport.write()).

The logic is a little complicated but it is ok for very low-level asyncio API.
Convenient user-facing wrappers like asyncio streams hide this logic by providing high-level primitives that support flow-control out of the box

I'm closing the issue, nothing to do here on asyncio low-level side.
msg343042 - (view) Author: (viocal) Date: 2019-05-21 12:58
for example 
the system free memory size is 512m
and filedata size is 500m
will transport Success
but filedata than 512m will be failed
msg343043 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-05-21 13:02
Please re-read my previous comment.
If you use asyncio incorrectly -- yes, you can run out of memory.
The proper usage of pause_readind()/resume_reading() resolves the issue.
msg343136 - (view) Author: (viocal) Date: 2019-05-22 00:42
thanks again
the environment:
filedata1<512M
filedata2>512M
filedata3>1G
this computer-------------peer computer
server(with asyncio)------clien(socket without asyncio)
memory<512M    -----------memory>512M
read filedata1 <--------- success
read filedata2 <--------- success
read filedata3 <--------- success
write filedata1 ---------> success
write filedata2 ---------> fail
write filedata2 ---------> fail

how todo  (function set_write_buffer_limits) is work
msg343141 - (view) Author: (viocal) Date: 2019-05-22 03:28
I have fixed it
modified selector_events.py

    def write(self, data):
        if not isinstance(data, (bytes, bytearray, memoryview)):
            raise TypeError(f'data argument must be a bytes-like object, '
                            f'not {type(data).__name__!r}')
         ...
        if not self._buffer:
            # Optimization: try to send now.
            while True: #########add by viocal
                try:
                    n = self._sock.send(data)
                except (BlockingIOError, InterruptedError):
                    pass
                except Exception as exc:
                    self._fatal_error(exc, 'Fatal write error on socket transport')
                    return
                else:
                    data = data[n:]
                    if not data:
                        return
                # Not all was written; register write handler.
                self._loop._add_writer(self._sock_fd, self._write_ready)

        # Add it to the buffer.
        self._buffer.extend(data)
        self._maybe_pause_protocol()
msg343142 - (view) Author: (viocal) Date: 2019-05-22 03:34
I have fixed it
modified selector_events.py

    def write(self, data):
        if not isinstance(data, (bytes, bytearray, memoryview)):
            raise TypeError(f'data argument must be a bytes-like object, '
                            f'not {type(data).__name__!r}')
         ...
        if not self._buffer:
            # Optimization: try to send now.
            while True: #########add by viocal
                try:
                    n = self._sock.send(data)
                except (BlockingIOError, InterruptedError):
                    pass
                except Exception as exc:
                    self._fatal_error(exc, 'Fatal write error on socket transport')
                    return
                else:
                    data = data[n:]
                    if not data:
                        return
                # Not all was written; register write handler.
                if not data
                    self._loop._add_writer(self._sock_fd, self._write_ready)

        # Add it to the buffer.
        self._buffer.extend(data)
        self._maybe_pause_protocol()
msg343150 - (view) Author: (viocal) Date: 2019-05-22 07:18
test:
os centos 7.4 
memory = 512M
filedata = 1.14G

this computer-------------peer site (same computer)
server(with asyncio)------clien(socket without asyncio)
write filedata ---------> success
before fix write buffers is storage free(memory)
after fix write buffers is storage cached(memory)
History
Date User Action Args
2022-04-11 14:59:15adminsetgithub: 81162
2019-05-22 09:05:19SilentGhostsetstatus: closed
resolution: fixed -> not a bug
stage: resolved
2019-05-22 07:18:37viocalsetstatus: pending -> (no value)

messages: + msg343150

2019-05-22 03:34:37viocalsetstatus: open -> pending

messages: + msg343142

2019-05-22 03:28:26viocalsetresolution: fixed
messages: + msg343141
2019-05-22 00:42:59viocalsetmessages: + msg343136
2019-05-21 13:02:30asvetlovsetmessages: + msg343043
2019-05-21 12:58:42viocalsetresolution: not a bug -> (no value)
messages: + msg343042
2019-05-21 12:19:13asvetlovsetresolution: not a bug
messages: + msg343032
2019-05-21 12:07:46viocalsetmessages: + msg343024
2019-05-21 11:42:27asvetlovsetmessages: + msg343017
2019-05-21 11:33:37viocalsetmessages: + msg343015
2019-05-21 11:18:23asvetlovsetmessages: + msg343014
2019-05-21 11:07:21viocalsetmessages: + msg343010
2019-05-21 06:37:37asvetlovsetmessages: + msg342982
2019-05-21 02:23:49viocalcreate