Issue36981
Created on 2019-05-21 02:23 by viocal, last changed 2022-04-11 14:59 by admin. This issue is now closed.
| Messages (14) | |||
|---|---|---|---|
| msg342973 - (view) | Author: (viocal) | Date: 2019-05-21 02:23 | |
in asyncio
when filedata than free memory(hardware)
will be memory out Or killed by OS
for buf in filedata:
transport.write(buf)
#to client
I try it todo:
abort transporting to protect application be killed by OS
modified selector_events.py
def _write_ready(self):
assert self._buffer, 'Data should not be empty'
if self._conn_lost:
return
try:
n = self._sock.send(self._buffer)
except (BlockingIOError, InterruptedError):
pass
except Exception as exc:
self._loop._remove_writer(self._sock_fd)
self._buffer.clear()
self._fatal_error(exc, 'Fatal write error on socket transport')
if self._empty_waiter is not None:
self._empty_waiter.set_exception(exc)
return
else:
try:
if n:
del self._buffer[:n]
self._maybe_resume_protocol() # May append to buffer.
if not self._buffer:
self._loop._remove_writer(self._sock_fd)
if self._empty_waiter is not None:
self._empty_waiter.set_result(None)
if self._closing:
self._call_connection_lost(None)
elif self._eof:
self._sock.shutdown(socket.SHUT_WR)
except Exception as exc: #(MemoryError)
self._buffer.clear()
self._loop._remove_writer(self._sock_fd)
self._fatal_error(exc, 'Fatal write error on Selector SocketTransport write ready')
if self._empty_waiter is not None:
self._empty_waiter.set_exception(exc)
return
|
|||
| msg342982 - (view) | Author: Andrew Svetlov (asvetlov) * ![]() |
Date: 2019-05-21 06:37 | |
The correct approach is using Protocol.pause_writing() / Protocol.resume_writing() callbacks. No trivial, though. See `StreamWriter.drain()` for example of implementation. |
|||
| msg343010 - (view) | Author: (viocal) | Date: 2019-05-21 11:07 | |
I use rotocol.pause_writing() / Protocol.resume_writing() but results is no change(memory out Or killed by OS) |
|||
| msg343014 - (view) | Author: Andrew Svetlov (asvetlov) * ![]() |
Date: 2019-05-21 11:18 | |
I'm sorry but I cannot tell why you are using `pause_writing` incorrectly without looking on the code. |
|||
| msg343015 - (view) | Author: (viocal) | Date: 2019-05-21 11:33 | |
for buf in filedata:
asc.resume_writing()
asc.transport.write(buf)
asc.pause_writing()
|
|||
| msg343017 - (view) | Author: Andrew Svetlov (asvetlov) * ![]() |
Date: 2019-05-21 11:42 | |
No. It doesn't work this way. pause_writing is a protocol callback called from transport when the internal buffer is full. In reaction to this callback the producer should stop calling transport.write() and resume writing after getting `resume_writing()`. Flow control is hard. As I wrote you can take a look at asyncio streams for example how to do it. |
|||
| msg343024 - (view) | Author: (viocal) | Date: 2019-05-21 12:07 | |
thanks you but I think protocol.resume_writing() / protocol.pause_writing() is auto called by Protocol because set transport.set_write_buffer_limits(high=65536*2, low=16384*2) #default (high=65536, low=16384) |
|||
| msg343032 - (view) | Author: Andrew Svetlov (asvetlov) * ![]() |
Date: 2019-05-21 12:19 | |
No. pause_writing/resume_writing are protocol callbacks called by transport. User code should respond to these callbacks by stopping sending data to transport (transport.write()). The logic is a little complicated but it is ok for very low-level asyncio API. Convenient user-facing wrappers like asyncio streams hide this logic by providing high-level primitives that support flow-control out of the box I'm closing the issue, nothing to do here on asyncio low-level side. |
|||
| msg343042 - (view) | Author: (viocal) | Date: 2019-05-21 12:58 | |
for example the system free memory size is 512m and filedata size is 500m will transport Success but filedata than 512m will be failed |
|||
| msg343043 - (view) | Author: Andrew Svetlov (asvetlov) * ![]() |
Date: 2019-05-21 13:02 | |
Please re-read my previous comment. If you use asyncio incorrectly -- yes, you can run out of memory. The proper usage of pause_readind()/resume_reading() resolves the issue. |
|||
| msg343136 - (view) | Author: (viocal) | Date: 2019-05-22 00:42 | |
thanks again the environment: filedata1<512M filedata2>512M filedata3>1G this computer-------------peer computer server(with asyncio)------clien(socket without asyncio) memory<512M -----------memory>512M read filedata1 <--------- success read filedata2 <--------- success read filedata3 <--------- success write filedata1 ---------> success write filedata2 ---------> fail write filedata2 ---------> fail how todo (function set_write_buffer_limits) is work |
|||
| msg343141 - (view) | Author: (viocal) | Date: 2019-05-22 03:28 | |
I have fixed it
modified selector_events.py
def write(self, data):
if not isinstance(data, (bytes, bytearray, memoryview)):
raise TypeError(f'data argument must be a bytes-like object, '
f'not {type(data).__name__!r}')
...
if not self._buffer:
# Optimization: try to send now.
while True: #########add by viocal
try:
n = self._sock.send(data)
except (BlockingIOError, InterruptedError):
pass
except Exception as exc:
self._fatal_error(exc, 'Fatal write error on socket transport')
return
else:
data = data[n:]
if not data:
return
# Not all was written; register write handler.
self._loop._add_writer(self._sock_fd, self._write_ready)
# Add it to the buffer.
self._buffer.extend(data)
self._maybe_pause_protocol()
|
|||
| msg343142 - (view) | Author: (viocal) | Date: 2019-05-22 03:34 | |
I have fixed it
modified selector_events.py
def write(self, data):
if not isinstance(data, (bytes, bytearray, memoryview)):
raise TypeError(f'data argument must be a bytes-like object, '
f'not {type(data).__name__!r}')
...
if not self._buffer:
# Optimization: try to send now.
while True: #########add by viocal
try:
n = self._sock.send(data)
except (BlockingIOError, InterruptedError):
pass
except Exception as exc:
self._fatal_error(exc, 'Fatal write error on socket transport')
return
else:
data = data[n:]
if not data:
return
# Not all was written; register write handler.
if not data
self._loop._add_writer(self._sock_fd, self._write_ready)
# Add it to the buffer.
self._buffer.extend(data)
self._maybe_pause_protocol()
|
|||
| msg343150 - (view) | Author: (viocal) | Date: 2019-05-22 07:18 | |
test: os centos 7.4 memory = 512M filedata = 1.14G this computer-------------peer site (same computer) server(with asyncio)------clien(socket without asyncio) write filedata ---------> success before fix write buffers is storage free(memory) after fix write buffers is storage cached(memory) |
|||
| History | |||
|---|---|---|---|
| Date | User | Action | Args |
| 2022-04-11 14:59:15 | admin | set | github: 81162 |
| 2019-05-22 09:05:19 | SilentGhost | set | status: closed resolution: fixed -> not a bug stage: resolved |
| 2019-05-22 07:18:37 | viocal | set | status: pending -> (no value) messages: + msg343150 |
| 2019-05-22 03:34:37 | viocal | set | status: open -> pending messages: + msg343142 |
| 2019-05-22 03:28:26 | viocal | set | resolution: fixed messages: + msg343141 |
| 2019-05-22 00:42:59 | viocal | set | messages: + msg343136 |
| 2019-05-21 13:02:30 | asvetlov | set | messages: + msg343043 |
| 2019-05-21 12:58:42 | viocal | set | resolution: not a bug -> (no value) messages: + msg343042 |
| 2019-05-21 12:19:13 | asvetlov | set | resolution: not a bug messages: + msg343032 |
| 2019-05-21 12:07:46 | viocal | set | messages: + msg343024 |
| 2019-05-21 11:42:27 | asvetlov | set | messages: + msg343017 |
| 2019-05-21 11:33:37 | viocal | set | messages: + msg343015 |
| 2019-05-21 11:18:23 | asvetlov | set | messages: + msg343014 |
| 2019-05-21 11:07:21 | viocal | set | messages: + msg343010 |
| 2019-05-21 06:37:37 | asvetlov | set | messages: + msg342982 |
| 2019-05-21 02:23:49 | viocal | create | |
