Add basic byte size estimation for batches by TheNeuralBit · Pull Request #17771 · apache/beam
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems very naive, but I'm not sure of another way to keep this consistent with the element-wise behaviour, which is tracking the distribution of element byte size estimates. Definitely open to suggestions here.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Look at adding an update_n to distribution counter
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added Counter.update_n to address this.
R: @robertwb PTAL, I've addressed the comments as we discussed offline
TheNeuralBit
changed the title
[BEAM-14408] Add basic byte size estimation for batches
Add basic byte size estimation for batches
Run Portable_Python PreCommit
Run PythonDocker PreCommit
Run Python_PVR_Flink PreCommit
Run Python_PVR_Flink PreCommit
1 similar comment
Run Python_PVR_Flink PreCommit
bullet03 pushed a commit to akvelon/beam that referenced this pull request
Jun 20, 2022
* BEAM-14408: Add Counter.update_n * BEAM-14408: Add basic byte size estimation for batches
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters