Feature request: Allow buffering all input to calculate total size before proceeding
Great, that's really helpful. So I think I see where the problem is, but I don't see why it happened or how to fix it.
In the past, PV would check whether it's in the foreground (not backgrounded with "^Z" "bg") by setting the TOSTOP terminal flag so that if it tried to write to the terminal while backgrounded, it would catch a SIGTTOU signal, and it would just keep trying to write every second or so until it no longer got SIGTTOU. This was not very robust and caused a few weird issues.
With the new version, before doing all that, PV checks whether it's in the foreground by checking that it is in the process group that its output terminal belongs to. If it's not, it suspends terminal output until that check succeeds again. It still does the TOSTOP/SIGTTOU thing to avoid race conditions (such as if it gets backgrounded just after it's done its check but just before it does the write to the terminal).
Anyway, looking at the debug logs, we can see that in the input phase, the check succeeds - PV's process group ID matches the terminal's owning process group ID:
[2024-10-09 18:50:20] (1567017) pv_in_foreground (src/pv/display.c:84): pv_in_foreground: true: our_process_group == tty_process_group
For reasons unknown, it stops being true in the output phase:
[2024-10-09 18:50:21] (1567017) pv_in_foreground (src/pv/display.c:88): pv_in_foreground: false: our_process_group=1567014, tty_process_group=3149893
I guess we could narrow it down a little by changing your command line:
timeout 5 ./pv --debug debug3.out -C -U - Makefile | { echo $$; while read -r line; do sleep 0.003; done; }
and then "grep process_group debug3.out" to see which parts the number that "echo $$" spits out matches up with, if any.
In the strace you sent this evening I can see similar behaviour. Input phase:
13:56:11.622749 getpgrp() = 1561723
13:56:11.622761 ioctl(2</dev/pts/49>, TIOCGPGRP, [1561723]) = 0
Output phase - TIOCGPGRP (find out the ID of the foreground process group in the terminal) - shows the foreground process group ID changed:
13:56:12.622983 getpgrp() = 1561723
13:56:12.622994 ioctl(2</dev/pts/49>, TIOCGPGRP, [3149893]) = 0
I'm guessing that in whichever version of the shell you're using, something odd's going on with process group handling in the pipeline. I'm a bit unsure of how to debug it though. You could try running dash, ash, ksh, or csh, and then re-running the above command within one of those shells, to see if they behave any differently. It would be weird if it was a bash bug, since I've tried this on CentOS 5 through 7, Rocky 8, and Alma 9, and that covers bash 3.2.25 to 5.1.8.
It could be something completely different, not a shell bug. I'm not too familiar with how "sessions" and "process groups" work and how they interact with the idea of a "controlling terminal". It could be that PV is missing an important job control / controlling terminal step I don't know about, not doing something it ought to, and for some reason it hasn't broken on any of my test systems yet.
If you don't find any difference in behaviour when running under other shells, you could also try redirecting stderr in the second part of the pipeline to see if that makes any difference:
timeout 5 ./pv --debug debug4.out -C -U - Makefile | { echo $$; while read -r line; do sleep 0.003; done; } 2>/dev/null
But really I'm grasping at straws there.
Great, that's really helpful. So I think I see where the problem is, but I don't see why it happened or how to fix it. In the past, PV would check whether it's in the foreground (not backgrounded with "^Z" "bg") by setting the TOSTOP terminal flag so that if it tried to write to the terminal while backgrounded, it would catch a SIGTTOU signal, and it would just keep trying to write every second or so until it no longer got SIGTTOU. This was not very robust and caused a few weird issues. With the new version, before doing all that, PV checks whether it's in the foreground by checking that it is in the process group that its output terminal belongs to. If it's not, it suspends terminal output until that check succeeds again. It still does the TOSTOP/SIGTTOU thing to avoid race conditions (such as if it gets backgrounded just after it's done its check but just before it does the write to the terminal). Anyway, looking at the debug logs, we can see that in the input phase, the check succeeds - PV's process group ID matches the terminal's owning process group ID: [2024-10-09 18:50:20] (1567017) pv_in_foreground (src/pv/display.c:84): pv_in_foreground: true: our_process_group == tty_process_group For reasons unknown, it stops being true in the output phase: [2024-10-09 18:50:21] (1567017) pv_in_foreground (src/pv/display.c:88): pv_in_foreground: false: our_process_group=1567014, tty_process_group=3149893 I guess we could narrow it down a little by changing your command line: timeout 5 ./pv --debug debug3.out -C -U - Makefile | { echo $$; while read -r line; do sleep 0.003; done; } and then "`grep process_group debug3.out`" to see which parts the number that "`echo $$`" spits out matches up with, if any. In the strace you sent this evening I can see similar behaviour. Input phase: 13:56:11.622749 getpgrp() = 1561723 13:56:11.622761 ioctl(2</dev/pts/49>, TIOCGPGRP, [1561723]) = 0 Output phase - TIOCGPGRP (find out the ID of the foreground process group in the terminal) - shows the foreground process group ID changed: 13:56:12.622983 getpgrp() = 1561723 13:56:12.622994 ioctl(2</dev/pts/49>, TIOCGPGRP, [3149893]) = 0 I'm guessing that in whichever version of the shell you're using, something odd's going on with process group handling in the pipeline. I'm a bit unsure of how to debug it though. You could try running dash, ash, ksh, or csh, and then re-running the above command within one of those shells, to see if they behave any differently. It would be weird if it was a bash bug, since I've tried this on CentOS 5 through 7, Rocky 8, and Alma 9, and that covers bash 3.2.25 to 5.1.8. It could be something completely different, not a shell bug. I'm not too familiar with how "sessions" and "process groups" work and how they interact with the idea of a "controlling terminal". It could be that PV is missing an important job control / controlling terminal step I don't know about, not doing something it ought to, and for some reason it hasn't broken on any of my test systems yet. If you don't find any difference in behaviour when running under other shells, you could also try redirecting stderr in the second part of the pipeline to see if that makes any difference: timeout 5 ./pv --debug debug4.out -C -U - Makefile | { echo $$; while read -r line; do sleep 0.003; done; } 2>/dev/null But really I'm grasping at straws there.