| View previous topic :: View next topic |
| Author |
Message |
guyuming
Apprentice
Joined: 19 Nov 2020
Posts: 242
|
 Posted: Thu Jul 18, 2024 1:41 am Post subject: tee not working as expected |
 |
|
with
| Code: | | locate .ebuild|grep coreutil|tee |
i can see the output of | Code: | | locate .ebuild|grep coreutil |
.
but with
| Code: | | locate .ebuild|grep coreutil | tee |xargs ls -lt |
i can only see the output of , I expect to see both this and the former part of the code pipeline
and if i write a simple script as follows:
| Code: | cat test_tee.sh
#!/bin/bash
while read line; do
echo $line | cut -n 1
done |
and run
| Code: | | locate .ebuild|grep coreutil | tee |test_tee.sh |
i won't see the .ebuild file name found as expected. |
|
| Back to top |
|
 |
Hu
Administrator
Joined: 06 Mar 2007
Posts: 22649
|
| Posted: Thu Jul 18, 2024 2:09 am Post subject: |
|
|
| Why would you see both? tee is writing to stdout (a pipe) and all zero files you named. A pipeline with | tee | as one of its components is a Useless Use of Tee. If you want tee to write to your terminal, you need to tell it that. |
|
| Back to top |
|
|
sublogic
Apprentice
Joined: 21 Mar 2022
Posts: 269
Location: Pennsylvania, USA
|
| Posted: Thu Jul 18, 2024 10:43 pm Post subject: |
|
|
| Hu wrote: | | If you want tee to write to your terminal, you need to tell it that. |
as in: | Code: | | locate .ebuild|grep coreutil | tee /dev/tty |xargs ls -lt |
But you'll get interleaved output from the grep and the xargs. It could be garbled beyond comprehension --although with line buffering you should at least get complete lines. For this example it's not too bad. Try it and see. |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 22649
|
| Posted: Thu Jul 18, 2024 10:57 pm Post subject: |
|
|
Although it would produce slightly different results, OP might be satisfied with removing tee entirely and instead asking xargs for some extra output: | Code: | | locate -0 '*coreutil*.ebuild' | xargs -0rt ls -lt |
Use of stdbuf could force tee to be line-buffered, but there would still be some risk of interleaving its output with xargs. We would need a description of OP's goal to evaluate other options. |
|
| Back to top |
|
|
guyuming
Apprentice
Joined: 19 Nov 2020
Posts: 242
|
| Posted: Thu Jul 18, 2024 11:18 pm Post subject: |
|
|
Thanks sublogic and Hu, especially for the /dev/tty. I tried |tee >1| but failed.
As to why i need this, the "locate grep ls" code i used above is just a simplified example.
The actual scenario is: I have a task to improve the trainning dataset for zinnia Chinese character handwrite recognition.
I use
| Code: | | grep -nH 七 handwriting-zh_Base_Simple_CN.s |
to query the trainning data , it return something like:
| Code: | handwriting-zh_Base_Simple_CN.s:297:(character (value 七)(width 1000)(height 1000)(strokes ((85 508)(895 338))((465 120)(483 802)(860 800)(905 667))))
handwriting-zh_Base_Simple_CN.s:298:(character (value 七)(width 900)(height 900)(strokes ((187 414)(511 362))((337 206)(344 575)(573 594)(574 540))))
...
|
And i use a program (fcitx-handwriting) to generate picture for the trainning data returned, this program output picture filenames
| Code: | grep -nH 七 handwriting-zh_Base_Simple_CN.s|fcitx-handwriting -p
/home/guyuming/handwriting-zh_CN_Community/七_3851964929.png
/home/guyuming/handwriting-zh_CN_Community/七_2175465799.png
... |
but i need a one-to-one correspondence between the trainning data line and the picture file, now i can use:
| Code: | | grep -nH 七 handwriting-zh_Base_Simple_CN.s | tee /dev/tty | fcitx-handwriting -p |
although, i still need a little manually counting to establish the one-to-one mapping, especially when the output list is long
before i know this tee /dev/tty skill, i modified the fcitx-handwriting program to print both the inputting trainning data and output filename in a line.
https://gitee.com/guyuming76/handwriting-doc/commit/6ee45dedf6c3c0fee58113e8e576c8d715b4ca59 |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 22649
|
| Posted: Fri Jul 19, 2024 12:20 am Post subject: |
|
|
| That gitee link appears to be login-restricted, so we cannot see your change. |
|
| Back to top |
|
|
guyuming
Apprentice
Joined: 19 Nov 2020
Posts: 242
|
| Posted: Fri Jul 19, 2024 12:46 am Post subject: |
|
|
| Hu wrote: | | That gitee link appears to be login-restricted, so we cannot see your change. |
I had set my gitee repository to public, but i see no setting to anonymous. you can login gitee with github or gitlab account through OAuth i think.
gitee provide the following benefits for me:
1. more stable to visit from China than github
2. It can render picture files in commit. The link i posted above is to show some screenshots.
3. It still support password login. I have to copy the long AcessCode stored somewhere to push to github.
If you still have any difficulty visiting the repository from outside China, please let me know and i will push a mirror to github.
Last edited by guyuming on Fri Jul 19, 2024 3:12 am; edited 1 time in total |
|
| Back to top |
|
|
guyuming
Apprentice
Joined: 19 Nov 2020
Posts: 242
|
| Posted: Fri Jul 19, 2024 1:00 am Post subject: |
|
|
| Quote: | | But you'll get interleaved output from the grep and the xargs. It could be garbled beyond comprehension --although with line buffering you should at least get complete lines. For this example it's not too bad. Try it and see. |
Hi, sublogic, in my test, all grep output precede all xargs output. I guess that my buffer is large enough.
I see in https://www.man7.org/linux/man-pages/man7/pipe.7.html that write to pipe can be blocked if the buffer is full.
but how can i set the buffer to just one line instead of a size or set the pipe to a blocked pipe , to see the interleaving you mentioned.
it would be great if it is one line of trainning data followed by one line of generated picture file name.
The pipeline is implemented in bash as far as i know. |
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20485
|
| Posted: Fri Jul 19, 2024 2:07 am Post subject: |
|
|
| guyuming wrote: | | I had set my gitee repository to public, but i see no setting to anonymous. you can login gitee with github or gitlab account through OAuth i think. |
Be aware that not everyone has or wants an account with those organizations, so it will limit who can offer help.
_________________
Quis separabit? Quo animo? |
|
| Back to top |
|
|
sublogic
Apprentice
Joined: 21 Mar 2022
Posts: 269
Location: Pennsylvania, USA
|
| Posted: Sat Jul 20, 2024 1:20 am Post subject: |
|
|
| guyuming wrote: | | before i know this tee /dev/tty skill, i modified the fcitx-handwriting program to print both the inputting trainning data and output filename in a line. |
If you control that program, that is by far the best solution. Dispense with tee altogether.
See the setvbuf(3) man page. Because grep is writing to a pipe, it vomits output whenever its buffer is full. The tee does the same thing on its output pipe, but what it writes to /dev/tty, a terminal, is line buffered so at least it comes out in blocks of complete lines. The final program fcitx-handwriting (unmodified) also vomits output when its buffer is full, but also as blocks of complete lines because its stdout is a terminal. I don't know of any way to control how the two outputs interleave. |
|
| Back to top |
|
|
guyuming
Apprentice
Joined: 19 Nov 2020
Posts: 242
|
| Posted: Thu Jul 25, 2024 2:52 am Post subject: |
|
|
@sublogic,
Yes, modifying the output of fcitx-handwriting is the best solution i can find now.
What i had thought was to block the first program in the pipe until ALL programs in the pipe have processed the first line of data and written their output.
But i should have thought about that the first program will unblock once the second program have finished reading. I have not tested this, but i think it should work this way. |
|
| Back to top |
|
|
|
Other Things Gentoo
