Shell cat [-n] <file> does not display the entire contents of script

FreeBSD-13.2p9 , bash-5.2.21

I have a script that when edited contains more text than is displayed when that file is cat'ted. Thus:
Code:
# vim  bin/pf_block_perm.sh
   1 #!/usr/bin/env bash                                                           
   2 # pf_blockperm.sh 2022-05-06 JBB                                              
   3 # -q quiet sft    2022-05-30 JBB$                                             
   4 # Generate PF blocklist from invalid user attempts on sshpipe hosts           
   5                                                                               
   6 #   Set BLOCKPERM table file name                                             
   7 BT='/var/db/pf/pf_block_perm'                                                 
   8 #   Set Whitelist file name                                                   
   9 WT='/var/db/pf/pf_white_list'                                                 
  10 #   Poll all sshpipe hosts                                                    
  11 for N in {1..3}                                                               
  12 do                                                                            
  13   ssh -q sshpipe-$N \                                                         
  14    "touch pf_block_perm ; \                                                   
  15     grep 'Invalid user' /var/log/auth.log | rev | cut -d ' ' -f 3 | rev | \   
  16       sort -u > pf_block_perm"                                                
  17   echo "get pf_block_perm pf_block_perm.sshpipe-$N" | sftp -q -b - sshpipe-$N 
  18 done                                                                          
  19                                                                               
  20 # Only have one webmail service host active ATM 2022-05-27                    
  21 # The webmail block_perm generation script is located on the service hosts    
  22 for N in {2..2}                                                               
  23 do                                                                            
  24   ssh -q webmail-$N.internal.harte-lyne.ca \                                  
  25     "touch pf_block_perm ; \                                                  
  26      /root/bin/pf_block_perm_webmail.sh"                                      
  27   echo "get pf_block_perm pf_block_perm.webmail-$N" | \                       
  28       sftp -q -b - webmail-$N.internal.harte-lyne.ca                         
  29 done                                                                          
  30                                                                               
  31                                                                               
  32 ### merge the new blocklists with the old                                     
  33                                                                               
. . .

When I use cat -n then I see this:
Code:
# cat -n bin/pf_block_perm.sh
     1  #!/usr/bin/env bash
     2  # pf_blockperm.sh 2022-05-06 JBB
     3  # -q quiet sft    2022-05-30 JBB
     4  # Generate PF blocklist from invalid user attempts on sshpipe hosts
     5
     6  #   Set BLOCKPERM table file name
     7  BT='/var/db/pf/pf_block_perm'

    15      grep 'Invalid user' /var/log/auth.log | rev | cut -d ' ' -f 3 | rev | \
    16        sort -u > pf_block_perm"
    17    echo "get pf_block_perm pf_block_perm.sshpipe-$N" | sftp -q -b - sshpipe-$N
    18  done
    19
    20  # Only have one webmail service host active ATM 2022-05-27
    21  # The webmail block_perm generation script is located on the service hosts
    22  for N in {2..2}
    23  do
    24    ssh -q webmail-$N.internal.harte-lyne.ca \
    25      "touch pf_block_perm ; \
    26       /root/bin/pf_block_perm_webmail.sh"
    27    echo "get pf_block_perm pf_block_perm.webmail-$N" | \
    28        sftp -q -b - webmail-$N.internal.harte-lyne.ca
    29  done
    30
    31
    32  ### merge the new blocklists with the old
    33
. . .

And if I leave out the -n option then I see this:
Code:
# cat  bin/pf_block_perm.sh

    "touch pf_block_perm ; \
     /root/bin/pf_block_perm_webmail.sh"
  echo "get pf_block_perm pf_block_perm.webmail-$N" | \
      sftp -q -b - webmail-$N.internal.harte-lyne.ca
done


### merge the new blocklists with the old

Why does not cat list all the contents of the file; and why does the output from cat -n <file>differ from cat <file>?
 
What does cat -v bin/pf_block_perm.sh output?
That gives the desired result. However, the man page for cat states that -v is used to display non-printing characters. None of the portions of the file that did not display contained any non-printing characters other than the end of line which is present in the lines that did display. What is happening here?
 
Yeah, I was wondering if there might have been some non-printable escape sequences there.

If I copy/paste the code above it just prints everything:
Code:
dice@molly:~/temp % cat -n test.sh
     1  #!/usr/bin/env bash
     2  # pf_blockperm.sh 2022-05-06 JBB
     3  # -q quiet sft    2022-05-30 JBB$
     4  # Generate PF blocklist from invalid user attempts on sshpipe hosts
     5
     6  #   Set BLOCKPERM table file name
     7  BT='/var/db/pf/pf_block_perm'
     8  #   Set Whitelist file name
     9  WT='/var/db/pf/pf_white_list'
    10  #   Poll all sshpipe hosts
    11  for N in {1..3}
    12  do
    13    ssh -q sshpipe-$N \
    14     "touch pf_block_perm ; \
    15      grep 'Invalid user' /var/log/auth.log | rev | cut -d ' ' -f 3 | rev | \
    16        sort -u > pf_block_perm"
    17    echo "get pf_block_perm pf_block_perm.sshpipe-$N" | sftp -q -b - sshpipe-$N
    18  done
    19
    20  # Only have one webmail service host active ATM 2022-05-27
    21  # The webmail block_perm generation script is located on the service hosts
    22  for N in {2..2}
    23  do
    24    ssh -q webmail-$N.internal.harte-lyne.ca \
    25      "touch pf_block_perm ; \
    26       /root/bin/pf_block_perm_webmail.sh"
    27    echo "get pf_block_perm pf_block_perm.webmail-$N" | \
    28        sftp -q -b - webmail-$N.internal.harte-lyne.ca
    29  done
    30
    31
    32  ### merge the new blocklists with the old
    33
 
Well, I cannot find anything using sed -n 'l' or vim -b which I thought would show anything that did not print.
 
[…] why does the output from cat -n <file>differ from cat <file>?
Now this you can look up yourself in /usr/src/bin/cat/cat.c: If nflag or certain other flags are set, scanfiles is called with a second actual parameter of 1 (≈ “true”), otherwise the second parameter is 0 (≈ “false”). This has the effect that in the scanfiles routine the​
C:
        } else if (cooked) {
test succeeds or fails, which in turn selects either the cook_cat or raw_cat implementation. Different implementations may exhibit different behaviors. Now I can’t tell you, though, why a getc(3) loop apparently works differently than a read(2) loop. They both (should) do essentially the same as evidenced by a truss(1) trace. 🤼‍♂️
 
See
od -h bin/pf_block_perm.sh
and look for "1a" in the middle of the output.
If exists, it indicates ^Z (EOF, 0x1A) is in the middle of the file and everything trailing are usually ignored if the file is treated as a text file.

See, for example, Control_character on Wikipedia.

Pasting / quoting texts here could make control characters like EOF be ignored. Posting the output of od(1) (or hexdump(1)) could help.
 
[…] look for "1a" in the middle of the output.
If exists, it indicates ^Z (EOF, 0x1A) is in the middle of the file and everything trailing are usually ignored if the file is treated as a text file. […]
This is preposterous. 🥸 Maybe in WinDoofus world or in some other bizarre non‑operating system an inband 1A byte has significance, but this is UNIX land:​
commandoutput
Bash:
printf 'Hello\n\032World!\n' > test
nothing
Bash:
# Confirm 0x1A byte has been written.
hd test
Code:
00000000  48 65 6c 6c 6f 0a 1a 57  6f 72 6c 64 21 0a        |Hello..World!.|
0000000e
Bash:
# `raw_cat` implementation
cat test
Code:
Hello
World!
Bash:
# `cook_cat` implementation
cat -n test
Code:
     1  Hello
     2  World!
 
I dislike bash and don't want to read its codes, but is cat command in bash world an internal command (thus, not calls /bin/cat unless specified with full path to it)?
If yes, how the internal cat handle 0x1a? DOS (including Windoze and OS/2) like? Or historical Unix like?

byrnejb, what happenes if you run /bin/cat instead of simple cat?
kai-burghardt's results in comment #10 shows that /bin/cat works as historical Unix.
 
Escape codes can quickly get you to open a bitter can of worms when it comes to the terminal, what it is and what all it can be and how it can interpret input [ just set it to raw with stty raw to have a quick preview ].

Assuming you have standard terminal setup most likely scenario is that you have an escape code, non ascii code(s), that is messing up your cat command in that script. And no, "cat" is not internal to shell.

Simple example:
Code:
perl -e 'print pack "H*", "68656c6c6f1b5b324a776f726c6421"' > foo
Inspect with hd:
Code:
$ hd foo
00000000  68 65 6c 6c 6f 1b 5b 32  4a 77 6f 72 6c 64 21     |hello.[2Jworld!|
0000000f

Check with cat (hard to show here as the escape sequence clears the screen). Check with cat and extra options:
Code:
cat -net foo
     1    hello^[[2Jworld!
 
I dislike bash and don't want to read its codes, but is cat command in bash world an internal command (thus, not calls /bin/cat unless specified with full path to it)?
If yes, how the internal cat handle 0x1a? DOS (including Windoze and OS/2) like? Or historical Unix like?

byrnejb, what happenes if you run /bin/cat instead of simple cat?
kai-burghardt's results in comment #10 shows that /bin/cat works as historical Unix.
/bin/cat is part of the FreeBSD world. I think the interpretation of <32 chars for ASCII output depends on your terminal. I use color sequences and cursor movement inside log files for better readability. Vt and Xterm have no problems with it. I remember colors were different in syscons.
I believe 0x1a is just the DOS EOF character but nothing in UNIX text output. It shouldn't show anything on screen in DOS either but programs might interpret it as something anyway. Where was this a high ASCII left-arrow symbol? I have seen that somewhere...
 
why does the output from cat -n <file>differ from cat <file>?
Regarding terminal influence: I don't know for sure if this bypasses the terminal alltogether:
Rich (BB code):
# cat -n tfile | sed -E 's/^[[:space:]]+[[:digit:]]+[[:space:]]//' > tfile2
# diff tfile tfile2
#

___
Edit: one "+" too many
Rich (BB code):
# cat -n tfile | sed -E 's/^[[:space:]]+[[:digit:]]+[[:space:]]+//' > tfile2
 
If there's no proprietary (or personal) problem you could always attach problematic script here for us to have a look.
 
There is no problem with posting the code but the portion that exhibits the problem is already provided above in the vim sample. Everything after that point displays without a problem.
 
There is no problem with posting the code but the portion that exhibits the problem is already provided above in the vim sample
Attach the original, unedited, file. The copy/paste process probably removed any of the escape sequences, that's why my copy/paste prints everything just fine.
 
Pragmatic solution: fix your problem by copying _from_ this forum and replacing it in your own script. Any control sequence that were there will be gone. Unless your really want to find out what happened that's probably the quickest solution. In that case diff/cmd] will be your friend as well.
 
Attached version is ok, I'm assuming you did the same here - you copy/pasted the contents and didn't attach the original file.

From what you've described it had to be non-ascii char/sequence interpreted as control characters in terminal.
 
You might have a multi-byte UTF-8 sequence in there that is not being properly handled because of your terminal's LANG setting:

Copy-pasting out of text in some document formats can get you into trouble:
View: https://www.youtube.com/watch?v=8KO0LobHZvY
 
Back
Top