Shell [Solved] What is causing this bash/jq behaviour?

Jim L.

Jim L.

I don't how to explain the output from the MRE below. I (believe I) understand that Bash is killing the left-hand Process 1 of the pipe line because the right-hand Process 2 has exited, and thus there is no one to read the standard out from Process 1.

What I don't understand is how the bash script that is Process 1 can detect that jq has failed, but then report that the Result Code returned is 0.

Code: 
$ cat foo.sh
#!/usr/bin/env bash

json='{
 "members": [
  "element 1",
  "element 2",
  "element 3"
 ]
}'

JQ="jq -Me" # M means Monochrome (no coloring), -e means fail on empty or null value retrieval

if ! $JQ '.["members"][]' <<< "${json}"
then
  rc=$?
  printf >&2 'jq failed to enumerate members[], rc="%s"\n' "$rc"
fi
$ ./foo.sh
"element 1"
"element 2"
"element 3"
$ ./foo.sh | false
jq failed to enumerate members[], rc="0"
$ ./foo.sh | cat -2
cat: illegal option -- 2
usage: cat [-belnstuv] [file ...]
jq failed to enumerate members[], rc="0"
 
i think $? is overwritten by if which is succesful

do
$JQ '.["members"][]' &lt;&lt;&lt; "${json}"
rc=$?
if [ $rc -ne 0 ]
then
echo "failed with $rc"
fi
 
Thanks, covacat. Yes, obviously your example will never report a failure with rc == 0. But my question is less about fixing and more about understanding.

I'm specifically using bash syntax, and I'm taking as given that the $JQ statement is failing. I'm also taken as given that the result code of the statement between if and then is in actuality the result code that is stored in variable rc by the command immediately following the then. Is one or the other of those statements incorrect? Is there a spot in the bash(1) man page I should read?

My question is, why is my if condition (the $JQ statement) failing, but yet the result code returned from the $JQ statement is 0. My (possibly mistaken) understanding is that in my example, because there are no commands performed between the execution of $JQ and the setting of variable rc, that the "rc=$?" will capture the result code of jq. In other words, the then clause doesn't alter the value of $? Is that incorrect? And for that matter, I would also expect to have the result code of the if condition available at the first line of the else clause. To state it another way, using if to test the success/failure of $JQ's result code $? should not change the value of $? that is available immediately following the if at the top of either the then or the else clause.

For quite some time, I've been using the bash idiom:

Code: 
if stdout="$(my-command)"
then
  echo success: "$stdout"
else
  rc=$?
  echo result code $rc
fi

This is the first time I can recall seeing unexpected behaviour.
 
Hmmm, a possible light bulb has gone off. You hinted that $? is overwritten by a successful if. So I inverted the boolean condition, and ran two tests such that one should fail and the other should succeed.

Code: 
$ cat foo.sh
#!/usr/bin/env bash

json='{
 "members": [
  "element 1",
  "element 2",
  "element 3"
 ]
}'

JQ="jq -Me" # M means Monochrome (no coloring), -e means fail on empty or null value retrieval

rc=255

if $JQ '.["members"][]' <<< "${json}"
then
  rc=$?
  echo success, rc=$rc
else
  rc=$?
  echo failure, rc=$rc
fi

rc=255

if $JQ '.["embers"][]' <<< "${json}"
then
  rc=$?
  echo success, rc=$rc
else
  rc=$?
  echo failure, rc=$rc
fi
$ ./foo.sh | cat
"element 1"
"element 2"
"element 3"
success, rc=0
jq: error (at <stdin>:7): Cannot iterate over null (null)
failure, rc=5
$ ./foo.sh | true
$ ./foo.sh | cat -2
cat: illegal option -- 2
usage: cat [-belnstuv] [file ...]

It looks like the problem my misunderstanding indeed goes away when the negation of the if condition is eliminated.

Thanks for the pointer!
 
Looks to me that it is pipe behaviour wrt std and error:
Code: 
$ ps -p $$
  PID TT  STAT    TIME COMMAND
62761  2  S    0:00.05 bash
$ date | false; echo $?
1
$ ddate | false; echo $?
bash: ddate: command not found
1
$ ddate 2> /dev/null | false; echo $?
1
$
 
Erichans Just to clarify whose result code I'm testing, the structure is closer to:

Code: 
(date; echo >&2 $?) | false
141

After improving my code, I can see that JQ was actually returning rc=141 (i.e., SIGPIPE), but my improper code was losing that value.

Thanks, Everyone! Marking as solved.
 
Thanks, all. To summarize:

Code: 
$ cat foo3.sh  
#!/usr/bin/env bash

condition() {
  date
}

if condition
then
  rc=$?
  echo take 1, \"then\" case: $rc >&2
else
  rc=$?
  echo take 1, \"else\" case: $rc >&2
fi

if ! condition
then
  rc=$?
  echo take 2, \"then\" case: $rc >&2
else
  rc=$?
  echo take 2, \"else\" case: $rc >&2
fi

$ ./foo3.sh
Mon Dec  8 15:52:13 PST 2025
take 1, "then" case: 0
Mon Dec  8 15:52:13 PST 2025
take 2, "else" case: 1
$ ./foo3.sh | false
take 1, "else" case: 141
take 2, "then" case: 0

Here we see that the "Take 1" code correctly returns rc=0 when condition succeeds, and rc=141 when it fails. That's the correct result.

Once we introduce boolean negation of condition, the "Take 2" code breaks down, because instead of returning the rc of condition, it tests the rc of ~ condition, and so $? in Take 2 becomes a test of whether the mathematical (boolean) evaluation succeeds or not. It either does succeed, or does not succeed, so the only possible results in Take 2 are 0 or 1.

This arose for me because sometimes in a validation loop, if an array element is found to be valid, there is nothing to be done except move on to the next. So I often test for the invalid case, and take action; and then ignore the case where the data is valid because no action needs to be taken. For example:

Code: 
#!/usr/bin/env bash

valid_data() {

# return 0 if $1 is valid, or an error code > 0 if not valid }

# for testing, we'll assume that the empty string is valid.
# any other string has an error equal to the length of the string.

  return ${#1}

}

echo Wrong way:

for value in "" jim paul frank
do

  if ! valid_data "$value"
  then
    rc=$?
    printf 'invalid: "%s", rc=%s\n' "$value" $rc
  fi

done

echo --

echo Right way:

for value in "" jim paul frank
do

  if valid_data "$value"
  then
    : # nothing to do for valid values
  else
    rc=$?
    printf 'invalid: "%s", rc=%s\n' "$value" $rc
  fi

done

Output:

Code: 
Wrong way:
invalid: "jim", rc=0
invalid: "paul", rc=0
invalid: "frank", rc=0
--
Right way:
invalid: "jim", rc=3
invalid: "paul", rc=4
invalid: "frank", rc=5

Both algorithms can tell which elements are valid or invalid, but only the second algorithm preserves the error code telling why the data is invalid.
 
You must log in or register to reply here.
Back
Top