Shell Shell scrip is not storing stings when running them &

amilis

amilis

For the simplicity... If i have a shell script as woring below on a FreeBSD 13.1 (nginx, php, mariadb) with 3 vCPUs.

Code: 
#!/bin/sh

t1=$(echo "1") &&
t2=$(echo "2") &&
t3=$(echo "3") &&
t4=$(echo "4") &&

wait

echo $t1
echo $t2
echo $t3
echo $t4

It runs without any problem.

But if I use &, no input at all or sometime only t4 is printed.

Code: 
#!/bin/sh

t1=$(echo "1") &
t2=$(echo "2") &
t3=$(echo "3") &
t4=$(echo "4") &

wait

echo $t1
echo $t2
echo $t3
echo $t4

If I change to 4 (or more) vCPU, all runs OK.

Why? And how can I store tX strings with (one) & if I have less vCPUs than tX?
I have many tXs and need to store all of them in memory (wait command) to process them later in the script, put them in DB etc. It works with && but not with &, and I need to run all of them in parallel as every tX take long time.

This work’s without any problem. But can’t work with files.
Code: 
echo "1" > file1.txt &
echo "2" > file2.txt &
echo "3" > file3.txt &
echo "4" > file4.txt &
wait
 
Move & to the command, i.e.
Code: 
t1=$(echo "1" &)
t2=$( (sleep 2 && echo 2) & )
2nd one is the example if you have more complex commands to run when you define variable. You can also debug with sh -x ./script.sh.
 
elgrande said:
Guess: When running with &, a new process is forked and the caller does not know about the sub-process variables.
Click to expand...
That’s what I thinking when you say that. But stuck on how to fix it. :)

_martin said:
Move & to the command, i.e.
Code: 
t1=$(echo "1" &)
t2=$( (sleep 2 && echo 2) & )
2nd one is the example if you have more complex commands to run when you define variable. You can also debug with sh -x ./script.sh.
Click to expand...
Just tried this and some other possibilities. Did not work.. or.. they worked but ”in serial” as no & or like &&.


I’am trying to get a lot of tshark going in parallel, like this below (the filter is mush more complicated on each tX, but this simple command is not working either so.. ). Yes, I have tshark, tcpdump etc. installed.

This is not working:
Code: 
t1=$(tshark -Y 'tcp.port==80' -r test.pcap |wc -l ) &

This works!:
Code: 
t1=$(tshark -Y 'tcp.port==80' -r test.pcap |wc -l ) &&

Just put 10 vCPUs on the VM (and only 4 tX in the script), but it’s not working as I sad on top. Maybe I was mistaken that it worked with more CPUs before (tested many thinks).
 
The "&" operator is completely unrelated to the "&&" operator.

When you use "&" after a command, the command (and the capture of its stdout) is performed in a sub-shell, which is a completely new process. When that sub-shell exits, all the environment variables belonging to that sub-shell (including t1, t2,...) are lost.

You have stumbled across one of the most common banes of shell scripting. Namely invoking a sub-shell, and losing changes it makes to its own environment when that sub-shell exits. Also, different shells behave differently in this regard. Older shells tend to invoke sub-shells more often than newer shells.

If you want to capture something from a sub-shell, you have to use some sort of inter-process communication. Writing to a file is the most common method of doing this.
 
Try executing this. It illustrates how to capture and manage output from sub-shells:
Code: 
MyPid=$$
O1=/tmp/out1.$MyPid
O2=/tmp/out2.$MyPid
O3=/tmp/out3.$MyPid
O4=/tmp/out4.$MyPid

trap 'rm -f $O1 $O2 $O3 $O4; trap "" 0; exit 0' 0 1 2 3 15

{ echo "1"; sleep 5; } >$O1 & Pid1=$!
{ echo "2"; sleep 1; } >$O2 & Pid2=$!
{ echo "3"; sleep 15; } >$O3 & Pid3=$!
{ echo "4"; sleep 1; } >$O4 & Pid4=$!

date
wait $Pid1; date; cat $O1
wait $Pid2; date; cat $O2
wait $Pid3; date; cat $O3
wait $Pid4; date; cat $O4

exit 0
 
Ok.. I need to use files or in you example Pid.
I have used files for years now. tshark > file. And then tail/cat the files to strings and then use all them inside the shell(s).

I tried to be clever and do some more effective workflow and not using files anymore... but.. well, you can’t get everything. :)

Thanks!
 
Shell scripting is programming of a kind. Usually, it is directly derived from interactive shell use "as one sits behind the terminal at the command line"; it is not designed as a fully-fledged programming language. Because of this it carries the burden of that interactivity when used for scripting where you have to be constantly aware of this interactivity aspect. Contrast that with normal programming languages where programming is its main goal but where you have to pay extra attention when incorporating command line instructions.

When scripting plays a significant role in your system you have of course the man pages but, a good overview and usage of its idioms and constructs such as "&" and "&&", sub-shells and redirection is useful. "&" is an operator that relates to command control at the command line (foreground execution versus background execution); "&&" can of course also be used at the command line but is more likely to be used extensively in scripting as it is more like the normal "logical and" operator used in normal programming languages. Perhaps this can help you further with your scripting: Sh - the POSIX Shell.
 
  • Like
Reactions: mer
I use shell as is background 24/7 stuff running and I use tshark, direct input in sql etc. in it.
php in frontend. I tried to be clever and make strings instead of printing to file and then make a string of if and save a minute or so (and unnecessary writings / reads). But i didn't get that “.. >> file &” worked and not “string=$(..) &”.

I will check the link. Seems that I can get some use of it. :)
 
amilis said:
I tried to be clever and ...
Click to expand...
If a problem is complex enough that it requires "clever" solutions to be solved in shell, then you should consider using different paradigm, like different programming languages. While writing long and complex scripts in shell languages (and I include things like DCL and Clist in there), they are all hamstrung by the fact that they are mostly intended for interactive use. There are other good programming languages (for example Perl, Python, Rexx, ...) that are better suited for solving complex problems.
 
Of corse, there is many other programming languages and I looking at Python for other stuff. But.. if I give up I will never learn to solve the problem in sh (but learn it in other languages..). Another thing is that I want a minimal installation on the server.

I went back some years and took back the idea there I solved this problem (just for now).. I have a main script and start other script inside that one (that have problem to run in parallell, like the above thing) and insert the results in the main script. Ugly, but it works.. Looking at bash to solve some above problems, but it’s not working fully jet.
 
It bothered me I gave you a bad advice. I remembered I once did something similar to what you need to one of my customers. I thought I did it in sh. I pulled the backup from 2012 and found out I did that in perl. It's been some time I did something meaningful in perl (if ever ;) ).

I can't share the actual script but for the demonstration purposes I recreated this:
Code: 
#!/usr/bin/env perl

use warnings;
use strict;
use threads;
use threads::shared;

our @thread_list = ();
our %logs : shared;

my $i;
my $workers = 5;

for($i=0; $i < $workers; $i++) {
        my @a : shared;
        $logs{$i} = \@a;
        push @thread_list, threads->new(\&worker_routine, $i);
}

# wait for threads
$_->join() foreach (@thread_list);

# collect data
for($i=0; $i < $workers; $i++) {
        print "log for jobid ".$i."\n";
        print foreach(@{$logs{"$i"}});
}

# $_[0] jobid
sub worker_routine {
        print "debug: worker_routine: jobid: ".$_[0]. "\n";
        my $jobid = $_[0];

        my $cmd = "sleep 1 && /usr/bin/id ".$jobid;

        if ((open (CMD, $cmd. "|")) == 0) {
                print "debug: worker_routine: ".$jobid.": unable to execute ".$cmd."\n";
                die("FATAL ERROR");
        }
        @{$logs{$jobid}} = $_ while(<CMD>);
        close (CMD);
}
It was originally meant for HP-UX but perl is perl most of the times. :)

Here I execute sleep and id command. Sleep here is just to delay the command on purpose. These 5 workers collect the logs in 1s, so all is done in parallel. Once done I show the output of the command. I'm addressing arrays in hash, jobid is key to them.
 
  • Like
Reactions: _al
As Erichans says "&&" is logical and. if A && B then...
But typically whitespace/end of line terminates the command in the script if I'm recalling correctly, so what is effect of the && in the original? I'm thinking it behaves like t1=$(echo "1").
 
The shell ignores new lines between list operators (e.g. "&&").
The original can be rewritten, without any change to the logic as:
Code: 
t1=$(echo "1") && t2=$(echo "2") && t3=$(echo "3") && t4=$(echo "4") && wait
So the command substitutions between the "&&" list operators proceed, in sequence, until one fails (i.e. exits with non-zero status).
If a failure occurs, the wait(1) will not be executed, which is likely to be a logic error.
 
  • Like
Reactions: mer
You must log in or register to reply here.
Back
Top