How can I replace string by using variables with special characters inside?

[Sopalajo]

Case example:

$  variable="$(ls -la)"

So the resulting variable stores some characters, like new line ones:

$ echo "$variable"
total 8
drwxr-xr-x   2 luis  wheel   512 May 27 04:15 .
drwxrwxrwt  15 root  wheel  1536 May 27 04:15 ..

When I try to replace a string with this variable:

$ echo "Replacement will be HERE after sed" | sed "s/HERE/$variable/g"
sed: 1: "s/HERE/total 8
drwxr-xr ...": unescaped newline inside substitute pattern

This is supposed to occur with any special character, like "\", "|", "*", "%"... etc.

How could I solve it, so no value inside the variable will become a problem and the replacement is done all right?

 

[Sopalajo]

My (still failed) attempt:

$ printf "$var=%q" "$variable"
=$'total 8\ndrwxr-xr-x   2 luis  wheel   512 May 27 04:15 .\ndrwxrwxrwt  15 root  wheel  1536 May 27 04:15 ..'

So, storing it in a variable:

$ printf "$var=%q" "$variable"
=$'total 8\ndrwxr-xr-x   2 luis  wheel   512 May 27 04:15 .\ndrwxrwxrwt  15 root  wheel  1536 May 27 04:15 ..'

But it doesn't work neither for the replacement:

$ echo "Replacement will be HERE after sed" | sed "s/HERE/$var2/g"
Replacement will be =$'total 8ndrwxr-xr-x   2 luis  wheel   512 May 27 04:15 .ndrwxrwxrwt  15 root  wheel  1536 May 27 04:15 ..' after sed

 

[ShelLuser]

The error message basically explains as much: "unescaped newline inside substitute pattern".

When you're working with special characters which the system should use literally then you need to escape them. By default this is done using the \ character. So if you set up an entire block of text in the way you demonstrated then that won't work because of that. As such you'd need to escape those characters somehow.

Can't really comment on what would be the best solution because I think your examples are a bit flawed. Normally you wouldn't compare a whole block of text such as the outpost of ls -la using sed. A better approach would be to process that output in a for loop and then iterate over each line and comparing that to the original.

What exactly are you trying to solve or do here though? Because I can't help wonder if you might be approaching whatever problem it is you have in a somewhat overcomplicated way.

 

[Deleted member 54719]

Might try to do your processing in perl instead of bash. perl is designed specifically for pattern matching and data extraction.

 

[Sopalajo]

I have had this problem on my shell scripts some times.
Of course, the case example on my question is a minimal case, because the original problem is a bit long to explain, and I will need to make some simplifications if I am going to give you some more details.

I will try to sumarize: I have developed a script that reads variables from a .config file. Some of them are small programs (like "ping 8.8.8.8 -c 1" to check internet, "curl checkip.dyndns.org" to get my public IP... etc).
The main script should be able to comunicate with the small programs by using some special variable, say "___Results___".

Example .config file to obtain the two first numbers of my public IP:

SmallProgram[1]: curl -s --interface hme1 checkip.dyndns.org | grep -Eo '[0-9\.]+'
SmallProgram[2]: echo "$___Results___" | cut -d "." -f 1,2

This means that, inside the code of the main program (by using arrays), I must do:

CommandToRun[2]="$( echo "${SmallProgram[2]}" | sed s/"___Results___"/"${ResultsOfOutput[1]}"/g )"

The above line means: "Replace, in the command (small program) 2 the «___Results___» substring by the (previously stored) results of the command (small program) 1".

As can be seen, it is hard to explain without showing the full script, and the script is too long to post it here.

By the way, I can workaround this in any manner, but finding the way (if possible) to make the automatic escaping of all characters in a variable could be useful in a future.

Thanks you for reading and sorry for the awkward problem.

 

[Spartrekus]

Case example:

$  variable="$(ls -la)"

So the resulting variable stores some characters, like new line ones:

$ echo "$variable"
total 8
drwxr-xr-x   2 luis  wheel   512 May 27 04:15 .
drwxrwxrwt  15 root  wheel  1536 May 27 04:15 ..

When I try to replace a string with this variable:

$ echo "Replacement will be HERE after sed" | sed "s/HERE/$variable/g"
sed: 1: "s/HERE/total 8
drwxr-xr ...": unescaped newline inside substitute pattern

This is supposed to occur with any special character, like "\", "|", "*", "%"... etc.

How could I solve it, so no value inside the variable will become a problem and the replacement is done all right?

You can have a look into the code below (the void interpreter does that in C with starting with '$').
https://github.com/spartrekus/nrpn