Solved Is it posssible to substitute a ' inside a string with sed?

[k.jacker]

I have some text files that contain ' (singleqoute) at the end of some lines, which I'd like to substitude/remove with sed.

Obviously, the following does not work (note the ' right after the word string and before the doubleqoutes)

echo "my foo string'" | sed 's/'$//g'

I tried to escape the ' and the $ and both with a backslash, without success.
sed will either complain Unmatched '''. or Illegal variable name.

For now I'm using a solution using cut and rev but editing the files in place with sed would be prefered.

Is this doable with sed?

 

[ralphbsz]

What shell are you using? The problem is not sed, the problem is that sed is not getting the quote, because the shell has eaten it.

I happen to use bash, and the following works great:

# echo "'abc'def'" | sed s/\'\$//
'abc'def

The other thing that works is to quote the whole argument to sed in double quotes (which then leaves the single quotes inside undisturbed):

echo "'abc'def'" | sed "s/'$//"

And remember, you don't have to use the slash to delimit sed's substitution expression:

echo "'abc'def'" | sed "s='$=="

Depending on your shell, your mileage may vary. When problems like this happen, you really need to study the quoting and substitution rules for your shell in detail. Very painful.

 

[Bobi B.]

Instead of quoting, you can escape special characters explicitly, as well: sed -e s/\'\$//g.

 

[olli@]

[…] Is this doable with sed?

As others have mentioned, it is really not a question of sed, but a question of proper quoting or escaping, so your shell doesn't interpret the special characters.
Apart from what the others mentioned, it is also possible to do that with “dollar-single quotes”, like this:

echo "my foo string'" | sed $'s/\x27$//'

The nice thing about it is that you can specify any ASCII code (\x27 is the single quote character, see the “hexadecimal” section of the ascii(7) manual page) and even Unicode. This kind of quoting works with any POSIX-compatible shell, including zsh, ksh and bash, and also with FreeBSD's /bin/sh (which is a bourne-compatible shell, but not POSIX-compliant).

 

[k.jacker]

What shell are you using?

This is part of a .sh script and I used sh interactiv as well for testing.

I really thought that the problem came from sed. I wasn't aware of, that sed the sed statement can be in in doubleqoutes or even without qoutes at all.

echo "'abc'def'" | sed s/\'\$//

This works in sh

echo "'abc'def'" | sed "s/'$//"

I could swear, I tried this with doubleqoutes around the sed statement yesterday and it didn't work, but it works now.
Maybe I had to many shell windows open yesterday and accidentally tried the second example in csh (which I use as my interactive shell)
Tested it now, and this works only in sh, but not csh.
Good to know.

When problems like this happen, you really need to study the quoting and substitution rules for your shell in detail. Very painful.

Yes. That's what I did a lot in the past. Aimed at getting better with that (mainly sh) and regular expressions in general.
In this regard, csh really hurt my brain much more then sh does, but it's fun. I suddenly can do so much more with the classical unix tools as well.

echo "my foo string'" | sed $'s/\x27$//'

Wow, that is very handy! Thanks as well for the tip with the ascii(7) manpage

Thanks guys for your help. I Learned something today.

 

[ralphbsz]

You say that you use csh interactively (probably tcsh). Yet you program your scripts in sh. This is quite a common situation. In general, csh makes for a bad "programming" language: for a variety of reasons, one should not write scripts in it, and I'm too lazy to rehash all the reasons here.

But then, as you noticed (the hard and painful way, BTDT), there are significant differences between sh and csh, in particular in how quoting works. And quoting and variable expansion is the #1 problem in using shells. And this is why I gave up on using (t)csh interactively: If I have to write scripts in sh, then I will also use it at the command line, so I only have to learn one set of rules, and there is perhaps a small chance that I will actually remember 10% of the rules. Old jokes: "Beware of the man who only has 1 gun, because he actually knows how to use it". Or "The man with two watches never knows what time it is."

So I settled on using bash interactively. The good news is that it is a true superset of the classic traditional V7 Unix Bourne shell (named after Mr. Bourne, who was part of the Bell Labs team that developed early Unix versions, together with Ritchie and Thompson, plus Kernighan and Pike). The POSIX shell standard is a close superset of the V7 Bourne shell too. In bash (and many other shells) you can run in V7/POSIX compatibility mode by starting your scripts with "/bin/sh" at the top. So using bash gives me (a) compatibility with V7/POSIX, meaning portability to fundamentally all systems, and (b) compatibility between interactive use and scripts, and (c) availability on fundamentally all systems, it exists on FreeBSD, Linux, MacOS, AIX, HP-UX and so on.

Exactly the same could be accomplished by using a modern Korn shell like pdksh or ksh93. Those are perhaps even better, because they are smaller and cleaner than bash (which has the Gnu/Linux problems of "second system effect" and "bells and whistles). I just settled on bash at the time due to random reasons.

 

[k.jacker]

one should not write scripts in it, and I'm too lazy to rehash all the reasons here.

Yes, I'm aware of this. I would say I did a lot of reading on that topic. Why csh is bad for programming, pros and cons of different shells, POSIX complience, portability, and so on.
I never planed on using csh for programming and never did. I didn't know about the "man with two watches or two guns" jokes, but felt like it all the years.
I've been a FreeBSD user for 25 years now (not professionally), and for the above reasons, I made several attempts to find me another/a better interactive shell but failed. There was allways something that was a dealbreaker in any shell that made me stick with tcsh (though thinking of the fact right now, that it took me 25 years to finally understand how to redirect stderr to another location as stdout I should have taken another attempt to switch).

Not beeing a pro though, I havn't spend 25 years on learning UNIX/FreeBSD or programming. There were many years I simply used it as a desktop OS.


Zsh e.g. was simply too much. Fish wasn't enough. Korn shells were very different, if I remember right, I didn't like the ksh93, but the pdksh was very good (it was this year I tried them for the second time or so). I guess I would have sticked with pdksh, if it hadn't the same chicken and egg problem you got with /bin/sh on FreeBSD. It won't read any configuration files when not invoked as a login shell, because, like for /bin/sh, you have to set ENV in /etc/profile or ~/.profile to something like ~/.shrc or ~/.kshrc for the pdksh to point to a configuration file. That doesn't work with Xorg, as .profile is never read .
To set my prompt for /bin/sh I set ENV from my ~/.tcshrc.

You make me consider giving pdksh another try.

bash (which has the Gnu/Linux problems of "second system effect" and "bells and whistles).

What do you mean by that? Is that the, too much linuxism that allways kept me from trying it.

Thanks for the historical input, that is allways a nice read.

 

[ralphbsz]

No, not too much "linuxism". It works just fine on non-Linus systems, and was actually written before Linux was even started.

It is just too complicated and has too many features. Now, for each of these features, there is someone who wanted it and found it useful, and someone who committed the time to implement it (sometimes the same person). So the features are not per se bad. But it makes it complicated to learn and hard to follow the documentation. It also tempts me to use obscure features once or twice (because I saw them in the documentation and they seem cool), and then forget them, and if I do that in scripts, those scripts become unmaintainable. For that reason, when I write scripts, I use /bin/sh (which on Linux machines is a soft-link to bash, but runs in old-fashioned POSIX-compatible mode), so I don't get tempted. The part I really don't like is when I find bash scripts written by other people, and they are unreadable because they use bizarre or obscure features.

The one bash feature I really like is "process >& file", for saving stdout and stderr at the same time. Much easier to type than "process > file 2>&1". But when writing shell scripts, the second form is just fine.

 

[k.jacker]

You have filled the gaps of what I read in the past years about that topic.
Thank you so much for your time ralphbsz , that was very revealing.