Solved Converting GNU to *BSD scripts takes a lot of time

maxmoon

Member

Reaction score: 4
Messages: 32

My current problem is, that I had a lot of shell scripts in Linux doing stuff the whole day and they helped me a lot.

But I have to rewrite every single script, because they are not working anymore, which takes a lot of time. I even think it could take days or even weeks and I am not sure if it's worth it to switch to FreeBSD if there is so much to do (and it will not work on Linux anymore)?

I am just curious how people who came from Linux to FreeBSD and used a lot of shell scripts are handling this problem?

Is rewriting scripts the best way or should be a better language used, like Python to avoid problems like this and to make it compatible with both operating systems?
 
Last edited:

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 8,050
Messages: 31,634

It depends on how you've written your shell scripts. A lot of Bash scripts I've seen people create don't really need Bash and will work just fine with sh(1). And that would be the best way to make your scripts more compatible. Don't use "bashisms" and stick to plain Bourne shell.

External commands, like date(1) are mostly compatible, but as you've already found out can differ in certain details. You either work around those or do things entirely differently (there's usually dozens of ways to come up with a solution).
 

vermaden

Son of Beastie

Reaction score: 1,234
Messages: 2,803

I do not want to sound harsh but If you would write these scripts in POSIX /bin/sh in the first place you would not have that problem.

You scripts are currently (probably) full of Bashisms and Linuxisms and to make them work on anything other then Bash or Linux you need to remove then. Each one one them.

In your place I would take that chance to rework these scripts to something better with a lot of case statements like this one below:

Code:
case $( uname ) in
  (Linux)
    # paste your old stuff here
    ;;
  (FreeBSD)
    # write new code here or copy with modifications if possible
    ;;
esac
Feel free to post the Bashisms and Linuxisms so we can help you to port these into proper POSIX /bin/sh code.
 

PMc

Aspiring Daemon

Reaction score: 180
Messages: 513

Purists may frown about it, but the linux tools can be installed from ports. Or /compat/linux can be installed.
Then it should just be a matter of changing the paths where the gnu tools are.

It depends: if these shell scripts do system maintenance work and might be needed in single-user mode, then they need to be converted. But if they rather form a kind of application, then they can be given the tools they want.
 
OP
OP
M

maxmoon

Member

Reaction score: 4
Messages: 32

It depends on how you've written your shell scripts. A lot of Bash scripts I've seen people create don't really need Bash and will work just fine with sh(1). And that would be the best way to make your scripts more compatible. Don't use "bashisms" and stick to plain Bourne shell.
I didn't know what bashism even mean, but after a little search it just mean not to use bash, like you said and if I get it right that everything I do in sh is compatible with bash but not vise versa. Am I right?

External commands, like date(1) are mostly compatible, but as you've already found out can differ in certain details. You either work around those or do things entirely differently (there's usually dozens of ways to come up with a solution).
Yes, date, awk and others made a lot of problems, but my ducktape solution for now is to use coreutils and replace every awk to gawk, date to gdate, etc.

I do not want to sound harsh but If you would write these scripts in POSIX /bin/sh in the first place you would not have that problem.
Being harsh is sometimes even efficient. I've learnt all the shell scripting by searching the net and that's what happens if you just do this. I have never heard of POSIX before and why it might be something good... and this might be a problem with many people learning shell scripting in Linux. But after searching the net it just says it is a standard for shell scripting. Am I getting it right?

You scripts are currently (probably) full of Bashisms and Linuxisms and to make them work on anything other then Bash or Linux you need to remove then. Each one one them.

In your place I would take that chance to rework these scripts to something better with a lot of case statements like this one below:
...
This might be a really good solution, but for testing the scripts I would need two operating systems. Is this the right way to have a compatibility with Linux and FreeBSD? It somehow sounds easier to use Python or other languages, which works on Linux and FreeBSD the same (hopefully it is like that, because I might be wrong about Python).

And on the other hand it just sounds really redundant to learn all the awk, grep, date and coreutils twice and might be really confusing if you switch between Linux and FreeBSD often.
Or is my way of thinking wrong to use a language, which works the same on FreeBSD and Linux?

Feel free to post the Bashisms and Linuxisms so we can help you to port these into proper POSIX /bin/sh code.
I will definitely create a POSIX /bin/sh version with FreeBSD and Linux switches for some scripts I've written and publish them and I really appreciate the help of this community.

Thanks a lot for your help

--------------------

P.S.:

balanga mentioned in another thread to install coreutils to get all the gnu versions, using gawk instead of awk, etc.

You can actually install GNU date if necessary, it is part of sysutils/coreutils.

Under FreeBSD it is run a gdate()
 

ralphbsz

Daemon

Reaction score: 1,269
Messages: 2,033

The real question is this: What is your long-term goal?

Are you developing software which will continue to be maintained and enhanced, and which needs to run on both *BSD and Linux? In that case, the best way forward is to find the common subset between the two operating systems, and code for it. My personal preference is to look at the POSIX standards, and use them as a guide, using only facilities standardized in POSIX. That includes using a POSIX shell, reading the POSIX documents instead of OS-specific man pages, and putting programs into POSIX conformance mode where possible.

Or do you have software that was developed for Linux, and will in the future run on BSD, and will continue to get development and maintenance? In that case, you don't want to live forever with the Gnu/Linux-style software installed in BSD, because it will cause extra hassle and confusion (awk versus gawk, never quite remembering the syntax of which version). In that case, you should port your software to use BSD native tools.

Or is running this software on BSD a one-time deal, and you don't have to worry about further development and maintenance on BSD? In that case the best solution is to do a one-time hack, and install bash and the Linux versions of the tools.

You asked whether Python works the same. In theory, yes. A pure python program, without OS interfacing, and using exactly the same python version, will work the same. But that's not the situation in the real world. To begin with, you may have different versions installed; for example, I'm using pythion 3.6 and 3.7 simultaneously, because on some of my machines 3.7 is not yet available. Second, there will be OS dependencies in corner cases. For example in the location of files (/usr/local/etc versus /etc, or /opt versus /usr/local), or device names (are disks called /dev/adaX or /dev/sdX, and don't even get me started on USB serial ports). Or in things that can't be done in python and require outside tools; for example, I just found out that implementing arp in python is very hard, so what I'm doing is running the OS-specific arp program from python and parsing the output, and at that point you would be OS specific.

So the answer to your question is: Programming in python will reduce your OS dependence, but not completely remove it.
 

vermaden

Son of Beastie

Reaction score: 1,234
Messages: 2,803

Being harsh is sometimes even efficient. I've learnt all the shell scripting by searching the net and that's what happens if you just do this. I have never heard of POSIX before and why it might be something good... and this might be a problem with many people learning shell scripting in Linux. But after searching the net it just says it is a standard for shell scripting. Am I getting it right?
Yes, if you write scripts with POSIX /bin/sh in mind they should work everywhere, not only on FreeBSD or Linux but also on HP-UX, AIX or Solaris, it takes more time thou.

This might be a really good solution, but for testing the scripts I would need two operating systems. Is this the right way to have a compatibility with Linux and FreeBSD? It somehow sounds easier to use Python or other languages, which works on Linux and FreeBSD the same (hopefully it is like that, because I might be wrong about Python).
Its one of the ways, sometimes its better to make such case ... esac statement and sometimes (probably more times) just to write portable (POSIX compatible) code. You can use Python but You will get yourself into another problem - Python2 versus Python3. While problem does not exists on FreeBSD or Linux where Python3 has been already made default but on AIX or HP-UX most things work around Python2 still. Also porting script from Python2 to Python3 is real PITA. IMHO - to have least PITA and most usable scripts for the future - just stick to POSIX /bin/sh standard.

And on the other hand it just sounds really redundant to learn all the awk, grep, date and coreutils twice and might be really confusing if you switch between Linux and FreeBSD often.
Or is my way of thinking wrong to use a language, which works the same on FreeBSD and Linux?
Simple/standard things work the same on awk/gawk or sed/gsed. The differences start to show when You use most advanced features. You may also use gsed (short for GNU sed) on FreeBSD, same for gawk or GNU ls.

Code:
% pkg which `which gls`
/usr/local/bin/gls was installed by package coreutils-8.31

% pkg which `which gawk`
/usr/local/bin/gawk was installed by package gawk-5.0.0

% pkg which `which gsed`
/usr/local/bin/gsed was installed by package gsed-4.7

I will definitely create a POSIX /bin/sh version with FreeBSD and Linux switches for some scripts I've written and publish them and I really appreciate the help of this community.
Not sure if this will be helpful for you, but here are mine POSIX /bin/sh compatible scripts - maybe you will find some constructs or parts of them helpful in your porting process - https://github.com/vermaden/scripts/ - available here.

Thanks a lot for your help
Welcome, good luck! :)
 

userxbw

Well-Known Member

Reaction score: 29
Messages: 294

if you got bash written scripts from Linux, change your shell to bash, then , for me, for the most part anyways, aside from a few glitches I've found, (thus far), all I've had to do is change
Code:
#!/bin/bash to #!/usr/bin/env bash
which I used find -exec sed comand on my dir full of bash scripts. For the most part I've not had to make a load of changes in my bash scripts. taking into concideration I'e only been running FreeBSD for a few weeks now.

as far as that goes, it should not matter what your shell is, sh or whatever bash is in FreeBSD and if you tell it to use it then it should still work.
 
OP
OP
M

maxmoon

Member

Reaction score: 4
Messages: 32

Ok, my solution is to just use bash and sysutils/coreutils (GNU versions of all the stuff I am using in my scripts), so I can use those scripts without changing much (only putting a "g" in front of all the commands, which don't work) and not investing a lot of time "fixing" it. But I really don't want to learn two versions of coreutils and make a switch case block at any point I use them. That's just too much work to provide compatibility for scripts I publish and I don't even know if any Linux user will ever download it. I will stick with Python, which should work on many systems and if there is stuff, which is more system specific (like /dev/sda & /dev/da0), then a switch case might be a good idea. I just think the effort is much lower by using Python instead of sh and two versions of coreutils, if the software is doing non-system stuff (like using a REST interface and providing data).

Thanks a lot to all the people who participated in this discussion. It was really interesting to know what FreeBSD users/admins think about Linuxism and Bashism. It was really an eye opener, because in the Linux world I didn't even know that I do something wrong.
 
B

badbrain

Guest


Ok, my solution is to just use bash and sysutils/coreutils (GNU versions of all the stuff I am using in my scripts), so I can use those scripts without changing much (only putting a "g" in front of all the commands, which don't work) and not investing a lot of time "fixing" it. But I really don't want to learn two versions of coreutils and make a switch case block at any point I use them. That's just too much work to provide compatibility for scripts I publish and I don't even know if any Linux user will ever download it.
You could just check the platform the script is running on using uname. If it's running on Linux then set TAR=tar. If it's running on FreeBSD then set TAR=gtar (check if the gtar binary exists, if not ask the user to install GNU coreutils). Don't invoke tar or gtar directly but use something like this: ${TAR} xf test.tar.gz. I use this trick myself :)
 
OP
OP
M

maxmoon

Member

Reaction score: 4
Messages: 32

You could just check the platform the script is running on using uname. If it's running on Linux then set TAR=tar. If it's running on FreeBSD then set TAR=gtar (check if the gtar binary exists, if not ask the user to install GNU coreutils). Don't invoke tar or gtar directly but use something like this: ${TAR} xf test.tar.gz. I use this trick myself :)
This is a nice duck tape solution for private scripts, so it will be still compatible with Linux and FreeBSD. I will try to use this instead of just replacing every coreutil with g[coreutilname] and I don't have to learn two versions of them.
 

userxbw

Well-Known Member

Reaction score: 29
Messages: 294

blacdog's post #12 lead me to believe/think that "everything" (sys tools) *bsd starts with a g. as I see that may not be the case.
gtar is not installed on my Fbsd.
Code:
userx@FreeBSD64.net:~
$ which gtar
......
$ which tar
/usr/bin/tar
....
$ which awk
/usr/bin/awk
...
which gawk
/usr/local/bin/gawk
...
$ which sed
/usr/bin/sed
...
$ which gsed
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 8,050
Messages: 31,634

No, it's the exact opposite. You can install the GNU variant, and those would be installed as g* to indicate the difference between FreeBSD's own tools and GNU tools with the same name.

tar(1) is FreeBSD's own implementation, archivers/gtar installs the GNU variant. To avoid name clashes and confusion it's installed as gtar(1).
 

olli@

Well-Known Member
Developer

Reaction score: 375
Messages: 486

This is a nice duck tape solution for private scripts, so it will be still compatible with Linux and FreeBSD. I will try to use this instead of just replacing every coreutil with g[coreutilname] and I don't have to learn two versions of them.
I think there's even a better way.

Create a directory /usr/local/gbin (or similar) and populate it with symlinks that point from the “standard” name to the GNU name, for example:
Code:
awk -> /usr/local/bin/gawk
date -> /usr/local/bin/gdate
sed -> /usr/local/bin/gsed
...
Then create a small snippet of shell code, like this:
Code:
if [ $(uname) = FreeBSD ]; then
        export PATH="/usr/local/gbin:$PATH"
fi
Save that snippet as /usr/local/gbin/gcompat or similar.

Then, in your scripts, source that snippet right at the beginning. In other words, the scripts should start with these two lines:
Code:
#!/usr/bin/env bash
. /usr/local/gbin/gcompat 2>/dev/null
You do not have to make any other changes to your scripts.
When you run the scripts on Linux, nothing will change. When you run them on FreeBSD, the directory you created above will be prepended at the front of the PATH, so the symlinks inside it will override the tools in FreeBSD's standard directories. Of course, if you have the linux_base package installed, you can also use that one.

Alternatively, the gcompat snippet could create aliases (e.g. alias date=gdate) or shell functions (e.g. date() { gdate "$@"; }), both of which would override the standard commands, too. In that case you don't need the special directory. However, that won't work for certain cases when a command is not called by the shell, but from another command that doesn't know about aliases or functions, for example with find -exec or xargs.
 
B

badbrain

Guest


I think there's even a better way.

Create a directory /usr/local/gbin (or similar) and populate it with symlinks that point from the “standard” name to the GNU name, for example:
Code:
awk -> /usr/local/bin/gawk
date -> /usr/local/bin/gdate
sed -> /usr/local/bin/gsed
...
Then create a small snippet of shell code, like this:
Code:
if [ $(uname) = FreeBSD ]; then
        export PATH="/usr/local/gbin:$PATH"
fi
Save that snippet as /usr/local/gbin/gcompat or similar.

Then, in your scripts, source that snippet right at the beginning. In other words, the scripts should start with these two lines:
Code:
#!/usr/bin/env bash
. /usr/local/gbin/gcompat 2>/dev/null
I think we should test if the file /usr/local/gbin/gcompat exists before attempt to source it.
 

olli@

Well-Known Member
Developer

Reaction score: 375
Messages: 486

I think we should test if the file /usr/local/gbin/gcompat exists before attempt to source it.
That's why I redirected stderr to /dev/null. If the file doesn't exist, nothing happens (it's just ignored) and the error message is suppressed.
Of course, you can also do that in a more sophisticated way if you like. It was just a simple proof-of-concept.
 
Top