Solved Makefile(s) for a "non-Intel-only" port

  • Thread starter Deleted member 67440
  • Start date
D

Deleted member 67440

Guest
I am not at all familiar writing ports for FreeBSD, so I ask for help from those who do it most often.

Of course there is documentation, but it is not exactly immediate in use, even the "quick one" is not so quick.

So these are beginner's questions, but on the other hand every system has its own port mechanism, until now (thanks to other users) I get the OpenBSD package, but I would also like to do it for FreeBSD

---
What I want: compile a single zpaqfranz.cpp into two executable (same file, different names)
/usr/local/bin/zpaqfranz
/usr/local/bin/dir


Why? Because change behavior if called "dir" (... something like... Windows' dir command)
---
Final target: pkg install zpaqfranz
---

I am confused by the existence of TWO Makefiles.
The first is the ..."normal" one, for compiling with make install clean or whatever

Something like that

Code:
CC?=        cc
INSTALL?=    install
RM?=        rm
PROG=        zpaqfranz
CFLAGS+=        -march=native -Dunix
LDADD=        -static -pthread -lstdc++ -lm
BINDIR=        /usr/local/bin
BSD_INSTALL_PROGRAM?=    install -m 0555
ONLY_FOR_ARCHS=    i386 amd64

all:    build

build:    ${PROG}

install:    ${PROG}
    ${BSD_INSTALL_PROGRAM} ${PROG} ${DESTDIR}${BINDIR}
    ${BSD_INSTALL_PROGRAM} ${PROG} ${DESTDIR}${BINDIR}/dir

${PROG}:    ${OBJECTS}
    ${CC}  ${CFLAGS} zpaqfranz.cpp -o ${PROG} ${LDADD}
clean:
    ${RM} -f ${PROG}

The problem is here: ONLY_FOR_ARCHS= i386 amd64

On "non Intel" CPUs it is necessary to #define a NOJIT, with something like (just a snippet)
Code:
.if ${ARCH} == "amd64" || ( ${ARCH} == "i386" && !empty(MACHINE_CPU:Msse2) )
ZPAQFLAGS=      -msse2
.else
ZPAQFLAGS=      -DNOJIT
.endif

But I am confused by "the second" Makefile (the port one), just about this one
Code:
PORTNAME=       zpaqfranz
DISTVERSION=    55.8
CATEGORIES=     archivers
MASTER_SITES=   http://www.francocorbelli.it/zpaqfranz/

MAINTAINER=     franco@francocorbelli.com
COMMENT=        Swiss army knife for the serious backup and disaster recovery ma
nager

LICENSE=        BSD2CLAUSE MIT
LICENSE_COMB=   dual

WRKSRC=         ${WRKDIR}

.include <bsd.port.mk>

do-build: SOMETHING????

=======
Recap (as in the attached file)

1) I made a "caricasito" folder, with the "normal" Makefile and zpaqfranz.cpp (the source)
2) compress everything into .tar.gz
tar -czvf zpaqfranz-55.8.tar.gz Makefile zpaqfranz.cpp
3) upload to http://www.francocorbelli.it/zpaqfranz/zpaqfranz-55.8.tar.gz
(it is the MASTER_SITES= http://www.francocorbelli.it/zpaqfranz/ of the port Makefile)
4) so far so good, then
make makesum in the zpaqfolder/ folder with
- Makefile "the 2nd"
- pkg-descr
- pkg-plist
getting the distinfo
5) created the .shar with
tar cf zpaqfranz.shar --format shar zpaqfranz
6) finally (just to be sure)
portlint zpaqfranz
---
Now the question: how to conditionally compile cutting
ONLY_FOR_ARCHS= i386 amd64 ?

If I understand correctly, it is possible to compile the source both using the Makefile inside the .tar.gz file (the "first" Makefile) or with the port's Makefile ("the second") and, in this case, with conditionality (i.e. define the constant NOJIT if the architecture is not Intel)
But how?

I understand this is a routine porting operation, but this is my very first on FreeBSD
Thank you
 

Attachments

  • 55_8_1.zip
    1.3 MB · Views: 75
I have to admit I didn't read all of your text because I had a hard time to follow it. But at least one (probably the central?) question I can answer:

Every software package distributed as source code is expected to contain some way to build and install it, and the installation is expected to be able to install into some prefix path (often called DESTDIR). In your example, the build system used for that is make, so you have a Makefile here.

Package building systems typically work by just using the build system of the original software to build it and install it to some "stage dir", where then all the files are added to a package. FreeBSD ports use make for that purpose, so that's your "second Makefile".

The key difference is: The "first Makefile" is maintained by the upstream software. The "second Makefile" is maintained as a part of FreeBSD ports.

If at all possible, you should only change the port Makefile.
 
Lets make it simply

I wouldn't want to make as many "first" Makefiles as the upported platforms
Code:
CC?=            cc
INSTALL?=       install
(...)
.include <bsd.port.options.mk>
.if ${ARCH} == "amd64" || ( ${ARCH} == "i386" && !empty(MACHINE_CPU:Msse2) )
ZPAQARCH=      -msse2
.else
ZPAQARCH=      -DNOJIT
.endif

all:    build
(...)

${PROG}:        ${OBJECTS}
        ${CC}  ${CFLAGS} ${ZPAQARCH} zpaqfranz.cpp -o ${PROG} ${LDADD}
(...)

Note .include <bsd.port.options.mk> to switch on ARCH

The "second" Makefile (the FreeBSD port) use a rather complex things like (I am learning how to write)
Code:
(...)
do-build:
        @cd ${WRKSRC};
        ${CXX} ${CXXFLAGS} ${ZPAQARCH} -o zpaqfranz  ${LDADD}

do-install:
.for prog in zpaqfranz
        ${INSTALL_PROGRAM} ${WRKSRC}/${prog} ${STAGEDIR}${PREFIX}/bin
.endfor

So the one-line question is:
is it possible to have a "SECOND" (aka: FreeBSD port) Makefile, that compile, install etc, that ignore the "first" Makefile (if any, in the .tar.gz), and HOW?

Why?
Because I do not want to maintain 10 different Makefiles with 10 differents .tar.gz (Linux, BSD, Solaris, QNAP, vSphere, whatever)
One generic "first Makefile" (that runs on Linux too, just to be clear) into a single .tar.gz (with the source code)
In the "second Makefile" (the FreeBSD one) I can put whatever .include needed

Do you have Linux (or something)?
Do this, with THE SAME .tar.gz
Code:
wget http://www.francocorbelli.it/zpaqfranz/zpaqfranz-55.8.tar.gz
tar -xvf zpaqfranz-55.8.tar.gz
make install clean
 
If, inside the SINGLE .tar.gz, you get a Makefile with a not present BSD .include (to check CPU type) in your system => kaputt
I hope now it is clear

Obviously no one forbids to compile "by hand", in fact make is useless for my software.
On the other hand, 90% of users are used to download, extract, make install clean.
Nothing more

Final step is, of course, submitting into ports
 
Stripped cd
Code:
PORTNAME=       zpaqfranz
DISTVERSION=    55.8
CATEGORIES=     archivers
MASTER_SITES=   http://www.francocorbelli.it/zpaqfranz/

MAINTAINER=     franco@francocorbelli.com
COMMENT=        Swiss army knife for the serious backup and disaster recovery manager

LICENSE=        BSD2CLAUSE MIT
LICENSE_COMB=   dual

WRKSRC=         ${WRKDIR}
.include <bsd.port.options.mk>

.if ${ARCH} == "amd64" || ( ${ARCH} == "i386" && !empty(MACHINE_CPU:Msse2) )
ZPAQFLAGS=      -Dunix -msse2
.else
ZPAQFLAGS=      -Dunix -DNOJIT
.endif

LDADD=          -static -pthread -lstdc++ -lm

do-build:
        ${CXX} ${CXXFLAGS} ${ZPAQFLAGS} ${WRKDIR}/zpaqfranz.cpp -o ${WRKDIR}/zpaqfranz ${LDADD
}

do-install:
        ${INSTALL_PROGRAM} ${WRKDIR}/zpaqfranz ${STAGEDIR}${PREFIX}/bin
        ${INSTALL_PROGRAM} ${WRKDIR}/zpaqfranz ${STAGEDIR}${PREFIX}/bin/dir

.include <bsd.port.mk>
 
without stripping
That would be an error. FreeBSD packages should only ever contain stripped binaries.

The exception is if WITH_DEBUG is defined. In that case, INSTALL_PROGRAM will do the correct thing and not strip the binary.

In general, avoid do-build and do-install targets and instead let the ports framework call upstream's way of doing it. They are only meant as a last resort.
 
In general, avoid do-build and do-install targets and instead let the ports framework call upstream's way of doing it. They are only meant as a last resort.
If I understand right, this is just about
let the "first" Makefile be called and DO NOT use the "second" Makefile to compile-install-whatever

I do not like very much, as explained before, because I will use the "first" Makefile mainly for Linux.

I will strip DEBUG at all, and "strip-without-mercy-everything" [ancient zpaq
.if defined(WITH_DEBUG)... not really necessary. Who debug?]

The created .pkg seems to work
 
You probably want to use NOJIT option as default and enable it only as OPTION for port. Because what is going to happen, when packages are build on modern CPU (I don't know, what exact HW is used by FreeBSD package builders) and someone then do pkg install your port on older one?

Also I have vague impression, that you can have multiple implementation using intrinsics and some generic code and the better one is selected during runtime if required CPU feature/instruction(s) is available. But I am not able to find any useful resource now.
 
The JIT (better no NOJIT) compile on the fly (translate in x86 opcodes) a custom language for compression , much faster than the interpreter

I see your point but "Intel" CPUs are way more widespread

If someone runs something "strange" I assume he knows how to compile by ports

But the average "linux" / bsd user just want a prebuilded x86 binary in 90% maybe

I send a candidate submit for the ports tree

I am curious to see if something will happen, I really do not know (yet) how pkg are made (by the FreeBSD team)
 
I have narrowed down to amd64 only, this should facilitate considerably
Always assuming the port is accepted, of course
The current Makefile is
Code:
CC?=        cc
INSTALL?=    install
RM?=        rm
PROG=        zpaqfranz
BINDIR=        /usr/local/bin
BSD_INSTALL_PROGRAM?=    install -m 0555
all:    build
build:    ${PROG}
install:    ${PROG}
    ${BSD_INSTALL_PROGRAM} ${PROG} ${DESTDIR}${BINDIR}
${PROG}:    ${OBJECTS}
    ${CC}  ${CFLAGS} zpaqfranz.cpp -o ${PROG} -static -pthread -lstdc++ -lm
clean:
    ${RM} -f ${PROG}

The port's Makefile
Code:
PORTNAME=       zpaqfranz
DISTVERSION=    55.9
CATEGORIES=    archivers
MASTER_SITES=    http://www.francocorbelli.it/zpaqfranz/freebsd/

MAINTAINER=     franco@francocorbelli.com
COMMENT=        Swiss army knife for the serious backup manager, on FreeBSD amd64 only

LICENSE=        BSD2CLAUSE MIT
LICENSE_COMB=   dual

NO_WRKSUBDIR=    yes

ONLY_FOR_ARCHS= amd64

.include <bsd.port.mk>
 
Hi fcorbelli, I just committed archivers/zpaqfranz (and adding you to additional FreeBSD contributors is pending).

Just two more things about it:
  • I'll have to add a CONFLICTS_INSTALL with misc/gnuls as both ports want to install bin/dir. You might want to pick a less generic name for the second functionality in future releases if you want to avoid such conflicts upfront.
  • If you think your software profits a lot from it, you can always add an OPTION to build it "optimized", like e.g. -march=native and also using SSE2 if it is available on the build machine. This would enable users of the port to build it that way. It just can't be a default option, because packages must work on any supported machine. I could assist you with that if you want.
 
diizzy to clarify, this is just about adding a non-default option for -march=native (which was present in upstream Makefile). This of course is only helpful for people compiling and installing ports locally. There are quite some ports in the tree doing this (typically with an option called NATIVE).

As I understand CPUTYPE, it should work transparently anyways, but therefore, everything you build will target this CPU type, which might not always be what you want. So that might be the reason some ports offer optional NATIVE.

I have no idea whether it makes sense for this software at all.
 
It is not easy to make a cross-platform program with an integrated JIT, while maintaining compatibility with gcc 3.4
Since other platforms (eg Debian) also require specific limitations I fear that I will have to do many different sources and Makefiles and packages

For now, for example, I have done a sed that transforms the source into a Debian-acceptable version (perhaps, it is not yet said)
 
diizzy to clarify, this is just about adding a non-default option for -march=native (which was present in upstream Makefile). This of course is only helpful for people compiling and installing ports locally. There are quite some ports in the tree doing this (typically with an option called NATIVE).

As I understand CPUTYPE, it should work transparently anyways, but therefore, everything you build will target this CPU type, which might not always be what you want. So that might be the reason some ports offer optional NATIVE.

I have no idea whether it makes sense for this software at all.
I know, it's still a bad and broken idea.
 
Can you please explain a concrete example of why you think it is bad and broken?
Thank you
 
I know, it's still a bad and broken idea.
It is certainly not broken (as a non-default option), just like setting CPUTYPE globally it gives the user an explicit choice to build a package that only works on some machines, plus it gives the port the chance to e.g. set some additional DEFINEs for the build.

Whether it's worthwhile to have such an option is a completely different discussion. I'm not a proponent of it, but it's a very opinionated area.
 
You're relying on the compiler to figure out what your CPU is capable of, while this wasn't an issue when all CPU families shared the same capatibility you may run into issues with broken binaries as some feature sets are removed from some models. There have been many reports of this being broken and/or breaking code in various fun ways. It may also misdetect if you're in a VM/Hypervisor.
 
Can you explain with some concrete example, please?

"if you compile on X with Y the Z will happen..."

And if this can happen with my software, and exactly how?

For example I really do not understand why a VM should or could be a problem

For this kind of software, you know, every single bit must be "mastered", and I think I do pretty well
Even with simd asm code, every single bit out of terabytes must be "driven" the right way
Therefore I am very courious to learn something new

Thank you very much!
 
Back
Top