Writing a port for my app and "release asset" meaning

[JohnK]

BACKGROUND NOTE:
I have a local headless git server (a jail on my server running git) which I normally push/pull to/from.

I created a tool, and I was nudged to create a port for it, and I started exploring the porter's handbook. I believe I have a fairly good handle on what my port's makefile will needs to be, but I was pulled into a side-tangent because of a warning in the handbook: "...Please switch to release assets...".

Q1: Clarification: 'release asset' means that I should provide a .tar.gz for my port's makefile to download? If so, I was going to push a release to a github mirror of my project repo for my port's makefile to download from.

In anticipation of a yes to this question, last night I created a few scripts to "build" and "package" which I plan to use to deploy to the github mirror with a server side git hook. I am now able to build a "FreeBSD package" (aka: tar.gz of my bin's and man pages) but I am starting to get to the point where I have to make other assumptions about my planned workflow.

EG 'package' layout:

    /bin/binary
    /share/man/man#/binary.#

And NOTE: My plan was to use (git) Example #17 and using the commit hash for a version number (still haven't worked/tested this aspect out fully yet though).

Q2: What happens after I submit the port? -i.e. do I become the maintainer of the port and does that mean I would have commit privileges?

Q3: Is that repo using git? If not do I need another application and/or just provide patches?

Here is a rough automation diagram I have in mind because of that 'release asset' note:

push -> build script -> pack script -> update port makefile -> ?
                                   |
                                   +-> push release to github.

I probably should have used jenkins or something like that but I'm here now...

Any help/info appreciated! Especially because I feel a little behind the eight-ball now/because of this side tangent--but at least I have a git hook based build infrastructure now.

 

[tingo]

Q2: maintainer = yes, commit privileges: I don't think you will get that at first. But I could be wrong - I'm not a port creator nor maintainer.
(To clarify the maintainer statement; I do not think that new ports will be allowed unless someone has agreed to be the maintainer. So the point of least resistance is to ask the port submitter to be maintainer.)

 

[JohnK]

Thanks. I have no problem being the maintainer (my project(s) has zero dependencies). Not sure I want commit access but a push would be a lot easier than a email patch (I prefer automated v manual). I'd like to set the port up for source download instead of binary but this "release asset" is sort of throwing me off track at the moment. -i.e. I'm about to create a GitHub release publishing script but I really should try to focus my attention on building and testing a port makefile first (but I suppose I have to get my thoughts down on paper before I forget them).

 

[jbo@]

Q1: Clarification: 'release asset' means that I should provide a .tar.gz for my port's makefile to download? If so, I was going to push a release to a github mirror of my project repo for my port's makefile to download from.

Upstream provided distfiles are preferred if they are available. If you control upstream and you don't provide any distfiles. You can simply use USE_GITHUB.

Q2: What happens after I submit the port? -i.e. do I become the maintainer of the port and does that mean I would have commit privileges?

Yes, you'll be the maintainer, but you won't be able to commit yourself - You'll need a committer for that.
However, you'll still be in control of the actual commit as we prefer submissions of git format-patch patches which represent the full commit (including commit message etc). You'll also be listed as the author in the commit. You just won't be able to actually push the commit. You'll create a PR on bugzilla and some committer will do the work.
Commit bits are not handed out easily (took me like 10 years to get mine - but your mileage may vary).

Q3: Is that repo using git? If not do I need another application and/or just provide patches?

Just clone the official ports repo locally, make your change(s), create a commit, then create a PR at https://bugs.freebsd.org containing your patch generated via git format-patch.

Additionally, keep in mind that we (the ports committers) tend to not accept just any new port. There should be a real "use case". We accumulated too much "dead weight" in the past. If it's just a tool that you use yourself, some committers might reject it.
However, that does not mean that you're out of luck. You can always create an overlay and have your poudriere build it. That way you can still pkg install my-cool-app just without it being part of the official ports tree. You can also make that available to 3rd-parties similar to Ubuntu's PPA.

 

[JohnK]

Nice info!

Additionally, keep in mind that we (the ports committers) tend to not accept just any new port. There should be a real "use case". We accumulated too much "dead weight" in the past. If it's just a tool that you use yourself, some committers might reject it.

whoa?! good point! Followup question: How do I prove that what I have written is worthy because I assume: "Ricky said I should make a port" will not be good enough. Should I create a thread here and ask for testers and/or opinions? Should I ask a current committer to test it?

I'm not worried about my end; "make install" works for me. I'm also happy not doing a port (less work for me :]) but I thought I'd try and "help" and contribute my-SUPER-cool-app (*sigh* you got the name wrong).

For the record, I don't know what Ubuntu's PPA is (I've always used a BSD and I don't think I've ever seen the inside of an Unbuntu OS) but I'll google that if you say I should.

I'm going to get my script for uploading a binary release to GitHub done and then create a port on my local server and see where that takes me (who know's I could get that far and find out I can't figure out the port makefile).

Thank you though! Any info you can spare, I'll take.

 

[JohnK]

So FWITW I briefly looked into Unbuntu PPA; I'm not sure I see the appeal i.e., I don't see how that is any better than:

git clone ....
make
doas make install

 

[Emrion]

In my personal experience, IIRC, it took about one month before my port has been committed in the main branch. The delay is pretty correct.

If you want to avoid common mistakes, take an existing port, technically close to yours, and plagiarize it!

One thing: don't make the same mistake than I. When you want to upgrade your port, don't post in the original PR, but create a new one. I found that somewhat stupid, but that the way it goes. If you don't comply to this, there is no deadline as noone is looking the original PR.

 

[T-Aoki]

For GitHub, you can obtain the equivalent of "Release Tarball" as Release Asset if the repo is tagged for specific commit. Maybe others would be, too.

And posting info to freebsd-ports ML to inform the PR (Bug *) would be a good idea.

Some committes would want Differential Revision on Phabricator, as it is easier to comment for specific line(s) of the diff.
I'm using Web Interface introduced here, as the primary way (using devel/arcanist) causes conflict, as I have archivers/arc installed as a dependency of security/clamav.

 

[astyle]

Q1: Clarification: 'release asset' means that I should provide a .tar.gz for my port's makefile to download? If so, I was going to push a release to a github mirror of my project repo for my port's makefile to download from.

I'm going to get my script for uploading a binary release to GitHub done and then create a port on my local server and see where that takes me (who know's I could get that far and find out I can't figure out the port makefile).

In any FreeBSD port, you have a standard file called distinfo. The Porter's Handbook should explain what that contains.

So basically, yes, the name and size of the tarball that you release on your github (or anywhere else) has to match the contents of what your distinfo says it is.

Next, if you have a link for downloading that tarball by hand, then that very link goes into the Makefile, iirc. Double-check with the Porter's Handbook for that.

The goal is to basically line up the details so that the existing Ports infrastructure on your machine doesn't complain.

If the download link is incorrect, you either point the Makefile to the correct link, or actually publish the file at the place where the Makefile expects to find it. If you placed too many authentication mechanisms around the file, and Ports cannot download it, consider moving the file to a place where it is reachable by the Ports and your Makefile.

Don't be afraid to make mistakes and experiment with it. But do be aware that sometimes, you explore all options available, only to realize that you need to go back up in the process to an earlier point, and make a different decision. Yeah, that's time-consuming. But that's what it takes to understand how the Ports infrastructure works. It's a whole chain of decisions, and you need to be ready to go back up many times, and reconsider your own decisions for every single step, even all the way back up to the beginning. Good luck!

 

[jbo@]

If you want to avoid common mistakes, take an existing port, technically close to yours, and plagiarize it!

The correct approach is to follow the porter's handbook closely.
Existing ports may have outdated structures or features in them or somebody simply got lazy over the years and never called out for it.

For GitHub, you can obtain the equivalent of "Release Tarball" as Release Asset if the repo is tagged for specific commit.

That is not recommended as GitHub announced a while back that they will purge/auto-delete automatically generated assets after a while which means that the port will fail with in the fetch stage eventually.
Either provide a release distfile manually or simply consume the git repository (usually a specific tag).

 

[JohnK]

This is sounding very complicated for something that only has 2,3,or less than or equal to 5 stars--and a fork I think--on github. I'm cool with either option (either creating a port or not) but I'd rather be writing code than filling out forms for something that will never be used. Here is a link to the repo; please glance at the readme and tell me if you think I should keep going down this path (please do not think I am asking for a ton of your time reviewing code and whatnot, just a 10,000-foot overview 'yeah' or 'nah' will suffice for me). https://github.com/JohnKaul/sysconf

I'm good with creating a makefile for the github sources or a tarball (I have already automated the tarball making and plan on testing my github upload script later). I was under the impression that the warning in the handbook said tarballs are preferred though (if that's wrong, awesome. if not, awesome).

I started by looking at some existing ports and I decided to read the handbook instead. My plan was to make a port on my server and test it, if that didnt work to go back and look at other ports and keep trying but I got sidetracked with this "release asset" thing (which I think is a specifically structured tarball).

At any rate, no matter what, thank you all very much for all the info you've provided so far. This is a lot to wrap your head around if your new to making ports.

 

[jbo@]

I'm good with creating a makefile for the github sources or a tarball (I have already automated the tarball making and plan on testing my github upload script later). I was under the impression that the warning in the handbook said tarballs are preferred though (if that's wrong, awesome. if not, awesome).

If you generate your own release distfile (tarball) and upload/attach it to the GitHub release - that is definitely the preferred method.
I was specifically addressing the issue of GitHub periodically deleting automatically generated tarballs (the ones generated by GitHub).
Also, a distfile is not necessarily the same as the automatically generated tarballs to begin with. Distfiles are crafted by upstream. It's the "known good" - what upstream wants you to use in a port. These can differ from "here's just a tarball of the repository at a given commit/tag/branch" as they can differ in content (such as including assets and whatnot).

As for creating the port: Following the porter's handbook is indeed what you should be doing. And using poudriere testport to test the port.

 

[JohnK]

I wrote a few scripts to "make" my project and tarball up the bin(s) and the man pages into a "package" (release asset) upon a post-receive hook (I have not tested an "upload to github release" script yet but that is waiting in the wings). ...I will continue down that path.

I will move on to creating my port makefile and other files and test. I've never used poudriere before but if that's a requirement then so be it.

 

[Emrion]

The correct approach is to follow the porter's handbook closely.
Existing ports may have outdated structures or features in them or somebody simply got lazy over the years and never called out for it.

Follow the porter's handbook isn't enough. Not all is explained. It's the best solution if you want to avoid most of the pitfalls and they are numerous. By the way, it's just common sense. Before to play guitar, listen guitar and play what you heard.

That said, everybody is free to do what he/she wants on that matter. It was just my advise.

 

[jbo@]

Emrion I am not disagreeing but if the porter's handbook isn't explaining something, the best approach would be to let us know so we can improve it

 

[JohnK]

Not sure if this would be of any value, but me trying to make a port for a program I wrote found a few things "difficult" right away. I mean, this will make me look like an absolute moron but if it helps another person or qualifies as "helping improve", I'll take the hit.

Here's the path I took through the book so far.

1. I found the initial decision between slow and quick confusing. It would be better to have an overview of what a port is or how it's structured. I can read the section and decide if my situation fits the quick/slow but I'm lost at this point so I just take the quick route to see if I can glean the port structure and make process.

2. I get to 3.1 and get hung up on 'MASTER_SITES'. "I'll have to use github, so I assume the makefiles have a github target...<ctrl-F>" Where I got hung up on github related variables for a while.

3. Back to 3.1. "Where is the rest of the makefile? How does it know where to install files to/from, etc.? Can I use this port makefile to just call my makefile and install?"

4. 3.2.2 Ah, a file list. Find but how does this tie into the port's makefile? "...lists all the files installed by the port" No, I think I want the port's makefile to install the files listed in the port pkg-plist. This must just be a file list. ...Wait, I can put the plist in the makefile. Done!

5. 'make makesum'?! I don't have all the info on what my port's makefile contains or what a typical port should look like yet. Why are we making a checksum?

6. Decide to mockup a makefile so I can follow along from 3.1.

6a. Get hung up on "release asset" and what is actually needed by the make targets to fetch a tarball, and extract it. I still haven't written anything or even created a directory yet because the process is still a bit too vague.

6b. Start creating a post-receive hook script to make and tarball up my project. Start reading about the GitHub API to get the tarball hosted somewhere a port makefile can fetch it from.

 

[astyle]

1. I found the initial decision between slow and quick confusing. It would be better to have an overview of what a port is or how it's structured. I can read the section and decide if my situation fits the quick/slow but I'm lost at this point so I just take the quick route to see if I can glean the port structure and make process.

Ok, "Slow" vs "Quick" porting: It's really more about how good a mental handle you have on the Ports infrastructure.

"Quick" porting is aimed at people who can troubleshoot just about anything they see, and identify options to resolve the situation. Those people just need a quick-and-dirty overview of the porting process, because that will guide the troubleshooting and resolution.

"Slow" porting goes step-by-step, outlining the available options at each step, referring to manpages, etc. Yeah, time-consuming to study, hence the "Slow" tag. It describes the exact same process as the "Quick" porting, just in more detail.

2. I get to 3.1 and get hung up on 'MASTER_SITES'. "I'll have to use github, so I assume the makefiles have a github target...<ctrl-F>" Where I got hung up on github related variables for a while.

Don't skip steps in the porting process. The "Slow" porting goes into way more detail than "Quick" porting. Make sure that step 1 is actually complete before moving on to step 2. If you get errors at step 1, you're stuck until all those errors are resolved.

One thing to know about Makefiles: They are merely compilation instructions, in plain text files. The Makefile for the port definitely has the ability to call/use the makefile it finds in the upstream tarball. A few times, I had a weird adventure when the upstream's makefile had a weird text encoding that FreeBSD's sh could not take - but csh handled that fine. (This is just an example of the kind of things one needs to be able to troubleshoot and resolve if they are to be able to port stuff to FreeBSD).

 

[kpedersen]

The correct approach is to follow the porter's handbook closely.
Existing ports may have outdated structures or features in them or somebody simply got lazy over the years and never called out for it.

In the past I have tended to do similar to Emrion's suggestion and indeed, I notice loads of ports accrue "cruft" over time that simply are not relevant to new developments.

So certainly great to use as inspiration if we get stuck, but following the handbook to build it up results in something that remains a little cleaner.

 

[JohnK]

Yes, I hear you and sorry, just frustrated.

I will/am doing the handbook route but the handbook seems to be written for those looking to port the software they found in the wild. Otherwise, it would say something like:

"Want to compile your program on BSD, tap into our make system by doing: "PROG=ACME SRCS=acme.c acme-common.c ..." Because when you define the PROG var it funnels through the following targets. And, now that your program compiles cleanly, place your proj in a directory called WORK, make a file-list, a description, etc. and create these entries in a makefile. Boom!

I know what a makefile is. ...however, BSD has always been "lacking" in makefile stuff as far as I'm concerned. I've always used BSD (started with Net in 1999 when I got an old, discarded PC from the company I was interning at) but when you try to learn more about the BSD makfiles you always get something like: "the best way to learn about the makefile system is to read the comments in the makefile." Sure, but it's even far easier to get a book on makefiles and just read it, instead; the only problem is that the book you get will be for GNU not BSD make. I chalk this up in the typical "BSD is stoic" column. As we know BSD make is different. -i.e., I can define functions in GNU makefiles but in BSD make I'm stuck with targets but with BSD make I can use the -v switch ("knock, knock. Who's there? ...automation!"). I know I need to set a var and that var gets used in a target but I don't know what/where/when is the issue because our make's are so cluttered with ifs and loops they can drive you crazy trying to find something "quickly". The fact you can define functions in GNU Make allows me to shoot myself in the foot 10 times designing my own kludgy system whereas the BSD Make targets force me to try to understand someone's--who is FAR smarter than I--actual engineered system.

And that's the thing. I don't want the port makefile to call my makefile. My makefile is generic enough (not only for GNU or BSD) but I want the system makefiles to do things it needs to do because it's built into the OS (mine isn't).

I mean, look about my situation. I'm going off on tangents building scripts to compile and package binaries because I'm not sure if I should be doing it one way or the other (so the tangent of creating a hacky automation server is the path of least resistance because I know I can control the build and probably just hack up enough in the port's makefile to copy a few files from here to there and spend less time fixing errors). ...ugh!! Sorry, again, frustrated because I still don't even know if my stupid-program is even worth the effort.

 

[Emrion]

Yes, I hear you and sorry, just frustrated.

I saw your program and I don't think it's complicated to port it. it's just a matter of a C compilation with a few source files and copy some files at the right places. I don't want to insist on the method I used myself, but... It's no headache guarantee. You adapt an existent Makefile of a port (a simple c program located on github), you follow the handbook (quick porting) and test it in your own port tree (you need to create a Release on github). Once it works, you generate the git patch as instructed and you're good to post the PR.

If there are some problems with deprecated ways you may use by copying an old port, the committer will ask you to correct it and tell you how to do it (it happened to me).

Tested your program and I think you should go farther in the idea and have a more universal utility that is able to deal with XML, JSON and others (a plugin system?). No matter if you use an existing code library. Also, it lacks a function to delete a var from the file. But Rome needed more than one day to be Rome. It's a decent first version.

 

[JohnK]

A HUGE thank you for the quick test! And you reminded me of a thought when I started writing it (I was frustrated with sysrc). My thought was: what happens if I write this thing and "jail.conf" switches to JSON or "XYZ" and this tool becomes defunct. Maybe this tool isn't meant to be ported (and kept as a "GitHub-make install" type thing). Humm...plugin system (a design criteria was no external libs so I'd like to roll something up from scratch but...). And the delete idea is good as well--especially since I adopted the -= and += operations--and I started off assuming a one value situation only [1]. Thank you much though. And, again, I apologize.

But I think I'll go through the motions--even if I don't submit the PR--just so I can say that I've done it. I'm going to create a port and jot down what I think my port's makefile needs to be (to work with a "release"). I've come this far so I'm going to at least try what knowledge I've gathered (keep the popcorn ready for the impending brain-explosion).

[1]:

[test]cat test.conf                       
/* This is a test configuration file.  */

keyone="value1"
[test]../sysconf -f test.conf keyone-=$(../sysconf -f test.conf keyone)
keyone: ->
Last value for key removed. Key removed from file.
[test]

 

[astyle]

And that's the thing. I don't want the port makefile to call my makefile. My makefile is generic enough (not only for GNU or BSD) but I want the system makefiles to do things it needs to do because it's built into the OS (mine isn't).

FreeBSD's Ports infrastructure is designed so that the port's Makefile calls the upstream (your) makefile. It's kinda important to be able to work with that.

And yeah, as Emrion pointed out, Rome was not built in a day...

Existing ports are a good example to follow in the sense that they have been built successfully. For example editors/nano is a successful port where details are cleanly laid out, and you'll see that they are in compliance with the Porter's Handboook. And yeah, there's a /usr/ports/editors/nano/Makefile that calls the upstream's /usr/ports/editors/nano/Makefile/work/nano-8.2/Makefile.

I was frustrated with sysrc

See the manpage: sysrc(8). That stuff is meant to be a command-line method for turning on daemons and loading kernel modules by editing /etc/rc.conf. The utility is probably based on automating the sed/awk commands. I haven't looked at the source code, so I wouldn't know for sure. But sysrc(8) is not meant for adjusting other .conf
files, only /etc/rc.conf.

Oh, and BTW, Makefiles do make extensive use of sed and awk. If you capture the compilation process (with the script(1) utility) of any port, you'll see evidence of that. That's an important troubleshooting skill to have in your toolbox if you wanna port software to FreeBSD.

 

[JohnK]

NOTICE: The beginning of the book--at least for me--shall now be: https://docs.freebsd.org/en/books/porters-handbook/book/#slow-porting

FEEDBACK: Just in the first section it explains the process and makes things clear(er). Although, I haven't gotten terribly far in the whole process, the read is very nice so far.

 

[JohnK]

... the port's Makefile calls the upstream (your) makefile. ...

You are correct (and I see that now). Thank you.

See the manpage: sysrc(8). That stuff is meant to be a command-line method for turning on daemons and loading kernel modules by editing /etc/rc.conf. The utility is probably based on automating the sed/awk commands. I haven't looked at the source code, so I wouldn't know for sure. But sysrc(8) is not meant for adjusting other .conf files, only /etc/rc.conf.

Of course I looked at the man page. You should look at the source; it's very nice. Very large lib but it's laid out nicely. And I want to steel some of those functions for use in other things!
I use sysrc a lot in my scripts to set variables in an environment file. For example: I typically create a /opt/info/environment file filled with vars like "ZROOOT=/blah" and "HOSTNAME=/blah" (I do this because I just have to adjust this one file on my servers, and I can share the setup and whatnot scripts-i.e. I have one set of scripts instead of writing multiple versions for different servers). Look at the -f flag for sysrc.

 

[astyle]

Look at the -f flag for sysrc.

I did. It only matters if you have a separate file that overrides what's set in /etc/rc.conf by default, like /usr/local/etc/rc.conf. What do you think sysrc expects to see when you tell it to write to a different file? It expects exactly the same structure as system defaults.

I use sysrc a lot in my scripts to set variables in an environment file. For example: I typically create a /opt/info/environment file filled with vars like "ZROOOT=/blah" and "HOSTNAME=/blah" (I do this because I just have to adjust this one file on my servers, and I can share the setup and whatnot scripts-i.e. I have one set of scripts instead of writing multiple versions for different servers). Look at the -f flag for sysrc.

Don't confuse env / setenv with sysrc(8). Those are very different things. The sysrc(8) does NOT mention ENV variables. And neither does rc.conf(5). (That was a pain point in getting Wayland to work correctly on FreeBSD, BTW!)

I guess it would help to remind of the UNIX philosophy: "Do one thing, and do it well". sysrc(8) does NOT bother with extra stuff like environment variables. It only concerns itself with /etc/rc.conf. That's what it means to actually read the manpages - it means to check to see if something is there or not. And yeah, that is some effort.