Ports transitioned to git.

dR3b

Member

Reaction score: 7
Messages: 40

Good question, possibly a bug? I tried to ping lwhsu@ about it…
That would be nice. This would also solve my problems with "poudriere".
Code:
poudriere ports -c -B 2021Q1 -m git+https -p 2021Q1
 

Zirias

Daemon

Reaction score: 1,406
Messages: 2,457

Any reason not to use the official repository? Is this "hardwired" to github in poudriere?
 

dR3b

Member

Reaction score: 7
Messages: 40

Any reason not to use the official repository? Is this "hardwired" to github in poudriere?
Yes it is: /usr/local/share/poudriere/common.sh
Code:
7581 : ${SVN_HOST="svn.freebsd.org"}
7582 : ${GIT_BASEURL="github.com/freebsd/freebsd.git"}
7583 : ${GIT_PORTSURL="github.com/freebsd/freebsd-ports.git"}
7584 : ${FREEBSD_HOST="https://download.FreeBSD.org"}
 

Trihexagonal

Son of Beastie

Reaction score: 2,299
Messages: 2,866

Sorry, but that's totally wrong. You do realize that it took FreeBSD 2 or so years before they came to this decision and carried out the change? See 3 years ago I wrote this guide on Git. Back then it was the next new cool thing but after 3 more years?
I was going to recommend someone participating in the thread make a Tutorial out of their efforts to save answering the same question over and over.

And since you have, saved me the work of figuring it out myself.


Ports good. Change bad. Like ports. Love FreeBSD. Make change. Dung flung. 🐒

Adapting to change and making it the new normal survival of the fittest.

Natures Way... I'm such a fan of it.
 

grahamperrin

Daemon

Reaction score: 519
Messages: 1,691

Discussions of gitup
… I suggest you take this subject to its own topic. …

👍

For those of you who are unaware, there's a gitup-specific topic (begun by the developer):

 

dR3b

Member

Reaction score: 7
Messages: 40

Currently this is my "workaround":
Code:
poudriere ports -c -B 2021Q2 -m git+https -p 2021Q1 -U https://git.freebsd.org/ports.git
 

Zirias

Daemon

Reaction score: 1,406
Messages: 2,457

Hm, for ports, I just always use the null method with poudriere. I don't see any drawback updating directly with git pull ;)
 

chrcol

Well-Known Member

Reaction score: 51
Messages: 479

I didn't have a problem with Ruby but the constant battling with a corrupt database made me seriously question my sanity. I was so happy when portmaster was released. Besides having no dependencies it worked much better than portupgrade ever did.
I prefer this doesnt become another what ports tool to use thread as lately on here there seems to be a campaign to get people to move over to the newer tools, but I will post a summary of why its my preference, although I dont use it on every server.

Essentially it boils down to I prefer how it deals with specific situations, the nice summary when its done, the automatic restore of old version of port if install fails. I do get however it is getting more and more dated and some do not like how it works.

I found a nasty looping problem with portmaster, where it can be made to go in a never ending loop with dependencies, there is a problem report on the freebsd website and a developer has taken the problem on, he is refactoring portmaster, but decided to start again so no ETA on that.

I personally dont want to have a duplicated port build system on my server which is why I am not using poudriere or synth, its not so much about workflow, I can change it to adjust to new tools, but rather just my preference in how the tools work.

My opinion on the index problem is likely an outspoken one, the way I see it, if someone changes the way ports work aka flavor's, its their responsibility to make sure all parts of the ports tree function correctly including the index feature. But I have already found a workaround for portupgrade on git lite, although over time as more and more ports switch to flavor's I may decide the work applying these workarounds to more and more ports isnt worth it, I am assuming at this point index wont be fixed as I expect there is a popular opinion of wanting to retire tools like portupgrade and keeping index broken is a way of doing that.
 

omarsidd

New Member

Reaction score: 5
Messages: 17

Would like to see more importance given to documenting (or even just announcing) major changes like this.
Not as an afterthought once a few dozen comments have piled up here. But as a common sense part of such transitions.

Eg:
* The FreeBSD handbook should have been updated immediately as the reference of record.
* The ports page (default handbook version) had no mentions of git when I checked earlier. There are still versions coming up in search (yes, on the docs.freebsd.org site) that don't have the current info.
* Check those top search results for likely strings like "updating freebsd ports" and make sure those resources are accurate?
* The UPDATING or README files in the subversion repo for ports don't have anything saying "this is now static / unsupported". Shouldn't that have been in the last commit?
* The front page news on the freebsd website has no mention. Ports affects a lot more people than, say, the newest RC being out. Spread awareness.
* The project twitter feed has no recent mentions.
* Perhaps a feedback link on the git/ports wiki to let users say "hey you missed these spots"?
* Based on other comments here, test common tools, not just developer favorites?

Yes, yay for progress. But it's 2021, updating matching resources should be a given, not an afterthought.
A newcomer to the community this week would certainly have been confused, and ports are not optional for most use cases.
 

grahamperrin

Daemon

Reaction score: 519
Messages: 1,691

Food for thought:
  1. the primary FreeBSD News page – <https://www.freebsd.org/news/>
  2. the subdirectory for news flashes – <https://www.freebsd.org/news/newsflash/>
How many of you were aware of the primary page?

The About sidebar at the FreeBSD site includes a link to News that misleads to <https://www.freebsd.org/news/newsflash/> with no way back to the News page.

Much of what's flashed is not truly news flash-worthy. Critically:
  • if every item of news is treated (or mistreated) as flashy, it can become difficult for an average reader to discern, at a glance, things that are of special or critical importance.
Would like to see more importance given to documenting (or even just announcing) major changes like this.

Warner Losh's FreeBSD mini-Git Primer (FreeBSD Journal, November/December 2020) began (emphasis: mine):

❝The FreeBSD project has begun its transition from Subversion to Git. This move has been over a year in the planning and represents the next step in FreeBSD’s continuing efforts to improve its workflow. …❞

Looking back a year or so, we'll probably find relevant announcements, however some of them might be nested in quarterly reports or away in mailing lists. Not treated as directly newsworthy (or news flash-worthy) at the time.
I'll probably edit this post to include some of what was relevant.

Not as an afterthought once a few dozen comments have piled up here. But as a common sense part of such transitions.

Eg:
* The FreeBSD handbook should have been updated immediately as the reference of record.
* The ports page (default handbook version) had no mentions of git when I checked earlier.

Please, how much earlier? <https://forums.FreeBSD.org/threads/ports-transitioned-to-git.79598/post-504060> around six days ago: "… At a glance, the current edition of the Handbook is similarly non-prepared for completion of the transition. …".

<https://docs.freebsd.org/en/books/handbook/ports/#ports-using-git-method> does mention Git.

<https://cgit.freebsd.org/doc/log/?qt=grep&q=Git> three days ago, Handbook: adjust for the various git migrations.

Your most recent view might have been an outdated edition of the Handbook, or prior to the adjustment.


There are still versions coming up in search (yes, on the docs.freebsd.org site) that don't have the current info.
* Check those top search results for likely strings like "updating freebsd ports" and make sure those resources are accurate?

<https://forums.FreeBSD.org/threads/ports-transitioned-to-git.79598/post-504067> "… In IRC, someone mentioned work to improve the situation. I assume that redirects will be involved."

* The UPDATING or README files in the subversion repo for ports don't have anything saying "this is now static / unsupported". Shouldn't that have been in the last commit?

Both <https://cgit.freebsd.org/ports/commit/?id=4010f7bbc03638d71781ce091bf40a0907fa12fe> (2021-03-31) and <https://cgit.freebsd.org/ports/commit/?id=ed8d3eda309dd863fb66e04bccaa513eee255cbf> (2021-04-06) made reference to <https://wiki.freebsd.org/git> where it's clear that write access to Subversion was disabled. At the foot of the page:

❝Documents at imp's github repo were drafted as material for the FreeBSD Handbook, which was converted to AsciiDoc/Hugo around the same time. There remain some items of interest.❞

– if I recall correctly, one of the most interesting items was FreeBSD moving to Git: Why. Subscribe to <https://github.com/bsdimp/freebsd-git-docs/issues/91>, if you like.

* The front page news on the freebsd website has no mention. Ports affects a lot more people than, say, the newest RC being out. Spread awareness.

See above, the food for thought.

* The project twitter feed has no recent mentions.
* Perhaps a feedback link on the git/ports wiki to let users say "hey you missed these spots"?
* Based on other comments here, test common tools, not just developer favorites?

Yes, yay for progress. But it's 2021, updating matching resources should be a given, not an afterthought.
A newcomer to the community this week would certainly have been confused, and ports are not optional for most use cases.

I suspect that in the noise, some people lost sight of the opening post (emphasis: mine):

Last SVN commit to the ports tree:


So make sure to change your local ports trees if you've been using subversion to update it. portsnap(8) users should not be impacted by this change.

Postscript

FreeBSD and Git a.k.a. Evaluating GIT for FreeBSD (Ed Maste) ◀ October 2018 FreeBSD Developer Summit | FreeBSD Foundation

Warner's Random Hacking Blog: FreeBSD Subversion to Git Migration: Pt 1 Why? (2020-09-19) ◀ https://old.reddit.com/r/freebsd/comments/iw2b69/-/
 
Last edited:

Trihexagonal

Son of Beastie

Reaction score: 2,299
Messages: 2,866

I never thought about what it will do to my Tutorial and the use of ports... Nothing till I change from using them.

Get to typing and don't let me hear those keys quiet till you rewrite it or Shakespeare.🐒🐒🐒🐒

Alas, poor portmaster! I knew him, Beasite, a fellow of infinite text, of most excellent fancy. He hath borne me on his back a thousand times, and now...

Monkey see? Monkey do.
 

mtu

Active Member

Reaction score: 111
Messages: 166

I'm also bugged by most of the annoyances mentioned here. I have a mental model that I think explains what's going on.

The basic hypothesis is: FreeBSD is geared towards the needs of those working on it. That's volunteer devs, employed devs and (by extension) vendors using FreeBSD for their businesses and contributing back (e.g. Netflix, but not Apple). FreeBSD specifically does not cater to the needs of whoever could be considered an "average user", which includes most people on these Forums.

Some examples that I think illustrate this point:
  1. For development and server usage, consumer hardware support and grapical desktops are not a priority. Hence, the sorry state of the installer, graphical desktop integration and WiFi.
  2. Poudriere is the favorite and best-suited tool for developers and professional deployment. Hence, other ways of using ports are poorly supported and documented: manual ports compilation, binary packages and related tools like portmaster. In the same vein, developers are likely to maintain and work on their own ports tree(s) with SVN/git, and so alternatives for users' convenience such as portsnap, INDEX or gitup are not a priority.
  3. There's almost no overlap between the users gathered on these Forums or on Reddit and the people working on FreeBSD. Naturally, the communication channels and tools relevant to the development of the project are those used in development. Mailing lists are key, as is a private Slack used by core devs, and IRC to some degree. Phabricator and the Bugzilla play important roles, issues and pull requests on Github do not. Pointing out flaws or suggesting improvements on these Forums is akin to screaming into the void.
  4. Documentation is recognized to be important, but not a priority when implementing changes – because those working on FreeBSD already know how to do things. Documentation therefore tends to lag behind current events and the latest-and-greatest ways of deployment/development.
This is emphatically not an attack on "the devs" in the name of "the users". Some people are devoting much of their lives to FreeBSD (or successfully using it to do business), and I am grateful for getting a great piece of software out of it without even paying any money.

I am, however, irritated by inconsistencies that arise from these realities in contrast with the stated goals of the project. For example:
  1. poudriere is the de-facto standard for dealing with ports—but it's not part of the base system, and not documented in accordance with its importance. What is in the base system, like portsnap and ports maintenance by raw Makefiles, gets little attention in comparison.
  2. The base system is meant to provide the tools needed for development, which is why it contains a compiler for example. The project also went to great pains to banish GPL'ed code from the base system. And yet it's practically impossible to do serious work on FreeBSD without git, which is GPLv2, and therefore cannot be part of the base system.
Now, there's always a voice that says "stop complaining and help out". Fair enough – I for one am trying to contribute by maintaining a small port, diligently reporting bugs and fighting my way through a Phabricator review every once in a while. But the reality is that not everyone that likes to use FreeBSD is in a position to dedicate as much time and energy as is needed to make a difference.

What to make of all this? I don't see a way to change this state of affairs, much as I would like to see some things done differently – it's an uphill battle I can't win with the time and energy on my hands. I'm trying to adapt my expectations, and tell myself: In any aspect of using FreeBSD, you can either do it the way most devs do, or be largely on your own.
 

PMc

Daemon

Reaction score: 671
Messages: 1,354

The basic hypothesis is: FreeBSD is geared towards the needs of those working on it.
I don't think You're much wrong - only, I would rather pronounce it as "working with it". It is the difference between business customers (B2B) and consumers (B2C).

It is the same when you buy a car, everything is explained in a very foolproof fashion. When you buy heavy-industry vehicle like forestry-machines, this is very different: you are expected to know how such equipment works, how it is to be handled and what are the important things (or have personnel who knows).
And certainly you can use your caterpillar for everyday driving, too - but then, maybe the stereo will not have all the perfect sound. (With real hardware, the segregation usually happes per price tag.)

Back in the days when FreeBSD got published, there was still a clear distinction between personal computer users (windows) and business applications (workstation/server=>unix). Now with Berkeley on the apples and linux on the gadgets the distinction is no longer clear.
 

olli@

Daemon
Developer

Reaction score: 1,253
Messages: 1,140

Back in the days when FreeBSD got published, there was still a clear distinction between personal computer users (windows) and business applications (workstation/server=>unix). Now with Berkeley on the apples and linux on the gadgets the distinction is no longer clear.
In this context, it is also interesting to see how the FreeBSD motto changed over the years.
In the 90s its motto was “Turning PCs into Workstations” – I still have a bunch of FreeBSD stickers and case plates made by Walnut Creek that have this motto on them.
For the past few years the motto has been “The Power to Serve”. This pretty much indicates the target audience and main intended use of FreeBSD.
 

ShelLuser

Son of Beastie

Reaction score: 2,091
Messages: 3,782

At the risk of going slightly offtopic (not my intention) but...

Some examples that I think illustrate this point:
  1. Poudriere is the favorite and best-suited tool for developers and professional deployment.
"Best suited tool"? I strongly beg to differ.

I maintain 3 VPS servers, one is leading and the others provide backups. I prefer full control and thus I'm using the Ports collection to install my software. I dislike overhead and therefor my main server provides a package repository for my other servers to use. This means I build once and install many.

All I'm using is ports-mgmt/portmaster (and a custom shell script to keep things in check). I simply install ("update") a new port, Portmaster places the package of the new port in /usr/ports/distfiles/All and a daily crontab makes sure that the repository databases get updated (" pkg repo /root/repo/pub.key"), all distributed by Apache.

In other words... I install a port upgrade, and the next day it'll be automatically available for pkg to install.

Just because something is popular doesn't automagically imply it's also the best.

  1. Hence, other ways of using ports are poorly supported and documented: manual ports compilation, binary packages and related tools like portmaster.
:rolleyes: Yeah right.

Lessee now... ports(7) which lists all build targets (you do realize that manual pages are actual manual pages on FreeBSD, right?), chapter 4.5 of the FreeBSD handbook (which even mentions Portmaster for crying out loud!) which obviously brings me to: portmaster(8) (remember my comment about manual pages?).

And about those binary packages: chapter 4.4 of said handbook.

This also brings us to pkg(8) which can refer you to pkg-install(8) and/or pkg-repo(8), if not from the command summary then surely from the SEE ALSO section.

(edit):

I probably shouldn't... oh well.

  1. Documentation is recognized to be important, but not a priority when implementing changes
So that's why Git got mentioned in the handbook long before the whole transition had been finished.

I can't help think you don't realize the massive effort that goes into some proper documentation. And I say that because of this:



See, if they really didn't give this any priority then they'd postpone the whole thing until after the transition was finished (like I did). It's a lot easier to add a referral to the project overview than to add actual documentation. But that's not what they did now, is it?

  1. – because those working on FreeBSD already know how to do things. Documentation therefore tends to lag behind current events and the latest-and-greatest ways of deployment/development.
See my point above.

It would really help your post if you would back up your unfounded claims by providing actual examples.

But... auch, that works both ways 😁 => Did you notice there's a "Git method" section in chapter 4.5 of the handbook?

Now, it's a real shame that they no longer provide a "last updated" section in the handbook but... I has source code:

Code:
commit a3ad91f11fea5cc457eb053c46c26e4dd43df599
Author: Rene Ladan <rene@FreeBSD.org>
Date:   Tue Apr 6 20:59:06 2021 +0200

    Handbook: adjust for the various git migrations.

    - prefer to install git as a package over building it oneself.
    - update dates in in the introduction
    - let some Subversion URLs move to the src repository, as it is the only
      repository with an Subversion exporter active.
    - let the user install git instead of subversion for doing ports work.

    Reviewed by:    emaste, mat
    Differential Revision:  https://reviews.freebsd.org/D29450
Your post: today (9th of April), these documentation changes: 6th of April.

Oh, it gets much better:

Code:
commit abff932fe818c144380fd5ebfdcdf44f93de46de
Author: Warner Losh <imp@FreeBSD.org>
>>> Date:   Tue Mar 16 21:55:44 2021 -0600

    Merge in the migration from subversion / github

    Add in the migrating from the old subversion / git docs. This section
    should be removed in a few months maybe.

    Contributions by: lwshu@, Alexander Richardson, kib@, Ceri Davies

[...]

commit cf77cf8be5142f3069e313f6eef3a36597b3dab3
Author: Li-Wen Hsu <lwhsu@FreeBSD.org>
>>> Date:   Tue Jan 12 00:46:00 2021 +0800

    Update information of source code repositories

    Doc and src repositories are in git now and ports tree is in progress.

    Reviewed by:    uqs, ygy
    Approved by:    ygy
    Differential Revision: https://reviews.freebsd.org/D28080

[...]

commit f3bf61d5614d54621681c2613edfa77c3eb659db
Author: Li-Wen Hsu <lwhsu@FreeBSD.org>
>>> Date:   Thu Jan 7 17:13:44 2021 +0800

    Add link to git pepository

    Approved by:    ygy
    Differential Revision:  https://reviews.freebsd.org/D28015

[...]

commit e98cc8974139ae77d1550f60dba75e3d7435915d
Author: John Baldwin <jhb@FreeBSD.org>
>>>> Date:   Mon Dec 21 11:07:24 2020 -0800

    Generate links to cgit instead of svn in document title blocks.

    PR:             251940
    Reviewed by:    emaste, gjb, ygy
    Differential Revision:  https://reviews.freebsd.org/D27704
Concluding: the documentation got updated long before the transition was even close to completion.

  1. poudriere is the de-facto standard for dealing with ports—but it's not part of the base system, and not documented in accordance with its importance. What is in the base system, like portsnap and ports maintenance by raw Makefiles, gets little attention in comparison.
I'd argue that Poudriere is a widely used tool when people want to maintain their own package repository. I don't see many people installing Poudriere just for the sake of installing a single port.

Got any arguments to back up your claim?

  1. The base system is meant to provide the tools needed for development, which is why it contains a compiler for example.
I'm actually not too sure about this to be honest. From my point of view the only reason we have a compiler onboard is because src is a part of the whole distribution. When installing FreeBSD you can opt to have the Ports collection and source tree installed as well. But what good would those be without a compiler?

And installing ports or building your own setup does not imply development.

  1. The project also went to great pains to banish GPL'ed code from the base system. And yet it's practically impossible to do serious work on FreeBSD without git, which is GPLv2, and therefore cannot be part of the base system.
Huh? Ever heard of freebsd-update(8)? It's "only" the de-facto way to upgrade your system, no biggie. It even allows you to customize its settings where you can opt-out from specific parts such as the ports collection and/or the source tree (as well as world/games 😧 )

Now, there's always a voice that says "stop complaining and help out".
Here's a voice that says: stop throwing wild statements around without any arguments to back up your claims.

But the reality is that not everyone that likes to use FreeBSD is in a position to dedicate as much time and energy as is needed to make a difference.
It's not about making a difference in the first place, it's about trying to help out. "Making a difference" makes it sound like a popularity contest, yech!

In any aspect of using FreeBSD, you can either do it the way most devs do, or be largely on your own.
What a load of nonsense.

What accounts for "being on your own" anyway? I mean... all the documentation is there, you only need to check it out, read and learn and then do things your way.

And I say this because, yah, I seriously dislike both Poudriere and Synth alike, both can take a hike for all I care. Don't read between the lines: I have the utmost respect for both projects, however that doesn't change the fact that "me, myself, and I" dislikes both programs with a passion (heck.. this even triggered my edit 😶). But my dislike doesn't automagically make both environment useless. What about all those other people who do enjoy them?

So all I did was check up with the Portmaster documentation, I studied pkg and learned about pkg-repo(8) and that resulted in my own "Poudrier & Synth"-free setup.

I even wrote a guide about that:


FreeBSD is NOT about "either the devs way or the highway". Sorry, I personally find that comment borderline to insulting. But once again: that's me, myself and I again.

All the documentation you might need is already out there. And there are a lot of people working really hard to make sure all of it stays up to date.


Instead of throwing accusations around you'd actually make a difference if you'd provide examples. Tell us what documentation section is bad according to you, give us links and then we (= the community; those are all which matters) actually have something to work on. Then we can actually look at those links and improve them.

Wanted to make a difference, now's your chance...

(edit: sorry for temporarily b0rking the thread).
 

bagas

Well-Known Member

Reaction score: 8
Messages: 276

Has this change propagated through yet? I'm still not seeing any changes to the ports tree through portsnap.
What are you about?
Ports are now received through git.
Portsnap is no longer up to date.
 

dal36

New Member

Reaction score: 8
Messages: 10

I think I misunderstood your message - sorry about that.
Thanks.
fixed gitup.conf.
The first post on the thread indicated that portsnap should not be impacted. I think I'm seeing the same thing as in your separate thread, i.e. portsnap not updating the ports tree. I take it you didn't find a solution to continue using Portsnap?
Last SVN commit to the ports tree:


So make sure to change your local ports trees if you've been using subversion to update it. portsnap(8) users should not be impacted by this change.
 

bagas

Well-Known Member

Reaction score: 8
Messages: 276

The first post on the thread indicated that portsnap should not be impacted. I think I'm seeing the same thing as in your separate thread, i.e. portsnap not updating the ports tree. I take it you didn't find a solution to continue using Portsnap?
Create an alias for the Portsnap command using the git in your environment.
But it's easier to use ports updates, git pull, or gitup ports.
 

olli@

Daemon
Developer

Reaction score: 1,253
Messages: 1,140

What are you about?
Ports are now received through git.
Portsnap is no longer up to date.
Portsnap will continue to be supported. It is deprecated, but it will continue to work for several years, probably. Note that FreeBSD 13 ships with portsnap, too (and it’s even still in 14-current).
Right now the portsnap data is still “frozen”. After the recent migration from Subversion to Git, the portsnap servers need to be adapted, too. Just have a little patience.

Edit: Alternatively, if you want to make the switch right now, net/gitup is a lightweight, easy-to-use replacement that covers all the typical use-cases of portsnap. Unlike git, it has zero dependencies, has a BSD license, and requires no knowledge about version control systems in general or git in particular.
 

grahamperrin

Daemon

Reaction score: 519
Messages: 1,691

Portsnap and new git ports tree

… not seeing any changes to the ports tree through portsnap. Or perhaps …

… (I shouldn't treat https://wiki.freebsd.org/git#Ports_Schedule as fully comprehensive) …

<https://lists.freebsd.org/pipermail/freebsd-questions/2021-April/293732.html> (2021-04-07):

❝Work is in progress to convert the portsnap build infrastructure to git. The changes are just about ready for testing, but it will probably be a couple of days yet before it is deployed.❞

FreeBSD mottos

… For the past few years the motto has been “The Power to Serve”. …

I recognise the phrase, but I can't recall when I last saw it.

There's a mixture in the Everyday FreeBSD Brochure (2018-02-20). The Get Involved Brochure (2016-04-06) appears fine without a motto. Both items referred from <https://freebsdfoundation.org/our-work/education-advocacy/>.

Overlaps

… almost no overlap between the users gathered on these Forums or on Reddit and the people working on FreeBSD. …

‥ I spend far more time in Reddit than here because sadly, discussion of -CURRENT is forbidden.


Documentation

… Documentation is recognized to be important, but not a priority when implementing changes …

I get the impression that things are suitably prioritised but sometimes, it's simply not possible (or appropriate) to have an updated document until after the dust has settled around something.

Posted yesterday:

 
Top