gitup

jmehr

Member

Reaction score: 55
Messages: 28

Hello everyone,

With the upcoming transition from Subversion to Git, I've been plugging away at building "gitup", a replacement for net/svnup and I'm happy to report that I've got a working "git clone" prototype.

Looking back at when I was writing net/svnup, one of the things I wished I had done differently was I didn't really engage the community in the process. So, as I start working on adding "git pull" functionality, I'd like to reach out to all of you for your thoughts and feedback about how I can make it best meet your needs.

For example, are there things about the git clone/pull process that you wish worked differently? If you've used net/svnup, are there things about it that you wish worked differently? If you tried using net/svnup and decided it wasn't for you, what could I do to fix that? Is your environment resource constrained and, if you don't have a lot of memory or disk space, how can gitup best accomodate your device's specifics?

Constructive criticism is welcome and appreciated. Thanks!
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 12,305
Messages: 38,827

This is too on-topic to be hiding in off-topic ;)
Was a little split between "Porting new software" and "Userland programming and scripting". But as this isn't about the port or porting process but the actual code itself "Userland.." seems more appropriate.
 
OP
J

jmehr

Member

Reaction score: 55
Messages: 28

I've implemented "git pull" functionality and the gitup prototype is now ready for wider testing. If you're interested, you can find the source code at:

https://github.com/johnmehr/gitup

Much like its predecessor net/svnup, gitup is intended for non-developers
who just need to synchronize a local repository without the additional
overhead required by the official git client and is not intended as a
feature-rich, drop-in replacement for it.

The code is beta, so the usual caveats/warnings (don't use it in a
production environment, make sure you've got backups, don't run as
root) apply.

I think I've got most of the common use cases taken care of but there's
always room for improvement so please don't hesitate to offer suggestions,
comments and/or constructive criticism. Thanks!
 

gnath

Active Member

Reaction score: 46
Messages: 187

Just for hands on I have tried to use "gitup ports" (forum codes are not working) without any problem. But there is error for making any ports. The old ports tree was fine with 'portsnap'. What else required to use this new ports tree ?
Portmaster also give the error "make: "/usr/ports/Mk/Uses/compiler.mk" line 78: warning: " --version" returned non-zero status"
 

unitrunker

Aspiring Daemon

Reaction score: 247
Messages: 547

I think portsnap always grabbed the latest ports tree - which was a problem for folks using quarterly pkg-s. It would be nice if this new tool used the same repo as pkg (quarterly or latest).
 
OP
J

jmehr

Member

Reaction score: 55
Messages: 28

When you have a moment, could you please run cat /usr/ports/.gituprevision and let me know what commit you pulled? You should see something like master:e5260f3eb.

You can track the quarterly ports branches by running gitup quarterly which should seamlessly follow the most recently available quarterly branch (I've only had one quarterly branch change to test and that one went well so it should be working as intended).
 

gnath

Active Member

Reaction score: 46
Messages: 187

I am following latest package repo for daily use. Old ports tree was used only for 'drm-kmod'. Output from /usr/ports/.gituprevision is 'master:e5260f3eb' . I shall try for quarterly and fresh tree for latest also.
 
OP
J

jmehr

Member

Reaction score: 55
Messages: 28

Using e5260f3eb, I did a quick test and I was able to compile /usr/ports/graphics/drm-fbsd11.2-kmod.

Looking at /usr/ports/Mk/Uses/compiler.mk, it looks like its trying to determine the version of the compiler you are using and can't for some reason. I'm not sure where specific compilers are specified (maybe /etc/make.conf?) so hopefully a ports tree expert will be able to lend a hand.
 

RypPn

New Member

Reaction score: 7
Messages: 15

Thanks for creating the tool, certainly makes the transition from svn to git relatively painless, one small observation, /usr/ports/.gituprevision wasn't created in my /usr/ports tree
 

sidetone

Daemon

Reaction score: 908
Messages: 1,859

Hi, when I ran psearch (ports-mgmt/psearch), I got this error:
Code:
Error reading index file "/usr/ports/INDEX-12": No such file or directory
From cat /usr/ports/.gituprevision, I got:
Code:
master:cc83265a6
psearch(1)
Code:
 -f FILE, --file=FILE
              Path to INDEX file. Defaults to the standard location of the
              INDEX file on the FreeBSD system that psearch runs on. Non-
              standard locations given in /etc/make.conf are ignored.
I removed the /usr/ports/ directory, ran gitup ports, and got some output plus:
Code:
gitup: build_repair_command: There are too many files to repair -- please re-clone the repository: Argument list too long
I tried, gitup -c ports, the command for cloning according to the manual, which put a few files in my ports directory, but got the same error message when trying to fill the ports directory.

Perhaps portsnap extract should be used first on an empty directory?
 
OP
J

jmehr

Member

Reaction score: 55
Messages: 28

I did some digging and it doesn't look like the INDEX-[os version] files are present in the ports tree. https://github.com/freebsd/freebsd-ports

I'm not a ports tree expert (and hopefully someone who is will chime in and correct me if I'm wrong) but I believe you can manually create these index files by running:

cd /usr/ports
make index

It looks like make index can take a long time to build so adding INDEX-12 to the "ignores" field of your ports section in gitup.conf should prevent it from being deleted during subsequent pulls.

gitup -c ports should not have gone into repair mode and I'll see if I can figure out what went wrong there. In the meantime, when you remove /usr/ports, you'll also want to remove /var/db/gitup/ports. Because net/gitup doesn't keep the packfiles it downloads from the repository, it uses the contents of this file to reconstruct what it needs from the local copy of the repository and will go into repair mode if it encounters any discrepancies. Removing both and then running gitup ports should get you back up and running.
 

Mjölnir

Daemon

Reaction score: 1,503
Messages: 2,114

Since some time now, gitup(1) fails to update my current sources.
gitup current
Code:
# Host: github.com
# Port: 443
# Repository: /freebsd/freebsd
# Target: /src/14-CUR
# Have: de1aa3dab23c06fec962a14da3e7b4755c5880cf
# Want: de1aa3dab23c06fec962a14da3e7b4755c5880cf
# Branch: refs/heads/master
gitup.conf(5):
# $FreeBSD$
#
# Default configuration options for gitup.conf.
# $Id: gitup.conf,v 1.1 2021/03/06 00:54:50 root Exp root $
{
"defaults" : {
"host" : "github.com",
"port" : "443",
"verbosity" : "1",
"work_directory" : "/var/db/gitup",
},

# "latest ports branch is done with portsnap(8) instead; see portsnap.conf(5)
"ports" : {
"repository" : "/freebsd/freebsd-ports",
"branch" : "quarterly",
"target" : "/ports/quarterly",
"ignore" : [
"./distfiles",
"./packages",
],
},

# release sources are updated with freebsd-update(8) instead; see freebsd-update.conf(5)
# "12.1-release" : {
# "repository" : "/freebsd/freebsd",
# "branch" : "releng/12.1",
# "target" : "/src/12.1-REL",
# },

"12-stable" : {
"repository" : "/freebsd/freebsd",
"branch" : "stable/12",
"target" : "/src/12-STABLE",
},

"13-stable" : {
"repository" : "/freebsd/freebsd",
"branch" : "stable/13",
"target" : "/src/13-STABLE",
},

"current" : {
"repository" : "/freebsd/freebsd",
"branch" : "master",
"target" : "/src/14-CUR",
}
}
Thx in advance for any hints.
 
OP
J

jmehr

Member

Reaction score: 55
Messages: 28

It looks like the "master" branch has been deprecated https://github.com/freebsd/freebsd-src/commit/de1aa3dab23c06fec962a14da3e7b4755c5880cf and switching to "main" should get you back up and running.

Also, we ran into some issues with repository paths that do not end in ".git". When you're pulling from github.com, if you change your repository to:

Code:
/freebsd/freebsd-src.git

you should be able to avoid those potential problems.
 

Mjölnir

Daemon

Reaction score: 1,503
Messages: 2,114

So I changed in gitup.conf(5)
Code:
        "current" : {
                "repository" : "/freebsd/freebsd-src.git",
                "branch"     : "main",
                "target"     : "/src/14-CUR",
        }
Correct? Do I also have to change the "repository": ... for my other source trees, although they seem to work fine? Of course, not for the quarterly ports(7) tree, right?
EDIT Just click Like if that's what you meant, else comment.
 

sidetone

Daemon

Reaction score: 908
Messages: 1,859

I had an error from compiling my kernel after using gitup for sources, but I'm unsure if it came from gitup or from another mistake. When I selected the custom compiled kernel, it wanted to boot from the cd drive. As if the sources were intended for CD instead of being on the harddisk. Going to a backup kernel, then a generic kernel update from freebsd-update made my system work.

It could also have been an error from when my custom KERNCONF files got erased by gitup, which I now know how to prevent, and my backup KERNCONF files may not have been as current.

This was weeks ago. I also haven't tried again to see.
 
OP
J

jmehr

Member

Reaction score: 55
Messages: 28

So I changed in gitup.conf(5)
Code:
        "current" : {
                "repository" : "/freebsd/freebsd-src.git",
                "branch"     : "main",
                "target"     : "/src/14-CUR",
        }
Correct? Do I also have to change the "repository": ... for my other source trees, although they seem to work fine? Of course, not for the quarterly ports(7) tree, right?
EDIT Just click Like if that's what you meant, else comment.

Your config looks good to me. The repository path problems occurred when using git.freebsd.org and if you continue using github.com, your other source trees should be good.

If you decide to switch over to using git.freebsd.org you'll want to change all of them to

Code:
/src.git
 
OP
J

jmehr

Member

Reaction score: 55
Messages: 28

I had an error from compiling my kernel after using gitup for sources, but I'm unsure if it came from gitup or from another mistake. When I selected the custom compiled kernel, it wanted to boot from the cd drive. As if the sources were intended for CD instead of being on the harddisk. Going to a backup kernel, then a generic kernel update from freebsd-update made my system work.

It could also have been an error from when my custom KERNCONF files got erased by gitup, which I now know how to prevent, and my backup KERNCONF files may not have been as current.

This was weeks ago. I also haven't tried again to see.
net/gitup should not have deleted your custom kernel config files (it is supposed to ignore the contents of the various sys/architecture/conf directories in your target directory when deleting files that no longer exist in the repository).

Were any of the directories in your repository symbolic links to other directories in your filesystem?
 

Mjölnir

Daemon

Reaction score: 1,503
Messages: 2,114

That was a heavy update...
Your config looks good to me. The repository path problems occurred when using git.freebsd.org and if you continue using github.com, your other source trees should be good.
So: yes, because it doesn't hurt when using github, but git.freebsd.org needs it. Yes, I'll stick to github until they start to act evilish, what I don't expect, frankly. Simply because it's nearer; IIUC the seem to have a mirror at the DE-CIX in Frankfurt/Main, Germany (derived from observing mtr(8)). I don't think git.freebsd.org has a geo-DNS'ed mirror in Europe; do they/we?
If you decide to switch over to using git.freebsd.org you'll want to change all of them to
Code:
/src.git
Ok, I took a note in my gitup.conf(5). Thx 4 chiming in so quickly. BTW a status or check command would be nice, as well as a -n / --dry-run do-nothing switch.
 

sidetone

Daemon

Reaction score: 908
Messages: 1,859

I might have deleted that line under ignore the time it got deleted, in gitup.conf, because I didn't understand it then. Then later it was explained, and I put it back, and updated the sources.
Were any of the directories in your repository symbolic links to other directories in your filesystem?
I didn't understand. There weren't symbolic files in /usr/src/sys/amd64/conf/. There were two custom files, that one had an include for a trimmed down CUSTOM copy of GENERIC. The error could have been on my end, from possibly using an outdated KERNCONF (that was previously a backup) than the one needed for this kernel.

I'll try again at a later day. I'll start over then.
 
OP
J

jmehr

Member

Reaction score: 55
Messages: 28

That was a heavy update...

So: yes, because it doesn't hurt when using github, but git.freebsd.org needs it. Yes, I'll stick to github until they start to act evilish, what I don't expect, frankly. Simply because it's nearer; IIUC the seem to have a mirror at the DE-CIX in Frankfurt/Main, Germany (derived from observing mtr(8)). I don't think git.freebsd.org has a geo-DNS'ed mirror in Europe; do they/we?

Ok, I took a note in my gitup.conf(5). Thx 4 chiming in so quickly. BTW a status or check command would be nice, as well as a -n / --dry-run do-nothing switch.

I'm not sure if there is or will be a mirror in Europe (I can't find any information about this) but an email to the freebsd-git mailing list should connect you with someone who can let you know about the plans for any additional mirrors. If I run across any, I'll be sure to post them here.

A dry-run command should be very easy to add (I'll put it on the to-do list). For the status command, what would be most helpful to see?
 

Mjölnir

Daemon

Reaction score: 1,503
Messages: 2,114

For the status command, what would be most helpful to see?
Files behind/deleted/new on the repository (indicators: */-/+); files up-to-date, but broken (checksum doesn't match; indicator: # or ?). Up-to-date files doesn't need to be reported IMHO.
There's one in Amsterdam at least, "This is gitmir.pkt.FreeBSD.org located at Amsterdam, The Netherlands."
git.freebsd.org is geo aware ;-)
Great - but then I found another reason to stick using github: let M$ pay for traffic & not FreeBSD :p
 

Yze

Member

Reaction score: 10
Messages: 22

jmehr any plans to add IPv6 support? We moved our build cluster long time ago to ipv6-only jails and so we can't evaluate it to provide you feedback
 
Top