gitup

grahamperrin

Beastie's Twin

Reaction score: 1,694
Messages: 5,067

… Cloning/pulling the source can take up to 2GB of memory. …

Is this the likeliest cause of "Killed", where the OS has no swap?

1629567620517.png
 

aafbsd

New Member

Reaction score: 4
Messages: 15

Just gave gitup a first try and started playing. I started with:

Code:
{
  "defaults" : {
    "proxy_host"        : "proxy",
    "proxy_port"        : "3128",
    "port"              : 443,
    "host"              : "git.freebsd.org",
    "low_memory"        : false,
    "display_depth"     : 0,
    "verbosity"         : 1,
    "work_directory"    : "/tmp/gu",
  },

  "ports" : {
    "repository_path"  : "/ports.git",
    "branch"           : "main",
    "target_directory" : "/tmp/g",
    "ignores"          : [ "INDEX-11", "INDEX-12", "INDEX-13", "INDEX-14" ],
  }

}

and non-existing dirs "/tmp/g" and "/tmp/gu". Now I run "gitup ports" and it fills "/tmp/g" with the
current ports-tree. Out of curiosity I now run "gitup -r ports" and stuff in "/tmp/g" gets deleted.
Either I didn't get the logic or this is somewhat buggy :).
 

astyle

Daemon

Reaction score: 1,001
Messages: 1,994

😩 Looks like I'll stick with portsnap until I see it no longer working. Or I might learn straight git (not a priority for me, got other stuff on my plate, I just like the idea) 😋
 

aafbsd

New Member

Reaction score: 4
Messages: 15

Not much to learn. "git clone" followed by regular "git fetch", "git diff" (to see what would be done) and "git merge".
Can be combined into "git pull" but I like to do the single steps.

However, this gives you a bloated ports directory. Mine is 2,2GB with 50% sitting in .git. You will laugh but I'm
still a ctm-user (since 25+ years). But ports is no more delivered via ctm and Julian has no time to fix it. So
I was happy to discover gitup which seems small but maybe not that mature yet :).

Another thing is that I have a few local modifications in my ports tree (stuff that can't be addresse via EXTRA_PATCH_TREE)
and I used to have a wrapper which restores the original files, applies the ctm files, and reapplies my own mods -- telling me
if their original versions were touched.

I have adopted this wrapper for the above git commands but I'd prefer to use something like gitup.
 

aafbsd

New Member

Reaction score: 4
Messages: 15

Now, what I'd like to have is an update mode where gitup leaves unknown files alone.
E.g., something like "ignores" for everything that does not get deleted or modified
actively from upstream. An example:

1. I run "gitup ports" to check out all for the first time

2. Now I add "/usr/ports/x/y/myfile"
3. Now I modify "/usr/ports/a/b/Makefile"

4. Someone modifies "/usr/ports/a/b/Makefile" upstream
5. Someone deletes"/usr/ports/c/p/distfile" upstream

6. Now I run "gitup ports" again. Normally it would now modify "Makefile",
delete "distfile" and "myfile". But I would like to have "myfile" left alone...
 

aafbsd

New Member

Reaction score: 4
Messages: 15

Because gitup will not let mentioned files alone. So I will have to live with git (and its bloat)
or hack gitup to do what I want. My hope was that gitup can be used as is for that but apparently
this is not the case.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 14,011
Messages: 40,746

gitup(1) is meant as a simple tool to only checkout the ports or source tree. It's not meant to do anything fancy.
 

aafbsd

New Member

Reaction score: 4
Messages: 15

add your file(s) to ignore list
As the ignore list doesn't honour wildcards (or even regexes) this would be quite a
mess in my case. But since my files follow a very simple naming scheme, I've simply
hardcoded a matching test to ignore_file() and this should be enough.
 

Jose

Daemon

Reaction score: 1,315
Messages: 1,535

Yep. Bloat was about 25% smaller but growing. I am happy now with the modified gitup ;-).
Not sure what you're talking about.
Code:
$ git clone --depth 1 -b 2021Q4 https://git.FreeBSD.org/ports.git
Cloning into 'ports'...
remote: Enumerating objects: 178456, done.
remote: Counting objects: 100% (178456/178456), done.
remote: Compressing objects: 100% (167126/167126), done.
remote: Total 178456 (delta 9323), reused 118397 (delta 6961), pack-reused 0
Receiving objects: 100% (178456/178456), 73.81 MiB | 9.18 MiB/s, done.
Resolving deltas: 100% (9323/9323), done.
Updating files: 100% (142020/142020), done.
$ du -sh ports                                                                          
1.0G    ports
$ cd ports
$ du -sh .git                                                                     
 94M    .git
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 14,011
Messages: 40,746

Make sure your /usr/ports isn't filling up with all the work/ directories.
 

aafbsd

New Member

Reaction score: 4
Messages: 15

Gitup gives you one branch at a time. Let's compare apples to apples. Not sure what doing it for 20 days would prove.
Yep, you suggested git shallow clones and I commented on it. To clarify:
I tried git shallow clones for approx. 20 days, syncing one or two times a day.
 
Top