Go application pulls wrong version of dependency

B

bschwand

Hi all,

I hit an issue where my port is pulling wrong sources for a dependency and I am not sure at which level the bug is. It is either in the build system, or something wrong with how I specify the GH_TUPLES in the Makefile, or something else entirely, but I don't seem to figure it out right now.

A bit of background:
I made the net-p2p/hostd port. I initially made it the "easy" way, but that did not work because the easy way pulls modules from some proxy site (pkg.go.dev or something similar) which is not updated often enough. The hostd devs recommend to build from github anyway. So in order to get the latest release sources, I switched to using the "full" process using GH_TUPLES.
This seemed to work fine for the currently checked in version of the port, except that ports-mgmt/modules2tuple has a bug where it does not parse correctly github tags of the form such as "hostd/v2.1.0", in this case it produces a GH_TUPLES line like this

SiaFoundation:web:v0.61.0:siafoundation_web_1/vendor/go.sia.tech/web/hostd
instead of

SiaFoundation:web:hostd/v0.61.0:siafoundation_web_1/vendor/go.sia.tech/web/hostd

so I fixed it manually in the Makefile, and that works for the current port.

Now, there is a new release and I updated the port accordingly, see my patch file here https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=286691

However I noticed that it pulls and builds the wrong version of go.sia.tech/web/hostd.
hostd is a daemon, which has a web UI, that web UI is some javascript and other files, which are in the go.sia.tech/web/hostd module. All the UI files and everything is part of the static hostd executable.

First I verified that the issue is with the port build only, not the application itself. If I do a standalone build of the application, not within ports, everything is fine and it builds with the proper files and modules version and hostd UI is correct.

I verified the version pulled by the FreeBSD ports system is incorrect because :
- when connecting to hostd, the UI has wrong files and tries to connect to deprecated http endpoints. Discussion with the devs of hostd confirmed the UI appeared to use old version of the UI JS files.
- when doing "make extract" on my port, sources for all dependencies are fetched as tarballs from github, extracted and I end up with a directory tree like this:
Code: 
/usr/ports/net-p2p/hostd $ ls work/
aws-sdk-go-1.55.7      go-jmespath-0.4.0     mod-0.24.0               time-0.11.0
bbolt-1.4.0            go-json-0.10.5        multierr-1.11.0          tools-0.32.0
bin                    go-querystring-1.1.0  mux-1.4.0                upnp-0.3.0
cloudflare-go-0.115.0  go-sqlite3-1.14.28    net-0.39.0               users-groups-deinstall.sh
core-0.12.0            golang-lru-2.0.7      pprof-407c9e7a662f       users-groups-install.sh
coreutils-0.13.2       goleak-1.3.0          qpack-0.5.1              web-5611d44a533e
crypto-0.37.0          hostd                 quic-go-0.51.0           web-hostd-v0.61.0
decimal-1.4.0          hostd-2.1.0           slim-sprig-52ccab3ef572  webtransport-go-4ac2c9250e66
flagg-1.1.1            hostd.conf            stage                    yaml-2.4.0
frand-1.5.1            hostd.yml             sync-0.13.0              yaml-3.0.1
ginkgo-2.12.0          httprouter-1.3.0      sys-0.32.0               zap-1.27.0
go                     jape-0.13.1           term-0.31.0
go-internal-1.12.0     mock-0.5.0            text-0.24.0
with all dependencies's source code in their own directory.

Well, if I now compare the contents of work/web-hostd-v0.61.0/ with the contents of the github repo tagged hostd/v0.61.0
( can be seen here: https://github.com/SiaFoundation/web/tree/hostd/v0.61.0/hostd )
the contents do not match at all. The sources that the port pulled are NOT the correct version and are effectively very old. BTW, to makes things clear: tag hostd/v0.61.0 is correct and what we want, but not what the sources fetched actually are.

I made sure to delete all distfiles, delete work/, etc made sure my environment is clean (I build and test all this in a jail created with appJail BTW), I made the port fetch and extract sources several times and it always pulls some old version from github.

So, what am I doing wrong here ? am I specifying the GH_TUPLES incorrectly ? Is there some bug in the port build system that fetches wrong versions ? Is github giving incorrect tarballs ?

Kind of at a loss right now on what next to try, to fix this. I mean it should just pull the right tagged version but it does not.

If anyone could try updating their port tree from my bug report and building hostd, at least I would know if it's reproducible and not just some issue on my end.

Any help and hints, advice, very much appreciated.

Cheers
 
Last edited:
noone has any ideas ? all I see is the wrong version is pulled by the port, how can this not be an issue in some other go application ?
 
monwarez said:
Have you tried to fetch using the port tree url ?
Click to expand...
good idea, I kind of assumed that the make file fetches the right thing.
so I deleted work/. deleted distfiles, then did
make fetch VERBOSE=y
that showed me

=> SiaFoundation-web-hostd-v0.61.0_GH0.tar.gz doesn't seem to exist in /usr/ports/distfiles/.
=> Attempting to fetch https://codeload.github.com/SiaFoun.../v0.61.0?dummy=/SiaFoundation-web-hostd-v0.6\
1.0_GH0.tar.gz
fetch: https://codeload.github.com/SiaFoun...y=/SiaFoundation-web-hostd-v0.61.0_GH0.tar.gz: \
size unknown
fetch: https://codeload.github.com/SiaFoun...y=/SiaFoundation-web-hostd-v0.61.0_GH0.tar.gz: \
size of remote file is not known

so I downloaded manually https://codeload.github.com/SiaFoun...y=/SiaFoundation-web-hostd-v0.61.0_GH0.tar.gz

and it matches what make fetch puts into distfiles. I untarred and it does contain all the right files, however when I make extract here is what it looks like

/usr/ports/net-p2p/hostd/work/web-hostd-v0.61.0 $ ls
assets go.mod go.sum hostd.go hostd_test.go web-hostd-v0.61.0
/usr/ports/net-p2p/hostd/work/web-hostd-v0.61.0 $ ls -l assets/_next/static/chunks/
total 337
-rw-r--r-- 1 bruno wheel 141003 2024-06-10 2024 framework-7e58418f45871195.js
-rw-r--r-- 1 bruno wheel 109406 2024-06-10 2024 main-2ab4491c439eb16a.js
drwxr-xr-x 2 bruno wheel 512 2024-06-10 2024 pages
-rw-r--r-- 1 bruno wheel 91460 2024-06-10 2024 polyfills-c67a75d1b6f99dc8.js
-rw-r--r-- 1 bruno wheel 2216 2024-06-10 2024 webpack-dbca7c9a1026ae6e.js
/usr/ports/net-p2p/hostd/work/web-hostd-v0.61.0 $ ls -l web-hostd-v0.61.0/hostd/assets/_next/static/chunks/
total 367
-rw-r--r-- 1 bruno wheel 9802 2025-03-28 22:09 546-4ec84ba77033f762.js
-rw-r--r-- 1 bruno wheel 140036 2025-03-28 22:09 framework-ab7ddb551c23d05a.js
-rw-r--r-- 1 bruno wheel 110991 2025-03-28 22:09 main-9a8933a9641dc604.js
drwxr-xr-x 2 bruno wheel 512 2025-03-28 22:09 pages
-rw-r--r-- 1 bruno wheel 112594 2025-03-28 22:09 polyfills-42372ed130431b0a.js
-rw-r--r-- 1 bruno wheel 2216 2025-03-28 22:09 webpack-dbca7c9a1026ae6e.js

it seems that the files that in the tar file are at hostd/ are to be copied to work/web-hostd-v0.61.0/ but instead there are other files version.
I mean definitely make extract does not just untar the archive as otherwise there would the whole repo which looks like this
~/tmp/web-hostd-v0.61.0 $ ls
README.md commitlint.config.js jest.config.ts nx.json renterd server walletd
apps hostd jest.preset.js package-lock.json scripts tools
babel.config.json internal libs package.json sdk tsconfig.base.json

so there is some file copying that happens with the make extract target, and I don't think it copies the right stuff
 
Is the patch in the bug report up-to-date? Can you lend me the latest one you're working on?

EDIT: I found the contents you want. Also, the hostd port needs go 1.24 toolchain (see go.mod), so it downloads it in fetch stage, you should just use go 1.24 in USES instead of go 1.23.

Code: 
# ls /usr/ports/net-p2p/hostd/work/web-5611d44a533e/
.changeset              .gitignore              README.md               internal                renterd
.editorconfig           .golangci.yml           apps                    jest.config.ts          scripts
.env.example            .husky                  babel.config.json       jest.preset.js          sdk
.eslintignore           .npmrc                  commitlint.config.js    libs                    server
.eslintrc.json          .prettierignore         go.work                 nx.json                 tools
.gitattributes          .prettierrc             go.work.sum             package-lock.json       tsconfig.base.json
.github                 .vscode                 hostd                   package.json            walletd
 
nxjoseph said:
Is the patch in the bug report up-to-date? Can you lend me the latest one you're working on?

EDIT: I found the contents you want. Also, the hostd port needs go 1.24 toolchain (see go.mod), so it downloads it in fetch stage, you should just use go 1.24 in USES instead of go 1.23.

Code: 
# ls /usr/ports/net-p2p/hostd/work/web-5611d44a533e/
.changeset              .gitignore              README.md               internal                renterd
.editorconfig           .golangci.yml           apps                    jest.config.ts          scripts
.env.example            .husky                  babel.config.json       jest.preset.js          sdk
.eslintignore           .npmrc                  commitlint.config.js    libs                    server
.eslintrc.json          .prettierignore         go.work                 nx.json                 tools
.gitattributes          .prettierrc             go.work.sum             package-lock.json       tsconfig.base.json
.github                 .vscode                 hostd                   package.json            walletd
Click to expand...
I set go 1.24 and that did not change anything
otherwise the patch is what I have been trying to make work

what do you mean you found the content you want ?
I tried using the git hash for web/hostd/v0.61.0 but that does not fix things.
do you agree there is a copy step that is not copying the right thing ?

also note that there is web and web/hostd, not the same thing
 
ok I found the issue, it's in the way the go modules are untarred by the port makefile.
This happens because two modules are actually coming from the same github repository, but one is in a subdirectory of the repo.
So, there are two tar files pulled out and placed in the distfiles, those have both the whole repository but at different tags/hashes.
Then the port makefile untars each, then moves them and makes link. However, for the second module it moves the whole directory(repo), not just the corresponding subdirectory.
Plus, the main directory already exists since it was just untarred, so we end up with the whole repo copied a second time into the wrong directory.
Then at build time, the wrong files are picked up because they are the ones coming from the first tag, not the second.

This is easily visible with the following port:

286691 – update to net-p2p/hostd , renterd, walletd: Offer and rent your storage space on the Sia network

bugs.freebsd.org bugs.freebsd.org

net-p2p/hostd
if you do "make extract", pay attention to
work/web-hostd-v0.61.0
work/web-5611d44a533e

same repo and directory structure. But what we want is to move work/web-hostd-v0.61.0/hostd/ to overwrite work/web-5611d44a533e/hostd
i.e. it should do
mv web-hostd-v0.61.0/hostd work/hostd-2.0.1/vendor/go.sia.tech/web/hostd

however the make file does instead
mv web-hostd-v0.61.0 work/hostd-2.0.1/vendor/go.sia.tech/web/hostd
so we end up with a whole copy of the repo in work/hostd-2.0.1/vendor/go.sia.tech/web/hostd/web-hostd-v0.61.0

I could do a post-extract "fix" in my port makefile, but I feel this really should be handled by the base makefiles, after all it does fetch the right stuff but moves it to the wrong place, so I think it's a case that should be added to the base make files, or something specific to go.mk

Any advice here ?
 
You must log in or register to reply here.
Back
Top