Service start error on poudriere-built gitlab

G

garrett

Hello, I am new to the forum.

I have been running www/gitlab (ce) in a jail for about 6 months, performing updates on the underlying system and on the app, and it's been working great.

Currently I'm on 14.1-RELEASE-p2.

At some point I went from a package install of gitlab-ce to a poudriere-built install - probably to get a head start on a fix. In order not to mismatch a poudriere build with a pre-built ports pkg, I reinstalled gitlab and pointed dependencies to the poudriere repo. I believe it worked fine before gitlab-ce v17 (just noting the version for context, I'm not suggesting there is a port problem).

I build with poudriere on server A. The Server B jail host pulls over the packages from server A. Packages are mounted into gitlab jail on Server B.

I recently performed updates on gitlab-ce and even double-checked with a re-compile (still no errors) and then even did a fresh install. I'm running into a problem when I attempt start the service:
Code: 
# service gitlab start
Don't run Bundler as root. Installing your bundle as root will break this application for all non-root
users on this machine.
Bundler::GenericSystemCallError: There was an error accessing
`/wrkdirs/usr/ports/www/gitlab/work-ce/gitlab-foss-v17.1.1/.bin`.
The underlying system error is Errno::EROFS: Read-only file system @ dir_s_mkdir - /wrkdirs
/s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler/shared_helpers.rb:119:in `rescue in
filesystem_access'
/s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler/shared_helpers.rb:104:in
`filesystem_access'
  /s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler.rb:511:in `mkdir_p'
  /s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler.rb:122:in `bin_path'
/s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler/installer.rb:116:in
`generate_bundler_executable_stubs'
/s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler/installer/gem_installer.rb:79:in
`generate_executable_stubs'
/s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler/installer/gem_installer.rb:18:in
`install_from_spec'
/s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler/installer/parallel_installer.rb:132:in
`do_install'
/s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler/installer/parallel_installer.rb:86:in
`call'
/s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler/installer/parallel_installer.rb:66:in
`call'
/s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler/installer.rb:244:in
`install_in_parallel'
  /s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler/installer.rb:201:in `install'
  /s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler/installer.rb:89:in `block in run'
  /s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler/process_lock.rb:12:in `block in lock'
  /s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler/process_lock.rb:9:in `open'
  /s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler/process_lock.rb:9:in `lock'
  /s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler/installer.rb:71:in `run'
  /s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler/installer.rb:23:in `install'
  /s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler/cli/install.rb:63:in `run'
  /s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler/cli.rb:247:in `block in install'
  /s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler/settings.rb:158:in `temporary'
  /s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler/cli.rb:246:in `install'
/s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler/vendor/thor/lib/thor/command.rb:28:in
`run'
/s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in
`invoke_command'
/s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler/vendor/thor/lib/thor.rb:527:in
`dispatch'
  /s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler/cli.rb:35:in `dispatch'
/s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler/vendor/thor/lib/thor/base.rb:584:in
`start'
  /s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler/cli.rb:29:in `start'
  /s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/exe/bundle:28:in `block in <top (required)>'
/s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler/friendly_errors.rb:117:in
`with_friendly_errors'
  /s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/exe/bundle:20:in `<top (required)>'
  /usr/local/bin/bundle:25:in `load'
  /usr/local/bin/bundle:25:in `<main>'

An error occurred while installing rake (13.2.1), and Bundler cannot continue.

In Gemfile:
  cssbundling-rails was resolved to 1.4.0, which depends on
    railties was resolved to 7.0.8.4, which depends on
      rake
Could not create Gemfile.lock for gitlab, please report this using FreeBSD Bugtracker, https://bugs.freebsd.org/

It appears (as seen near the top of the results) that it is looking for
`/wrkdirs/usr/ports/www/gitlab/work-ce/gitlab-foss-v17.1.1/.bin` and as best I can tell that's actually a path involved in the poudriere build process.

I'm failing to understand what I may have changed (I would say "nothing"). I also don't understand the build process enough to explain why it's searching for a relative path, or what "it" is that is doing the searching for it.

I'm also suprised that Gemfile.lock is not already created. I would have expected that to occur during the update compilation process. I did confirm it is missing from /usr/local/www/gitlab/.

If I had to guess what the problem is, I would say I should not use poudriere to build a particular ruby/rubygem package. But that's the limit of my understanding.
 
If I recall correctly you have to set BUILD_AS_NON_ROOT=no in poudriere.conf for it to build correctly with poudriere. I ran into this issue too. There's a PR about it, I'll try to find it.

garrett said:
I'm also suprised that Gemfile.lock is not already created.
Click to expand...
You're supposed to remove it, make sure you follow the upgrade instructions correctly. All the gems are installed through ports/packages, you don't want bundler to mess this up. But every now and then there's a minor mismatch that's usually quickly resolved. It's a huge and complicated port, with dozens of gems that all need to line up perfectly.

garrett said:
I would say I should not use poudriere to build a particular ruby/rubygem package.
Click to expand...
I've been building gitlab with poudriere for quite some time now. The only problem I had was when the default Ruby changed to 3.2 and gitlab wasn't ready for that yet. That was quickly remedied by changing the default ruby back to the old version until the issue was resolved with gitlab itself.
 
Thank you SirDice

Here is what my poudriere.conf shows:
Code: 
# Define to yes to build and stage as a regular user
# Default: yes, unless CCACHE_DIR is set and CCACHE_DIR_NON_ROOT_SAFE is not
# set.  Note that to use ccache with BUILD_AS_NON_ROOT you will need to
# use a non-shared CCACHE_DIR that is only built by PORTBUILD_USER and chowned
# to that user.  Then set CCACHE_DIR_NON_ROOT_SAFE to yes.
#BUILD_AS_NON_ROOT=no

It's disabled, but I am assuming the setting of no is the default. I checked and have CCACHE_DIR set as well.
 
Remember to set BUILD_AS_NON_ROOT=no because it defaults to 'yes' if you don't use ccache.
 
Alright, did the upgrade on my install, was kind of dragging my feet. Upgraded the host from 13.2 to 14.1. Upgraded the jail where gitlab is running too. Then upgraded gitlab from 17.0.1 to 17.1.1. Everything was built with poudriere. No issues, everything starts, everything works.
 
Well, no luck for me yet. After rebuilding the packages with the changes suggested, I am getting the same error. I have pkg forced-upgraded the gitlab jail just to make sure I was getting the new packages. I don't see any errors at all during the compilation of gitlab - and Gemfile.lock is created successfully after the compilation step (at least that outcome is different that the original scenario).

Is it understood why the relative path reference is being sought in this error?
Code: 
Bundler::GenericSystemCallError: There was an error accessing
`/wrkdirs/usr/ports/www/gitlab/work-ce/gitlab-foss-v17.1.1/.bin`.
The underlying system error is Errno::EROFS: Read-only file system @ dir_s_mkdir - /wrkdirs
/s/usr-local/lib/ruby/gems/3.2/gems/bundler-2.5.13/lib/bundler/shared_helpers.rb:119:in `rescue in
filesystem_access'

I still don't understand the concept of what might be causing it - I would expect this error during a poudriere build but the error is occurring in the gitlab jail (with a totally separate host from the poudriere jails). It's not really a "read-only filesystrm" issue - it's actually not even a path that exists and shouldn't. Maybe I am misunderstanding the error? Is service gitlab start actually used to build additional files?

So many aspects of this just don't make sense to me. I am currently re-compiling gitlab once more just to try it.

EDIT: still same error. Will try a restore next.
 
OK, I just found in .bundle/config:
---
BUNDLE_BIN: "/wrkdirs/usr/ports/www/gitlab/work-ce/gitlab-foss-v17.1.1/.bin"
Click to expand...
So this must be the reason it is calling the path. Is this what it should be?


EDIT:
I found a config.pkgsave in the gitlab project root .bundle directory with contents of:
---
BUNDLE_JOBS: "2"
Click to expand...
I replaced the existing config and it's starting up now.

I have no idea why it changed. But it's working!
 
garrett said:
So this must be the reason it is calling the path. Is this what it should be?
Click to expand...
Looks the same on my system:
Code: 
root@gitlab:/usr/local/www/gitlab # cat .bundle/config
---
BUNDLE_BIN: "/wrkdirs/usr/ports/www/gitlab/work-ce/gitlab-foss-v17.1.1/.bin"

garrett said:
Code: 
An error occurred while installing rake (13.2.1)
Click to expand...
Interesting error. What does gem list rake and pkg info -x rake output?
Code: 
root@gitlab:/usr/local/www/gitlab # gem list rake

*** LOCAL GEMS ***

opentelemetry-instrumentation-rake (0.2.2)
rake (13.2.1)
root@gitlab:/usr/local/www/gitlab # pkg info -x rake
rubygem-opentelemetry-instrumentation-rake-0.2.2
rubygem-rake-13.2.1

Although gitlab runs bundle to 'install' various gems, it should pick them up from the system, that's why gitlab has a huge dependency list, most of them gems.
 
Oh, I seem to have that directory:
Code: 
root@gitlab:/usr/local/www/gitlab # ll /wrkdirs/usr/ports/www/gitlab/work-ce/gitlab-foss-v17.1.1/
total 9
drwxr-xr-x  2 root wheel 63 Jul  6 14:21 .bin/
Yours is giving a 'read-only' error. I suspect this is due to the way the jail has been set up? Is the root filesystem writable? It is on my system (using bastille for jail management):
Code: 
root@gitlab:/usr/local/www/gitlab # ll /
total 208
drwxr-xr-x   18 root wheel   22 Jul  6 08:24 .bastille/
-rw-r--r--    1 root wheel 1011 Jul  7 00:59 .cshrc
-rw-r--r--    1 root wheel  495 Jul  7 00:59 .profile
drwxr-xr-x    2 root wheel    2 Dec 21  2021 .template/
-r--r--r--    1 root wheel 6109 Jul  7 00:58 COPYRIGHT
lrwxr-xr-x    1 root wheel   14 Dec 21  2021 bin@ -> /.bastille/bin
lrwxr-xr-x    1 root wheel   15 Dec 21  2021 boot@ -> /.bastille/boot
dr-xr-xr-x   10 root wheel  512 Jul  6 08:55 dev/
drwxr-xr-x   27 root wheel  106 Jul  7 00:59 etc/
lrwxr-xr-x    1 root wheel    8 Dec 21  2021 home@ -> usr/home
lrwxr-xr-x    1 root wheel   14 Dec 21  2021 lib@ -> /.bastille/lib
lrwxr-xr-x    1 root wheel   18 Dec 21  2021 libexec@ -> /.bastille/libexec
drwxr-xr-x    2 root wheel    2 Apr  9  2021 media/
drwxr-xr-x    2 root wheel    2 Apr  9  2021 mnt/
drwxr-xr-x    2 root wheel    2 Apr  9  2021 net/
dr-xr-xr-x    2 root wheel    2 Apr  9  2021 proc/
lrwxr-xr-x    1 root wheel   17 Dec 21  2021 rescue@ -> /.bastille/rescue
drwxr-x---    3 root wheel   13 Jul  6 14:29 root/
lrwxr-xr-x    1 root wheel   15 Dec 21  2021 sbin@ -> /.bastille/sbin
drwxrwxrwt  713 root wheel  715 Jul  7 11:58 tmp/
drwxr-xr-x    6 root wheel   15 May  8  2021 usr/
drwxr-xr-x   24 root wheel   24 Jul  6 08:55 var/
drwxr-xr-x    3 root wheel    3 Jul  6 14:21 wrkdirs/

Honestly that seems like a weird place to put those binaries. Don't know why or where it's coming from but this is not where I would expect them.
 
Yes! My system files are read-only - that yours are not might be a good clue to narrow down what changed. From my error it seems it actually wanted to create the wrkdirs folder at the jail root:
Read-only file system @ dir_s_mkdir - /wrkdirs
Click to expand...
I think that means to run mkdir from s.

Code: 
/ # ll
total 82
drwxr-xr-x  15 root wheel uarch  20 Jul  3 00:29 ./
drwxr-xr-x  15 root wheel uarch  20 Jul  3 00:29 ../
drwxr-xr-x   2 root wheel uarch  49 Jul  3 00:25 bin/
drwxr-xr-x  14 root wheel uarch  67 Jul  3 00:26 boot/
dr-xr-xr-x   9 root wheel -     512 Jul  4 20:53 dev/
lrwxr-xr-x   1 root wheel uarch   5 Jul  3 00:29 etc@ -> s/etc
lrwxr-xr-x   1 root wheel uarch   6 Jul  3 00:29 home@ -> s/home
drwxr-xr-x   4 root wheel uarch  78 Jul  3 00:26 lib/
drwxr-xr-x   3 root wheel uarch   5 Jul  3 00:25 libexec/
drwxr-xr-x   2 root wheel uarch   2 Jul  3 00:24 media/
drwxr-xr-x   2 root wheel uarch   2 Jul  3 00:24 mnt/
drwxr-xr-x   2 root wheel uarch   2 Jul  3 00:24 net/
dr-xr-xr-x   2 root wheel uarch   2 Jul  3 00:24 proc/
drwxr-xr-x   2 root wheel uarch 150 Jul  3 00:25 rescue/
lrwxr-xr-x   1 root wheel uarch   6 Jul  3 00:29 root@ -> s/root
drwxr-xr-x  12 root wheel uarch  15 Jul  8  2023 s/
drwxr-xr-x   2 root wheel uarch 150 Jul  3 00:27 sbin/
lrwxr-xr-x   1 root wheel uarch   5 Jul  3 00:29 tmp@ -> s/tmp
drwxr-xr-x  14 root wheel uarch  16 Jul  3 00:29 usr/
lrwxr-xr-x   1 root wheel uarch   5 Jul  3 00:29 var@ -> s/var

Do you also have a /usr/local/www/gitlab/.bin directory? Is it possible that those are the binaries already in place? All of the binaries in that directory for me are timestamped the same and from the day before the gitlab package was built...possibly from the poudriere build of another dependency. (a rebuild of all packages took over a day)

According to the bundler docs the BUNDLE_BIN variable directs where to store the executables from bundle gems (the default is false).

Interesting error. What does gem list rake and pkg info -x rake output?
Click to expand...
Code: 
# gem list rake

*** LOCAL GEMS ***

opentelemetry-instrumentation-rake (0.2.2)
rake (13.2.1)

# pkg info -x rake
rubygem-opentelemetry-instrumentation-rake-0.2.2
rubygem-rake-13.2.1

At the time of the error, I recall searching for them then and they were installed even though the error suggested they weren't.

The Gemfile.lock has a timestamp reflecting the first start of the gitlab service. It's curious to me that it's overwriting the Gemfile.lock previously created during the bundle compilation step of su -l git -c "cd /usr/local/www/gitlab && RAILS_ENV=production NODE_ENV=production USE_DB=false SKIP_STORAGE_VALIDATION=true bundle exec rake gitlab:assets:compile"

This morning I am recalling two additional things - I think after switching to poudriere I did a fresh install...and then I switched back to the package because the port wasn't listed for some reason and was no longer available via poudriere for a short time. This might be the reason for the existence of .bundle/config.pkgsave

Secondly, (faceslap) I left out in the first post that I am running poudriere-devel. Not sure how that plays into any of this.

I'm sort of leaning toward that the .bundle/config contents are what changed. If you had a read-only system it would have produced the same error I would think.
 
Ran into the same problem.
My instance is also running in a jail (13.3-RELEASE-p4), but I am not building from ports but using pkg (FreeBSD: { url: "http://pkg0.sjb.freebsd.org/${ABI}/latest", mirror_type: "NONE" }).
Went through the update steps (https://gitlab.fechner.net/mfechner/Gitlab-docu/-/blob/master/update/17.0-17.1-freebsd.md) without an issue up until starting the instance results in the same error as post #1 described.
Code: 
I'm sort of leaning toward that the .bundle/config contents are what changed. If you had a read-only system it would have produced the same error I would think.
Just checked in a backup. The folders .bundle and .bin did not exist before the upgrade.
 
You must log in or register to reply here.
Back
Top