Synth: Introducing new custom package repository builder for FreeBSD and DragonFly

I have a few questions if you'll indulge me.

Say one wants to build a custom repository using synth, what is the recommended way to connect to it? Setup a webserver I guess? If I wanted to build a custom package repository in one of my servers jails and then connect to it and use it as my pkg repository, is synth the right tool for this?

I noticed you said it is not intended to be running to jail, but is there any reason it isn't recommended? Or is it just not necessary?

I have a small server no one's using, an old Windows Home Server HP470 with 2GB of RAM I was recently given, that I could use where the only thing it would be used for is building ports. Or I could run in my much faster server, a FreeNAS server with 32GB of RAM and a Xeon E3, but it would be run in a jail alongside several other jails I use for other things. Or, would I be better off just running synth on the system I want to update itself? What would you recommend?
 
STREBLO said:
Say one wants to build a custom repository using synth, what is the recommended way to connect to it? Setup a webserver I guess?
Click to expand...

see pkg.conf(5). The URL takes several protocols including http://, https://, ftp://, file://, and ssh://. You have many options.

If I wanted to build a custom package repository in one of my servers jails and then connect to it and use it as my pkg repository, is synth the right tool for this?
Click to expand...

It's capable of doing it.

I noticed you said it is not intended to be running to jail, but is there any reason it isn't recommended? Or is it just not necessary?
Click to expand...

It's not necessary and there's no benefit to do it. At the time, it didn't work, but on the github/jrmarino/synth README.md, there's a FAQ entry on how to configure the jail so Synth can run inside one. It's only for people that refuse to run Synth in anything other than a jail for no particular reason.

I have a small server no one's using, an old Windows Home Server HP470 with 2GB of RAM I was recently given, that I could use where the only thing it would be used for is building ports. Or I could run in my much faster server, a FreeNAS server with 32GB of RAM and a Xeon E3, but it would be run in a jail alongside several other jails I use for other things. Or, would I be better off just running synth on the system I want to update itself? What would you recommend?
Click to expand...

Memory is the #1 consideration, the more the better. Then it's number of cores. Then it's newness. ;).
Use the FreeNAS server but be conservative on the number of builders in case the other jails need lots of resources too.

I've never had a problem multitasking on DragonFly but some people with older FreeBSD systems (probably very old?) claim that the system gets unusable while Synth is running. I assume that's a combination of a taxed system and the performance edge of DragonFly showing as well.
 
Using ZFS, 8GB of RAM but an i3 processor (and that's what compiling taxes), I've not yet run into any issues running synth, even with things like libre office. This will be while running, generally, a browser, usually Firefox, (but several sessions--this is a multimonitor setup and I might have it in two or three windows, and before adding swap, Firefox or Thunderbird would sometimes just die, but that was memory), and Thunderbird. So, I won't say I've really been taxing resources, but anyway, I've run synth in a different tag (dwm's version of workspaces, more or less), while getting on with my work and had no resource issues at all on FreeBSD-10.3
 
scottro said:
Using ZFS, 8GB of RAM but an i3 processor (and that's what compiling taxes)...
Click to expand...

Until you run out of memory and swapping to a mechanic drive kicks in. Agreed that CPU limits, but only if memory is plentiful with tmpfs option.
8Gb is a good amount though, good for 4x4 (4 builders by 4 jobs) if the swap is an SSD (4x4 will dip into swap with 8GB, but usually less than 5%)
 
STREBLO said:
I have a small server no one's using, an old Windows Home Server HP470 with 2GB of RAM I was recently given, that I could use where the only thing it would be used for is building ports. Or I could run in my much faster server, a FreeNAS server with 32GB of RAM and a Xeon E3, but it would be run in a jail alongside several other jails I use for other things. Or, would I be better off just running synth on the system I want to update itself? What would you recommend?
Click to expand...

So do I. See https://forums.freebsd.org/threads/47319/

I would say your HP470, as dear as it is to you and me will be slowwwwwwwww for ports building efforts. I use it as a miniNAS (running FreeBSD) and I use net-p2p/btsync to sync my files with my main FreeBSD based NAS. I do have Synth on it, but only for keeping it up to date. For bigger port building efforts I currently use a Dell PowerEdge 1950 ( see https://forums.freebsd.org/threads/54413/ )and it works real well with FreeBSD on bare metal.
 
PacketMan said:
So do I. See https://forums.freebsd.org/threads/47319/

I would say your HP470, as dear as it is to you and me will be slowwwwwwwww for ports building efforts. I use it as a miniNAS (running FreeBSD) and I use net-p2p/btsync to sync my files with my main FreeBSD based NAS. I do have Synth on it, but only for keeping it up to date. For bigger port building efforts I currently use a Dell PowerEdge 1950 ( see https://forums.freebsd.org/threads/54413/ )and it works real well with FreeBSD on bare metal.
Click to expand...
When you say slow, how slow exactly do you mean? To compile all the ports I might use on a desktop, what timeframe would I be looking at?

Also, is it possible to use a combination ports from a private repository and the official repositories? So only compile the parts that aren't available in the official repository. Then tell my computer to look to my private repo, then if its not there fallback to the official repo.
 
STREBLO said:
To compile all the ports I might use on a desktop, what timeframe would I be looking at?
Click to expand...

Your wording isn't very clear. We don't know what ports you would personally use on a desktop, and it's probably a bad idea to try and build all the ports you might use on a desktop. :P But compiling is a very resource-intensive process, particularly on the CPU. If you're using a low-powered machine to build an entire desktop environment and complementary ports, it's not unusual for it to take a full 24 hours or more.

STREBLO said:
Also, is it possible to use a combination ports from a private repository and the official repositories? So only compile the parts that aren't available in the official repository.
Click to expand...

Yes, Synth can do this. It will build only those ports that (a) you customize yourself, and (b) are dependencies of those customized ports which, due to your own customizations, must then be custom-built themselves. For everything else, Synth will simply download the packages from the repo, place them in your local repository, and add them to your local repository's database. See the synth configure section of the man page.
 
ANOKNUSA said:
But compiling is a very resource-intensive process, particularly on the CPU. If you're using a low-powered machine to build an entire desktop environment and complementary ports, it's not unusual for it to take a full 24 hours or more.
Click to expand...
I just built a ports list essentially including kde4, libreoffice and chromium on an older desktop machine using poudriere: 51 hours. This is just for testing, I'll soon have my new server for such things ;) So yes, it takes a very long time. It's probably a bit faster with synth (without the jails overhead, preparing them here, especially without ZFS, takes some time) but I wouldn't expect a huge difference.
 
Zirias said:
I just built a ports list essentially including kde4, libreoffice and chromium on an older desktop machine using poudriere: 51 hours. This is just for testing, I'll soon have my new server for such things ;) So yes, it takes a very long time. It's probably a bit faster with synth (without the jails overhead, preparing them here, especially without ZFS, takes some time) but I wouldn't expect a huge difference.
Click to expand...

Jail runtime overhead is totally negligible and synth uses the same kind of techniques anyway, setting up chroot(8) is about as fast as jail cloning with ZFS and mount_nullfs(5) works the same on both. Where poudriere loses is when you are not using ZFS, the jail cloning becomes an expensive operation because it's done with cpdup(1).
 
kpa said:
Jail runtime overhead is totally negligible and synth uses the same kind of techniques anyway, setting up chroot(8) is about as fast as jail cloning with ZFS and mount_nullfs(5) works the same on both. Where poudriere loses is when you are not using ZFS, the jail cloning becomes an expensive operation because it's done with cpdup(1).
Click to expand...
This is actually not true. Chroots are much, much faster than using jails. We have anecdotes that using Synth cuts 25% off the time. Earlier in this thread, someone reported a job that reproducibly takes 24 hours on poudriere only takes 18 hours on synth. The same exact task. Others have reported noticibly shorter build times.

Once the jails are up, operations are similar, but the jail starting/stopping, even with persistence, costs significant time.
 
Maybe, Synth is faster with the same number of packages, but this does not matter, cause synth compiles everytimes more packages than poudriere.
 
talsamon said:
Maybe, Synth is faster with the same number of packages, but this does not matter, cause synth compiles everytimes more packages than poudriere.
Click to expand...
This is a false statement. Both programs use the exact same logic to determine rebuilding and both are equally aggressive at removing packages for rebuilding. Your statement is simply incorrect.
 
You are wrong. If you continue to disagree, please show the code and not anecdotes to prove your point. Before you start, note that I am very well informed on poudriere code too (I've contributed to the code base).
 
I have not looked in the poudriere code, I had made a quick look in the synth code. But it does not mater, I only know synth had made problems, that poudriere never does. (I have give up to work with synth a month ago, after it deletes 30 packages from the system without reason - someone on pipermail reported synth has deleted 290 packages on his system).
 
You are making a allegation, proof matters. Since you have no data backing up your statement, can you just not comment anymore? I thought you stopped using synth a long time ago. That "pipermail" post was also full of issues and missing tons of details, so no one can make conclusions (note that nobody else is providing confirming stories or data). I responded to that person and he never responded back even though 5 people were participating in the conversation.

Bottom line: Your stories are worthless. it's not like you have two machines identically set up and you make the same exact request to both and compare. You're just making your own conclusions with limited (and inaccurate) data and assumptions.

I get you don't like Synth -- but others do so just let others answer the questions instead of spreading false information. Thank you.
 
STREBLO said:
When you say slow, how slow exactly do you mean? To compile all the ports I might use on a desktop, what timeframe would I be looking at?
Click to expand...

Based on what I know, a x11/gnome3 desktop, not counting any other applications (ports) you have installed I would say take easy 48 hours at least on a HP470, maybe even closer to 72 hours after you toss in a few ports. Use it for what it was intended (a NAS) and you should love FreeBSD on it. Using it as a builder box and it will take quite some time.
 
Last edited by a moderator:
I'm trying to set up a custom repo built with Synth which is available to remove machines. I followed this guide for poudriere: https://www.digitalocean.com/commun...m-to-create-packages-for-your-freebsd-servers. I can see the list of live_packages on the client in a browser. But when I try to use pkg I get the following error:

Code: 
root@bclinton:~# pkg update
Updating bobama repository catalogue...
Repository bobama has a wrong packagesite, need to re-create database
pkg: http:///bobama:8080/live_packages/meta.txz: No address record
repository bobama has no meta file, using default settings
pkg: http:///bobama:8080/live_packages/packagesite.txz: No address record
Unable to update repository bobama


/usr/local/etc/nginx/nginx.conf
Code: 
root@bobama:~# cat /usr/local/etc/nginx/nginx.conf
load_module modules/ngx_mail_module.so;
load_module modules/ngx_stream_module.so;

worker_processes  1;

events {
  worker_connections  1024;
}

http {
  include  mime.types;
  default_type  application/octet-stream;

  sendfile  on;
  keepalive_timeout  65;

  server {
  listen  8080;
  server_name  192.168.1.6;

  location / {
  root  /usr/local/www/nginx;
  index  index.html index.htm;
  }

  location /live_packages {
  root  /var/synth/;
  autoindex on;
  }

  error_page  500 502 503 504  /50x.html;
  location = /50x.html {
  root  /usr/local/www/nginx-dist;
  }
  }
}

/usr/local/etc/pkg/repos/bobama.conf
Code: 
root@bclinton:~# cat /usr/local/etc/pkg/repos/bobama.conf
bobama: {
  url  : http:///bobama:8080/live_packages,
  mirror_type: "http",
  enabled  : yes,
}

All other pkg repos are disabled.
 
Last edited by a moderator:
google FreeBSD + "no address record". I might be something misconfigured in your /etc/resolv.conf file. It's not really a synth issue and barely a pkg(8) issue, it's a generic network configuration issue I think.
 
I searched and didn't find anything relevant. I'm not an expert in networking, but everything else network-related works just fine on my 2 computers, except pkg + custom repo. So I changed sharing of my custom repo from HTTP to NFS + file. Worked without any problem.
 
Have a strange failure when building news/pan with option NLS off:

Code: 
synth prepare-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Stand by, updating external repository catalogs ... done.
Scanning existing packages.
pan-0.140.txz NLS is ON but port says it must be OFF
00:00:17 => [02] Builder launched
00:00:17 => [04] Builder launched
00:00:17 => [01] Builder launched
00:00:17 => [03] Builder launched
00:00:17 => [02]  Shutting down
00:00:17 => [04]  Shutting down
00:00:17 => [03]  Shutting down
piped_mute_command failure:
=> umount: unmount of /usr/obj/synth-live/SL01/construction failed: Device busy

piped_mute_command failure:
=> umount: unmount of /usr/obj/synth-live/SL01/dev failed: Device busy

piped_mute_command failure:
=> umount: unmount of /usr/obj/synth-live/SL01/bin failed: Device busy

piped_mute_command failure:
=> umount: unmount of /usr/obj/synth-live/SL01 failed: Device busy

piped_mute_command failure:
=> rm: /usr/obj/synth-live/SL01/bin/pax: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/echo: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/hostname: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/cp: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/cat: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/pwait: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/[: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/realpath: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/rcp: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/chmod: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/rm: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/chflags: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/sync: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/dd: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/sleep: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/kill: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/chio: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/ln: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/getfacl: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/red: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/setfacl: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/csh: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/pgrep: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/stty: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/unlink: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/kenv: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/rmdir: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/ls: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/ed: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/ps: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/expr: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/date: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/tcsh: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/domainname: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/uuidgen: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/rmail: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/sh: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/pwd: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/pkill: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/mkdir: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/test: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/df: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/mv: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/link: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/freebsd-version: Read-only file system
rm: /usr/obj/synth-live/SL01/bin: Device busy
rm: /usr/obj/synth-live/SL01/dev/reroot: Operation not supported
rm: /usr/obj/synth-live/SL01/dev/fd: Operation not supported
rm: /usr/obj/synth-live/SL01/dev/led: Operation not supported
rm: /usr/obj/synth-live/SL01/dev/usb: Operation not supported
rm: /usr/obj/synth-live/SL01/dev/msdosfs: Operation not supported
rm: /usr/obj/synth-live/SL01/dev/gpt: Operation not supported
rm: /usr/obj/synth-live/SL01/dev/pts: Operation not supported
rm: /usr/obj/synth-live/SL01/dev: Device busy
rm: /usr/obj/synth-live/SL01/construction: Device busy
rm: /usr/obj/synth-live/SL01: Device busy

00:00:50 => [01] 00:00:32 Failure news/pan
00:00:50 => [01]  Shutting down



The task is complete.  Final tally:
Initial queue size: 1
  packages built: 0
  ignored: 0
  skipped: 0
  failed: 1

Duration: 00:00:49
The build logs can be found at: /var/log/synth
Stand by, prescanning existing packages.
Stand by, recursively scanning 879 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt

/usr/local/etc/synth/synth.ini:
Code: 
; This Synth configuration file is automatically generated
; Take care when hand editing!

[Global Configuration]
profile_selected= LiveSystem

[LiveSystem]
Operating_system= FreeBSD
Directory_packages= /data/live_packages
Directory_repository= /data/live_packages/All
Directory_portsdir= /usr/ports
Directory_options= /var/db/ports
Directory_distfiles= /usr/ports/distfiles
Directory_buildbase= /usr/obj/synth-live
Directory_logs= /var/log/synth
Directory_ccache= /root/.ccache
Directory_system= /
Number_of_builders= 4
Max_jobs_per_builder= 4
Tmpfs_workdir= true
Tmpfs_localbase= true
Display_with_ncurses= false
leverage_prebuilt= true

System:
Code: 
uname -a
FreeBSD bobama 10.3-RELEASE FreeBSD 10.3-RELEASE #0 r297264: Fri Mar 25 02:10:02 UTC 2016  root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64

Added static libs to news/pan to fix startup crash:
Code: 
root@bobama:pan# svn diff
Index: Makefile
===================================================================
--- Makefile   (revision 414425)
+++ Makefile   (working copy)
@@ -21,7 +21,7 @@
USE_GCC=   any
GNU_CONFIGURE=   yes
CPPFLAGS+=   -I${LOCALBASE}/include
-LDFLAGS+=   -L${LOCALBASE}/lib -lgnuregex ${ICONV_LIB}
+LDFLAGS+=   -L${LOCALBASE}/lib -lgnuregex ${ICONV_LIB} -static-libgcc -static-libstdc++
OPTIONS_DEFINE=   GTKSPELL GNUTLS NLS
OPTIONS_RADIO=   GTK
 
Is it reproducible? If you reboot your computer and try just to build pan separately ( synth force news/pan), does it still happen?
 
You must log in or register to reply here.
Back
Top