ports-mgmt/synth: Upgrade version 1.54 => 1.60

marino

Daemon

Reaction score: 436
Messages: 1,091

ports-mgmt/synth: Upgrade version 1.54 => 1.60
Code:
Major new feature: Web-based front-end

  A dynamic html report is now generated for each build.  The report is
  placed at <profile log directory>/Report/index.html.

  The report is updated 10 times per minute.  The entire build history
  is retained and is searchable, but that history is reset at the start
  of the next run.

  In addition, several fields are clickable and trigger a quick-filter
  on the history log including the "Built", "Failed", "Ignored",
  "Skipped", "No." column and "ID" column (see tiptool for action hint)
  and the "Total" field clears the search pattern quickly.

  An earlier but representative version of the web report can be seen at:
  https://github.com/jrmarino/synth#web-interface

  The Synth web report pulls 2 orders of magnitude less data than the
  equivalent poudriere report because the json data is split into
  multiple files limited to 500 ports each (e.g. the poudriere data file
  might be 1.8 Mb on a big bulk run polled every 8 seconds where as
  Synth only pulls the latest segment file (50kb avg) every 6 seconds.
  Synth also stops polling when the run is complete.

Minor enhancement: The log files were cleaned up

  The data/phase headers and footers were modified so that the build
  log appears a lot cleaner now (subjective of course).


This functionality has been requested a few times and the initial implementation has been well received. Hopefully you guys will find it useful as well.
 

kpa

Beastie's Twin

Reaction score: 1,806
Messages: 6,318

He is not including a web server into Synth, you're supposed to set up one yourself just like you would with Poudriere.
 

jrm@

Daemon
Developer

Reaction score: 473
Messages: 1,205

marino@, the interface looks nice. Off-topic question: Why did the build of editors/emacs fail?
 
OP
OP
marino

marino

Daemon

Reaction score: 436
Messages: 1,091

He is not including a web server into Synth, you're supposed to set up one yourself just like you would with Poudriere.
It's actually a good question.
I had intended to create 2 versions:
  1. a "dumb" version (this one)
  2. a "slick" version using websockets, basically real time. There are 2-3 possible implementations for this (e.g. aws, simple components, roll-your-own). It would be its own web server, there would need to be a configuration option for the listening port and perhaps ipv4 vs ipv6 and max connections though.

However, given how nice the "dumb" version is already, I may never get around to making the slick version.
 
OP
OP
marino

marino

Daemon

Reaction score: 436
Messages: 1,091

marino@, the interface looks nice. Off-topic question: Why did the build of editors/emacs fail?
emacs got updated to the emacs-devel without me realizing it and the patches in dports needed to be copied from emacs-devel back to emacs (along with some tweaking to Makefile.DragonFly). It was definitely DragonFly specific issue.
 
OP
OP
marino

marino

Daemon

Reaction score: 436
Messages: 1,091

Code:
ports-mgmt/synth: Upgrade version 1.60 => 1.61

Address "[N] Fetch prebuilt packages" bug:

  Once the /usr/local/etc/pkg/repos/00_synth.conf file was created, there
  was a good change that fetching prebuilt packages would stop with a
  fetch error that the checksum didn't match.  This seems to be caused by
  pkg(8) using both the FreeBSD and the Synth repositories and often
  deferring to the latter.  This is incorrect behavior as only the FreeBSD
  repository should be used for prefetching.  This issue is hopefully
  resolved now.

Web report behavior change:

  Any existing ??_history.json files are now deleted at the start of
  each run.  They were only litter as the javascript did not pull in old
  files, so this change just removes them.

  While here, rename the "Elapsed" column on the builders table to
  "Duration" to match the history table lable and also adjust the widths
  of the origin and information columns on the history table.
Those using the "fetch prebuilt packages" option will want this update.
 
  • Thanks
Reactions: ASX
A

ASX

Guest


Those using the "fetch prebuilt packages" option will want this update.
It is nice to find the answer before posting the question! I was experiencing strange results from "synth status", in that there was a disagreement between portupgrade -a -n looking to rebuild 5 pkgs, and synth status looking to rebuild 144 pkgs. synth-1.61 appear to have solved the problem. Thanks!
 

xtaz

Well-Known Member

Reaction score: 109
Messages: 411

Just seen the web front end. Nice work! I have to ask now though. In the curses front end the skipped and swap labels are in lower case compared to the other labels. I always figured this was because you were infering that they were lower priority or something. But in the web front end they are all capitalised. Minor typos or intentional? :)
 

xtaz

Well-Known Member

Reaction score: 109
Messages: 411

Heh. I guess you're so used to such things that you become blind to them. I've also noticed this whilst I am using the web front end in my nginx error log. I guess it's trying to read the json file whilst it's being written, or emptied or something? Not sure if there's anything you can do there.

Code:
2016/10/18 21:25:50 [alert] 27076#0: *8402 pread() read only 732 of 736 from "/var/log/synth/Report/summary.json" while sending response to client, client: 10.0.0.10, server: example.com, request: "GET /synth/Report/summary.json?_=1476821965721 HTTP/2.0", host: "example.com", referrer: "https://example.com/synth/Report/"
2016/10/18 21:25:53 [alert] 27076#0: *8402 pread() read only 734 of 736 from "/var/log/synth/Report/summary.json" while sending response to client, client: 10.0.0.10, server: example.com, request: "GET /synth/Report/summary.json?_=1476821965722 HTTP/2.0", host: "example.com", referrer: "https://example.com/synth/Report/"
2016/10/18 21:25:56 [alert] 27076#0: *8402 pread() read only 702 of 736 from "/var/log/synth/Report/summary.json" while sending response to client, client: 10.0.0.10, server: example.com, request: "GET /synth/Report/summary.json?_=1476821965723 HTTP/2.0", host: "example.com", referrer: "https://example.com/synth/Report/"
2016/10/18 21:26:11 [alert] 27076#0: *8402 pread() read only 699 of 702 from "/var/log/synth/Report/summary.json" while sending response to client, client: 10.0.0.10, server: example.com, request: "GET /synth/Report/summary.json?_=1476821965728 HTTP/2.0", host: "example.com", referrer: "https://example.com/synth/Report/"
2016/10/18 21:26:14 [alert] 27076#0: *8402 pread() read only 699 of 702 from "/var/log/synth/Report/summary.json" while sending response to client, client: 10.0.0.10, server: example.com, request: "GET /synth/Report/summary.json?_=1476821965729 HTTP/2.0", host: "example.com", referrer: "https://example.com/synth/Report/"
2016/10/18 21:26:38 [alert] 27076#0: *8402 pread() read only 681 of 701 from "/var/log/synth/Report/summary.json" while sending response to client, client: 10.0.0.10, server: example.com, request: "GET /synth/Report/summary.json?_=1476821965737 HTTP/2.0", host: "example.com", referrer: "https://example.com/synth/Report/"
 
OP
OP
marino

marino

Daemon

Reaction score: 436
Messages: 1,091

I guess so. I checked the nginx error log on the DF build server and there's 1 of those for summary.json and a few of them for history.json. I guess the file is getting ripped out from under the web server and rewritten.
 
OP
OP
marino

marino

Daemon

Reaction score: 436
Messages: 1,091

Heh. I guess you're so used to such things that you become blind to them. I've also noticed this whilst I am using the web front end in my nginx error log. I guess it's trying to read the json file whilst it's being written, or emptied or something? Not sure if there's anything you can do there.
Code:
ports-mgmt/synth: Upgrade version 1.61 => 1.62

Cosmetic: curses display

  Capitalize "Swap" and "Skipped" field labels

Cosmetic: web report

  After the build completes, fade out the builders table in 2.5 seconds.
  Nobody needs to see a table showing all the builders in "shutdown" mode.

Bandwidth: json files

  A history.json file holds up to 500 log entries.  16 bytes of spaces were
  removed which reduces the history file size by 8000 bytes.  Spaces were
  also removed from summary.json which can account for 20%+ of the size.

Only rewrite history files upon change:

  The last history.json file was being written/rewritten every 3 seconds
  regardless if there were changes to it or not.  Now it's only written
  if it's changed since the previous write.  This might help reduce errors
  seen by in nginx access log which are thought to occur when file changes
  as the file is being served.
xtaz@,
Version 1.62 fixes the display and now Synth significantly reduces the amount of times it rewrites the latest history file. That should reduce the number of nginx errors by a lot (but it won't eliminate them).
 
A

ASX

Guest


May be I spoke too soon. A little Intro:
On my system I'm using practically only pkgs, with two exceptions:
- packages marked as RESTRICTED, like astro/google-earth and audio/lame
- packages coming from outside FreeBSD repos, (specifically from GhostBSD project).

In this moment there are 778 pkgs installed on my system, and pkg say there are no new pkgs.

portupgrade -a -n also report no new packages:
# portupgrade -a -n
---> Session started at: Thu, 20 Oct 2016 21:38:20 +0200
[Reading data from pkg(8) ... - 778 packages found - done]
** Port directory not found: sysutils/ghostbsd-grub2-settings
** Port directory not found: x11-themes/ghostbsd-icons
** Port directory not found: x11/ghostbsd-installed-common-settings
** Port directory not found: x11-themes/ghostbsd-mate-themes
** Port directory not found: x11/ghostbsd-slim-theme
** Port directory not found: x11-themes/ghostbsd-wallpapers
** Port directory not found: sysutils/inxi
** Port directory not found: net-mgmt/networkmgr
** Port directory not found: devel/ssft
** Port directory not found: x11/xfce-installed-settings
---> Listing the results (+:done / -:ignored / *:skipped / !:failed)
- sysutils/ghostbsd-grub2-settings (port directory error)
- x11-themes/ghostbsd-icons (port directory error)
- x11/ghostbsd-installed-common-settings (port directory error)
- x11-themes/ghostbsd-mate-themes (port directory error)
- x11/ghostbsd-slim-theme (port directory error)
- x11-themes/ghostbsd-wallpapers (port directory error)
- sysutils/inxi (port directory error)
- net-mgmt/networkmgr (port directory error)
- devel/ssft (port directory error)
- x11/xfce-installed-settings (port directory error)
---> Packages processed: 0 done, 10 ignored, 0 skipped and 0 failed
---> Session ended at: Thu, 20 Oct 2016 21:39:28 +0200 (consumed 00:01:07)
portmaster -a -n also report similar results:
===>>> All ports are up to date

synth is configured to fetch packages, and report different figures:
Total packages that would be fetched: 902
The complete build list can also be found at:
/tmp/synth_prefetch_list.txt
These are the ports that would be built ([N]ew, [R]ebuild, [ U]pgrade):
N => astro/google-earth
Total packages that would be built: 1
The complete build list can also be found at:
/tmp/synth_status_results.txt
Now, I'm fairly new to ports and ports management but I would like to understand why the different results, or if I'm doing something wrong:

google-earth has been patched, because as available in ports it didn't worked .. and that could be the reason why synth status report about, even if there is not a new version of google-earth in ports.

The total number of packages to be fetched is larger than the number of installed packages, I suspect synth want fetch also build-dependencies, if so it is probably an unwanted side effect.
 
OP
OP
marino

marino

Daemon

Reaction score: 436
Messages: 1,091

ASX@ It looks correct to me.
Remember what Synth is trying to do: It's trying to create a local repository that your localhost system needs. You start with zero packages. When you run synth status, it automatically adds those 778 packages to the build list. It would build them all except you're fetching prebuilt packages so it wants to fetch them instead. You could fetch 900 packages and pkg(8) wouldn't upgrade any of them. It's just creating the baseline repository that matches your current up-to-date installed packages.

You could run the same command twice. The first time it fetches 900 packages and builds 1, the second time it does nothing because it's up to date.

It is fetching build dependencies. Remember its trying to build all of them, so the initial list is the full set of ports required to accomplish that. It then checks each one and sees if it could use the prebuilt version. You're right in the observation that approximately 100 packages are getting downloaded that will not be installed so I guess it is a "side effect" but it's definitely expected. To prune those would require some kind of additional pass to determine that A) the end package is getting downloaded and B) the dependency package isn't a LIB or RUNS depends for anything else. It's not impossible but it is a bit complex.
 
  • Thanks
Reactions: ASX
A

ASX

Guest


My target is to be able to build the packages, configured to fit my goals, so this is not a problem, also you have now clarified those additional packages will not be installed and I was just unsure about that.

Your answer also imply that google-earth is being rebuilt for the same reason, it is not already in the local repository, for those reading audio/lame actually IS in the local repository and is not listed in the packages to be rebuilt.

Thanks for clarifying.
 
OP
OP
marino

marino

Daemon

Reaction score: 436
Messages: 1,091

Correct. Basically pkg(8) has two repositories to choose from. Say you have package "X" of version y.z installed. Both repositories have "X" version y.z in the them (the same). When you run pkg upgrade command, pkg(8) checks both repositories, sees that the installed version is the same as what's in the repository so that it considers package "X" to be up-to-date and there will be no attempt to reinstall it. It's the same case for 777 packages assuming the version numbers between installed and repository match.
 

freebuser

Member

Reaction score: 5
Messages: 97

Recently I noticed synth didn't count a PORTREVISION as a port which requires build ( OpenSSL from memory which has had a security patch added).

Has this been resolved?
 
OP
OP
marino

marino

Daemon

Reaction score: 436
Messages: 1,091

That's never been the case. You must have been mistaken about the case in question. A portrevision bump causes a version mismatch against the existing package and that's always been detected.
 

freebuser

Member

Reaction score: 5
Messages: 97

It definitely happened to me. When I checked out the updated ports tree with the patched version synth didn't detect the update. I had to delete the packages for synth to rebuild the updated port.
There may be other reasons but the one I noticed was the portrevision.
 
OP
OP
marino

marino

Daemon

Reaction score: 436
Messages: 1,091

AFAIK, what you assert isn't possible. the version comparison works. There's no way that synth would fail to detect a difference in versions between the port and the package. whatever you saw, there was another explanation. FYI, nobody has mentioned anything like this until now.
 
A

ASX

Guest


synth has worked well here, and I managed to fetch / rebuild all packages installed on my system. I did this a few at a time, using several of the available commands (build, just-build, status ...), only for my own knowledge.

When using synth rebuild-repository I got the following message:
Note: This system's pkg(8) is configured with CONSERVATIVE_UPGRADE = true
You may wish to toggle that setting if this is a local repository.
I wasn't aware of that option until then, which I needed to set to "false". May be a quick note about that in synth man page would be useful, that is for ports newbies like me, of course.
 
OP
OP
marino

marino

Daemon

Reaction score: 436
Messages: 1,091

That's the purpose of the check that Synth just did for you. The pkg(8) setting doesn't come into play until there's a synth repository, so checking for it when the repo is built means you were advised in time. I would also say "need" is strong. I think most people would want the setting to be false but not necessarily everyone (especially those building a repository for multiple machines). Also, if all the packages are removed from the system and then reinstalled via the Synth repository, the setting doesn't even matter. It only comes into play when packages already exist (which is the usual case) but over time most will get replaced by the Synth versions.
 
  • Thanks
Reactions: ASX
A

ASX

Guest


That's the purpose of the check that Synth just did for you
And I'm thankful for that, yet, I felt like I did something wrong ... anyway, I was not suggesting that Synth man page should direct people to set the flag to false, only to suggest that CONSERVATIVE_UPGRADE flag exists, point being "I was not aware of that option".
 
Top