Synth 1.30 has been released

Now this makes no sense. If I want to disable a default in a port config, or if I have to enable a different setting in a port, then config files are absolutely needed. If I didn't change the configs, I'd rely solely on pkg and not even bother with Synth.

These replies are crazy long.
Okay, let's say you have 500 ports installed.
you didn't customize 500 ports. You might have customized 8 ports, 8 of 500.
So you need 8 customized options. You have 492 ports wthout options.
A port without options never has a bad config. A port with cached options can go invalid with changes to the tree. ergo, a ports wthout config files are in a better state.




Even if I'm not describing the process properly, Synth is clearly doing something wrong here. Here's another example, and I'll try to make it as clear as possible.

ddclient-3.8.2_1.txz failed option check.
progress: 86.72%

why do you think the failed check is wrong?

So dns/ddclient failed the option check when I disabled its one option, and the package should have been rebuilt with the new options selected (or deselected in this case). But the newly-built package is missing from the repository and cannot be installed by pkg

hmmm? it's not missing, it got deleted.

This, as I understand it, is wrong. Why is the package now missing from the repository? Why do I have to run Synth twice to get the package file back in the repo and ready to be installed?

It just deleted, you see it on the screen.

I'm not understanding your point. You change the port options which immediately invalidate the package, which gets removed because of the invalidation. When you build it a second time, the package's stored options match the current ports tree so it passes the validation check.
 
why do you think the failed check is wrong?
It's not wrong. That's not the problem. Synth correctly detects the options have changed and deletes the old package file. That's fine.

What's not fine is that Synth should then rebuild the package with the new option settings on the first invocation of synth prepare-system. Yes, it should delete the old package file, because those options are now outdated, but then it proceeds to rebuild the package using the new option settings, but doesn't actually repopulate the repo with the new package file it just built a few seconds ago! The newly-built package just disappears!

I'm not understanding your point. You change the port options which immediately invalidate the package, which gets removed because of the invalidation. When you build it a second time, the package's stored options match the current ports tree so it passes the validation check.
OK, here's my point. The deletion is fine. When I run synth prepare-system after changing a port option, it should do two things: (1) Delete the old package with the outddated option setting, (2) Build the new package using the new option settings and place the new package file in the repo. Running synth prepare-system once should accomplish both steps. But it doesn't. Only (1) gets completed. I have to run synth prepare-system a second time to complete (2). Are you saying that's by design?
 
i just looked at your dns_ddclient client example again. The output isn't matching what you describe.
  1. you ran synth prepare-system
  2. It deleted the bad dns_ddclient package
  3. It rebuilt the package
  4. It rebuilt the repository
One pass. It did 1) and 2). Correct?
 
i just looked at your dns_ddclient client example again. The output isn't matching what you describe.
  1. you ran synth prepare-system
  2. It deleted the bad dns_ddclient package
  3. It rebuilt the package
  4. It rebuilt the repository
One pass. It did 1) and 2). Correct?
No, because the repository was missing the new dns/ddclient package file after the first invocation of synth prepare-system completed. Here, I'll show you again by starting from scratch. Bear with me on this, please -- I know this is long, but please follow along through all steps to reproduce:

Getting rid of dns/ddclient:

Code:
root@wintermute-pts/2(3:13pm)~/:pkg delete ddclient
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):

Installed packages to be REMOVED:
  ddclient-3.8.2_1

The operation will free 178 KiB.

Proceed with deinstalling packages? [y/N]: y
[1/1] Deinstalling ddclient-3.8.2_1...
[1/1] Deleting files for ddclient-3.8.2_1: 100%
root@wintermute-pts/2(3:48pm)~/:rm /var/synth/live_packages/All/ddclient-3.8.2_1.txz
remove /var/synth/live_packages/All/ddclient-3.8.2_1.txz? y
/var/synth/live_packages/All/ddclient-3.8.2_1.txz
root@wintermute-pts/2(3:48pm)~/:make -C /usr/ports/dns/ddclient/ rmconfig
===> Removing user-configured options for ddclient-3.8.2_1
root@wintermute-pts/2(3:49pm)~/:
The package is now removed from the system, the options file has been deleted, and I manually deleted the package file from the repository.

Now we reinstall with synth install dns/ddclient:

Code:
root@wintermute-pts/2(3:49pm)~/:synth install dns/ddclient



The task is complete.  Final tally:
Initial queue size: 1
  packages built: 1
  ignored: 0
  skipped: 0
  failed: 0

Duration: 00:00:10
The build logs can be found at: /var/log/synth
Stand by, prescanning existing packages.
Stand by, recursively scanning 246 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
Updating Synth repository catalogue...
Fetching meta.txz: 100%  264 B  0.3kB/s  00:01
Fetching packagesite.txz: 100%  71 KiB  72.2kB/s  00:01
Processing entries: 100%
Synth repository update completed. 246 packages processed.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
  ddclient: 3.8.2_1

The process will require 178 KiB more space.
[1/1] Installing ddclient-3.8.2_1...
[1/1] Extracting ddclient-3.8.2_1: 100%
Message from ddclient-3.8.2_1:
***********************************************************

Copy
  /usr/local/etc/ddclient.conf.sample
to
  /usr/local/etc/ddclient.conf

and edit it to fit your needs.

If you would like to run ddclient as a daemon add the
following line to /etc/rc.conf

  ddclient_enable="YES"

If you would like to force ddclient to update your account
daily regardless of IP changes add the following line to
your /etc/periodic.conf

  daily_ddclient_force_enable="YES"

***********************************************************
root@wintermute-pts/2(3:50pm)~/:ll /var/synth/live_packages/All/ddclient*
-rw-r--r--  1 root  wheel  40640 Mar 19 15:50 /var/synth/live_packages/All/ddclient-3.8.2_1.txz
root@wintermute-pts/2(3:51pm)~/:
The package is now installed, and ddclient-3.8.2_1.txz is in the repo.

Time to change the option:

Code:
root@wintermute-pts/2(3:51pm)~/:make -C /usr/ports/dns/ddclient config

root@wintermute-pts/2(3:52pm)~/:cat /var/db/ports/dns_ddclient/options
# This file is auto-generated by 'make config'.
# Options for ddclient-3.8.2_1
_OPTIONS_READ=ddclient-3.8.2_1
_FILE_COMPLETE_OPTIONS_LIST=SSL
OPTIONS_FILE_UNSET+=SSL
root@wintermute-pts/2(3:53pm)~/:
The option is now changed. Now we run synth prepare-system to see if both (1) and (2) above are completed in one run:

Code:
root@wintermute-pts/2(3:53pm)~/:synth prepare-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
ddclient-3.8.2_1.txz failed option check.
progress: 89.93%


The task is complete.  Final tally:
Initial queue size: 1
  packages built: 1
  ignored: 0
  skipped: 0
  failed: 0

Duration: 00:00:13
The build logs can be found at: /var/log/synth
Stand by, prescanning existing packages.
Stand by, recursively scanning 246 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
root@wintermute-pts/2(3:54pm)~/:ll /var/synth/live_packages/All/ddclient*
ls: No match.
root@wintermute-pts/2(3:55pm)~/:
And we see that only (1) happened -- the old package was deleted, but the new package was not put in the repo -- the new version of ddclient-3.8.2_1.txz is not in /var/synth/live_packages/All. But we did build it during this run -- note that "packages built: 1" is in the final tally. But where's the new package I just built? We have to run synth prepare-system a second time for the new package to appear in the repo:

Code:
root@wintermute-pts/2(3:56pm)~/:synth prepare-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
progress: 85.50%


The task is complete.  Final tally:
Initial queue size: 1
  packages built: 1
  ignored: 0
  skipped: 0
  failed: 0

Duration: 00:00:13
The build logs can be found at: /var/log/synth
Stand by, prescanning existing packages.
Stand by, recursively scanning 246 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
root@wintermute-pts/2(4:02pm)~/:ll /var/synth/live_packages/All/ddclient*
-rw-r--r--  1 root  wheel  40592 Mar 19 16:01 /var/synth/live_packages/All/ddclient-3.8.2_1.txz
root@wintermute-pts/2(4:02pm)~/:
The package was rebuilt again (packages built: 1), but now we see (finally!) that (2) has been completed -- the new package is in the repo and is ready to install. I shouldn't have to run synth prepare-system twice for that to happen, right?
 
And we see that only (1) happened -- the old package was deleted, but the new package was not put in the repo -- the new version of ddclient-3.8.2_1.txz is not in /var/synth/live_packages/All. But we did build it during this run -- note that "packages built: 1" is in the final tally. But where's the new package I just built?

That's what I was saying. Something was built, I assume ddclient in this case, so
/var/synth/live_packages/All/ddclient-3.8.2_1.txz should exist at this point.

It also should be accounted for after pkg repo command completes.


We have to run synth prepare-system a second time for the new package to appear in the repo:

Code:
root@wintermute-pts/2(3:56pm)~/:synth prepare-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
progress: 85.50%
The task is complete.  Final tally:
Initial queue size: 1
  packages built: 1

What was just built? ddclient should already be present so nothing should be rebuilt.
We didn't see it deleted before the repo build command.

The package was rebuilt again (packages built: 1), but now we see (finally!) that (2) has been completed -- the new package is in the repo and is ready to install. I shouldn't have to run synth prepare-system twice for that to happen, right?

no, you should not have to do that. The question is did the package from the first build go? it clearly built and was not deleted so it should still be there after the first run.[/code]
 
That's what I was saying. Something was built, I assume ddclient in this case, so
/var/synth/live_packages/All/ddclient-3.8.2_1.txz should exist at this point.

It also should be accounted for after pkg repo command completes.
That's the bug I'm trying to report -- it doesn't exist at this point! Look at the bottom of my log after the first time I run synth prepare-system: the file /var/synth/live_packages/All/ddclient-3.8.2_1.txz does not exist!

What was just built? ddclient should already be present so nothing should be rebuilt. We didn't see it deleted before the repo build command.

dns/ddclient was built a second time.

no, you should not have to do that. The question is did the package from the first build go? it clearly built and was not deleted so it should still be there after the first run.
Right, but it's not there!

I'm not sure how else to explain this now. Would you please try to reproduce the problem using the steps I outlined and let me know if the same thing happens to you?
 
Maybe tomorrow. it's late here and the machine is busy with something else.

To reword this, you're saying that the first build of ddclient fails to package. I don't see how that's possible if it's listed as a successful build. The only other possibility is that it does successfully package, but then is immediately deleted, silently, before the repo command is executed. Both are hard to believe.

After 1), the ddclient build log and the 00*.log doesn't have anything interesting in it?
 
After 1), the ddclient build log and the 00*.log doesn't have anything interesting in it?
I just restarted the whole process again, and both look normal after (1). Here's 00_last_results.log after the first synth prepare-system:

Code:
-=>  Chronology of last build  <=-
Started: Saturday, 19 MAR 2016 at 23:47:40 UTC
Ports to build: 241

Purging any ignored/broken ports first ...
00:00:05 Sanity check complete. Ports remaining to build: 1
00:00:05 [01] => dns/ddclient
00:00:14 dns/ddclient success
Finished: Saturday, 19 MAR 2016 at 23:47:54 UTC
Duration: 00:00:13

And after the second synth prepare-system:
Code:
-=>  Chronology of last build  <=-
Started: Saturday, 19 MAR 2016 at 00:05:11 UTC
Ports to build: 241

Purging any ignored/broken ports first ...
00:00:05 Sanity check complete. Ports remaining to build: 1
00:00:05 [01] => dns/ddclient
00:00:14 dns/ddclient success
Finished: Saturday, 19 MAR 2016 at 00:05:25 UTC
Duration: 00:00:13
 
I just restarted the whole process again, and both look normal after (1).

I was able to reproduce this using your test case. Indeed the newly-built package was immediately deleted and therefore not packaged. Thanks for being persistent. I think version 1.32 will resolve this for you.

Code:
ports-mgmt/synth: Upgrade version 1.31 => 1.32

prepare-system, upgrade-system package deletion bug:
  These two commands had a serious logic flaw that resulted in newly-built
  packages to be immediately deleted prior the pkg(8) repo command.  (And
  the obvious result of the system not getting updated without re-running
  the same command.)

  The cause was that the first package validity scan marked the package
  for deletion, and this setting was never reset.  The obsolete package
  was correctly deleted and rebuilt, but then it was quickly deleted
  again during the second package scan because the previous deletion mark
  was still present.  The fix was simple: Reset the deletion marks before
  rebuilding the repository so the second scan proceeds accurately.

  Test case provided by Wapcaplet in FreeBSD forums
 
I was able to reproduce this using your test case. Indeed the newly-built package was immediately deleted and therefore not packaged. Thanks for being persistent. I think version 1.32 will resolve this for you.
Yes, that fixed it. Thanks again for your help!
 
ports-mgmt/synth: Upgrade version 1.32 => 1.33
Code:
Prefetching option fix:
  Until now, if the "prefetch suitable packages" option was enabled, it
  could happen that "synth force" and "synth test" commands wouldn't do
  anything.  Moreover, even though it wouldn't build, it would delete the
  existing package and refetch it because force/test always pre-deletes
  existing packages.

  Now, ports listed with "force" and "test" commands are always built,
  even if suitable remote packages are available.

Improvements with test mode:
  1) Changes to /var/tmp are now ignored.  These are legal modifications
  2) Access time modification to the builder mount are now ignored.  This
     was possible if changes to /tmp or /var propagates upward (legally).
  3) A new environment variable is added for test command, "LOCK".  If it
     defined to any value, the builder's localbase will be remounted as
     read-only during the configure and build phases.  This is a
     diagnostic tool to help pinpoint where ports are committing file
     system violations by writing to localbase during these phases.

New notice to user regard CONSERVATIVE_UPGRADE:
  At the end of the "synth rebuild-repository" command, Synth will probe
  the host pkg(8) to see if it's configured with CONSERVATIVE_UPGRADE.
  If so, Synth will emit a notice because chances are good that the
  user built repository for the local system and may try to upgrade the
  system with pkg(8), and in this case, they may wish to turn off the
  CONSERVATIVE_UPGRADE setting.
 
Is this meant to happen now with this new version? It used to just say something about no packages needing to be upgraded so nothing to do. Now it exits with what looks like an error?

Code:
# synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Stand by, updating external repository catalogs ... done.
Scanning existing packages.
No packages qualify for prefetching from official package repository.
Unfortunately, the system upgrade failed.

Code:
# synth upgrade-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Stand by, updating external repository catalogs ... done.
Scanning existing packages.
Unfortunately, the system upgrade failed.
 
it looks like fixing one issue caused a regression.
Can you test something for me?
do a make clean; make patch in ports-mgmt/synth, then find src/portscan-pilot.adb in the $WRKDIR and remove lines 278-280, then build/reinstall. That should put it back to how it was. I was trying to make it not ask to rebuild repository if nothing built.
 
I went ahead and commited that patch to ports. Please let us know if that restores the previous behavior.
 
Apologies for me taking a while to test this. Been a bit busy today. I've just tested it with the same two commands as I mentioned in the post above. They both look good now. I get the original message about having nothing left to do. Thanks!
 
Back
Top