[...] My main question at this point would be: Where exactly is that notion of an "overlay" coming from? From my understanding of the overall system architecture this would just be an "additional" ports tree.
Maybe I'm missing the essence of your question but, if you see it as 'just be an "additional" ports tree' how would you easily integrate your private ports tree with the standard ports tree, other than via this overlay option? The
-O option enables you to append your private ports tree during a bulk build.
The principle of
Overlay (programming)* allows for working with multiple units or segments treated as one—more or less (restrictions apply). Overlay programming was needed when main memory is too small to have a program completely loaded; this was before the era of virtual memory. For environments without virtual memory management this still applies today, especially in embedded applications.
I don't know to what extent poudriere implements overlays however, I think I can say that you don't have to bother yourself with possible memory restrictions when adding one or more separate private ports trees as overlays.
A specification of overlays defines how segments of a program are related to each other and is usually passed to the linker; overlays can be defined recursively. The specification is used to allow for an overlay segment to be loaded automatically from secondary memory (=disk) to main memory when execution must continue in that segment. Responsible for this automatic loading at runtime is an overlay manager. Thus, after the puzzle of selecting all segments and integrating them into an overlay design, the programmer does not have to inject extra code that explicitly loads segments at runtime when they are needed. On the other hand, when an overlay manager is absent, it's the programmer's responsibility to explicitly call an overlay, for example from
FORTRAN source code.
The picture below shows an example of the logical structure of individual segments. Because of memory constraints, segments A, B and C can not to be in main memory at the same time. In this tree structure the main program functions as root. As the program is running, execution moves from one segment to another; this is visualised by the bended green arrows which represent calls (returns from calls are not shown). When execution starts in the root and following successive calls from one segment to the next, the sequence of segments from the root to the segment where the code is executing is called a path. During execution, calls must be made along those paths defined by the logical structure. A call across the tree (an exclusive call), for example from segment A to segment B is not possible. The layer of segments can be extended by another level as can be seen by segments D & E and, as before, the segments D & E cannot be in main memory at the same time.
This layout means that when segment C is loaded in main memory A & B are not. When a call is made from segment C into either D or E, the overlay manager loads the appropriate segment into main memory, if it is not already loaded. Suppose we have the situation that code of segment D is already executing. That means that segments C and D are in main memory, in addition to the main program. When the call returns from segment D to C and when, at some point in time, a call is made into segment E then E is loaded into main memory. Of course segment D is discarded from main memory by this action.
The specification defines how segments are related to each other and defines the overlay structure. I now do not consider segments D and E any further. By defining segments A, B and C as
Overlay 1,
Overlay 2,
Overlay 3 respectively, the overlay structure is shown (from
Overlay (programmierung)) in the picture below (the overlay manager is not shown).
It is important to realise that this tree structure formed by overlays is not the tree structure inside either the standard ports tree or your private ports tree.
Suppose you have the ports
accessibility/sct &
www/firefox and in addition to that you have your private port
tools/tool1 in your
privateportstree, structured in the same way as the FreeBSD ports tree as
SirDice suggested.
Without overlays:
Code:
# poudriere bulk accessibility/sct www/firefox -p default; poudriere bulk tools/tool1 -p privateportstree
With overlays:
Code:
# poudriere bulk accessibility/sct www/firefox tools/tool1 -p default -O privateportstree
I think the advantage will be more obvious when this
-O option is used in combination with the
-f option to specify the ports in a separate file.
Note that if a port in your
privateportstree depends on a port in the default ports tree, the first option will not work. For that you have to integrate the two trees, for example via git; but then you'll have created a new ports tree. Another tree that you'll have to maintain as the default ports tree gets updates, as well as your
privateportstree, when you produce more code.
___
* Have a look at the Wikipedia entries for other examples and references of overlays. The subject of overlays is related to linkers and loaders. For an in depth look at linking (for example dynamic linking in C++) and loading, I'd suggest you have a look at
Linkers and Loaders by John R. Levine. From the summary of Chapter 8
Loading and Overlays:
[...] [Overlays] require a great deal of manual programmer work to design and specify the overlay structure, generally with a lot of trial and error "digital origami", but they were a very effective way to squeeze a large program into limited memory.
Overlays originated the important technique of "wrapping" call instructions in the linker to turn a simple procedure call into one that did more work, in this case, loading the required overlay. Linkers have used wrapping in a variety of ways. The most important is dynamic linking, which we cover in chapter 10, to link to a called routine in a library that may not have been loaded yet. Wrapping is also useful for testing and debugging, to insert checking or validation code in front of a suspect routine without changing or recompiling the source file.
