I'm in the process of setting up a new box for myself to use as a desktop. In the process, I've had a few ideas for new make targets/options that would significantly help reduce the overall build time and hassle.
1. make configdeps | make configusedefaults
To compile x11/kde4 on a new system, there are a whole slew of dependencies that need to be built/installed - not a big deal. However, whenever one of these dependencies pops up a config screen to set options for that port, the compile stalls until I pick the options and hit OK. I constantly have to click over to the server on my KVM throughout the day to set options and continue the compile. Given the number of dependencies and the fact that I let it run overnight, this is a problem; it is now Thursday afternoon, I've been compiling since Friday, and it is still not done. It's almost a certainty that the computer has spent the vast majority of that time waiting for me to confirm what options to use; the vast majority of the time, I simply hit [TAB][ENTER] to accept the defaults and the computer continues compiling. (I don't know what the majority of the options even are or what they do, so most of the time I just accept the defaults.)
Why not have a make target of configdeps that would go through each of the build/run dependencies and run make config on them, one after another, without compiling them? This way, I could run make configdeps and pick all the options for every port, then just run a simple make. The computer would be busy and stay busy until it was done.
Another option that might be of use is make configusedefaults. This would simply bypass the config screen and use the default options, exactly as if you were to use [TAB][ENTER] on each config screen. I'm sure people change options here and there so it wouldn't be good for everybody, but it would certainly be applicable for many people.
2. make rmdistafter / make cleanafter | make savespace
I allocated only 10 gig for my /usr partition. While more than large enough for my mail server, I did not foresee that I'd run short of space on a kde compile. When it happens, I simply run make clean in the x11/kde4 directory and the system cleans all the work files out of the dependency directories. I then re-run make and it picks up where it left off. (Thank God.
) The distfiles are also consuming a significant amount of space. These ideas are designed to eliminate these from causing problems.
make rmdistafter would simply remove the distfile for a given port after the install is complete.
make cleanafter would simply run make clean in a given port after the install is complete.
make savespace would simply enable both rmdistafter and cleanafter.
While both of these are easy enough to accomplish manually after building a port, the advantage would come when there are a large number of dependencies involved. After each dependency is installed, the system would run make clean, remove the distfile, and continue. These would probably be better as options rather than make targets, but could be either.
3. make concurrent
This is another item that might be better served as an option. It's also the only one I can see that might be problematic to implement.
When a port is built, the distfile is fetched, then extracted, patched, and compiled. Some of the distfiles are large and take a while to download. Even with my 6 meg internet connection, some take several minutes to finish downloading, and many people have slower Internet than I. This results in the system waiting for the download to finish, then pegging the CPU, basically waiting for it to finish compiling. These resources are not tied together and could be used concurrently, but the present structure does not allow for this.
This option would start a separate thread for up to n dependencies, running each concurrently. It would more effectively utilize the CPU/network resources of the machine because while one port is waiting for the network, another could use the CPU. However, if a dependency is dependent on another port that is also a dependency of the original, things could get hairy with the timing of things. I do not believe this to be an insurmountable problem, but it is more complicated that any of the others presented here.
Another option that may work better is to have the main make process simply download all the distfiles for the dependencies of the port you're in, one after the other. After the first is done, a thread is spawned and the compile process for the port is started. As soon as the thread finishes, control is returned to the main/download process. If another port's distfile is done, a new thread is immediately started to compile/install that one. If the distfile isn't done downloading, the system waits until it is and then fires it. This structure would serve to concurrently utilize both the CPU and network resources so that time waiting for either while the other is idle is kept to a bare minimum. The timing problem when compiling/installing multiple ports would no longer be a concern with this design because only one compile thread would be active at any given time, even though the next port's distfile is being downloaded at the time.
4. make rmolddist
This one is applicable to any system that's run for any length of time that has had the ports updated. All the distfiles for the old port versions are stored. I don't know how the specifics of the ports system work and therefore don't know how involved it would be, but a make rmolddist target to simply remove all the old distfiles for the current port, keeping the current one, would be very nice. Of course one can always rm -rf /usr/ports/distfiles/* and just let the distfiles re-download as needed, but this would be a much more elegant solution.
I believe that these additions would benefit the software installation process. Quite frankly, not many people are going to try BSD if they have to wait as long as I have just to build/install KDE. I know that binary packages are available, but then I'd be in a similar situation as what I'm in with my Slackware box right now; I'm moving to BSD because I don't like relying on other people to generate binary packages for me and don't have the patience to jump through all the hoops needed to make them myself. (I've done a few and it seems like every piece of software that exists has it's own process that is different from every other piece of software. Also, most require some specific tweaking to work at all. Even some of the binary packages don't work correctly...) After using ports on my mail server, I decided that BSD would be more optimal for my desktop as well.
I don't know if this is the correct place to make these suggestions, but since it's on freebsd.org I'm guessing that developers sometimes look at the forum. If there's a better place, please let me know.
(Apologies in advance to DD - couldn't find a way to make the cmd tag not line break before/after. Because it would really break up the above, I used the file tag instead.)
1. make configdeps | make configusedefaults
To compile x11/kde4 on a new system, there are a whole slew of dependencies that need to be built/installed - not a big deal. However, whenever one of these dependencies pops up a config screen to set options for that port, the compile stalls until I pick the options and hit OK. I constantly have to click over to the server on my KVM throughout the day to set options and continue the compile. Given the number of dependencies and the fact that I let it run overnight, this is a problem; it is now Thursday afternoon, I've been compiling since Friday, and it is still not done. It's almost a certainty that the computer has spent the vast majority of that time waiting for me to confirm what options to use; the vast majority of the time, I simply hit [TAB][ENTER] to accept the defaults and the computer continues compiling. (I don't know what the majority of the options even are or what they do, so most of the time I just accept the defaults.)
Why not have a make target of configdeps that would go through each of the build/run dependencies and run make config on them, one after another, without compiling them? This way, I could run make configdeps and pick all the options for every port, then just run a simple make. The computer would be busy and stay busy until it was done.
Another option that might be of use is make configusedefaults. This would simply bypass the config screen and use the default options, exactly as if you were to use [TAB][ENTER] on each config screen. I'm sure people change options here and there so it wouldn't be good for everybody, but it would certainly be applicable for many people.
2. make rmdistafter / make cleanafter | make savespace
I allocated only 10 gig for my /usr partition. While more than large enough for my mail server, I did not foresee that I'd run short of space on a kde compile. When it happens, I simply run make clean in the x11/kde4 directory and the system cleans all the work files out of the dependency directories. I then re-run make and it picks up where it left off. (Thank God.
) The distfiles are also consuming a significant amount of space. These ideas are designed to eliminate these from causing problems.make rmdistafter would simply remove the distfile for a given port after the install is complete.
make cleanafter would simply run make clean in a given port after the install is complete.
make savespace would simply enable both rmdistafter and cleanafter.
While both of these are easy enough to accomplish manually after building a port, the advantage would come when there are a large number of dependencies involved. After each dependency is installed, the system would run make clean, remove the distfile, and continue. These would probably be better as options rather than make targets, but could be either.
3. make concurrent
This is another item that might be better served as an option. It's also the only one I can see that might be problematic to implement.
When a port is built, the distfile is fetched, then extracted, patched, and compiled. Some of the distfiles are large and take a while to download. Even with my 6 meg internet connection, some take several minutes to finish downloading, and many people have slower Internet than I. This results in the system waiting for the download to finish, then pegging the CPU, basically waiting for it to finish compiling. These resources are not tied together and could be used concurrently, but the present structure does not allow for this.
This option would start a separate thread for up to n dependencies, running each concurrently. It would more effectively utilize the CPU/network resources of the machine because while one port is waiting for the network, another could use the CPU. However, if a dependency is dependent on another port that is also a dependency of the original, things could get hairy with the timing of things. I do not believe this to be an insurmountable problem, but it is more complicated that any of the others presented here.
Another option that may work better is to have the main make process simply download all the distfiles for the dependencies of the port you're in, one after the other. After the first is done, a thread is spawned and the compile process for the port is started. As soon as the thread finishes, control is returned to the main/download process. If another port's distfile is done, a new thread is immediately started to compile/install that one. If the distfile isn't done downloading, the system waits until it is and then fires it. This structure would serve to concurrently utilize both the CPU and network resources so that time waiting for either while the other is idle is kept to a bare minimum. The timing problem when compiling/installing multiple ports would no longer be a concern with this design because only one compile thread would be active at any given time, even though the next port's distfile is being downloaded at the time.
4. make rmolddist
This one is applicable to any system that's run for any length of time that has had the ports updated. All the distfiles for the old port versions are stored. I don't know how the specifics of the ports system work and therefore don't know how involved it would be, but a make rmolddist target to simply remove all the old distfiles for the current port, keeping the current one, would be very nice. Of course one can always rm -rf /usr/ports/distfiles/* and just let the distfiles re-download as needed, but this would be a much more elegant solution.
I believe that these additions would benefit the software installation process. Quite frankly, not many people are going to try BSD if they have to wait as long as I have just to build/install KDE. I know that binary packages are available, but then I'd be in a similar situation as what I'm in with my Slackware box right now; I'm moving to BSD because I don't like relying on other people to generate binary packages for me and don't have the patience to jump through all the hoops needed to make them myself. (I've done a few and it seems like every piece of software that exists has it's own process that is different from every other piece of software. Also, most require some specific tweaking to work at all. Even some of the binary packages don't work correctly...) After using ports on my mail server, I decided that BSD would be more optimal for my desktop as well.
I don't know if this is the correct place to make these suggestions, but since it's on freebsd.org I'm guessing that developers sometimes look at the forum. If there's a better place, please let me know.
(Apologies in advance to DD - couldn't find a way to make the cmd tag not line break before/after. Because it would really break up the above, I used the file tag instead.)
