CRUX

I've tested the proposed change with one port.

The pkgmk change: https://github.com/therealfun/crux-ports/blob/master/port-on-rails/enable-diff_footprint_filter

The port using this change: https://github.com/therealfun/crux-ports/blob/master/libvirt/Pkgfile

I don't like it, it's too complicated.

It is too complex, but most likely we do need to consider optionals in the future in one or the other way.

Upvoting for awareness.

By proposing that the extra code is placed in the Pkgfile, you make it the port maintainer's responsibility to keep track of all the possible side effects that might arise by having other ports installed. For small and simple projects, the list of side effects is easily identified, but larger codebases will inherit a messy web of interwoven dependencies that can quickly consume the limited time of a port maintainer. One commentary on this complexity brings up the phenomenon of "spooky action at a distance" resulting from such a super-connected digraph: https://sunshowers.io/posts/dependencies/

The cited disadvantages of workarounds 1 and 3 are not so strong as to warrant this patch, imho.
1. Setting PKGMK_IGNORE_NEW=yes (but keeping PKGMK_IGNORE_MISSING=no) will preserve most of the benefits of a .footprint file (as long as the port maintainer generated the footprint on a minimal system).
3. Making a custom port is the workaround chosen by someone who sees a need for additional tests that the original port maintainer did not envision. Such a user is gaining experience that puts them further along the path to "contrib" maintainer. If they choose not to make use of this experience and apply to join the "contrib" team, it is more likely because of other factors than because of having a customized port in their personal overlay.

The Pkgfiles of long ago were not in the habit of performing filesystem tests (or `prt-get isinst $SOFT_DEP` tests). Usually they aimed for complete reproducibility by explicitly disabling any optional features, never giving ./configure a chance to auto-detect anything. If you wanted a feature that the original port maintainer did not enable, you just forked the port and tweaked the flags passed to ./configure. Port standards these days are a lot less rigid, perhaps to prevent the proliferation of duplicates in our portdb: https://crux.nu/portdb/?a=dups But if we really wanted to avoid dups, it would be better to advertise the prt-get.aliases feature more widely (if the forked port happens to be a dependency of something else), and get people in the habit of giving their forked port a name that indicates exactly which feature was toggled. That way a search of the portdb would turn up more descriptive names, so that the new CRUX user could quickly grab the one best suited to their system.

I am making up this example: having a port named rust-clang for example that overlays opt/rust because a user wanted a rust binary compiled by clang and linked by lld is not what I ideally want to see. We don't need to reproduce Archs AUR where people add things like that. It could be a concious decision by opt/rust to detect both clang and lld, then set the flags to make that happen.

My problem with user repositories is that most of them are not cared for. A lot of times they are one timers, or worse: they overlay core/opt/contrib ports and steer away with their design decisions, potentially leaving you with a broken system when you decide to drop their overlay and want to continue with your normal day to day life.
Personally, I do not use user-repositories except for jue and jaeger, because I trust them. I have had bad experiences when I started out using CRUX in ~2007. Looking at our current portdb entries, a lot of them are absolutely not cared for at all. I would always advise to not use them, no questions asked.

In an ideal world, I would want people to propose patches for Pkgfiles, which would enable whichever feature they are looking for, which could be reviewed and merged as soon as they meet the quality criterias needed. Of course, this would need core/opt/contrib maintainers to actually review and merge stuff, which the past already proofed to be difficult for some.
We also could use more helping hands. Those that actually hands on take a swoop of ports in contrib off my and Dannys hands (if he ever comes back), since we are the top maintainers my total ports number there, which would free up our time giving us more chance to care about core/opt as well as the website, tickets, and other shenanigans.

So in short:
I am very much pro "fluid Pkgfiles" which can suit all use-cases. It minimises the need to bump a ton of dups. It just needs to be a good, clean patch and prt-utils needs to understand that too, e.g. running pkgfoster would need to check if the port A in question is an optional dependency somewhere for port B, and if A is removed, it should prompt a rebuild of B.

And as far as more user contributed patches go, we might be using something soon which would allow that and a bit more. When I have time to finish setting things up, basically, which I rarely do, because I have a real job and real, other hobbies and such outside of CRUX.

Thanks for the feedback, Tim. I expect that the average CRUX user is a little more discerning that the average user of Archs AUR, and follows the advice of inspecting carefully the behaviour of any user-contributed Pkgfile.

Meanwhile the problem of soft dependencies can arise even without triggering a footprint mismatch. Here's one example that happened to me within the past hour:
After scanning a handful of documents into JPG files, I wanted to resize them all by 50% in order to save bandwidth for the server that would be hosting them. I tried running `gm convert infile.jpg -resize 50% outfile.jpg`, but the command failed with error message "no decoder delegated for infile.jpg". I inspected the graphicsmagick Pkgfile and discovered that there was not even a list of soft dependencies, so I could not tell by inspecting package timestamps whether graphicsmagick had been built before any of its soft dependencies. Running `revdep -vvv graphicsmagick` also reported a properly-linked port. Yet rebuilding graphicsmagick detected the presence of libjpeg-turbo and correctly installed the decoder, so I was able to perform the resizing operation as originally planned.

The point of this anecdote is that no matter how closely the port maintainers pay attention to optional dependencies, the ecosystem is just too large for every combination of installed ports to be accounted for in our current design of Pkgfiles. Something as simple as graphicsmagick's optional dependency on jasper or libjpeg-turbo would not have been detected by the pkgfoster changes Tim proposed, because the Pkgfile doesn't list any optional dependencies.

By making "fluid Pkgfiles" the norm (rather than explicitly toggling the exact combination of features that ./configure recognizes), we create a ports ecosystem that's ultimately less transparent; the user has no idea by reading the Pkgfile which features were enabled and which were disabled. At best they might have a build log saved somewhere, which could tell them what was autodetected by ./configure, but even this kind of transparency is lost when /etc/prt-get.conf has the line "rmlog_on_success yes".

> I expect that the average CRUX user is a little more discerning that the average user of Archs AUR, and follows the advice of inspecting carefully the behaviour of any user-contributed Pkgfile.
lol, okay.

> Something as simple as graphicsmagick's optional dependency on jasper or libjpeg-turbo would not have been detected by the pkgfoster changes Tim proposed, because the Pkgfile doesn't list any optional dependencies.
Typical "overworked and underpaid" situation.

> By making "fluid Pkgfiles" the norm (rather than explicitly toggling the exact combination of features that ./configure recognizes), we create a ports ecosystem that's ultimately less transparent
I think you misinterpret what I mean with "fluid Pkgfiles" - of course they should list optional dependencies, as far as they are known. With romster having a few hundred ports in his name, and me with a few hundred ports in my name, you just can't expect it to be perfect, ever.
A lot of times software has a way to tell you what you linked against (e.g. IIRC the about page in wireshark), and even if it didn't, ldd or looking at what files were installed through /var/lib/pkg/db might be able to solve that riddle - though granted, thats no ultimate solution. But withouth bigger changes to our tools, this is as good as it gets, IMO.

> I think you misinterpret what I mean with "fluid Pkgfiles" - of course they should list optional dependencies, as far as they are known.
In retrospect I chose a poor example; graphicsmagick ultimately does depend on libjpeg-turbo, but it shows up two levels into the dependency tree, not as a first-order dependency. How deeply into the dependency tree should a port maintainer go, to account for all the optional ports that could have been present when the hard dependencies were built? "Overworked and underpaid" would be severe understatement, if we expected port maintainers to inspect the entire dependency tree for the optional ports that could affect compile-time options.

Pkgfiles that allow the builds to be non-reproducible across different CRUX systems, but still robust enough to satisfy the majority of use cases, is a fine goal in theory. But then port X in the dependency tree of port Y might have been built in a way that the maintainer of port Y did not anticipate, and "spooky action at a distance" is the result. I concede your point that the other extreme (Arch AUR with all its variants) is rather unwieldy in practice, and I'm not sure how to reconcile the conflicting goals here.

This issue is from a time when I thought I can help "improve" CRUX.

Regarding the optional line (or "nice to have", how I like to use), I think it is OK how it is now and any "improvement" will make ports more complex (more or less). I haven't close this issue because I think that (1) is not my business and (2) no idea is hammered enough until it fits nicely/perfectly with the current system.

A few years ago I was curious about how can a CRUX inspired distro (Arch Linux) diverge so much with regard to the build files (PKGBUILD). Just compare QEMU files: [1] vs [2]. At that time, I found the moment/commit when PKGBUILD files almost doubled in size. It was after Judd Vinet stepped down and Aaron Griffin started to make substantial changes (~2008).

I see that CRUX is on the same path: it adds complexity, but removes none. Others may call this: improving or syncing with the reality. I'm not complaining about this trajectory, but things go were you point them to go. I've read the old mailing list archive and I've seen when pkgmk got its "improvements" (starting with the shared directories). Once I've re-implemented pkgmk in 100 lines of bash (upkgmk). My point is, when you want to make it KISS, it will be "more-KISS". When you want to "adapt" it, most probably it will be "less-KISS".

[1]: https://crux.nu/ports/crux-3.6/opt/qemu-all/Pkgfile
[2]: https://github.com/archlinux/svntogit-packages/blob/packages/qemu/trunk/PKGBUILD

> PKGBUILD files almost doubled in size after Judd Vinet stepped down and Aaron Griffin started to make substantial changes (~2008).

Thanks Fun, for providing more context to the cautionary tale of ArchLinux evolution. I don't quite share your pessimism about the trajectory that CRUX is on. With our niche userbase valuing simplicity more highly than the average Arch user, the trend toward unreadable Pkgfiles or unmaintainable tools will always face stiff resistance.

> I see that CRUX is on the same path: it adds complexity, but removes none. Others may call this: improving or syncing with the reality.

Tim addressed this point in FS#1410: we don't write every piece of software that appears in the ports tree. Any complexity chosen by the upstream project will filter down into CRUX, unless we miraculously had the developer hours to clean it up and make it conform to KISS. Such patching efforts are already at the threshold of impossibility for a decently-staffed project like OpenBSD; it's well outside the realm of possibility to expect this level of patching from the four-person team that is CRUX.

> Once I've re-implemented pkgmk in 100 lines of bash (upkgmk).

Here I suspect you may be onto something. The complexity introduced upstream is outside our control (barring the extreme measure of dropping all the ports that make non-KISS decisions regarding their toolchains and dependencies), but we don't have to compound it by sticking with unmaintainable scripts, or programs that nobody feels competent enough to tinker with. A few months ago I took your 100-line upkgmk and added just enough code to bring it to feature-parity with pkgmk [1]. Then I went on to write a drop-in replacement for prt-get in 855 lines of Perl [2]. These projects demonstrate that there are viable routes away from legacy codebases toward simpler solutions, without giving up some of the more useful features that CRUX packaging has acquired over the years.

[1] https://git.sdf.org/jmq/Documentation/src/branch/master/scripts/pkgmeek
[2] https://git.sdf.org/jmq/Documentation/src/branch/master/scripts/prt-auf