I was trying to install CRUX on a system with NVME drives managed by Intel VMD (motherboard hardware raid related). To access the drives the VMD kernel module is needed. I was not able to modprobe it. Would it be possible to add it to the ISO?

# prt-get diff
Differences between installed packages and ports tree:

Port Installed Available in the ports tree

gobject-introspection1.74.0-1 1.76.1-1
grep 3.9-1 3.10-1
tzdata 2022g-1 2023a-1

There should be at least one space after 'gobject-introspection' for normal parsing.

instead of

/usr/bin/setterm -blank 15

it could very well be -blank ${BLANKTIME-15}, with an additional optional BLANKTIME in /etc/rc.conf

The

/bin/cat /var/lib/urandom/seed > /dev/urandom

of /etc/rc mixes the pool, but no longer counts so as to unlock the kernel's CRNG.
For this my entropy saver has been used, or Donendings thing, that has for example been committed to busybox a few months back.

(credit to beerman for discovering this behaviour) When prt-get install (or update) is given an argument list with more than one port, it skips the preliminary check of whether those ports are installed. This useful safeguard is only exercised when exactly one port is given on the command line.

I wrote a patch that reinterprets such a malformed command in the most charitable way: any ports for which install (or update) makes sense are retained; the rest are discarded with a warning. If none of the ports on the command line are compatible with the requested action, then the command fails in the same manner that `prt-get update foo` would fail, when foo is not installed.

TODO: the output would be even nicer if the warnings about ignored (incompatible) arguments could be postponed until the post-transaction summary, rather than getting lost in the scrollback buffer as the usable arguments get processed by pkgmk and pkgadd. But prt-get doesn't provide a straightforward public method for manipulating the contents of m_ignore (populated by argparser.cpp), which would be the most natural place to preserve the information for a post-transaction summary.

Security model of the ports system relies on the fact that only properly signed ports can be built by pkgmk. However, pkgmk sources not yet verified Pkgfile at the very beginning of its operation (see line 813 for details). This allows malicious Pkgfile to alter the behavior of pkgmk making all subsequent security measures unreliable. For example, malicious Pkgfile can define signify() { return 0; } which will be called by pkgmk instead of /usr/bin/signify (local symbols have precedence over calls to external programs) and allow modified files to pass signature/checksum verification by returning 0 (check passed) all the time. This is especially worrisome because rsync-based delivery of the ports tree is not protected against mitm attacks and any successful attack of this sort can lead to full control over the system by modifying the source code of one or more ports. Please note that using fakeroot to build ports doesn't protect against this attack because modified source code will be built, packaged and installed to the system as usual. Please find below a patch to /usr/ports/core/which/Pkgfile which makes changes to which.c and therefore /usr/bin/which but doesn't require .signature to be updated.

See  FS#1644  first.
compress_manpages() is called when:
gpg.1 exists
gpg.1.gz symlink to non-existing gpg.2.gz

The first loop from that function tries to compress all .N manuals to .N.gz. In this case, it fails with the following message:

gzip: ./usr/share/man/man1/gpg.1.gz already exists; not overwritten

The second loop, which tries to rename X.N -> Y.N functions to X.N.gz -> Y.N.gz, ignores this case because the target (gpg.2.gz) is missing.

Ignoring the gzip error code from the first loop in this case is harmless, apart from prtcheckmissing showing /usr/share/man/man1/gpg.1.gz as missing, but I wonder if we should exit on such errors (it might break other builds).

These patches should be self-explanatory.

The improvements are:
- smaller pkgmk code
- more download options for users

In one release lifetime, the software can obtain new dependencies (libinput for example) and it would be good if 'prt-get' can show that. Also, some packages can be manually deleted (after installing for example), and there is no need to remember whether this package is a dependency of other port or not. We can always check where we've mistaken.

I made the patch to prt-get to check for missing dependencies of currently installed ports.
By the way, I'm not familiar with C++, and the code can look awful))

There is a usage example: https://paste.pound-python.org/show/8DqIKor5T4PegZUB2cxH/

I've attached a patch to run 'file' in parallel (using xargs -P$(nproc)).

The time spent changed from 30 seconds to .2 seconds on a 5500 files port (most of them 'data') with a faster drive. I didn't test more ports.

Every port has a "hard" dependencies list ("Depends on" line, which is reflected in the .footprint file) and a "soft" dependencies list (partially found in the "Optional/Nice to have" line). Building the port while some soft dependency is installed may trigger a "footprint mismatch" error. The current workarounds for this case are: (1) set PKGMK_IGNORE_NEW, (2) force the installation with an explicit pkgadd or (3) make your own port. All have their disadvantages. (3) reduces the likelihood for a new user to become a "contrib" maintainer, (2) stops the "depinst" process and (1) reduces the utility of the .footprint file. I'll add here another workaround with its own disadvantages :)

Let's say some port has "/usr/bin/A" in its footprint when built without some soft dependency and "/usr/bin/A" plus "/usr/bin/B" when built with the soft dependency. The Pkgfile will be something like:

pre_diff_footprint_filter() { cat; if [ -f /soft-depX ]; then echo "drwxr-xr-x root/root usr/bin/B"; fi; }

build() { ...; if [ -f /soft-depX ]; then cflags+=--enable-x; fi; ./configure ... }

The changes to pkgmk are minimal: replace the sort line from check_footprint() with:

sort -k 3 $PKGMK_FOOTPRINT | pre_diff_footprint_filter > $FILE.footprint.orig

and have a default "pre_diff_footprint_filter() {cat;}"

bsdtar has the --use-compress-program option which can be used with compressors running with multiple threads. I've attached two patches to allow this kind of usage, with two different approaches.

I've summarily tested them with opt/mupdf (using a NVMe drive).
The decompression showed no visible improvements in this case, but the compression did:
110s (-J)
20s (--use-compress-program pxz)
25s (-j)
3s (--use-compress-program lbzip2)
4s (--use-compress-program pbzip2)
11s (-z)
1s (--use-compress-program pigz)

The deepthought repo has the pigz port and mine the others.

I might add a /etc/pkgmk.conf 'plugin' in my pkgmk.conf port in the future.

I've tested the patch summarily:
- force MISMATCH on some file
- change the public key from .signature (forcing a verbose error message)
- change the .signature file (forcing the old/new error message)

While the server was down, more people expressed the preference for an offline version of the Handbook, not just on the CRUX ISO, and a manual version will be even better.

A handbook command in the core/filesystem port (where the crux command is) that will dump/page the handbook.txt will be nice, not to mention that it could be used when installing/upgrading from ISO too.

For the manual version, I've tried more tools to convert from HTML format to man. I didn't tried go-md2man from "virtualisation" repository yet.

The "easiest" path found is to convert HTML to Markdown with html2text.py, run a couple of sed and convert to man format with "ronn" (a ruby gem). The tables aren't pretty, but readable.

I've attached the script and the generated manual.

PKGMK_PACKAGE_CACHE_REGEX="(wxgtk|firefox)#"
PKGMK_PACKAGE_CACHE=(http://crux.ster.zone/packages/3.3/)

The best practice for every port update is to rebuild the port (prt-get depinst) in a pristine container, with only core packages, and no sexy CFLAGS. Doing this, the maintainer has "the package", and it will be pity not to put it somewhere, for others to use it, especially when it will take 15min*10 to build it on a slow machine.

From day one I've asked why do people leave CRUX. And I've asked for me, to know what to avoid. While I didn't get a clear answer, I've seen messages like: tired of building packages. Even these days, on the #crux channel.

I've attached the patch 0002 a second time, with a fix to avoid downloading packages when "force build" option is used.

Patch 0003 try all the extensions (gz, bz2, xz) regardless of PKGMK_COMPRESSION_MODE, and use a symlink if needed.
Patch 0004 add signature support.

I'd like to propose a patch that adds an option to keep work directory only if the build fails:
/etc/pkgmk.conf: PKGMK_KEEP_WORK="failed"

In this patch, I've added +# PKGMK_KEEP_WORK="no" to the end of pkgmk.conf because otherwise it conflict with my other pkgutils patches. If you plan to merge, you can move it closer to PKGMK_WORK_DIR.

Thanks to Alan Mizrahi, I took it from his 'big' patch.

The checks of directory permissions of pkgmk are too strict.

what I've stumbled upon so far:
-uf -um -us only need read permission on source directory and write permission on port directory
-rs only needs write permission on port directory
-do needs write permission on source directory and read permission on port directory

regards, deepthought

P.S. The user building my own ports is not the one who owns the ports tree. So I have to "pkgmk -uf", "pkgmk -um" and "pkgmk -us" as root and chown the files afterwards.

Hello, the git ports driver (/etc/ports/drivers/git) runs a 'git clean' stage during a ports -u that deletes downloaded sources, built packages, work dirs, and so on from the ports tree. The default crux configuration is to store all that stuff in the ports tree, and the other ports drivers don't behave in this manner. As a workaround, all that stuff can be stored outside the tree, but that's not the default setting...

I recommend rethinking the git clean stage (assuming the crux defaults for storing downloads/work/packages won't change). The git driver didn't always have this line in it, and I'm not sure why it was added.

I just realised that we are setting the *local* repo dir in each .httpup file
shipped by crux.nu/portdb, like: ROOT_DIR=/usr/ports/mechaniputer etc.

/usr/bin/ports explicitly sets PORTS_DIR="/usr/ports" and uses it for everything
but ports -u, afaics, e.g.:

port_version=`cd $PORTS_DIR/$port; . Pkgfile; echo $version-$release`

I propose to

a) make PORTS_DIR configurable. For now this could happen in /etc/pkgmk.conf
b) comment out the destination/ROOT_DIR parts in the .rsync/.httpup files served
and let the system default be /usr/ports but allow users to override this on a
per repo basis (just like it is today)
c) update the ports man page, post to the ML
d) adjust /etc/ports/drivers/{rsync,httpup}
e) adjust ports(1) to honor the different base dirs for --list and --diff

The only issue I see is with older versions of ports and recently downloaded
repos files from crux.nu/portdb, where the neither the repo file nor ports
sets the target directory for the synced repository during ports -u.

Please comment.

pkgadd -u removes files if they don't exist in the new package, regardless of the rejection rules in pkgadd.conf.

As a test I create package for app-1.0 with /etc/app.conf, and then app-2.0 without it:

[root@koji ~]# mkdir etc
[root@koji ~]# echo default configuration > etc/app.conf
[root@koji ~]# tar -czf app#1.0-1.pkg.tar.gz etc
[root@koji ~]# rm etc/app.conf
[root@koji ~]# tar -czf app#2.0-1.pkg.tar.gz etc

I will install app-1.0, and then customize my configuration file:

[root@koji ~]# pkgadd app#1.0-1.pkg.tar.gz
[root@koji tmp]# cat /etc/app.conf
default configuration
[root@koji tmp]# echo customized configuration > /etc/app.conf

Now I will upgrade to app-2.0, and my custom configuration file will be gone:

[root@koji tmp]# pkgadd -u app#2.0-1.pkg.tar.gz
[root@koji tmp]# ls -l /etc/app.conf
ls: cannot access /etc/app.conf: No such file or directory

My pkgadd.conf contains:
UPGRADE ^etc/.*$ NO

I noticed strange behavior of synchronization httpup collections that looks like a bug: ACL mask is always set to r-x on port dir creation, ignoring default mask from parent.

For example, I create directory /usr/ports/jw where I want to sync Johannes's port collection and setup default rwx rights for the group wheel (so default ACL mask is rwx):

alexta /usr/ports $ getfacl jw
# file: jw
# owner: root
# group: root
user::rwx
group::r-x
mask::r-x
other::r-x
default:user::rwx
default:group::r-x
default:group:wheel:rwx
default:mask::rwx
default:other::r-x

After ports syncing by calling sudo ports -u I get following ACL attributes for port glew (effective ACL mask is r-x, while expecting rwx):

alexta /usr/ports/jw $ getfacl glew
# file: glew
# owner: root
# group: root
user::rwx
group::r-x
group:wheel:rwx #effective:r-x
mask::r-x #SHOULD BE rwx
other::r-x
default:user::rwx
default:group::r-x
default:group:wheel:rwx
default:mask::rwx
default:other::r-x

I looked into the code and found, that httpup calls mkdir with creation mode set to 0755, so this ends up changing the effective mask ACL entry on the directory. It seems, that httpup should call mkdir(dir, 0777) and let the umask take care of the default permissions.

Attached patch fixes this behavior.

Implement a depupdate command as described here: http://thread.gmane.org/gmane.linux.distributions.crux.devel/2266

getPkgmkSetting() could do with a cache, since usually within install transactions these values don't change.

Look into providing an elegant solution avoiding losing all the const/static-ness

if pkgadd writes an entry to /var/lib/pkg/rejected the directory owner is ignored if only a file in that directory is rejected.

The above sound a bit crazy, let's show an example:

fcron owns the following directories in /var:

drwxr-xr-x root/root var/spool/
drwxrwx--- daemon/daemon var/spool/fcron/
-rw------- root/root var/spool/fcron/root.orig

if root.orig is modified and therefore rejected by pkgadd we got the following in /var/lib/rejected:

drwxr-xr-x root/root var/spool/
drwxrwx--- root/root var/spool/fcron/
-rw------- root/root var/spool/fcron/root.orig

if a directory, rejected by pkgadd, has new/other permissions or user/group than the installed one, rejmerge should do an update which the current version refused to do.

How to reproduce:

1. echo >> /var/spool/cron/crontabs/root
2. chown daemon /var/spool/cron/crontabs
3. cd /usr/ports/core/dcron
4. pkgmk -u

pkgadd reports something like that:

pkgadd: rejecting var/spool/cron/crontabs/, keeping existing version
pkgadd: rejecting var/spool/cron/crontabs/root, keeping existing'version

5. run rejmerge

Note:
step 1. above is necessary because pkgadd doesn't add empty directories to /var/lib/pkg/rejected, which might be considered as bug as well.

Attached patch adds a check of directory permissions to rejmerge.