CRUX

Welcome to CRUX bug tracking.
Tasklist

FS#1313 - git ports driver wipes built packages, downloaded sources, etc from ports tree

Attached to Project: CRUX
Opened by Ryan Mullen (rmull) - Thursday, 05 May 2016, 11:53 GMT
Last edited by Tim Biermann (tb) - Thursday, 24 June 2021, 11:24 GMT
Task Type Bug Report
Category ports
Status New
Assigned To CRUX Developers (crux)
Operating System CRUX
Severity Low
Priority Normal
Reported Version 3.2
Due in Version Undecided
Due Date Undecided
Percent Complete 0%
Votes 1
Private No

Details

Hello, the git ports driver (/etc/ports/drivers/git) runs a 'git clean' stage during a ports -u that deletes downloaded sources, built packages, work dirs, and so on from the ports tree. The default crux configuration is to store all that stuff in the ports tree, and the other ports drivers don't behave in this manner. As a workaround, all that stuff can be stored outside the tree, but that's not the default setting...

I recommend rethinking the git clean stage (assuming the crux defaults for storing downloads/work/packages won't change). The git driver didn't always have this line in it, and I'm not sure why it was added.
This task depends upon

Comment by Fun (fun) - Tuesday, 28 February 2017, 23:08 GMT
Is this still happening?
Comment by Ryan Mullen (rmull) - Tuesday, 28 February 2017, 23:47 GMT
Yes, it is.
Comment by Fun (fun) - Wednesday, 01 March 2017, 08:11 GMT
It might do that if there is a some .git file without NAME=... in /etc/ports/.
Right?

Comment by Ryan Mullen (rmull) - Wednesday, 01 March 2017, 12:47 GMT
If any of the git repos in /etc/ports were missing their $NAME settings, the driver would error out prior to running any of the git commands, since it performs a check to make sure $NAME is set.

The root of the problem is that the driver runs a 'git clean' on the repo, which will remove all untracked files (which includes built packages and sources). It didn't always do that, so someone must have intended it to work that way and I'm not sure why.
Comment by Fun (fun) - Wednesday, 01 March 2017, 15:39 GMT
Maybe I'm missing something. For sure I've missed the NAME check.

I wasn't aware that ports will keep the packages and sources there, but in /usr/ports/{distfiles,packages}, like this: https://github.com/6c37/crux-ports-git/blob/3.3/bspwm/Pkgfile
Comment by Ryan Mullen (rmull) - Wednesday, 01 March 2017, 16:28 GMT
I think we may be talking about two different things. My report was regarding the git ports driver at /etc/ports/drivers/git that is installed when the opt/git port is installed. These drivers are used for syncing the ports tree (with 'ports -u'), not for installing packages. The Pkgfile you linked is from a collection of third-party ports that use git repositories for downloading their sources rather than the usual tarball. Fetching sources from a git repo is not something handled by the git driver in question.

Furthermore, /usr/ports/{distfiles,packages} is not the default location for downloads and built packages - the port's directory is. Keeping distfiles and packages stored outside of the ports tree is one possible mitigation for the issue (because then it's outside the git repo that will get cleaned during ports -u), but it's not the default setting.
Comment by Fun (fun) - Wednesday, 01 March 2017, 19:24 GMT
I didn't realized that default is
PKGMK_SOURCE_DIR="$PWD"
PKGMK_PACKAGE_DIR="$PWD"
PKGMK_WORK_DIR="$PWD/work"

Like a NOOB, I blindly followed the wiki instructions. It looks like most people use the default way (code) or the recommended way (comments/wiki/fake news). Most of them, judging by the popularity of this issue, follow the comments. It is psychological interesting.

LOL.
It is confusing to have the docs so different than the code/defaults.
And all the silly waves I did in the last days around here...
LOL.
Crux is so libertarian... Only one rule: everybody does what s/he wants with her/his installation. Defaults are just a starting point. Fight Club.
LOL

Anyway, I vote for you on this, but I also vote for defaults based on popularity, because this is the best way to avoid confusion. Of course, simplicity should be on top, but clarity should follow, because confusion makes the simple be less simple, at best.
Comment by John McQuah (farkuhar) - Thursday, 14 April 2022, 00:33 GMT
> The git driver didn't always have this line in it, and I'm not sure why it was added.

I might be misreading the commit history, but according to https://crux.nu/gitweb/?p=ports/opt.git;a=commitdiff;h=bd86ca2544f8de84b455ad0e7e9b801abc51dc05 the "git clean" command was there from the very beginning. If you want to shield from deletion the downloaded sources, work directory, and built packages when running `ports -u`, there are two possibilities that come to mind:
1) have pkgmk detect when the repository is under git management, and create a .gitignore file based on the source array, the package name, and the setting for $PKGMK_WORK_DIR.
2) assemble a list of excluded files in the git driver itself, and pass this list to the -e flag of the git clean command.
Option 1 might encounter strong opposition because pkgmk is already more complicated than it was five years ago. Every new feature in pkgmk seems to need greater justification than the last feature added. Option 2 should be easier to persuade the core maintainers to adopt, and I'd be willing to put together a patch if there's enough interest.

On an unrelated note, does anyone actually use the LOCAL_REPOSITORY feature that closed Flyspray issue 1364? I can see when it was added in the commit history of the git driver, but there's no corresponding update to the ports(8) manpage illustrating how this feature might be used. It's a shame when a good idea fades from institutional memory just because nobody bothered to add some examples to the documentation.
Comment by John McQuah (farkuhar) - Tuesday, 10 May 2022, 11:48 GMT
I inserted a few lines in the git driver to shield pkgmk artifacts from deletion. See attached. After several weeks of testing, it looks like a viable solution to the original issue.

This updated driver does NOT inspect its environment to see whether pkgmk artifacts are actually saved in the ports tree. A more optimal refactoring would bundle the manipulations of .git/info/exclude into a separate function, which is only called when /etc/pkgmk.conf indicates an unchanged default location for downloads and built packages.

Loading...