CRUX

Welcome to CRUX bug tracking.
Tasklist

FS#1060 - pkgmk: Add support for curl

Attached to Project: CRUX
Opened by Alan Mizrahi (alancio) - Tuesday, 16 September 2014, 02:15 GMT
Last edited by Juergen Daubert (jue) - Saturday, 25 July 2015, 11:01 GMT
Task Type Feature Request
Category tools → pkgutils
Status Closed
Assigned To No-one
Operating System CRUX
Severity Low
Priority Normal
Reported Version 3.1
Due in Version 3.2
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

I have made some changes to pkgmk to support curl in addition to wget.

I think this makes pkgmk more robust, and allows using features present in curl that are missing in wget.

New variables in /etc/pkgmk.conf:

PKGMK_DOWNLOAD_PROG="" (curl or wget)
PKGMK_CURL_OPTS="" (parameters for curl, when using curl)

Please check if the changes are worthy of being merged.
This task depends upon

Closed by  Juergen Daubert (jue)
Saturday, 25 July 2015, 11:01 GMT
Reason for closing:  Implemented
Additional comments about closing:  Thanks for the patch! Implemented for CRUX 3.2
Comment by Fredrik Rinnestam (frinnst) - Tuesday, 16 September 2014, 21:28 GMT
Is there a reason why keeping both is a good idea? I imagine there might be some scenarios where a url might be handled differently by wget and curl (where redirects etc are involved). I'd say replacing wget with curl might be a better option.
Comment by Danny Rawlins (Romster) - Tuesday, 16 September 2014, 22:51 GMT
I did try curl a year or two ago on pkgmk, and I found it to be unreliable on timeouts and resuming as in it would create multiple resume files or end up clobbering existing downloaded data. This may have changed now. ALso i did see a bug ages ago (not sure if it still exists) that a DNS timeout actually gave a false error.

Back then I also used aria2c in pkgmk which is more robust and has support for meta4 files that I'd really like curl and wget to support. not sure if they do yet.
Comment by Alan Mizrahi (alancio) - Sunday, 21 September 2014, 15:32 GMT
I have tested curl with pkgmk for a while and have found no problem with it.

The patch I proposed will keep using wget by default, and only use curl if the user sets it in pkgmk.conf.

Other options are:
- Use curl by default (in the code), and make sure everybody is aware of this by announcing it loudly.
- Enable curl by default in pkgmk.conf, so that new users will use curl by default, while old ones stay with wget.

I would like to have both curl and wget as options, but if I had to choose I would use curl instead of wget.

I haven't checked aria2. It looks interesting, but I don't know how useful it is since most people are behind firewalls/proxies.
Comment by Fredrik Rinnestam (frinnst) - Saturday, 27 September 2014, 19:41 GMT
I've been running with the patch and curl as download app for about a week now. No issues so far :)
Comment by Juergen Daubert (jue) - Tuesday, 09 December 2014, 11:04 GMT
Please explain why using curl instead of wget makes it more 'robust'? Which exclusive curl features do we need for pkgmk? I don't remember that we ever had any problems with wget, but I might have missed something.

Generally I'm not a friend of adding options to pkgmk that are not really necessary, but I've no problem if we switch to curl instead of wget if that improves pkgmk. We need libcurl anyway for httpup, so the switch would make wget no longer a core requirement.
Comment by Alan Mizrahi (alancio) - Wednesday, 10 December 2014, 05:06 GMT
Curl has better proxy authentication support.
Wget supports only basic with clear-text passwords.
Curl supports basic, digest and negotiate using clear-text, NTLM and Kerberos.

I also prefer curl's progress output better.

I have made some additional changes (attached to this message), feel free to try and comment:
# 1. curl support (see http://crux.nu/bugs/index.php?do=details&task_id=1060)
# 2. add comment about PKGMK_KEEP_WORK=fail to keep work dir only when build fails in pkgmk.conf
# 3. add DEBUG variable to build a debug version of a package easily.
# The package will not be stripped and will use different CFLAGS: -O0 -ggdb
# Use it like this:
# sudo -H -u pkgmk DEBUG=1 fakeroot pkgmk
# 4. PKGMK_CREATE_DIRS to create missing dist, pkg and work directories

Comment by Jose V Beneyto (sepen) - Wednesday, 07 January 2015, 02:09 GMT
#1.
we don't use auth for port sources

#3.
currently you can use an alternative config file, with your debug CFLAGS, then 'pkgmk -cf <pkgmk-debug.conf>' will work
And you can use 'pkgmk -ns' (or .nostrip files)
Comment by Alan Mizrahi (alancio) - Wednesday, 07 January 2015, 05:45 GMT
Hi Jose,

#1. I was talking about Proxy authentication, not HTTP server authentication.

#3. I know, but this is about making it easy. I seldom have to build a "debug" package, but when I do, I'd rather run DEBUG=1 pkgmk than the other options.
In particular, I don't see myself using .nostrip files ever.
Comment by Fredrik Rinnestam (frinnst) - Thursday, 19 March 2015, 21:24 GMT
I have been running into some issues with the original patch and ftp.

curl -L --fail --ftp-pasv --retry 3 --retry-delay 3 -o /usr/ports/src/nss-3.18.tar.gz.partial --insecure ftp://ftp.mozilla.org/pub/mozilla.org/security/nss/releases/NSS_3_18_RTM/src/nss-3.18.tar.gz

curl uses 100% cpu on one core and fails to download anything. Guess it's a curl issue though. wget & browsers download the file just fine.

Loading...