CRUX

Welcome to CRUX bug tracking.
Tasklist

FS#1484 - pkgmk: parallel compression/decompression

Attached to Project: CRUX
Opened by Fun (fun) - Saturday, 07 October 2017, 18:39 GMT
Last edited by Fun (fun) - Tuesday, 17 October 2017, 07:16 GMT
Task Type Improvement
Category tools → pkgutils
Status New
Assigned To No-one
Operating System CRUX
Severity Low
Priority Normal
Reported Version 3.3
Due in Version Undecided
Due Date Undecided
Percent Complete 0%
Votes 1
Private No

Details

bsdtar has the --use-compress-program option which can be used with compressors running with multiple threads. I've attached two patches to allow this kind of usage, with two different approaches.

I've summarily tested them with opt/mupdf (using a NVMe drive).
The decompression showed no visible improvements in this case, but the compression did:
110s (-J)
20s (--use-compress-program pxz)
25s (-j)
3s (--use-compress-program lbzip2)
4s (--use-compress-program pbzip2)
11s (-z)
1s (--use-compress-program pigz)

The deepthought repo has the pigz port and mine the others.

I might add a /etc/pkgmk.conf 'plugin' in my pkgmk.conf port in the future.
This task depends upon

Comment by Fun (fun) - Sunday, 15 October 2017, 21:29 GMT
A pkgmk.conf patch for the second approach is here (https://github.com/therealfun/crux-ports/blob/master/pkgmk.conf/FS1484.orig) with the bsdtar_X functions changed to use the parallel compressors.

Loading...