CRUX

Welcome to CRUX bug tracking.
Tasklist

FS#1484 - pkgmk: parallel compression/decompression

Attached to Project: CRUX
Opened by Fun (fun) - Saturday, 07 October 2017, 18:39 GMT
Last edited by Predrag Ivanovic (pedja) - Monday, 18 March 2019, 15:03 GMT
Task Type Improvement
Category tools → pkgutils
Status New
Assigned To CRUX Developers (crux)
Operating System CRUX
Severity Low
Priority Normal
Reported Version 3.3
Due in Version Undecided
Due Date Undecided
Percent Complete 0%
Votes 1
Private No

Details

bsdtar has the --use-compress-program option which can be used with compressors running with multiple threads. I've attached two patches to allow this kind of usage, with two different approaches.

I've summarily tested them with opt/mupdf (using a NVMe drive).
The decompression showed no visible improvements in this case, but the compression did:
110s (-J)
20s (--use-compress-program pxz)
25s (-j)
3s (--use-compress-program lbzip2)
4s (--use-compress-program pbzip2)
11s (-z)
1s (--use-compress-program pigz)

The deepthought repo has the pigz port and mine the others.

I might add a /etc/pkgmk.conf 'plugin' in my pkgmk.conf port in the future.
   0001-Add-PKGMK_COMPRESSION_PR... (1.6 KiB)
   0001-Add-functions-for-bsdtar... (1.7 KiB)
This task depends upon

Comment by Fun (fun) - Sunday, 15 October 2017, 21:29 GMT
A pkgmk.conf patch for the second approach is here (https://github.com/therealfun/crux-ports/blob/master/pkgmk.conf/FS1484.orig) with the bsdtar_X functions changed to use the parallel compressors.
Comment by Danny Rawlins (Romster) - Monday, 08 March 2021, 04:31 GMT
You can use xz parallel compression without patching pkgmk https://crux.nu/Wiki/HowToSpeedUpBuildingPackages#xz-threads

Loading...