• Status New
  • Percent Complete
    0%
  • Task Type Improvement
  • Category tools → pkgutils
  • Assigned To
    crux
  • Operating System CRUX
  • Severity Low
  • Priority Very Low
  • Reported Version 3.3
  • Due in Version Undecided
  • Due Date Undecided
  • Votes 1
  • Private
Attached to Project: CRUX
Opened by fun - 07.10.2017
Last edited by pedja - 18.03.2019

FS#1484 - pkgmk: parallel compression/decompression

bsdtar has the --use-compress-program option which can be used with compressors running with multiple threads. I've attached two patches to allow this kind of usage, with two different approaches.

I've summarily tested them with opt/mupdf (using a NVMe drive).
The decompression showed no visible improvements in this case, but the compression did:
110s (-J)
20s (--use-compress-program pxz)
25s (-j)
3s (--use-compress-program lbzip2)
4s (--use-compress-program pbzip2)
11s (-z)
1s (--use-compress-program pigz)

The deepthought repo has the pigz port and mine the others.

I might add a /etc/pkgmk.conf 'plugin' in my pkgmk.conf port in the future.

   0001-Add-PKGMK_COMPRESSION_PR... (1.6 KiB)
   0001-Add-functions-for-bsdtar... (1.7 KiB)
fun commented on 15.10.2017 21:29

A pkgmk.conf patch for the second approach is here (https://github.com/therealfun/crux-ports/blob/master/pkgmk.conf/FS1484.orig) with the bsdtar_X functions changed to use the parallel compressors.

Romster commented on 08.03.2021 04:31

You can use xz parallel compression without patching pkgmk https://crux.nu/Wiki/HowToSpeedUpBuildingPackages#xz-threads

Loading...

Available keyboard shortcuts

Tasklist

Task Details

Task Editing