Categories: Ports

How to speed up building packages.

Author

Danny Rawlins

Description

Here I list the common ways to speed up building packages. You can use one or more of these methods if you choose to.

Contents

Compile in ram

Concurrent build jobs

Concurrent xz compression threads

ccache compiles

Compile in RAM

An easy way to reduce disk access and prolonging the life of the disk, by using faster ram for compilation time.

Instructions

Note you need a decent amount of ram that is dynamically used so this figure won't be used when you are not building a package.

Minimum 500MB for small packages. For example flac, xscreensaver and x264.

Average of 1GB is recommended. For example firefox, wesnoth and mesa3d.

Best performance 2GB or more. For big ports like qt4 and boost.

Recommended but not required. Add a pkgmk user and fakeroot to prt-get.conf as on FakerootPorts then return to this page.

If you are using a pkgmk user for building find your user ID for pkgmk

$ id pkgmk

Edit /etc/fstab and add this line. Setting your preferred size size=... in M for megabytes or G for gigabytes and user ID of the previous command uid=...

Note that you don't have to even have that much ram, it just means the bigger ports may go to swap space and diminish the time savings of this for the larger packages, but is still worthwhile to do even with 500MB of ram available to compile in.

pkgmk /usr/ports/work tmpfs size=1G,uid=XX,defaults 0 0

Mount the tmpfs, (saves rebooting).

# mount -t tmpfs -o size=1G,uid=XX,defaults /usr/ports/work

Edit /etc/pkgmk.conf and set where you put your ram file system from your /etc/fstab file.

PKGMK_WORK_DIR="/usr/ports/work/$name"

Note since you are now compiling in memory there is no point in keeping -pipe as this will decrease performance.

Edit /etc/pkgmk.conf and remove -pipe from all your *FLAGS=.

CFLAGS='-O2 -march=i686 -fomit-frame-pointer'
CXXFLAGS='-O2 -march=i686 -fomit-frame-pointer'

Using concurrency for more than one build job

Another way to speed up compiling is to set the number of parallel jobs to do at the same time, set the number of jobs to the number of processors you have. In this example I have chosen 4 concurrent jobs for a quad core processor. Set this in pkgmk.conf

export MAKEFLAGS='-j4'

Alternatively use this to set the number of jobs to how many processors you have running.

export MAKEFLAGS="-j$(/usr/bin/getconf _NPROCESSORS_ONLN)"

Using concurrency for more than one compression thread in xz

Another optimization for rather large files in packages is to set more than one thread count for compression.

Note this does have a side effect of using more ram and slightly worse compression!

Note not all packages will use the max number of threads depending how large the package size is.

Note this will only take effect if you set PKGMK_COMPRESSION_MODE="xz" in pkgmk.conf

PKGMK_COMPRESSION_MODE="xz"
export TAR_WRITER_OPTIONS="xz:threads=${JOBS}"
export XZ_OPT="-T${JOBS}"

Using ccache to cache builds

This is how you can cache C and C++ objects to speed up compiles, particularly useful if you rebuild the same package over and over, or bumping the version regularly.

ports -u opt
prt-get depinst ccache

Add these lines to /etc/profile

export PATH="/usr/lib/ccache/:$PATH"
export CCACHE_DIR="/var/cache/ccache"
export CCACHE_COMPILERCHECK="%compiler% -dumpversion; crux"

For distcc users see Distcc on how to set the number of jobs to suit your distcc cluster.

Inspired by treach for giving me the idea and cptn for suggesting I write wiki pages.

CRUX : Home