3 steps to speed your KDE3.3 build

I’m sure by now that all KDEophiles know about the new KDE 3.3 Release. For those of us using Gentoo or just plain like building from source (you know who you are), a new KDE release fills us with excitement, but also a slight annoyance that we need to leave our machine overnight (or over weekend) building in the back ground. Particularly those of us with old machines. I use an ancient Dell Inspiron 5000e with a 700Mhz P3.

The good news is that there are now 3 little tools that you can use to speed up the build process. If you use Gentoo it’s super easy. If not, it’s not really that much harder. Actually this should speed up the build process of most larger c/c++ based unix projects as well.

Step one - Unsermake

Unsermake is a new addition to the build process of kde. I think it should be made the default option and my guess is it probably will at some point.

Unsermake replaces automake and all the billions of tiny Makefile’s that get called recursively with one large Makefile. It is completely transparent to the kde build process. I’m sure you could use this for other projects, but I’ll leave that to you.

There are several benefits to this single Makefile approach:

• It only has to recurse through everything once and not for each phase of the build process.
• It allows make to do what it does best, handle dependencies globally.
• It allows make to handle the forking of compile process in a much more efficient way, which we will see helps us when we get to distcc below.

Gentoo approach

# emerge unsermake
#  echo "UNSERMAKE=\"/usr/kde/unsermake/unsermake\"" >/etc/env.d/46unsermake
# export UNSERMAKE="/usr/kde/unsermake/unsermake"

The second step adds an environment variable for future use, letting KDE know where it is. While the last one just quickly sets the value for use right now.

All others

I haven’t tried this, but just I think it’s pretty simple. You just have to dowload the latest unsermake build and unpack it somewhere. It is written in Python and doesn’t have any need for it’s own build process. So try this:

# wget http://msquadrat.de/pub/softlib/unsermake/unsermake-0.3.0.4224.0.tar.bz2
# tar xjvf unsermake-0.3.0.4224.0.tar.bz2
#  export UNSERMAKE=[CURRENTDIR]/unsermake-0.3.0.4224.0/unsermake

Then start the KDE Build proces.

Step 2. Don’t cough, ccache

You know the annoying feeling when you are 3 hours into a build process and it dies on you, because of a missing dependency. Or let’s say you want to change the configuration of it and rebuild it. ccache is a lifesaver in these situations.

You can basically think of it as a caching proxy c compiler. It just sits in front of your gcc and caches our returns a previously cached version of the output of the real compiler. I believe it builds up a hash of the compiler flags as well as the source files. If it’s a match to something already in the cache it returns it, thus skipping the recompile. This itself can cut masses of time off a recompile.

Gentoo approach

# emerge ccache

Then edit your /etc/make.conf file so it has ccache listed under the FEATURES section:

FEATURES="{OTHER ENTRIES} ccache"

Thats all ther is to it.

Others

# wget http://ccache.samba.org/ftp/ccache/ccache-2.3.tar.gz
# tar xzvf ccache-2.3.tar.gz
# ./configure
# make
# su
# make install
# mkdir /usr/local/ccache
# ln -s /usr/local/bin/ccache /usr/local/ccache/gcc
# ln -s /usr/local/bin/ccache /usr/local/ccache/cc
# ln -s /usr/local/bin/ccache /usr/local/ccache/g++

Now it is installed. To actually use it you must either change the CC variable in the Makefile you’re using to use /usr/local/ccache/gcc or put the /usr/local/ccache in your path before the real gcc. eg.

# export PATH=/usr/local/ccache:$PATH

Now configure and build to your hearts content.

Step Three - Take a load off your back with distcc

My poor notebook. Great screen, new harddrive maxed out memory but only 700MHz. Thankfully I have an Athlon desktop sitting beneath my desk (actually dining room table) without a monitor (no space). This is my local server which I use to offload lots of stuff.

Enter distcc which comes from the same team that developed ccache above. It basically offloads compiler processes to other machines in the network. Thus speeding things up drastically. In particularly if you use it with unsermake.

You do need to install distcc on all your machines and a fastish net is probably a good idea. I wouldn’t necessarily recommend using it over a WIFI connection, but thats just an educated hunch.

Planning

First you need to write down the ip addresses of the machines you want in your super compiler cluster. I for one have only 2. Let’s for sake of argument say my notebook has IP 192.168.0.10 and my server 192.168.0.20

Gentoo Approach

1. emerge distcc
2. /etc/init.d/distccd start
3. distcc-config —set-hosts “192.168.0.10 192.168.0.20”

As my main machine is slower than the server I list the server last, so it gets most of the load.

Now do the same on the other machine[s] in your network.

Edit your /etc/make.conf file like you did with ccache above and add distcc. Your file should now look something like this:

FEATURES="ccache distcc"

Now everytime you emerge something it gets sent first via ccache and onwards to distcc which distributes the load around your network.

Others

Very similar to installing ccache:

# wget http://distcc.samba.org/ftp/distcc/distcc-2.17.tar.bz2
# tar xjvf distcc-2.17.tar.bz2
# ./configure
# make
# su
# make install
# mkdir /usr/local/distcc
# ln -s /usr/local/bin/distcc /usr/local/distcc/gcc
# ln -s /usr/local/bin/distcc /usr/local/distcc/cc
# ln -s /usr/local/bin/distcc /usr/local/distcc/g++

Now you can start it up on the servers with:

1. distccd —daemon

Put the list of hosts in /etc/distcc/hosts eg.:

1. echo “192.168.0.10 192.168.0.10”>/etc/distcc/hosts

To use it do the same as for ccache, just make sure that ccache’s cc is infront of distcc in the path:

# export PATH=/usr/local/ccache:/usr/local/distcc:$PATH

An easier way to distribute load without installing.

I haven’t tried this these guys have a new bootable cd with distcc, that you can download, burn onto a cdrom and boot up on your various pc’s that aren’t running linux. No install needed. Pretty cool.

Disclaimer

I only really use Gentoo, so I haven’t tested out the non gentoo instructions thoroughly. If you see any glaring errors let me know.