Distcc

A forum for the discussion and development of programs to assist working on or playing with the Vegastrike engine and data sets.
Post Reply
strook
ISO Party Member
ISO Party Member
Posts: 461
Joined: Fri Sep 03, 2010 12:10 pm

Distcc

Post by strook »

Ever tried distcc?
Great tool!
it improves the compile speed by a factor of 2 or more!
:mrgreen:
plz visit my vegastrike project branch here

plz support VegaOgre by donating to it!

My systems: Mac mini 1, 4gig RAM;
i5 Quad Core 2400, 300mbit WLAN, 1,3Tbyte HD, 60 GB SSD,
nvidia geforce 8400gs 512MB, 6gig RAM with Ubuntu 11.4,
win7 and hackintosh installed
klauss
Elite
Elite
Posts: 7243
Joined: Mon Apr 18, 2005 2:40 pm
Location: LS87, Buenos Aires, República Argentina

Re: Distcc

Post by klauss »

Wow... cool...
Oíd mortales, el grito sagrado...
Call me "Menes, lord of Cats"
Wing Commander Universe
breese
Bounty Hunter
Bounty Hunter
Posts: 152
Joined: Thu Sep 02, 2010 8:00 pm

Re: Distcc

Post by breese »

strook wrote:it improves the compile speed by a factor of 2 or more!
distcc is mainly useful if you have more machines to compile on.

If you simply want to use all the cores on a single machine, then use the "make -j2" command (replace 2 with the number of cores your machine has.)
strook
ISO Party Member
ISO Party Member
Posts: 461
Joined: Fri Sep 03, 2010 12:10 pm

Re: Distcc

Post by strook »

maybe (i haven't tried out it yet) you would maximize the speedup on for example a machine with 2 cores with -j3, cause the processor makes often noops cause the pipeline is not full, cause of data dependencies (code that must be executed in serial order).
but on modern cpus there are several workarounds.
but maybe you get a speedup if you have a very long pipeline and many execution units on the cpu.
today the cpus are optimized in trying more and more parallel.
plz visit my vegastrike project branch here

plz support VegaOgre by donating to it!

My systems: Mac mini 1, 4gig RAM;
i5 Quad Core 2400, 300mbit WLAN, 1,3Tbyte HD, 60 GB SSD,
nvidia geforce 8400gs 512MB, 6gig RAM with Ubuntu 11.4,
win7 and hackintosh installed
klauss
Elite
Elite
Posts: 7243
Joined: Mon Apr 18, 2005 2:40 pm
Location: LS87, Buenos Aires, República Argentina

Re: Distcc

Post by klauss »

breese wrote:
strook wrote:it improves the compile speed by a factor of 2 or more!
distcc is mainly useful if you have more machines to compile on.
I know... hence

Wow... cool!!

;)

But ya, I don't have several computers at my disposal atm ;)

But at home, ya, I do have about 3. An old one, an even older one, and a "mature" one. Perhaps together they can make one "not so old" one :D

I bet my energy company will disagree :lol:
Oíd mortales, el grito sagrado...
Call me "Menes, lord of Cats"
Wing Commander Universe
klauss
Elite
Elite
Posts: 7243
Joined: Mon Apr 18, 2005 2:40 pm
Location: LS87, Buenos Aires, República Argentina

Re: Distcc

Post by klauss »

strook wrote:maybe (i haven't tried out it yet) you would maximize the speedup on for example a machine with 2 cores with -j3, cause the processor makes often noops cause the pipeline is not full, cause of data dependencies (code that must be executed in serial order).
Well, not true.

In order to get advantage of that (you're talking of hyperthreading), your processor actually has to support hyperthreading (core 2's don't for instance), and furthermore, your tasks have to be taxing different execution units in the CPU. Which is rare in compile workloads.

Compilation is usually memory-bound, and even with 4 cores, a memory-bound process with -j4 won't perform at 4x. AMD processors cannot use the entire bandwidth of the memory bus on one core, so yes, -j4 may work nicely on a 4-core CPU, but -j5 will probably just increase cache pressure and be counterproductive. On intel processors, however, single-core memory bandwidth is much better, and 2 cores can easily saturate a 4-core's memory bus. So -j4 would be hard-pressed to outperform -j3, say.

The thing is rather complex, so a good rule of thumb is -jn with n=number of processors. Going over it is dubious, and a smaller number is probably worth a try.

So... with complex things as these, better not speculate and benchmark instead. I know for instance HT alone does shave off about 5-10% of build time on a P4 I have with -j2. So it's kind of worth it. And -j4 on i5 works wonders. -i8 on i7 is the same, HT in i7 is a whopping lot better than on the P4, it really performs a lot better, but still, the i7 is suffering from cache pressure already with -i8, so I usually use -i6 there.
Oíd mortales, el grito sagrado...
Call me "Menes, lord of Cats"
Wing Commander Universe
strook
ISO Party Member
ISO Party Member
Posts: 461
Joined: Fri Sep 03, 2010 12:10 pm

Re: Distcc

Post by strook »

compiling a linux app on macosx didn't work,
there were too much linker errors.
also xcode does multithreading in compiling already.
speedup in compiling would be also only if i buy a second mac.
plz visit my vegastrike project branch here

plz support VegaOgre by donating to it!

My systems: Mac mini 1, 4gig RAM;
i5 Quad Core 2400, 300mbit WLAN, 1,3Tbyte HD, 60 GB SSD,
nvidia geforce 8400gs 512MB, 6gig RAM with Ubuntu 11.4,
win7 and hackintosh installed
strook
ISO Party Member
ISO Party Member
Posts: 461
Joined: Fri Sep 03, 2010 12:10 pm

Re: Distcc

Post by strook »

I noticed, in eclipse you can too enable parallel build when Using a makefile imported project.
plz visit my vegastrike project branch here

plz support VegaOgre by donating to it!

My systems: Mac mini 1, 4gig RAM;
i5 Quad Core 2400, 300mbit WLAN, 1,3Tbyte HD, 60 GB SSD,
nvidia geforce 8400gs 512MB, 6gig RAM with Ubuntu 11.4,
win7 and hackintosh installed
pheonixstorm
Elite
Elite
Posts: 1567
Joined: Tue Jan 26, 2010 2:03 am

Re: Distcc

Post by pheonixstorm »

While i'm thinking about it.. There is (or was) a linux livecd that had this feature. I used it extensively when building my gentoo boxes since I liked doing the stage 1 build from scratch
Because of YOU Arbiter, MY kids? can't get enough gas. OR NIPPLE! How does that mkae you feeeel? ~ Halo
Post Reply