0.6 plans

[ace123]

Yeah, configure is really touchy about things that cause a recompile.
In theory Precompiled headers could help compile time, and if someone can figure out how to make it efficiently be created it might be a good idea.

Anyway, as to boost versions, as long as we 1.35 supports GCC 3.3 we might even be able to avoid needing 1.31 or 1.33 at all..

That only leaves 1.28--however I think as long as we allow ./confiugre to accept --with-boost=1.28 and find the correct files (we have to assume someone will have checked out the 1.28 directory with an external) then it should be fine.

EDIT: I see you already included 1.28 and 1.35--that's perfect. I'll look at fixing the windows code and build scripts to make them use the right version.

[safemode]

Thought I should clear up why the boost changes were done and why the last minute change was done.

I'm aware that playing with boost isn't on the roadmap. It was talked about before 0.5.0, and I felt that it fell within Bugfixing and Prepping for a cross platform build system. It's not a new feature, or even a modification to the game. It's simply a shuffling of svn repository side changes. It _shouldn't_ even be in the game at all, we're not modifying the boost code, users should have to get it on their own via direct download or thier distro, as a dependency, but we're nice.

The problem with being nice about boost is that it forces everyone to get amost twice as much data as they would need to in order to compile VS. Ugh. Enter, external repositories. Everyone _should_ download the vegastrike module with --ignore-externals, then update only the boost rev they want, if they need to. via svn update ./boost/rev

The reason why I had to put 1.28 up there is tha Apple likes to think they know best and thier users are stupid, so they make changing anything extremely hard. Old PPC mac's use gcc 3.1 and it's some pre-release that borks on all boost but 1.28.. So, seeing as how everyone else should be fine with 1.35, 1.33 isn't needed for anyone, so i swapped it for 1.28.

So again. We should start pulling the vegastrike module with --ignore-externals and then explicily update a boost dir if we need it to minimize svn downloads. Of course, some devs will need to retrieve all revs to test, most should be fine with only 1 though, or using the system copy.

That should conclude any major svn repo changes for 0.5.1.

I got more char warning fixes to patch up, (the make ai fly better is pushed till after ray collider), hqtextures additions, vssetup hacking to get hqtextures configurable in there until new config app is made.
We're still looking into a good cross platform installer, some promising ones extist. I think one thing that is necessary is the ability to remotely retrieve necessary files. I'd prefer the installer to just pack the OS bins, then all reference a single archive of data and download it after the bins are installed. Then have an update feature.

with that taken care of, we'd be about 70-80% to 0.5.1 . not bad for only a month or so since 0.5.0

[klauss]

I disabled precompiled headers for now until we can figure out how to make it stop compiling everything multiple times for Vegastrike and vegaserver targets, etc. It increases compile time. Which is exactly what it's supposed to lower.

Did you do make precompiled?
If not, then it's probable compiling the headers when they're not needed might be the responsible of all that increased time.

In my system, it decreased compile times significantly. But you have to "make precompiled" first. I couldn't get automake to do it automatically (bummer). If you know how, though, any target should make precompiled first (I could add it as extra dependency, but it often built precompiled headers after, rather than before, building all the object files).

[safemode]

the problem had nothing to do with actual compile time per object. It was the different CXXFLAGS cause automake to build objects for each target that has different CXXFLAGS. This caused build time to increase over 50-100% as much of the source had to be built twice for vegaserver and vegastrike .
even making vegaserver have the same CXXFLAGS had no change in automake building things twice.

the only way i could see it decreasing compile time for you is if you specify a target for make (dont make all). The times i posted were for make all after a distclean.

[klauss]

Oh, yes, I always specify a target.
Then why not add the -include precompiled.h flag on vegaserver too rather than removing precompiled headers?

Just curious.

[safemode]

Like i said, i did. It still built everything twice.

something about that argument changes the way the objects get compiled (or how automake things they will be), so that even though the gcc argument is the same for vegaserver and vegastrike, with that -include line, they're treated as if they would cause changes for different targets.


the right way to do this is to make it part of ./configure. via an --enable-precompiled

then modify the Makefile.am to handle the condition of if PRECOMP is set or not set,wrapped around the line you would set vegastrike_CXXFLAGS.

so if i dont set the enable option in configure, no special CXXFLAGS is set for vegastrike. If i do, then we get what you had, Though i can probably find a way to get precompiled to compile as a dep to "all" and targets.


that would be the preferred way. It'd be great if we can figure out how to not make it compile objects multiple times with the precomp headers. It wont be of any advantage to me until then, since i always build everything to make sure changes dont leave any of our other bins uncompileable.

[safemode]

I can do the autoconf idea if you're not keen on playing with it. I didn't remove precomp, just the vegastrike_CXXFLAGS is commented out.

getting ./configure to play nice with precomp headers shouldn't be hard.

[klauss]

I could probably do the configure thingy. Although I've been trying to make 'precompiled' a dep of 'all' for a while now, I've read automake's docs once and again (not in full, granted) and I didn't find how to do that in a way that will make it build 'precompiled' before anything else.

The way to make 'make' do it, is to add 'precompiled.gch' as a dependency to every .o, but I can't make automake output such a rule without rewriting by hand the default rule for compiling (which is not recommended according to automake guys).

I'll toy with that a bit more I guess.

[ace123]

I think the reason for it recompiling everything is that vegastrike_CXXFLAGS is set at all.... I don't remember exactly what I did when we first moved to this new automake system.

Though it would be a bit annoying to force all the other projects to depend on that precompiled header as well.

[klauss]

Well, the header only has reeeeally common stuff. Standard headers only. <set>, <map>, <hash_map>, etc... they're usually a bitch to compile and makes up for a great deal of compile time. I tried putting boost there but it didn't help as much because less files actually use boost.

So, making the projects depend on standard headers wouldn't be a big deal I imagine. Right?

[safemode]

when i setup the build to avoid compiling doubles of everything the first time, we made sure to not give Vegaserver and vegastrike special CXXFLAGS. I suppose having anything set, regardless if it is the same or not, causes separate builds.


In any case, the autoconf way is the best way to implement this. That way, the user is in control of what gets used.

[safemode]

I looked it up. people have tested boost 1.35 on mac and it compiled fine
http://www.boost.org/doc/libs/1_35_0/li ... rview.html

The problem would then be to install gcc4 on the mac, which apparently requires Xcode, or fink. Would be cool though for someone to go through the trouble to see if it works. It's one step closer to a unified conf/build system for all supported OS's.

[charlieg]

Thought I should clear up why the boost changes were done and why the last minute change was done.

I'm aware that playing with boost isn't on the roadmap.

Hence why roadmaps are notoriously inaccurate and are best used as guidelines

[safemode]

Some points are intentionally vague like "bugfixes" to catch things we dont anticipate... But the explicit points of the road map are not merely suggestions, but rules. X Y Z must be complete before release. The only way around that is to change the roadmap, and that usually requires very good reason. I consider the boost changes bugfixes because it didn't really require any code changes and it was related to cleaning the source, which should be listed with "bugfixes".

this roadmap to 0.6 isn't a wish list, it's not a todo list. It's not going to be something that people ignore/add to/remove however they feel like as time goes by like previous roadmaps. Hopefully.

Now, it's still open for modification because it's still fairly new and all, but i think once 0.5.1 is out, we'll see much less need to change anything. Though, there are always exceptions.

[safemode]

I was playing with the precompiled headers thing. I found that precompiled.h didn't effect my compile time barely at all. This may have to do with how no source includes it and maybe the -include line doesn't do what's intended (at least not on gcc 4.2 ).

So i thought, hrm, what is the biggest header that we include in many places .... unit.h

I precompiled that header (unit.gch) ...commented out PRECOMPILED so build is the same as without precompiled headers. See, gcc searches for precompiled headers all the time, so long as they're named the same except the suffix is .gch.

Doing this drops my compile time by 50%. make -j2 vegastrike went from 10 minutes (make all only takes about 2-3 minutse longer), to 5 minutes.

So what i was thinking was that ./configure could take a --enable-headers to enable precompiled headers. This would tell automake to build a few specific knit picked headers in the source prior to building anything else, so for instance, we'd build unit.h and mabye configxml.h and Stdafx.h in collide2 ..

this would not require any special cflags, and despite not precompiling some standard headers, it would provide a massive decrease in build time for those compilers that support precompiled headers.


anyways, that's my idea. Smacking down unit.h alone provides a huge decrease in compile time. if we just compiled that the ./configure option would be worthwhile.

[ace123]

Would it be possible to precompile all headers we use into one file--that is, make a file that includes every ".h" in the project?

That way it could maybe cache just about every class you might need...

[safemode]

something about having that done just doesn't sit right.

source should only include what it needs.

Another thing, that's a massively intrusive change that is geared towards a compiler specific feature.

This is very strange, i can't recreate the time decrease i saw the first time i used unit.gch. I'll have to investigate tomorrow. Whatever we end up doing, it's gotta be ./configurable because mileage seems to vary a lot.

[ace123]

unit_generic is included by a whole lot more than unit.h (which is only for graphics)... might want to test with that.

[klauss]

Precompiling unit is hard since there must be no C statement before the include (only preprocessor statements are allowed), and all defined macros at the point of inclusion must be the same (which is likely to not work if unit is included at different places in different sequences).

But yea, precompiling that one would be cool.

Moving any include of unit_generic.h to the beginning of the file and then building unit_generic.gch would do the trick.

Also, this can only be done with a handful of headers, since in any compile unit (cpp) only one of those may be used.

[safemode]

I've been unable to reproduce compile time decreases no matter what headers (your set, or individual ones in game) I choose to precompile.

But, as long as it's a ./configure controlled feature, i dont care.

Basically, make all takes about 12 minutes, no matter what. make vegastrike takes 10 min. That's with -ftree-vectorize and -O2.

I dont know where that 5 min vegastrike compile came from that i initially reported, I haven't been able to reproduce that.

[safemode]

sidestepping precompiled headers.... since that's ongoing, it's a guess and check on what headers produce a noticable gain.

I'd like to suggest that not updating ~/.vegastrike generated textures, that use textures in /data as source, when a source texture is changed is a bug. That is to say, if I update the space background, ~/.vegastrike/textures/backgrounds should create a new sphere map based on the new files. This does not happen and requires manual deletion to get VS to regen the file.

Since it's a bug, I'm hoping that I can create a fix prior to 0.5.1.

Basically, the only way to really do this is to utilize filesystem modify/creation times. If the source textures are newer than the generated texture, regen type stuff. This would probably require a new function in vsfilesystem that retrieves the modification date of the file (probably using some boost function so we auto support FS's on all OS's).

Hopefully then, we can push this to all access to generated content, so the days of whiping out ~/.vegastrike can be over. It could also be used for the config file, either prompting vssetup to be used or failing with an error that the user config file is no up to date anymore, whenever the default /data config file is changed.

[ace123]

Filesystem times should be fine... the only problem is if the system clock is off by more than a day you might have issues with it regenerating a few textures each time.

If you go creation times the system clock problem will be fixed, but creation times tend to be different on each platform and are in general confusing to work with.

I changed the setup program to use the mtime, and if the system clock is wrong, well then you just have to reconfigure every option each time you open it.

[safemode]

anyone not using NTP deserves to have to wait while the textures regen every time they run

[safemode]

Couple explanations about the recent commits.

1. I made vsfilestystem traverse our set of root dirs. Let me explain what root dirs are for anyone lucky enough to not have worked with vsfilesystem.

Upon load, we have a set of hardcoded paths for finding the data dir + a compile time data dir path + a config directive data dir path. This includes all the relative paths to data from the current dir you execute vegastrike, in addition to the absolute unix paths. The directory that is found to contain the data is the root dir.

Now We have special directories.
We have the homedir. This is where generated content go, like the config file, bsp's, sphere maps, everything Dynamic universe creates. etc etc.
We have the /mods dir. This is a largely unused special directory that is supposed to be utilized by mods that intend to only alter parts of vegastrike, rather than replace it entirely.
We finally have hqtextures. This is the same thing as /mods, but is allowed to exist in a config defined directory and does not stop the user from having a /mods dir.

Now, the order of search should be homedir -> hqtextures -> mods -> data

That is, any file requested by the game, first gets looked for in homedir, if not found, then hqtextures, then mods, then data. Every file. That is, those directories that get enabled get searched, if /mods doesn't exist, it wont be searched, if hqtextures isn't activated, it wont be searched.

Now, on to the patch.

The problem was we have files, (.spr, and .bfxm, and .ani and such), that reference files via a path relative to the location of that special file. That is to say, usually we ask for files with a path that would be relative to /data/, but these files listed inside these special files were listed relative to the special file's directory.

Now, vsfilesystem has hooks to handle special files, and it saves the subdirectory they are found in, so that subsequent requests for files can use that subdirectory to complete the filename + path that doesn't exist for files listed in the special files.

The issue we were having was that vsfilesystem not only saved the subdirectory, but it saved the entire absolute path and used it. This broke the whole multiple data dir paths setup we have and use for non-referenced files. a blast001.png listed in orange_exposion.ani would only be looked for in the absolute path that orange_explosion.ani was found.

What this means is that if we wanted to replace a single blastNNN.png image in our hqtextures or mods dir, we'd have to put the .ani file in our directory as well as _ALL_ of the images the .ani listed, or vegastrike would not find the images at all.

Obviously that is wrong. All the patch does is fix the behavior of vsfilesystem to use the relative path to /data for the references files as well as any other file we look for. hqtextures now functions correctly.

Ideally, we would want to really fix things such that these special files referenced to the relative path of /data/ so that there needn't be any "special" files. But for now, this wont affect anyone detrimentally, and requires no data changes.


2. I made opcode's Stdafx.h file able to be precompiled and used if in the future, such a feature could be used to an advantage. It does not affect compile speed or function in the meantime. Basically, to use precompiled Stdafx.h, you'd precompile it to Stdafx.h.gch. Since only opcode uses that header, it is ideal for use. The only issue is, opcode is very easy to compile, so precompiling that header wont likely make any difference in overall compile time.

[safemode]

OK, Since i didn't think all of Klauss's work was extensive enough of a change and big enough to give everyone a fun time debugging, I started the work to get cmake going in VS.

I'm still learning the fairly simple syntax, but I'm pretty sure i have auto-selection and detection of boost and ffmpeg working

Obviously that's not tested on windows and mac, but all i need to do to support them is add some standard paths to the search paths (or the user could supply their own)

Once that's perfected though, I'm on to python. basically, replacing the m4 scripts with special little cmake files that will search out and find the headers and paths of libraries for all archs. These files are in the same language as the regular Cmake files.

After i get the m4scripts done being replaced, it should be fairly simple to construct the main CMakeLists.txt (config file) file and get some cross platform Build file generating on. (i'm actually doing both at the same time)

Also, I think we should look seriously at CPack to package under the Cmake build system. CPack supports apple, windows, unix. All CPack requires is cmake....though you have the option of using a compression program native to your OS (7zip or such), or pkging to a native format (deb,rpm,etc), rather than just tarballs or such.

When this is done, it should radically reduce the amount of work involved in constructing a release and maintaining the build system. Though, we'll keep the autoconf setup until everything is perfected on the cmake system. They are not mutually exclusive, it would just suck to maintain the autoconf system once the cmake one is fully functional.

Then i'll get to move back to playing with hqtextures.