the ancient stuttering problem

[chuck_starchaser]

Just make sure if you're developing on the data4.x dir in svn, you dont commit changes to those image files

Why not? It would accelerate adoption of your dds patch

That's excellent! Really good news!

How easy is compression going to be? Here's a c++ library that has dds dxtn compression:
http://www.gamedev.net/community/forums ... _id=457960

Can compress to DXT1 or DXT5 (if EXT_texture_compression_s3tc is available)

[safemode]

using the gimp dds plugin, compression is as easy as saving a file.

both gimp and the dds plugin are open source, and cross platform (i think there are dds plugin binaries out there) so it should be easy to have both windows and linux dev's able to generate dds files.


Eventually, I want to give vegastrike the ability to cache compressed textures to the hdd so it doesn't have to recompress those everytime and generate mipmaps. This is in addition to changing some textures in the release to dds files rather than jpg and png. So depending on the quality level set, those textures that are compressed by VS now can be saved for future use as dds files in some cache dir. The feature would be entirely optional.

So basically, if 10% of the textures being shipped are dds files already, but your quality setting compresses 80% of the loaded textures, you could cache those 80% and then have 90% of the textures as dds files the next time you run the game...theoretically drastically decreasing load time and memory footprint.

that's down the road though. For now, I want to get dds file loading working correctly for all valid inputs and outputs and have it be endian safe. Then get software fallbacks working. Then, after all that, worry about the caching feature.

[Halleck]

Certainly, it sounds good in theory.

But before we go converting all the textures in the data set, is there any way to benchmark the DDS loading to see if it really does speed everything up?

Or would we need a converted dataset to benchmark in the first place?

[safemode]

that'll be done soon of course.

I'll insert a cerr statement prior to asknig to open an image and after it's loaded by gl . that should tell us pretty much hands down which way is faster and by how much on an image to image basis.

[hellcatv]

the performance savings will be huge...
loading the textures currently requires going through each byte of the png or jpeg and decompressing it...then going through each byte and compressing it doing all sorts of heuristics...

in the future it will be doing a sequential read from disk... copying that memory to a gpu buffer and saying "go"

and if there's one thing modern systems are great at---it's sequential reads and copies....
basically we're taking something computers are mediocre at...and replacing it with something they rock at...and will get better at over time

[chuck_starchaser]

Good point there, Hellcat; wasn't thinking about the aspect of burst mode dma and all that; probably the gpu goes directly to file cache and grabs the thing with minimal cpu involvement, right?

And the other thing that rocks about this, Halleck, is the larger amount of textures you can fit in the videocard. Will be like day and night; mark my words.

[Halleck]

Sounds good then.

Is there a way we could hack up a batch conversion method so we don't have to use GIMP to convert them all?

Before the DDS files are committed I think we should branch, or at least tag, the data4.x with all PNG/Jpeg images.

[charlieg]

Ooo... do I sense a quickfire 0.5.1 release with this DDS support implemented?

[chuck_starchaser]

Rather than have a batch program for use by developers, what would be better is a batch program included with the download, that does all the image conversions and cacheing after you run setup. That way the download can be made a lot smaller, by using jpg most of the time. DXT is not a high compression ratio format; it's rather a compression format for ultra-fast decompression. I think having dds images shipped with the game would fatten up the download unnecessarily.
A bit more programming effort, yes; but then *nothing* would need changing, on the data side, which is a Good Thing (TM).

[safemode]

indeed, it was always intended by me to have the creation of the dds files be done on the user end. I actually wanted the creation of the dds files to be done per-quality setting. Meaning, it wouldn't generate the dds files until you changed a setting to minimize memory usage. Then depending on the level of minimization, it would select another pre-defined list of textures to compress and cache as dds files.

It was thought that this would consume too much disk space (more than users would find acceptable) so it was then decided that we convert over a decent number of textures to dds at the distribution pkg level.

This would undoubtedly sound like a bad idea because the dds files are so much bigger than png and jpg. The good thing is dds files can be compressed by bzip2 and the like EXTREMELY well. To the point that the file is smaller than png compressed. You'll probably never beat jpg, but a number of really large textures are png and thus, we should pretty much even out even if we convert a large number of textures to dds.

A batch program would be nice, but i dont have the time to write up a cross platform one.


In any case, I really would like it if we would branch data4.x to data5.x (get rid of the old data5.x). All of the textures should go into a master branch, so we dont lose the originals.

[chuck_starchaser]

indeed, it was always intended by me to have the creation of the dds files be done on the user end. I actually wanted the creation of the dds files to be done per-quality setting. Meaning, it wouldn't generate the dds files until you changed a setting to minimize memory usage. Then depending on the level of minimization, it would select another pre-defined list of textures to compress and cache as dds files.

I'd do it on a quality basis. Most users wouldn't get the meaning of "minimization". I'd have a compression quality settings like

• No compression
• Best quality compression (DXT5)
• Some quality (DXT3)
• No quality (DXT1) ((but really fast))

and I'd apply that quality to ALL textures. Much simpler.

It was thought that this would consume too much disk space (more than users would find acceptable) so it was then decided that we convert over a decent number of textures to dds at the distribution pkg level.

Disk space is not an issue at all, any more; my drives are 250 gigs each and they are a year old... Just warn in the settings that using compression will require additional disk space.

The good thing is dds files can be compressed by bzip2 and the like EXTREMELY well. To the point that the file is smaller than png compressed. You'll probably never beat jpg, but a number of really large textures are png and thus, we should pretty much even out even if we convert a large number of textures to dds.

This would kill the advantage Hellcat was mentioning, but on the other hand would reduce file system accesses and increase disk locality: How about using one huge rar file to cache all dds textures, then keeping the file open at all times, decompressing dds's as they are needed?

A batch program would be nice, but i dont have the time to write up a cross platform one.

Exactly. The biggest problem I see is recursing through all the folders and files, in a cross-platform way. I think boost's filesystem library is the thing for this, but I dread boost.

In any case, I really would like it if we would branch data4.x to data5.x (get rid of the old data5.x). All of the textures should go into a master branch, so we dont lose the originals.

I think I heard data5x is the ogre branch test data; if so I would skip to data6x.

EDIT:
Sorry for my ignorance: DXTn designations are more about alpha channel handling philosophies than about compression levels. Found this document explaining dxt compression pretty clearly:
http://wiki.beyondunreal.com/wiki/DXT

EDIT2:
Seems to me DXT1 is all we need, with or without the 1-bit alpha, depending on whether the original texture has alpha channel.

[safemode]

I meant compressing them (bzipping the dds files) for installation, after that they are as they are now.

tarballing up linked textures was another idea that was thrown around.

It's still a possibility if it turns out that disk access is the bottleneck. Though, i somehow doubt the disk access is the constraining factor.

If we get things using dds files mostly where possible and it still causes the stuttering effect when a unit is put into the game, then we need to look elsewhere as to the cause of that problem.


I found a script-fu plugin for gimp that does batch conversions. I made a small hack in the gimp_dds plugin to use the last save options for dds files. So what i do is save a file with the dds options i want, then use the batch convertor to save tons of them that way. Then i use a perl script called rename to rename all the files to the original filename.

Eventually we're going to have to do something about the filenames. They are basically meaningless. We could create our own extension for them in the tree, and keep the real extension in our master texture branch so we can visually see what format they're in. VS doesn't care about extensions except that they're hardcoded into various unit files and such.

Perhaps make all textures in VS .tex. We would copy the source texture from the master texture tree to the data5.x dir and rename it to .tex. This way we dont cause confusion by having .png's really be .dds files and etc. Everything will be a .tex no matter what codec it uses.

[safemode]

Right now i converted over a bunch of planet textures, a unit (llama), and a half dozen of the biggest animation textures.

Load time from start of VS to finish of loading a save game was 1 minute exactly on a non-altered data4.x dir.

Load time on my altered data4.x dir was 52 seconds.

Obviously, many more textures need to be converted to make a decent dent in the load time. And also obviously, load time is dominated by perl code cpu time, so optimizing texture loading isn't going to cut your load time in half. I need to convert over the units so i can see if it stops the stuttering in total_war_python

I somehow doubt it will, python is outputting messages like "launched 197 ships" so i'm guessing if that means it just inserted 197 ships, there is gonna be a lag.


EDIT:

loading the python total war mission yields a load without dynuniverse really getting in the way


load time without altered data4.x 19 seconds
load time with altered data4.x 4 seconds

That's some serious speed improvement there. Just goes to show you though, that the main hiccup in loading a regular game is not in the texture loading so much as the python.

[Halleck]

Give Klauss a chance to sync data5.x with the one in ogre_branch... I guess this will have to become data6.x based on ogre's new roadmap target.

[chuck_starchaser]

Yeah, I wouldn't touch data5x if I were you, safemode; that's Klauss' stuff, and he might send the Kilrathi to get you.

EDIT:

loading the python total war mission yields a load without dynuniverse really getting in the way


load time without altered data4.x 19 seconds
load time with altered data4.x 4 seconds

That's some serious speed improvement there. Just goes to show you though, that the main hiccup in loading a regular game is not in the texture loading so much as the python.

Okay, so, if I'm reading this correctly, pythonless loading time gets sped up 5 times ?!?!
But normal game loading doesn't benefit as much because pythons crawling time dominates?

[klauss]

nVidia has free (though not opensource) DXT compression tools that can run in batch mode. They're great, produce high-quality textures (DXT compression is very hard, and the realtime heuristics used in OpenGL drivers for on-the-fly compression are really crappy compared to good offline tools). I'd recommend using them - their only drawback is that they run only on Windows, and they use up a lot of RAM, and take a long time to compress stuff, but the quality output is worth it I guess. And maybe they run with wine.
That's what I used to create the few dds present in data5.x (I'm not sure whether I commited them - but if I did, ya, that's what I used).

If you want to rename the ogre dataset into ogre_branch/data5.x (or ogre_branch/data6.x) I'm ok with that. In any case, don't touch it much, as it's governed by a radically different filesystem.

About that fast loading directly off kernel buffers chuck was talking about... that is what I think happens (note the "I think" - I'm not absolutely certain, but I think so), but only with DirectX API. DX has an API for loading DDS files directly off disk, and what it does is map the file into virtual memory (mmap-style). If the graphics card is AGP, then that's it - any access off those virtual addressess will make the virtual memory system to page the file in, thanks to the MTRR (the famous AGPGART). If it's not AGP, or if it has to be loaded on graphics memory, it still gets loaded by the paging system through a memory lock, followed by bus transfer into the graphics card. In any case, the data is never copied around, so it's very fast. But OpenGL, with its OS-agnostic interface, can't do that. And we use OpenGL - that's yet another thing Ogre might help with, if it becomes an issue, but I doubt it, even copying data, loading off precompressed DDS is a big improvement, and anything further is just nitpicking.

And, as safemode said, DDS compresses very well with gzip/bzip2. I'd compress all textures offline, and distribute them compressed with gzip/bzip2, and have setup decompress them on install. Maybe VS could even load them compressed if the .dds texture is missing, since zlib can handle it rather easy (although it won't be as fast, it will be great for developers - they won't have to "install" vs to debug stuff).

I had settled on that for Ogre, at least - ogre can be extended to handle that automatic loading of .gzipped stuff rather easily - it can even load the dds files directly off a .zip, but that would be harder for SVN - if you change one texture, you have to commit the entire .zipped folder - bad stuff.

[safemode]

we're using the on the fly compression for all the textures as it is now anyway. unless you're running in highest quality mode with no compression at all, everything you look at in the game is compressed to the same compression codec that the dds files are compressed in using the same routines. Probably not with the GL_NICEST flag though like gimp-dds will use.

gimp will do batch recompression to dds, it does it cross platform, with open source and thus can be maintained indefinitely and edited if the need be.

take a png, compress it with your nvidia compressor. Send the compressed dds to me, i'll compress the same texture with gimp and see if there is any difference, positive or negative.


I vote for renaming the current data5.x to something else. The next release is going to be a 5.x release, it only makes sense that the data module is data5.x.


And as for this fast loading off of disk. Read this about mmaping

http://lists.freebsd.org/pipermail/free ... 50371.html

That being said, our cpu use seems to be stuck in python much more than in loading textures. This should decrease pauses some, but not enough i think. We need to work on either optimizing the python or moving key functions into C++ to speed it up.

[klauss]

we're using the on the fly compression for all the textures as it is now anyway. unless you're running in highest quality mode with no compression at all, everything you look at in the game is compressed to the same compression codec that the dds files are compressed in using the same routines. Probably not with the GL_NICEST flag though like gimp-dds will use.

I've seen many cases of users disabling compression because of the quality decrease, mostly with modern cards that have more than enough video memory. I'm talking about PR and privateer-related mods, which use much smaller texture sets.

gimp will do batch recompression to dds, it does it cross platform, with open source and thus can be maintained indefinitely and edited if the need be.

Maybe gimp can do high-quality compression too - I've never seen that plugin - but the point is still clear: precompressing dds offline is always better, quality-wise, than letting the driver do it. I don't think users would like the wait for that to happen during setup... so I'd rather distribute them in dds form already, and you said it yourself: dds + gz does a very good job (and sometimes better than png), compression-ratio-wise.
Anyway... I never ever considered the possibility of having the dds be alone, it's absolutely necessary to have a separate repository with editing-friendly media. I'd prefer xcf (or similar editable formats, with layers still separate and all), but whatever we have will do - but certainly in a lossless format.

take a png, compress it with your nvidia compressor. Send the compressed dds to me, i'll compress the same texture with gimp and see if there is any difference, positive or negative.

It almost sounds as if you didn't believe me. Almost hostile... I hope I'm misunderstanding you.
If you don't believe me... ok... your loss. Do the test if you want.

I vote for renaming the current data5.x to something else. The next release is going to be a 5.x release, it only makes sense that the data module is data5.x.

I'm ok with that. Move the current 5.x to ogre_branch/data6.x if you will.

And as for this fast loading off of disk. Read this about mmaping

http://lists.freebsd.org/pipermail/free ... 50371.html

Huh... cool stuff.
Anyway, that's completely not to the point. We (the article and I) are talking totally different things. In their case, the data doesn't fit in memory. In our case, it must fit in memory. So all the overheads are present in the read() call version too - since the buffer must hold the entire file anyway, and you can't disable paging. Also, it's the kernel itself doing the thing... it can optimize however it wants, contrary to linux - we're talking Windows here (I don't know any linux-based DirectX), and they're talking FreeBSD.

That being said, our cpu use seems to be stuck in python much more than in loading textures. This should decrease pauses some, but not enough i think. We need to work on either optimizing the python or moving key functions into C++ to speed it up.

I agree - I tried to say that, only with less details. Loading, provided on-the-fly compression is taken out of the equation, is not the issue.
Though back in the days I did profile VS, I noticed there's a relatively fair amount of time spent at loading sprites - they produce expensive calls to open() / close() (opening and closing files is rather expensive, no matter the cache). At the time I tried adding caches for the most important classes, but a unified resource cache would go a long way into fixing that (and better, more readable and modular code).

[safemode]

in regards to the mmap issue. The freebsd article describes how mmap in reality works, rather than in theory, which the guy who the article is responding to was trying to make. He was saying exactly what you're saying. That the kernel should be able to do X Y Z about mmap. In reality though, hardware doesn't allow mmap to work as nicely as you'd think. mmap is complex to the point that it's benefits are highly reliant on the size of data in addition to your specific computer hardware. It's not just a BSD thing, it's not just a linux thing. It's any kernel. If it seems to work better in windows to do it that way, then the windows code can do it that way (when the graphics backend supports it). But i wouldn't expect the same results across different cpu/memory controller/disk controller combinations. mmap would seem highly affected by the performance of cpu's dealing with page faults and kernels in predetermining them.

In any case, it's not like one method negates the other from existing.


In regards to the dds nvidia compressor vs gimp dds I wasn't being hostile. i really want you to use your nvidia compressor program to create a dds so that i can see how the gimp dds compressor performs in comparison.

The way i imagine quality levels in the future VS is like this:
We distribute VS with jpg's and dds files for all the images. The dds files are textures that do not have fine line details that would be readily apparent to have artifacts from compression during gameplay at high quality modes. JPG's are everything else.
We allow the user to pick varying levels of quality settings, these settings would contain directory paths to various groups of textures. The high qulity mode would include none, all textures are loaded as they are distributed.
The medium quality mode would include large textures, it would have paths that would lead to bases and the such.
etc etc
the low quality mode would include all the textures.

What would happen is, upon setting a new mode, the first run of VS will generate cached dds copies of the textures in it's path. These would be used in all subsequent runs until the quality level is changed.

In this way, people wanting low quality and super memory/cpu savings, can do so, and fast (after the first run), without having people who want higher quality get stuck with bad looking textures.

[safemode]

in regards to the mmap issue. The freebsd article describes how mmap in reality works, rather than in theory, which the guy who the article is responding to was trying to make. He was saying exactly what you're saying. That the kernel should be able to do X Y Z about mmap. In reality though, hardware doesn't allow mmap to work as nicely as you'd think. mmap is complex to the point that it's benefits are highly reliant on the size of data in addition to your specific computer hardware. It's not just a BSD thing, it's not just a linux thing. It's any kernel. If it seems to work better in windows to do it that way, then the windows code can do it that way (when the graphics backend supports it). But i wouldn't expect the same results across different cpu/memory controller/disk controller combinations. mmap would seem highly affected by the performance of cpu's dealing with page faults and kernels in predetermining them.

In any case, it's not like one method negates the other from existing.


In regards to the dds nvidia compressor vs gimp dds I wasn't being hostile. i really want you to use your nvidia compressor program to create a dds so that i can see how the gimp dds compressor performs in comparison. I dont have windows.

The way i imagine quality levels in the future VS is like this:
We distribute VS with jpg's and dds files for all the images. The dds files are textures that do not have fine line details that would be readily apparent to have artifacts from compression during gameplay at high quality modes. JPG's are everything else.
We allow the user to pick varying levels of quality settings, these settings would contain directory paths to various groups of textures. The high qulity mode would include none, all textures are loaded as they are distributed.
The medium quality mode would include large textures, it would have paths that would lead to bases and the such.
etc etc
the low quality mode would include all the textures.

What would happen is, upon setting a new mode, the first run of VS will generate cached dds copies of the textures in it's path. These would be used in all subsequent runs until the quality level is changed.

In this way, people wanting low quality and super memory/cpu savings, can do so, and fast (after the first run), without having people who want higher quality get stuck with bad looking textures.

This has to beat recompressing them every single load... i mean, it's the same quality. Either every texture gets compresed by the driver everytime you reload the texture or every texture gets compressed by the driver once. Which is better? I haven't studied it closely enough, but there are GL_HINT's to alter the methods used by a driver to compress a texture, and every video card chipset compresses it at differing qualities at those settings, i'm not sure if VS uses those hints on a per-quality basis, or at all.

Note, i'm talking about the non-dds distributed textures above.

[chuck_starchaser]

I don't think users would like the wait for that to happen during setup... so I'd rather distribute them in dds form already, and you said it yourself: dds + gz does a very good job (and sometimes better than png), compression-ratio-wise.

I highly doubt that the wait to generate dds's during setup would even compare to the added time to download them. I'm still of the opinion that textures without alpha should all be jpg in the download. We're talking a good 10 or 20 to 1 difference in size. Maybe sprites with alpha channel that would have to come through as png, those might as well be deliverd as gzipped dds's; but large textures (512+), specially nebula backgrounds, many of which are bmp's (?!?!?!), should be jpg.

[safemode]

Eventually that may be how things are done. but it requires a cross platform installer that can handle determining how to compress a given input into the correct DDS output. There's nothing wrong with ust shipping things premade for now (it isn't going to be any larger of a distribution pkg than it is now ) until such an installer is created.

[hellcatv]

Apparently some of you aren't on Internet2...compression of DDS files would take me way longer than downloading the installer

Besides then you get a double-lossy sort of problem.... you lose once from raw->jpg and another from jpg->dds it'd be like recompressing all your mp3 files to ogg...

now you could store everything as png...but then it will take you folks on Internet1 even longer to download--- and it will increase compression time to boot (not too significantly, but alas)

Now you could argue that most artists save stuff as jpg as a matter of course---but ideally in the future we'd have all our art as layered xcf or psd files without any sort of lossy compression so we can always convert to whatever newfangled compression cards will support in the future

[klauss]

You should really reread the FreeBSD post. All that overhead relates to mmap's usage of the paging subsystem. The paging subsystem happens to incurr the overhead whenever you access a new page - new in the sense that you haven't accessed before (or often). With static application buffers (32k processing buffers, for instance), that's nonexistent, as you only allocate 8 pages (assuming 4k pages) once and never again incurr in the overhead of the paging system since those 8 pages get accessed too often to ever get paged out in normal conditions. However, in the way the guy used mmap, he would incurr into that overhead for every 4k of data to be processed. That's what made mmap-style memory mapping inherently slow. With DDS textures that's not so - you load the entire DDS to memory, and you need it all there. You usually can't process it in chunks, and if you happen to load big enough textures I bet DX will switch methods "intelligently". In any case, that kernel-based loading procedure is faster since it can select the best method.
Again, let me repeat myself, I'm talking about DirectX - Windows always, and I'm not talking about implementing anything, I'm saying that using DX's API to load DDS files directly is better because it's implemented on the kernel side, where it can (and does) use that stuff. Heck, even file copying uses that - ever wondered why you get an "error performing inpage operation" while copying small files off defective disks?

So... any discussion is an academic excercise, there's nothing to do, and I'm only commenting this to justify the usage of DirectX rendermanagers with Ogre when in Windows. That's all.

Now... about nVidia vs Gimp, I just checked nVidia's site and now they distribute the source too - still they don't run in Linux I think. I'll see if I can post a few comparison. I just compiled the dds plugin for gimp, and in fact it uses OpenGL's realtime compression so it's bound to be low-quality. So you obviously haven't ever used high-quality compression tools. Let me clarify something - they take a long time to run... they are slow... and memory hungry. You don't want a setup procedure that runs a batch of those - certainly not.
If I were you, I'd concentrate on getting the highest-quality options working, and then work on lesser-quality options. Why? Because I've worked on ways to load only the lowest MIPs off a DDS with Ogre, and it somewhat works. Even if it didn't work in the end, because it isn't as fast, it can be worked on later. Thing is, I've been working on efficient ways to get quality settings without going the brute-force route of simply distributing several datasets, and it is possible.

BTW: For comparison, high-quality compression of a rather normal base background image takes about 1 minute (ok, super-quality settings which probably are overkill most of the time). It's a long time. The quality improvement is subtle, and only really important on some bad cases for DXT - I can't find any good example now, but I remember, for instance, that Gold's splash screen suffered greatly from compression artifacts, some starry backgrounds in vanilla Vega Strike too. In those cases, high-quality compression did a great job of minimizing those artifacts. In most other cases the differences were small, but in general smoother.

[safemode]

I'd just like to get a dds compresed with the nvidia tool using any texture in VS currently made as dds ...since it's basically agreed that the currently compressed textures are ok to be shipped as dds. Then compare it to my own rather than argue back and forth. The textures you compress with gimp-dds wont look the same as the ones I compress.

I want to see the difference before i go about hacking the gimp-dds plugin to always use software compression and produce "higher" quality dds textures either using nvidia code or some other code if the code in gimp-dds for software compression isn't sufficient.

The software compression is in there, you just dont use it because the plugin defaults to hardware if it's available.

In any case, just compress up an explosion texture or one of the other ones i've already changed. send it to me at safemode2@comcast.net or do up one of the base unit textures since i was eying those up too for compression due to their size.