Image Types and Image Extensions

Development directions, tasks, and features being actively implemented or pursued by the development team.
Post Reply
pheonixstorm
Elite
Elite
Posts: 1567
Joined: Tue Jan 26, 2010 2:03 am

Image Types and Image Extensions

Post by pheonixstorm »

I've been thinking on this for a few days and was wondering.. why do we use both jpg and png? Was the initial design for everything done in jpg then slowly switched over to png or was it a design decision to use both? Why not jpg or tga or png and tga for that matter?

On to the next issue. Since ogre does not like having dds files with our current naming convention (.image and .texture) would it not be wiser to go through and rename all the image files back to dds? I know it would require some work to get them all changed over and changed in svn but it would make life easier for adding new files into the ogre fork/branch and back from ogre. Since vegaogre shares the same dataset with vegastrike this just seems to be more logical as well as allowing dds readers the chance to actually READ and preview the files (even with a batch file renaming is irritating when I want to view a file).

Anyway, thats my idiot idea for the nigt.
Because of YOU Arbiter, MY kids? can't get enough gas. OR NIPPLE! How does that mkae you feeeel? ~ Halo
charlieg
Elite Mercenary
Elite Mercenary
Posts: 1329
Joined: Thu Mar 27, 2003 11:51 pm
Location: Manchester, UK
Contact:

Re: Image Types and Image Extensions

Post by charlieg »

I'm guessing JPG has been used for file size consideration. PNG is usually lossless but this means photographic-y images are massive as PNG. It might be that JPG images could be stored as DDS but I'm not knowledgeable on the differences between those two formats.
Free Gamer - free software games compendium and commentary!
FreeGameDev forum - open source game development community
breese
Bounty Hunter
Bounty Hunter
Posts: 152
Joined: Thu Sep 02, 2010 8:00 pm

Re: Image Types and Image Extensions

Post by breese »

pheonixstorm wrote:On to the next issue. Since ogre does not like having dds files with our current naming convention (.image and .texture) would it not be wiser to go through and rename all the image files back to dds?
Concur.
klauss
Elite
Elite
Posts: 7243
Joined: Mon Apr 18, 2005 2:40 pm
Location: LS87, Buenos Aires, República Argentina

Re: Image Types and Image Extensions

Post by klauss »

No renaming, it's a huge hassle.

Ogre should use magic numbers to detect file formats, if it doesn't... well... we could contribute it :p

In any case, if renaming results a necessity for Ogre, let the ogre branch do it. Doing it in trunk now would be both premature and a huge hassle.
Oíd mortales, el grito sagrado...
Call me "Menes, lord of Cats"
Wing Commander Universe
charlieg
Elite Mercenary
Elite Mercenary
Posts: 1329
Joined: Thu Mar 27, 2003 11:51 pm
Location: Manchester, UK
Contact:

Re: Image Types and Image Extensions

Post by charlieg »

I agree with klauss. It's insane to rename all the files in trunk just for the sake of it.
Free Gamer - free software games compendium and commentary!
FreeGameDev forum - open source game development community
pheonixstorm
Elite
Elite
Posts: 1567
Joined: Tue Jan 26, 2010 2:03 am

Re: Image Types and Image Extensions

Post by pheonixstorm »

@klauss actually its not a huge hassle to rename them all. I use a nice batch file for when I want to view them in the nvidia dds viewer. The hassle is switching from .image to dds then from dds to .image (or .texture).

simple batch
rn *.image *.dds
rn *.texture *.dds

I think the real trick will be getting svn to figure out what we have done. Oh well, I can try at home and check the results.

If ogre doesn't know which format it is.. well, thats bad design but then again how many games rename thier file extensions??

Let the ogre branch do it? That might be well and dandy but there are 2 ogres. And what if another fork shows up using some other back end that doesnt supported magic numbers? What about the various dds viewers that don't recognize that our .image and .texture files are really dds files (and its not just the nvidia viewer that does this).

Well, something to discuss at least.. there are far more important projects on the board than this.

@charlieg not for the sake of it. For the sake of our forks that use a different 3d object format and may or may not be able to use our custom image extension.
Because of YOU Arbiter, MY kids? can't get enough gas. OR NIPPLE! How does that mkae you feeeel? ~ Halo
breese
Bounty Hunter
Bounty Hunter
Posts: 152
Joined: Thu Sep 02, 2010 8:00 pm

Re: Image Types and Image Extensions

Post by breese »

pheonixstorm wrote:I think the real trick will be getting svn to figure out what we have done.
"svn move" should work just fine. It even retains the history.
klauss
Elite
Elite
Posts: 7243
Joined: Mon Apr 18, 2005 2:40 pm
Location: LS87, Buenos Aires, República Argentina

Re: Image Types and Image Extensions

Post by klauss »

pheonixstorm wrote:@klauss actually its not a huge hassle to rename them all. I use a nice batch file for when I want to view them in the nvidia dds viewer. The hassle is switching from .image to dds then from dds to .image (or .texture).

simple batch
rn *.image *.dds
rn *.texture *.dds
You forgot:
  • To update every bfxm that reference those textures/images
  • To update all base scripts that reference those texture/images
  • To update the code, since some texture/image defaults are hardcoded
  • To update all technique xml, units.csv, etc... that reference those textures/images
  • To make sure they're DDS before renaming (not all of them are DDS, there are PNGs and JPEGs)
Furthermore, when you develop content, you may want to switch from DDS to PNG to JPEG, testing which works best. You'd have to update all references when you do if you were to use the proper extension. With image/texture, the extension (file name) doesn't change, and no references need updating.

Again... ask safemode, he's been through this mess before. Renaming isn't simple, it IS a hassle, it requires every user using SVN to re-download everything (svn move is benign server-side, but not client-side).
Let the ogre branch do it? That might be well and dandy but there are 2 ogres. And what if another fork shows up using some other back end that doesnt supported magic numbers?
Then that other fork does the renaming too.
We can't be expected to cater to other engine's needs.
What about the various dds viewers that don't recognize that our .image and .texture files are really dds files (and its not just the nvidia viewer that does this).
Well, either
  • don't use them (pick one that does detect the type), or...
  • rename them when you want to view them but only on your working copy, or...
  • configure your browser to treat .texture as .dds (it's usually possible and OK).
@charlieg not for the sake of it. For the sake of our forks that use a different 3d object format and may or may not be able to use our custom image extension.
Forks are forks. We don't have to be compatible with them, it's the other way around. What if there were two forks with incompatible requirements? would we be required to support both file formats/naming conventions?

No.

Simple and plain no.
Oíd mortales, el grito sagrado...
Call me "Menes, lord of Cats"
Wing Commander Universe
pheonixstorm
Elite
Elite
Posts: 1567
Joined: Tue Jan 26, 2010 2:03 am

Re: Image Types and Image Extensions

Post by pheonixstorm »

klauss wrote:[*] To update the code, since some texture/image defaults are hardcoded
This I did not know... why are they hardcoded?? Oh well...

So what about my other question, why png and jpg?
Because of YOU Arbiter, MY kids? can't get enough gas. OR NIPPLE! How does that mkae you feeeel? ~ Halo
charlieg
Elite Mercenary
Elite Mercenary
Posts: 1329
Joined: Thu Mar 27, 2003 11:51 pm
Location: Manchester, UK
Contact:

Re: Image Types and Image Extensions

Post by charlieg »

PNG is lossless. JPG looks better for photographs. TBH you could probably drop JPG since (my understanding is) it has some crossover (as a compression techinque) with DDS.

You can't drop PNG though.

To be honest I totally disagree with the way it has been done (.image/.texture extensions) when really it should have been a simple search for .dds, .png, or .jpg and pick them up in that order (i.e. some kind of function loadImageFile("filename") and you omit the extensions) - the engine looks for any 1 of the 3.

However, saying that, creating extra work at the moment in trunk is just a bad idea. Release. Where's 0.6? Or 0.5.1? How many years between releases?

Release early, release often. You never know, it might actually bring more developers to the project as the game will get in front of more eyes.
Free Gamer - free software games compendium and commentary!
FreeGameDev forum - open source game development community
klauss
Elite
Elite
Posts: 7243
Joined: Mon Apr 18, 2005 2:40 pm
Location: LS87, Buenos Aires, República Argentina

Re: Image Types and Image Extensions

Post by klauss »

charlieg wrote:To be honest I totally disagree with the way it has been done (.image/.texture extensions) when really it should have been a simple search for .dds, .png, or .jpg and pick them up in that order (i.e. some kind of function loadImageFile("filename") and you omit the extensions) - the engine looks for any 1 of the 3.
That's an option. In fact, it should also look for .dds.gz (because DDS is highly compressible by deflate), but the .image rationale isn't half as bad as it seems: Looking for a file is a rather expensive operation, load time increases significantly with these kind of automatic probes.

Going straight to the .image/.texture file is, literally, 4 times faster than trying .dds, then .png, then .bmp, then .jpg. Even worse if you consider that you have to try .dds in every mod folder before proceeding to .png in every mod folder... etc.

All those probes end up wasting a lot of time, I've profiled it. To the point I had to add a "probe miss cache" (make VS remember which files aren't there) to speed up stuff back into reasonable levels.

Furthermore, VS already probes several locations (the unit folder, plus the textures folder, etc...), maning each time VS looks for a texture, it already probes about a dozen places. If you make it probe for each supported extension, that would be 4 or 5 dozen places. To load a single file.

A bit too inefficient for my taste.
charlieg wrote:However, saying that, creating extra work at the moment in trunk is just a bad idea. Release. Where's 0.6? Or 0.5.1? How many years between releases?
Ya... lets remember that. Feature-wise, VS is almost ready for 0.5.1.

Lets forget the minutia and push forward.
Oíd mortales, el grito sagrado...
Call me "Menes, lord of Cats"
Wing Commander Universe
Deus Siddis
Elite
Elite
Posts: 1363
Joined: Sat Aug 04, 2007 3:42 pm

Re: Image Types and Image Extensions

Post by Deus Siddis »

Also keep in mind that our present content set is a bloated outdated mess. When the time comes to move over to OGRE completely, it would be best to leave most of the old content behind anyway.

The stuff is impossible to maintain in present quantity and lowers the project's visual standards, expectations and potential in present quality.
charlieg
Elite Mercenary
Elite Mercenary
Posts: 1329
Joined: Thu Mar 27, 2003 11:51 pm
Location: Manchester, UK
Contact:

Re: Image Types and Image Extensions

Post by charlieg »

klauss wrote:That's an option. In fact, it should also look for .dds.gz (because DDS is highly compressible by deflate), but the .image rationale isn't half as bad as it seems: Looking for a file is a rather expensive operation, load time increases significantly with these kind of automatic probes.

Going straight to the .image/.texture file is, literally, 4 times faster than trying .dds, then .png, then .bmp, then .jpg. Even worse if you consider that you have to try .dds in every mod folder before proceeding to .png in every mod folder... etc.
Normally you are right on technical matters. Here, I suspect you are wrong. Performing file checks on modern file systems is very quick. It has been for several years. 10 years ago this could have been a valid viewpoint, but now it's a bit archaic.

I remember 5s being knocked off my bootup time ~2003 because it was discovered that a particular file was being stat()-ed 10,000 times. That was 2003. 5s. 10,000 'does file exist' operations.

Also, there's going to be a majority file format, DDS.gz I presume. So for most files it's 2:1 since the DDS.gz exists (2:1 as check exists before load). 3:1 on PNG, 4:1 on JPG. Worst case is 5:1 (plain DDS - mandate official DDS files must be compressed with gzip but support plain .dds for testing).

Iterating over files and checking their contents, yes that may take time. Simply checking whether a file exists? Microseconds. The benchmarks for boost on this do 1 million operations.

I am trying to find some actual benchmark results but it's tough. I guess somebody could try it?

Code: Select all

boost::filesystem::exists( "myfile.txt" )
Just run it a lot of times, against a file that exists and a file that doesn't. I know that's simplistic, that there are all kinds of factors such as IO caching and file system implementation, but I don't think it's going to have the hugely detrimental effect that klauss is talking about.

There's a software development saying: don't optimize early. If you avoid doing things because you think they might be slow you go on to make decisions that have future ramifications, such as this.
Free Gamer - free software games compendium and commentary!
FreeGameDev forum - open source game development community
pheonixstorm
Elite
Elite
Posts: 1567
Joined: Tue Jan 26, 2010 2:03 am

Re: Image Types and Image Extensions

Post by pheonixstorm »

Considering how old the engine is its not suprising some things aren't or weren't done a certain way. If we had the people to modernize the code I way say that we could probably get at least a 10% speed increase

I think the majority of our time lost is going into our current data directory structure. We place all images in hundreds of different directories when it would take less time (even having three image formats) to load all files from 1 central directory or file such as biowares erf, Ids wad (i think its a wad), .forge for Assassins creek, mpq for blizzard... I think what takes so long for VS is the fact we have over 1gb of data spread over so many folders.

This itself (with or w/o discussion of changing the images) should be looked into at some point. If ogre is .6 we should standardize our loading and directory structure. Currently we have images in cockpits, textures, bases, units (each unit directory), etc. This is a lot of searching for the games images not to mention loading the content files which are also scattered about.

Anyone feel the same??
Because of YOU Arbiter, MY kids? can't get enough gas. OR NIPPLE! How does that mkae you feeeel? ~ Halo
charlieg
Elite Mercenary
Elite Mercenary
Posts: 1329
Joined: Thu Mar 27, 2003 11:51 pm
Location: Manchester, UK
Contact:

Re: Image Types and Image Extensions

Post by charlieg »

No, I think you're all making assumptions about what is taking a long time without accurate profiling data.

I don't think the folder structure will particularly matter. Why would it? Let's suppose there's 5000 files in 1000 folders. The length of time to get file handles for all those files can't be more than a few seconds. I mean, for example, in Eclipse I regularly perform searches across thousands of files in no more than 10-20s. (I'm being conservative in that estimate too.) That is searching the files, as in going through every byte and tokenizing them. Those files in are deep* nested folders too.

* Example: /org.vexi-build.jsdoc/gen/main/java/org/vexi/tools/autodoc/parse/Parser.java

You're worrying about the wrong things, thinking of how it will be optimal without real data backing the decisions. What is truly optimal will be what makes Vega Strike 1) easier to work with for developers and 2) easier to work with for Artists. If the code base and code paths are clearer, then the real optimization work can begin because you can understand what is going on and realistically profile.

Don't create lots of work by guessing what the fastest-for-IO layout is. Do things for a tangible reason and not based on assumptions.
Free Gamer - free software games compendium and commentary!
FreeGameDev forum - open source game development community
Philip
Atmospheric Pilot
Atmospheric Pilot
Posts: 1
Joined: Tue Jan 25, 2011 2:55 pm

Re: Image Types and Image Extensions

Post by Philip »

On 0 A.D. we have some IO profiling data - apparently the performance when loading ~500 files (on Windows (NTFS), from a cold cache) was about 3 MB/sec, while bundling everything into a compressed archive and using async IO reached about 30 MB/sec, with most of the benefit coming from the bundling. (I've seen similar (but not accurately measured) speedups recently with the same code on Linux (ext4).)

The main difference sounds like the reduction of disk seeks (which take ~10ms each). The archive's list of files can be loaded as a single chunk on startup and then file existence checks are simply a hash lookup, and reading a file is simply a seek+read (with no internal fragmentation (i.e. wasted space when you have lots of tiny files) and little external fragmentation (since the OS likely used contiguous disk blocks for the archive) so disk caches will work well). With loose files instead, the OS would have to read all the directory metadata and check access rights (and the virus scanner will probably chug around in the background too), and it's painfully slow due to the random access patterns (at least until the second time you run the game, when the OS has cached everything in RAM - the IO performance matters more for players (who only run once per reboot) than for developers).

So I think there's evidence that it can make a big difference in practice. It's probably not a good use of time to reimplement a system like this from scratch, though - there's e.g. PhysFS which already does this. (Also 0 A.D.'s 'lib' code (including the virtual filesystem stuff, which apparently is more flexible and efficient than PhysFS) is designed to be reusable in other projects - currently it's only been reused by its original programmer but it ought to be feasible to make it work elsewhere.)
greenfreedom10
Hunter
Hunter
Posts: 67
Joined: Wed Sep 29, 2010 3:06 am
Location: Lost in an adventure

Re: Image Types and Image Extensions

Post by greenfreedom10 »

charlieg wrote:What is truly optimal will be what makes Vega Strike 1) easier to work with for developers and 2) easier to work with for Artists. If the code base and code paths are clearer, then the real optimization work can begin because you can understand what is going on and realistically profile.
Agreed! I am guessing that most users are not significantly interested in how long it takes for VegaStrike to start. What does matter is the game experience after it is running, and to this end, that the code/data structure be understandable and maintainable. So new contributors can jump in and start cranking out contributions quickly! I would much rather write/see code that is slow but easy to understand/maintain than fast code that I can not understand when I am trying to squash a bug.
klauss
Elite
Elite
Posts: 7243
Joined: Mon Apr 18, 2005 2:40 pm
Location: LS87, Buenos Aires, República Argentina

Re: Image Types and Image Extensions

Post by klauss »

charlieg wrote:No, I think you're all making assumptions about what is taking a long time without accurate profiling data.
And you're assuming I didn't profile :P

Which I did. Ya, years ago... sure... but still, profiling identified file lookups as a significant bottleneck, mostly because every time a unit is launched hundreds of such checks have to be made, for customized faction-textures and whatnot.

Each check also did several checks... so the thing was pure madness.

In any case, true, modern computers might be a lot better at that. But still... why change things to be less efficient?
charlieg wrote:I mean, for example, in Eclipse I regularly perform searches across thousands of files in no more than 10-20s. (I'm being conservative in that estimate too.) That is searching the files, as in going through every byte and tokenizing them. Those files in are deep* nested folders too.
Eclipse does intensive preindexing though... it cheats.

Which, I'd say, VS should too. VS should build a catalogue of all available content at load time and never again.
charlieg wrote:You're worrying about the wrong things, thinking of how it will be optimal without real data backing the decisions. What is truly optimal will be what makes Vega Strike 1) easier to work with for developers and 2) easier to work with for Artists. If the code base and code paths are clearer, then the real optimization work can begin because you can understand what is going on and realistically profile.
I guess people don't remember all the intrusive changes brought about when the exact reverse rename was made. That is, when .bmp/.jpg/.whatever was renamed into .texture/.image.

There was a rationale there (there's even a blog post about it), and agree or not with it, the process was massively intrusive to all developers and many users. Not to name mods.

Repeating the same process to UNDO it is insane.

Saying that using .jpg/.dds is easier to work with is ignoring safemode's blog post about why the rename was done on the first place.

Saying that statting takes microsecods is wishful thinking:

Code: Select all

claudiofreire@klauss:~$ time stat memmap.py
  File: `memmap.py'
  Size: 4822      	Blocks: 16         IO Block: 4096   regular file
Device: 809h/2057d	Inode: 11452617    Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/claudiofreire)   Gid: ( 1000/claudiofreire)
Access: 2010-11-04 17:21:57.000000000 -0300
Modify: 2010-10-19 14:56:11.000000000 -0300
Change: 2010-11-18 15:46:39.000000000 -0300

real	0m0.040s
user	0m0.000s
sys	0m0.004s
(on a core i5)

Pick your number. 4ms. 40ms. Times 30 (average number of probes VS makes without a cache to create a texture). Times 1000 (number of units spawned when loading a heavily populated system). Times 2 (number of textures per unit on average).

That's 4ms x 30 x 1000 x 2 = 240s = 4 minutes

So...
...a cache helps.
...not doing unnecessary probes matters.

(ya, I know... fstat() must be faster, but I don't expect it to be a lot faster, it still requires a requires several syscalls)
Oíd mortales, el grito sagrado...
Call me "Menes, lord of Cats"
Wing Commander Universe
breese
Bounty Hunter
Bounty Hunter
Posts: 152
Joined: Thu Sep 02, 2010 8:00 pm

Re: Image Types and Image Extensions

Post by breese »

charlieg wrote:No, I think you're all making assumptions about what is taking a long time without accurate profiling data.
But it is more fun to guess than to measure :wink:

I profiled the VS initialization a few weeks ago, and the vast majority of time is spent reading and converting sound formats, and parsing XML files.

VSFileSystem::FileExists() spends 1% on stat() and 99% on string manipulation.
charlieg wrote: You're worrying about the wrong things, thinking of how it will be optimal without real data backing the decisions. What is truly optimal will be what makes Vega Strike 1) easier to work with for developers and 2) easier to work with for Artists. If the code base and code paths are clearer, then the real optimization work can begin because you can understand what is going on and realistically profile.
Well spoken.
ghoulsblade2
Bounty Hunter
Bounty Hunter
Posts: 136
Joined: Tue Jul 27, 2010 7:41 pm
Location: germany
Contact:

Re: Image Types and Image Extensions

Post by ghoulsblade2 »

Times 1000 (number of units)
why reload already loaded files ?

anyway, i'd suggest not changing anything there for now, there sure are more pressing problems than file-names.
klauss
Elite
Elite
Posts: 7243
Joined: Mon Apr 18, 2005 2:40 pm
Location: LS87, Buenos Aires, República Argentina

Re: Image Types and Image Extensions

Post by klauss »

ghoulsblade2 wrote:
Times 1000 (number of units)
why reload already loaded files ?

anyway, i'd suggest not changing anything there for now, there sure are more pressing problems than file-names.
Thing is, if it failed to load it, it will try again. And again. Well... would... past...

Also, there are tons of variants that are already being checked every time. Like textures appended with faction names (so you can customize the painting on ships of each faction), and things like that.

If you take a look, it's really amazing how a simple texture load explodes into more file operations that one would have imagined.

Making the operation even more complex doesn't sit well with me, I'd rather try to simplify it.
Oíd mortales, el grito sagrado...
Call me "Menes, lord of Cats"
Wing Commander Universe
charlieg
Elite Mercenary
Elite Mercenary
Posts: 1329
Joined: Thu Mar 27, 2003 11:51 pm
Location: Manchester, UK
Contact:

Re: Image Types and Image Extensions

Post by charlieg »

klauss wrote:Eclipse does intensive preindexing though... it cheats.

Which, I'd say, VS should too. VS should build a catalogue of all available content at load time and never again

....

Pick your number. 4ms. 40ms. Times 30 (average number of probes VS makes without a cache to create a texture). Times 1000 (number of units spawned when loading a heavily populated system). Times 2 (number of textures per unit on average).

That's 4ms x 30 x 1000 x 2 = 240s = 4 minutes

So...
...a cache helps.
...not doing unnecessary probes matters.

(ya, I know... fstat() must be faster, but I don't expect it to be a lot faster, it still requires a requires several syscalls)
Nail meet head.

Early optimization (avoidance of file lookups) is the wrong approach.

Optimizing the 1000 unit creations by caching the file lookups is the correct approach.

Don't optimize early. Optimize correctly.

(Btw the 'time stat' is unfair. It's timing the output too. That's why it takes so much time.)

Completely agree on how intrusive this was and would be and why doing it to trunk is unwise.

Btw, here's the blog post klauss referenced:
http://vegastrike.sourceforge.net/devblog/?p=34
Last edited by charlieg on Tue Jan 25, 2011 10:56 pm, edited 1 time in total.
Free Gamer - free software games compendium and commentary!
FreeGameDev forum - open source game development community
klauss
Elite
Elite
Posts: 7243
Joined: Mon Apr 18, 2005 2:40 pm
Location: LS87, Buenos Aires, República Argentina

Re: Image Types and Image Extensions

Post by klauss »

breese wrote:I profiled the VS initialization a few weeks ago, and the vast majority of time is spent reading and converting sound formats, and parsing XML files.
I'm not talking about VS initialization that happens once, but what happens when you launch a unit, which is a major source of stutter.
breese wrote:VSFileSystem::FileExists() spends 1% on stat() and 99% on string manipulation.
guess why

Not the harmles stat() you'd expect.
Oíd mortales, el grito sagrado...
Call me "Menes, lord of Cats"
Wing Commander Universe
safemode
Developer
Developer
Posts: 2150
Joined: Mon Apr 23, 2007 1:17 am
Location: Pennsylvania
Contact:

Re: Image Types and Image Extensions

Post by safemode »

The filename change was 100% a means to help artists and contributors to deal with the fact that certain data files (not usually C++ or python scripts) had explicit names for certain files and that changing these was rarely ever done in accordance with the changing of the actual image codec. As time progresses png and jpeg may be supplanted by something better suited (dds) or something newer and hunting down everything that called that particular filename with that particular extension would be time consuming and thus probably not done. So the person would just save the new file as the old filename extension and thus the cluster F was born that was the vs Data repo not so long ago.

by making it something not linked to codec/compression etc we simplify things for contributors by making it so that updating images or having multiple versions of the same image in different formats (for high quality and low quality etc) doesn't require any non-local changes to anything.

Almost any image editor or viewer wont use extensions to identify content you explictly give it to open, they use mime-types, which is what VS uses too. Windows users may suffer the most as so much is extension oriented in the shell and such that most actions with "unrecognized" files will have to be manual. But that's a shortcoming of that architecture. Relying on extensions to identify a file is a broken means and i think we had upwards of 10 years of evidence of that prior to renaming.

sorry for the rant. now about unit profiling

You have to be really careful about how you read the profiling output. String conversions occur a metric ton compared to most other functions because of the heavy scripting we do.

FS seeks for things like custom textures and faction oriented textures could be pre-fetched in a way by doing the seek at startup and appending an attribute in the in-memory units.csv map telling the engine whether or not any additional textures exist other than the standard one so that additional seeks never need to be made for those requests for every unit created. Also something that may be useful is run-time path culling. VS has many paths it looks for for content, especially in the case of mods or hq-textures being enabled. An initial pass can be made on all paths and any that return o file not found error or contain nothing can be eliminated, reducing misses in subsequent searches.

This would make creating units without custom textures take about as much time as the initialization of the unit class as textures would usually already be loaded in memory, and python wouldn't waste time looking for anything extra.

i like the idea of clustering into a big single file, but that kind of thing would be something done on the end-user's machine during install. It would be insane to maintain that in source control or via updates.
Ed Sweetman endorses this message.
ghoulsblade2
Bounty Hunter
Bounty Hunter
Posts: 136
Joined: Tue Jul 27, 2010 7:41 pm
Location: germany
Contact:

Re: Image Types and Image Extensions

Post by ghoulsblade2 »

so the original reason of the file-renaming was that a lot of files referred the texture-filename directly.

just some food for thought on that : in ogre they use material-names rather than texture-filenames directly. so all the different files could refer to the material name,
and there'd be a plaintext .material file detailing what textures files (and shaders etc) should be used for that material.
That might also reduce some of the planet/atmosphere/shader-packed-into-texture-attribute hacking in universe/sectors etc.
Post Reply