11447 crashes on texture load

Post by **klauss** » Fri Aug 31, 2007 5:45 pm

Hi... I wish I had the time to be more useful (ie: patch), but I guess a detailed report will have to suffice:

SVN 11447 crashes on load (it has done that for a while now, not just 11447).
SuSE 10.1, linux.

Stderr/out w/ gdb backtrace attached.
Seems like a DDS texture isn't loading properly, and the "graceful recovery" isn't so graceful.

Post by **loki1950** » Fri Aug 31, 2007 10:50 pm

Thx klauss you are not alone as the saying goes.

Enjoy the Choice

Post by **klauss** » Sat Sep 01, 2007 3:53 am

Good news: found that bug (I got tired of schoolwork, and decided to give 10 minutes to this problem

)
Bad news: this DXT thing makes it impossible for now (ie: hard, not going to do it, not yet) to do automatic downsampling of textures. So the "max_texture_dimension" config option which is so crucial for supporting lesser-memory cards and systems isn't working

I'll be commiting the fixes shortly...

Post by **hellcatv** » Sat Sep 01, 2007 4:39 am

oooh you fixed the bug?

when it crashed on my machine just now it was a 512x512 image that was not compressed with DDS in the forst place

my impression was that it was calling gluBuild2DMipmaps with a buffer 1/3 the size required for a 24 bit uncompressed texture...

I haven't figured out why it's getting a buffer so small yet--but did you see the same thing? I think the downsampling problem is a different one than the one I'm seeing on windows

Post by **safemode** » Sat Sep 01, 2007 6:36 am

the problem is non power of two sizes will still convert to a dds file but it is not compressed. This is valid. but i dont support it because it doesn't do any good and makes no sense.

We really should just enforce a power of two policy with textures. they all must be scaled in either the X or Y independently if need be to be a power of two. Using gimp or photoshop to do this is much better than the in-game scaling that's done.

I've been converting some images i've found that are 1280 by something, and things are working much nicer.

there are unit textures like this, sprites, menu images.
They all must be converted to power of two sizes. I assumed nvcompress would scale that automatically, rather than produce a dds that was uncompressed.

In any case, my data5.x-test branch will be getting these updated textures

Post by **hellcatv** » Sat Sep 01, 2007 10:07 am

Firstly---at least it loads now under windows thanks to the fact that klauss fixed grayscale uncompressed jpeg files! So folks should update

secondly--safem0de could you commit actual individual texture power of two resize FIXES to the svn HEAD? it would be nice to have anything other than glacial framerates when playing

I find it hard to even click Campaign on the menu screen since it's 1280...and since I'm on a mac right now it's difficult to fix the texture myself...

but i DEFINITELY agree---power of two textures only---
we should make a tool to go through and make sure

Post by **klauss** » Sat Sep 01, 2007 4:33 pm

nvdxt (at least this old version I had) had a parameter for rescaling:

Code: Select all

nvdxt -dxt5 -rescale hi -RescaleGaussian -file something.jpg -output something.dds

The new one must have something similar.

(the "hi" is for nearest highest, but you could use "lo" or "nearest")

Post by **safemode** » Sat Sep 01, 2007 4:53 pm

i'm going to apply all texture fixes to the data5.x-test branch. I want them to be tested by someone else prior to merging to svn head.

The reason being, I never experienced any glacial framerates, even prior to changing textures to power of two. I dont experience a lot of the crashes and problems other people seem to be experiencing and I usually dont have the time to immediately respond to people's complaints about X Y or Z causing their system to crash. It may take a day for me to get the time to deal with it.

So, from now on i'll be committing all my texture fixes to the data5.x-test dir, after someone verifies that a given texture from there is good, it can be merged into svn head. I'll also be scaling back my merges to svn head from my safemode branch. Because my system likes to hide memory corruption and the like from me, I want at least one other person to test-run the changes in my tree.

If we want to keep the svn in a constant state of readiness or everyone flips out, then there has to be some peer review to the commits to head.

Post by **safemode** » Sat Sep 01, 2007 5:17 pm

nvcompress (nvidia's dds compressor) seems to be either broken as it is on their site, or people just dont use the bpp settings anymore. I'm not sure. It looks as if none of the textures i compressed with nvcompress have bpp or depth set. The gimp compressor does set those values.

This really sucks.

It doesn't set other parts of the header as well and i dont know why.

http://code.google.com/p/nvidia-texture ... loads/list

anyways, i'm off for today. I really need to think about what compresser we can use ...and i may have to redo all the textures because of it.

this looks like either a change in api format or a bug in nvcompress. I'll have to look into the code for nvcompress to see how it writes the header, it may be different from older dds specifications.

Post by **klauss** » Sat Sep 01, 2007 9:18 pm

To me, it makes sense that depth isn't set. DXT is a special compressed format, there is no "depth". Usually "depth" refers to the buffer format after loading (which is usually uncompressed, but not with DXT), and DXT loading outputs a buffer in a totally different and depth-less format. Maybe that's what they were thinking. And who knows... maybe that's how it's supposed to be. I know for instance that with RIFF WAV files, many special codecs also set the bits-per-sample to 0.
You can guess which channels are present in the encoded stream by the DXT version (1, 3, 5) and that's all you need.
I'd simply ignore it silently and if we need such a depth value, guess it from the format.

Post by **safemode** » Sat Sep 01, 2007 9:50 pm

depth is part of the dds header, to say it's supposed to always be 0 is pretty strange. Why have it in the header at all?

bpp is the format of the internal uncompressed image. This is also a part of the header. The color masks are also part of the header. All of this is not being set by nvcompress (either all the time or only sometimes, i havnt checked all of them) .. If none of this is supposed to be set, then why have it? gimp's dds compressor sets them.

we dont really need any of that info to load the dds to GL. We do need it somewhat for software decompression... though to what extent i'm not sure (not at my computer).

I'm fine with just ignoring the bpp/depth silently and making an educated guess default.

our real crashing problem is textures that arent power of 2 dimensions. I"m amazed nvcompress doesn't scale them. Also nvcompress seems to put out uncompressed dds files when the dimension is bad, which we currently dont catch.

So, some dds files will need to be fixed, lots of bad images (not power of 2) need to be fixed. All will get done in the data5.x-test branch for people to test. when i get home that is.

Post by **ace123** » Sat Sep 01, 2007 11:04 pm

I'm looking at the kimgio source for reading DDS:
http://developer.kde.org/documentation/ ... ource.html

That one does not listen to depth. It seems that there are a number of predefined image formats.
To test this, I wrote a simple script for "file" in /etc/magic

They are all DXT1 or DXT5 formats, and they all have a depth of 0.
It should be easy to calculate based on the image pixel format, however.

/etc/magic

Code: Select all

# DDS Texture format.
0        string         DDS\              DDS Image data
>4	 lelong		!124		  \b, ERROR: Nonstandard Header Size %d
>12	 lelong		x		  \b, %hd x
>16	 lelong		x		  %hd
>24	 lelong		>0		  \b, Depth %d
>28	 lelong		x		  \b, %d Mipmaps
>76	 lelong		!32		  \b, ERROR: Nonstandard PixelFormat Size %d
>80	 lelong		&0x4		  \b, 
>>84	 byte		>0		  \b%c
>>>85	 byte		>0		  \b%c
>>>86	 byte		>0		  \b%c
>>>87	 byte		>0		  \b%c
>80	 lelong		&0x401		  \b, Format
>>80	 lelong		&0x11
>>>88	 lelong		16
>>>>104	 lelong		&0x8000            A1R5G5B5
>>>>104	 lelong		^0x8000		   A4R4G4B4
>>>88	 lelong		32		   A8R8G8B8
>>80	 lelong		^0x11
>>>88	 lelong		16                 R5G6B5
>>>88	 lelong		24		   R8G8B8
>84	 byte		0
>>88	 lelong		>0		  \b, BitCount %d
>>88	 lelong		0		  \b, ERROR: No FourCC and no BitCount
>92	 lelong		>0		  \b, RMask %08X
>96	 lelong		>0		  \b, GMask %08X
>100	 lelong		>0		  \b, BMask %08X
>104	 lelong		>0		  \b, AMask %08X

Post by **klauss** » Sun Sep 02, 2007 12:24 am

safemode wrote:depth is part of the dds header, to say it's supposed to always be 0 is pretty strange. Why have it in the header at all?

DDS can have uncompressed data as well, I think those are there for the uncompressed case. I somewhere read that DDS was supposed to handle JPEG and stuff like that too, but it never got implemented. So... it's there because MS always wants to beef up formats into uselessness. Luckily DDS never went as far as its mesh counterpart... X

BTW - make sure you generate the DDS with mipmaps (and that you load them) - I'm not sure what you do, I'm just talking.

BTW2: That's some mighty useful script - thanks ace!

Post by **safemode** » Sun Sep 02, 2007 3:02 am

I have no plans on supporting uncompressed textures in any functionality other than to throw an illegal format error.

the textures that nvcompress is not writing bpp and depth and mask info are compressed however, since they do load in the game (most of them), which wouldn't be the case if they weren't compressed, since i load all the dds files into GL with their DXTN type flag....uncompressed would not be in such a format.

all dds files have to have mipmaps in VS. We dont like it when they dont (I dont want to have to generate them in game when we can generate them beforehand, that's just stupid.

Post by **klauss** » Sun Sep 02, 2007 3:34 am

Ya, I'm not sure whether you're just saying or you think we're saying different things and you're arguing while I'm not.
Ie: I'm perfectly OK with your implementation - of course we'll use DDS only for compressed textures, and loading uncompressed DDS is... well... not required (though would be nice, but too much work probably, nah...). I was just telling you why the existence of that depth field, and that I think it's supposed to be 0 for compressed textures.

I think we should just revert the ones that weren't power of two and got borked by nvcompress-ing them, and if someone would be so kind as to resample them (or if nvcompress has such an option) and commit compressed versions (in power-of-two form), it would rock.

BTW - you have data5.x free for your tests, I moved it into branches/ogre_branch/data6.x already. Just post when you need something checked, if I have time I'll check it out and... check it out (

- co-svn, co-take a look

)

Oh... forgot to mention: I'm noticing a sever quality drop in base art. DDS-compressing base art isn't a good idea IMO, since compression of base art forces it into 16-bit color, loosing a lot of smoothness (shades get all... solarized... whack). I think base art should be reverted as well. It doesn't heavily impact performance anyway.

HINT: You can revert binary changes without wasting a lot of bandwidth by using SVN copy:

Code: Select all

svn rm <bad file>
svn cp -r <good revision> https://vegastrike.svn.... ../file.jpg file.jpg

That is, svn cp -r <source_revision> <source_url> <target_url_or_wcpath>

Post by **safemode** » Sun Sep 02, 2007 4:22 am

I'm just saying ...not arguing against your theory.

the reason i said it was stupid was because there are bettter ways to designate an uncompressed texture, like with the FOURCC value. If it's uncompressed, it's obvious that the fact that it's "dxt1" or "dxt3" "dxt5" doesn't matter.

set the depth, then set the fourcc to something like "RAW_" or something.

I think it's really stupid on the tool's part to not set depth but keep putting something useless like a fourcc of a compressed datatype that the file isn't even compressed in.

I'm currently resampling the textures in lanz cosine. Then recompressing them with nvcompress.. I'm still using the tool despite it's apparently retarded header writing.

data5.x can be used for the release. Fixes to data5.x-test can be merged into data5.x after it's been verified by at least one other developer/person. when it comes to this type of graphics work, things apparently _NEED_ to be verified on different hardware. Something may work just fine for me and then be hosed for 80% of the rest of the userbase.

what base art are you talking about? The images inside the bases or the images of the bases from space? I suspect you're talking about the images inside the base. If we want base images to be good looking, we should make them png and then we wont compress them. It would take much too long to pick and choose what to make dds, it's quicker to just make it all dds and then pick and choose what we want to be uncompressed and high quality.

That's the point of the test branch, I want people to tell me what is needed as a non-dds. I want people to tell me what textures look _wrong_ as dds. I want people to tell me if the dds file is unusable in an area. I'd like to avoid a surprise after release or after a commit to head that "oh, half the textures dont work" or something similar. I'm testing here, but with 1400 textures changed, i'm going to miss some.

Post by **klauss** » Sun Sep 02, 2007 7:12 am

safemode wrote:what base art are you talking about? The images inside the bases or the images of the bases from space? I suspect you're talking about the images inside the base.
Yep.

safemode wrote:If we want base images to be good looking, we should make them png and then we wont compress them. It would take much too long to pick and choose what to make dds, it's quicker to just make it all dds and then pick and choose what we want to be uncompressed and high quality.

It's really easy - all base art is located in the same place - sprites/bases, IIRC.
And yes, leaving them png is enough.