Bug report: DDS breaks with 2048 x 1024 [SOLVED]

For collaboration on developing the mod capabilities of VS; request new features, report bugs, or suggest improvements

Moderator: Mod Contributor

Post Reply
chuck_starchaser
Elite
Elite
Posts: 8014
Joined: Fri Sep 05, 2003 4:03 am
Location: Montreal
Contact:

Bug report: DDS breaks with 2048 x 1024 [SOLVED]

Post by chuck_starchaser »

Any time the python code tries to load a 2048 x 1024 DXT1, it reports "RuntimeError: unidentifiable C++ exception"...

Code: Select all

Loading Star System Gemini/Troy
 Next To: Gemini/Regallis
 Next To: Gemini/Freyjadot -0.000490warning: Not importing directory 'K:\SVN\spirit187\bases\fixers': missing __init__.py
Traceback (most recent call last):
  File "mining_basesunsetmerchant.py", line 37, in <module>
  File "K:\SVN\spirit187\bases\merchant_guild.py", line 9, in MakeMerchantGuild
    Base.Texture (room, 'background', 'bases/merchant_guild/merchantguild.spr', 0, 0)
RuntimeError: unidentifiable C++ exception
I was getting a black screen when loading a game with a 2k dds landing pad, so I reverted to jpg. Then I had a landing pad, but couldn't go to the main concourse; so I reverted the main concourse to jpg. Now I get the landing pad and main concourse, but can't go into the merchant's guild or the bar or the ship upgrades...
So far I had never had any trouble with smaller sizes of DDS; used DXT1 and DXT5 compression with no problem.
Did the conversions with the Gimp plugin; DXT1 without mipmaps. The images open fine in Gimp.
The sooner this could be fixed, the better; I converted ALL backgrounds for PU, and luckily I didn't commit to trunk...
Last edited by chuck_starchaser on Tue Mar 18, 2008 1:12 am, edited 1 time in total.
pyramid
Expert Mercenary
Expert Mercenary
Posts: 988
Joined: Thu Jun 15, 2006 1:02 am
Location: Somewhere in the vastness of space
Contact:

Re: Bug report: DDS breaks with 2048 x 1024

Post by pyramid »

Base backgrounds should be square (1024x1024 or 2048x2048), not sure why however, maybe for being closest to a 4:3 ratio and POT. In this case 2048x1024 should work as well. I tried other resolutions, even non-POT and they worked, but only as png not as dds. That's probably due to the conversion/compression step for non-dds textures that happens on loading.

Anyway, my gut feeling is telling me that something is strange, at least so fundamentally different since the last dds changes done to the code that it's not compatible with certain dds conversions. For example I had to replace 7 planet textures in svn yesterday. Being already DXT1+mipmaps they simply didn't load and planets showed up as blank. After recompression with nvcompress they loaded fine. nvddsinfo showed absolutely nil difference between the previous and the newly compressed ones. Maybe the method/code used to compress them was different (previous was from safemode, though he might use the same nvcompress version as me 0.9.3) or they simply got corrupted during the upload when there were the svn and general sf access problems, or when moving them from sol2 to sol, or they were always bad, not sure, since i never tested them before (except one).

Also, all the 'segfault' and 'camera not found' errors we are getting in the bug forum lead me to believe that something strange is going on. I had some segfaults and camera errors myself, apparently without a reason, mostly after mingling with the config or home .vegastrike dir. Somehow I managed to get rid of them, however not sure how exactly nor what is the cause.

Some of the causes I can imagine for the above problems:
* vegastrike.config is not compatible with the new code (wither it was not updated by the user or was updated and code was not recompiled)
* textures are corrupted (gimp 'should' work but nvcompress works for sure. try wiki> deveopment> 2d graphic tools>nv texture tools. there is a link to a compiled binary for win)
* some textures were deleted / or not deleted from the .vegastrike homedir (had some segfaults here)
* incompatibility with graphic drivers (see wiki>development>compatibility list)
* some supposedly unrelated save game was removed from ~/.vegastrike (happened to me removing savegame+corresponding serialized_xml file, loading another savegame and, voilà, segfault; did a new campaign load, save, exit, load savegame and, pronto, all was ok)

Probably we should hang out some big blinking neon sign somewhere for the svn getters with something like: AFTER DOWNLOAD PLEASE RECOMPILE AND DELETE ALL HOME .VEGASTRIKE DIR CONTENTS.

Where the ALL is really questionable, might be only the texture folder, or maybe also the milkyway and meshes folders.
safemode
Developer
Developer
Posts: 2150
Joined: Mon Apr 23, 2007 1:17 am
Location: Pennsylvania
Contact:

Post by safemode »

gimp dds uses hardware. I extremely suggest not using gimp to make dds files. It'll compress different per card you happen to have.


You were having issues loading 2048x1024 dds textures? I've never seen such issues. Can you be sure that you were using hardware dds decompression?

A C++ exception leads me to believe that either the GL call failed, and we dont deal with failures well, or we were using software mode and the software code is bugging out. I need to know which is going on.

I strongly recommend removing all config related items not svn updated. Get rid of it all. savegames etc. Just remove it all and start new.

I'd like to know if a DDS file is not working before things get reverted in trunk or updated, until it's determined for sure to be a dds file issue. I'd like to get the DDS code more cleaned up, and file juggling just hides it.

I'm guessing pyramid would be the first to hear something, so if you could throw a pm over my way or aim message, i'll go ahead and fix the code.

ps, finding out exactly what function threw the exception would indicate if we were in hardware mode or software, and make it obvious to me what exactly is failing.
Ed Sweetman endorses this message.
safemode
Developer
Developer
Posts: 2150
Joined: Mon Apr 23, 2007 1:17 am
Location: Pennsylvania
Contact:

Post by safemode »

i am 99.9% sure the issue was due to Sol not really being used before, since Sol2 was, and that masked the fact that i missed some broken dds files back from before we realized nvcompress wasn't fully making mipmaps for unsquare textures. I then patched nvcompress to not be broken and made fixes.... but apparently these guys slipped through.

in addition to that, i previously had a workaround to hide these broken files from error, by faking the last 1x mipmaps. I would output when one was found, but there is so much output when starting vs that nobody caught it. That workaround has been removed, making these issues much more obvious. Perhaps it would be good to get every system looked at now, to make sure no surprises hit us after 0.5 release.

so this was a data file problem, not a code related one. These files were missing their tiny 1x mipmaps.
Ed Sweetman endorses this message.
pyramid
Expert Mercenary
Expert Mercenary
Posts: 988
Joined: Thu Jun 15, 2006 1:02 am
Location: Somewhere in the vastness of space
Contact:

Post by pyramid »

I've checked the old files. They indeed have the 1x mipmap missing. Whatever the reason, this was corrected with my upload yesterday.

The textures were actually sol2 textures which were moved to sol. I did join both folders on the last weekend and there is no sol2 anymore.

It would be best to get every system looked at now, to make sure no surprises hit us after 0.5 release just not sure I can handle that entirely and systematically. Maybe I'll just call for help in the bug forum.
safemode
Developer
Developer
Posts: 2150
Joined: Mon Apr 23, 2007 1:17 am
Location: Pennsylvania
Contact:

Post by safemode »

The problem with making a check in code, is that we only come around such a bad texture if we happen to load it, and that may never happen.

about the only sure way i know of doing the check is to make a script that calculates how many mipmaps a given res combo should have, and checks the nvddsinfo output of every texture in the game to see if it matches.

That's not really hard to make esp in perl or such.
# mipmaps calulated would be choosing the maximum of the height or width, then looping that value counting the divisions of 2 until the value is 1. you could even set up a table of the various Power of 2 resolutions and their corresponding mipmap count.

simple.


As a side-note, we still have to fix rings. I think the ring texture needs an alpha channel, and maybe now it doesn't have one. star_system_xml.cpp seems to play with setting ring options up, something about alpha layers are mentioned.

A good test would be to put the ring master in data4.x, then test to see if the ring looks right. If so, then see if it has an alpha channel, if so, then we make them DXT5. If not,either the master doesn't work or the image has no alpha channel), something is wrong in the code, with the latter being DDS code centric.

The planetary rings is the only other big graphics regression i know about.
Ed Sweetman endorses this message.
pyramid
Expert Mercenary
Expert Mercenary
Posts: 988
Joined: Thu Jun 15, 2006 1:02 am
Location: Somewhere in the vastness of space
Contact:

Post by pyramid »

Yes, the rings. I'll try to pull it in for 0.5. Besides transparency, the problem with rings is their low resolution, which makes them ugly, and their grainy texture which makes them look unreal.

And the mipmap check script...
chuck_starchaser
Elite
Elite
Posts: 8014
Joined: Fri Sep 05, 2003 4:03 am
Location: Montreal
Contact:

Post by chuck_starchaser »

LOL; I didn't know about Gimp using hardware. I will try with nvcompress; never used it before. (DAMN! I've converted lots of stuff using Gimp...)

Anyways, these 2048 x 1024 backgrounds are for bases; they don't have mipmaps. Just to clarify.

And, I'm not sure if this is related; someone just reported slowdowns after updating to the latest PU revision, with all the new DDS textures.
http://wcjunction.com/phpBB2/viewtopic.php?p=7266#7266
Any way to know if software mode is being used from stderr or stdout?

And, BTW, I did instruct users to delete /.pu/textures/backgrounds
Is there a reason they should delete anything else? I have NOT updated any unit textures to DDS. Only sky backgrounds and base textures and animations ... and planet textures.
safemode
Developer
Developer
Posts: 2150
Joined: Mon Apr 23, 2007 1:17 am
Location: Pennsylvania
Contact:

Post by safemode »

i would just set a table of all texture resolutions (we dont go bigger than 2048)
so we have enum blah[12][2] {2048, 1024, 512, 256, 128, 64, 32, 16, 8, 4, 2, 1}, {12,11,10,9,8,7,6,5,4,3,2,1};

or whatever the synax is for static initilization of a 2d int array in perl

then we go through every file and retrieve the name and resolution, number of mipmaps . Pick the bigger side, do if (mipmaps == blah[res]) good, else, bad.

I'd probably use my code in vsimage.cpp to make a small utility that lets me return data in a very easy to use format so the script can be made easily. But a lot of nifty grepping and sed'ing can be used with nvddsinfo.
Ed Sweetman endorses this message.
safemode
Developer
Developer
Posts: 2150
Joined: Mon Apr 23, 2007 1:17 am
Location: Pennsylvania
Contact:

Post by safemode »

chuck_starchaser wrote:LOL; I didn't know about Gimp using hardware. I will try with nvcompress; never used it before. (DAMN! I've converted lots of stuff using Gimp...)

Anyways, these 2048 x 1024 backgrounds are for bases; they don't have mipmaps. Just to clarify.

And, I'm not sure if this is related; someone just reported slowdowns after updating to the latest PU revision, with all the new DDS textures.
http://wcjunction.com/phpBB2/viewtopic.php?p=7266#7266
Any way to know if software mode is being used from stderr or stdout?

And, BTW, I did instruct users to delete /.pu/textures/backgrounds
Is there a reason they should delete anything else? I have NOT updated any unit textures to DDS. Only sky backgrounds and base textures and animations ... and planet textures.
light maps are generated, this is always going to be png. They are not Seen in the game, but they make the background ambient light on your ship. The dds textures are used for all visual display of backgrounds. The generation of the light map uses software decoding to create the png. This is a one time process per backgound loaded.

There is an easy way to tell if he's hitting software dds, just check ram usage. Software dds for dxt1 only occurs if dds hardware is not found, so we'll be using a TON of ram. If he's got close to 1GB or more allocated for VS, then he's all software.

If your cockpit images are dxt5, then he may have an nvidia card that is blacklisted. This will kick back to software. This will be significantly slower if we're software decoding large images.


The only other situation i can think of that would slow things down significantly, is if his hdd is so slow, that it takes more time to read the larger dds file than to decode the more complicated png/jpg, then generate mipmaps, then re-compress to dds. This could happen if your hdd was very slow, but you had a crazy fast video card and cpu/ram. Strange combo.

Ask him if how fast pre-dds loaded. Maybe even 0.4 (though it's not a direct comparison). DDS brought load times down from minutes to less than half a minute.. If the DDS file size is to blame, he should have actually measured a slower start time after dds files were added.

Keep in mind also, the texture sizes you are using are huge. If the after update he used images were much larger in resolution, than that would be significant.

Check his ram usage by VS, with an all stock pull (no Fing around with directories like he's doing). If it's high then this is solved. If it's low, then lets get some real numbers. He may just be imagining it, thinking the filesize must take longer to load, cuz it's bigger.

In no way have i heard that dds takes longer to load than pre-dds. Hell, even software should be faster, as it's simpler than png.

I can alleviate some software dds time by not reading in mipmaps if we have no s3tc support. But that would be a rare situation. Most people support s3tc.

make him run glxinfo too and see if it's saying he has s3tc compression support.

ps: I didn't hear anyone complain about VS taking longer to load after i made things DDS. If the larger filesize of dds is to blame (and it's always larger because we have mipmaps), then we would have seen many people complain that it's taking longer to load than pre-dds did. Never happened. The idea is almost impossible to imagine. Maybe he's loading the game over nfs, or has such little amount of ram his OS can't precache anything and his hdd is slow as hell. I'll stop speculating till we have real info.
Ed Sweetman endorses this message.
safemode
Developer
Developer
Posts: 2150
Joined: Mon Apr 23, 2007 1:17 am
Location: Pennsylvania
Contact:

Post by safemode »

you said you had a slowdown with the cockpit. I'd wonder if you are running our of video ram with such a huge texture (esp if you have mipmaps (doesn't make sense to have mipmaps for a cockpit though).

Also, this may have something to do with the slowdown when a unit or planet texture is directly in front of the ship and filling the screen. Perhaps some sort of linear alpha blending or color to alpha blending option that is taking up cycles on the gpu.

If things get slower with the cockpit in than without during gameplay, then we have a different issue at work here. Either something unwanted is being done to the cockpit, slowing crap down, or we're running out of video ram and we're wasting time bouncing from vram to system ram and back.
Ed Sweetman endorses this message.
loki1950
The Shepherd
Posts: 5841
Joined: Fri May 13, 2005 8:37 pm
Location: Ottawa
Contact:

Post by loki1950 »

make him run glxinfo too and see if it's saying he has s3tc compression support.
Kinda hard to do on a windows only system :wink: which is what the problem user has the GPU is an ATI Radeon 9600 so it should have s3tc support.

Enjoy the Choice :)
my box::HP Envy i5-6400 @2Q70GHzx4 8 Gb ram/1 Tb(Win10 64)/3 Tb Mint 19.2/GTX745 4Gb acer S243HL K222HQL
Q8200/Asus P5QDLX/8 Gb ram/WD 2Tb 2-500 G HD/GF GT640 2Gb Mint 17.3 64 bit Win 10 32 bit acer and Lenovo ideapad 320-15ARB Win 10/Mint 19.2
chuck_starchaser
Elite
Elite
Posts: 8014
Joined: Fri Sep 05, 2003 4:03 am
Location: Montreal
Contact:

Post by chuck_starchaser »

He tried back with the old jpg cockpit texture, and it didn't improve; so apparently this Centurion problem is not related to dds.

Quite puzzling, since he's the second person to report trouble with this ship when the cockpit is visible. First report was pre-dds; but the ship hasn't been touched.

He did report longer load times. Myself I've noticed no difference, but then again, I don't think I've converted 20% of all the textures yet.
safemode
Developer
Developer
Posts: 2150
Joined: Mon Apr 23, 2007 1:17 am
Location: Pennsylvania
Contact:

Post by safemode »

damn, it's gonna be hard to compare a pre-dds to full dds system then.

I noticed a massive load time decrease with VS from pre dds to when i replaced all the textures with dds, which i did mostly at once.
The other devs and people noticed the same decrease.

Maybe try pulling the pre_dds tag and comparing (note that many of the textures have also been updated with larger resolution textures since then). I cant imagine the time not being reduced significantly.


but never-the-less, see how much ram he's using, 20% wont be enought to make the ram difference huge, but it should still be noticable if those 20% are all big textures. Also, windows may have a utility to check vid ram usage for radeon cards.

In any case, VS dumps the GL symbols on load, so you should be able to grep for s3tc from there to make sure that vs is detecting it's on. I know the driver supports it.

a slowdown in cockpit mode specific to a ship doesn't make much sense. but it has nothing to do with dds.
Ed Sweetman endorses this message.
chuck_starchaser
Elite
Elite
Posts: 8014
Joined: Fri Sep 05, 2003 4:03 am
Location: Montreal
Contact:

Post by chuck_starchaser »

Agreed. That Centurion problem is a mystery on its own.

I downloaded NVIDIA's texture utilities 2. Got nvcompress sitting there; but no docs come with it. I found,
http://code.google.com/p/nvidia-texture ... dLineTools
but I'm not sure I understand enough of it.

What's the command to use for DXT1 + mipmaps, DXT1 no mipmaps, DXt5 + mipmaps, and DXT5 without mipmaps? Ok, I see --nomips; but the rest I don't know.
I'm only interested in highest quality. Don't care if it takes a week.

BTW, I see there mentioned nvassemble for putting together dds cubemaps...
pyramid
Expert Mercenary
Expert Mercenary
Posts: 988
Joined: Thu Jun 15, 2006 1:02 am
Location: Somewhere in the vastness of space
Contact:

Post by pyramid »

It's all in here http://vegastrike.sourceforge.net/wiki/ ... nvcompress

Well, I'll update whatever isn't.

There is a small difference with texture tools v2.x. It has more compression options, e.g. -bc1a, which texture tools 0.9.x doesn't have (it uses -rgb option instead). But this is just a minor detail. You get a full options listing with nvcompress --help.
chuck_starchaser
Elite
Elite
Posts: 8014
Joined: Fri Sep 05, 2003 4:03 am
Location: Montreal
Contact:

Post by chuck_starchaser »

Thanks! Problem solved. It was Gimp's dds tool. Using nvcompress the 2k x 1k textures work fine.

(Now, if I can figure out how to setup the path in xp so that I can use nvcompress from anywhere... This business of moving nvcompress and all the dll's to the folder where I need them is not sustainable...)
ace123
Lead Network Developer
Lead Network Developer
Posts: 2560
Joined: Sun Jan 12, 2003 9:13 am
Location: Palo Alto CA
Contact:

Post by ace123 »

I usually just copy all EXE's and DLL's into C:\windows :lol: but I suppose setting up a separate path directory would be better

I am wondering: did you use safemode's patched version of nvcompress? or the stock one from nvidia? Is the patch still necessary?
safemode
Developer
Developer
Posts: 2150
Joined: Mon Apr 23, 2007 1:17 am
Location: Pennsylvania
Contact:

Post by safemode »

-bc1 is dxt1 -bc2 is dxt3 -bc3 is dxt5 etc nvcompress -h tells you.


be very careful about making any commits with nvcompress2 make sure you get the same number of mipmaps and that the compressor isn't using hardware gl (it should take a while to compress unless you're using cuda).
Ed Sweetman endorses this message.
chuck_starchaser
Elite
Elite
Posts: 8014
Joined: Fri Sep 05, 2003 4:03 am
Location: Montreal
Contact:

Post by chuck_starchaser »

I'm using the stock one (didn't even know there was a patched one).

Was careful not to use cuda. My line goes:

Code: Select all

nvcompress -bc1 -nomips -nocuda a.jpg a.dds
EDIT: Found it. Right click on my computer, advanced, advanced, advanced, then environment variables, int the lower screen is path.
charlieg
Elite Mercenary
Elite Mercenary
Posts: 1329
Joined: Thu Mar 27, 2003 11:51 pm
Location: Manchester, UK
Contact:

Post by charlieg »

Found it. You have a rubbish OS. Better switch before more goes wrong on you. :lol:
Free Gamer - free software games compendium and commentary!
FreeGameDev forum - open source game development community
chuck_starchaser
Elite
Elite
Posts: 8014
Joined: Fri Sep 05, 2003 4:03 am
Location: Montreal
Contact:

Post by chuck_starchaser »

You're preaching to the converted, CharlieG. I'm just waiting for my chance to escape; coming real soon; I can feel it.
pyramid
Expert Mercenary
Expert Mercenary
Posts: 988
Joined: Thu Jun 15, 2006 1:02 am
Location: Somewhere in the vastness of space
Contact:

Post by pyramid »

ace123 wrote: I am wondering: did you use safemode's patched version of nvcompress? or the stock one from nvidia? Is the patch still necessary?
I've tested texture tools v2. The patch is not required. Mipmaps are created correctly and even on non-square textures the 1x mipmap is created.
safemode
Developer
Developer
Posts: 2150
Joined: Mon Apr 23, 2007 1:17 am
Location: Pennsylvania
Contact:

Post by safemode »

i think i stopped using it for fear that it created less nice looking textures than the 0.9.. you'll have to judge for yourself comparing the two outputs (one from 0.9 and the other 2.0)
Ed Sweetman endorses this message.
Post Reply