UberShader

[charlieg]

The glass looks fantastic and the rest is starting to look a lot better as well (often a lot of specular/white in previous screenshots).

[chuck_starchaser]

Thanks. Ya, I believe the shader is correct; the texturing is another story... Beautiful job, but whoever did it did not put the bump master in the master repo; but instead made a static bump shading with Gimp. There's no way to recover height from it, so the normalmap I did is garbage. So is the specular texture. There was no specular with the model, so I created one ad-hoc by doing crazy curves and blendings in Gimp, then I put shininess in alpha that's basically a gray fill and a bit of noise. And the AO wouldn't bake well because of the stupid practice of overlapping islands in the unwrap.
IOW, garbage in, garbage out.

[Phlogios]

Cockpits are gonna look fabulous.

[chuck_starchaser]

They will, indeed; --much better than the current Demon cockpit in PU. The three major
improvements are:

• Far more correct Fresnel computation
• Dynamic lights' reflections look like white discs, as they should for stars
• Reflections are suppressed for the far wall of the glass (near and far sides rendered
  differently in separate passes).

Having said that, there's still bugs in the shader that I can't figure out yet, like the
darkened peripheries of glass objects; but it's good enough for now; just trying to
finish up the Diplomatic Center before officially committing. One more day of work.

[JsnMtth]

Those might be good for the screenshot gallery too. Especially since they are rendered in-game.

[safemode]

in unrelated news... http://ptex.us/overview.html probably not suited for realtime use, but since it's bsd licensed, anything they got going on can be knitpicked if there is anything useful to us in there.

edit: maybe i'll add the version check thing in /data tonight. Then windows users wrongly using the SVN of data with their 0.5.0 release binary will get a nice error message telling them to either revert to an earlier version or update their binary. This will allow us to at least avoid ambiguity in what versions of the engine and data people are using when they come to us with problems. The /data will always have a fixed version it is compatible with, and wont work with any other version.

for instance, /vegastrike engine may have a version of 0.6.0.1 the 4th subversion is for SVN revisions. not necessarily a release, though it could be. We increment this number when the engine changes how the /data works. It may not crash, it could just cause different functionality, in both cases, we increment it. Then /data is always set to a single version it is compatible with. This way we can keep them in sync at runtime and ensure that the /data being used is what is meant for the version of the engine being used. We could say, a released version is one which ends in .0, so like 0.6.1.0 would be a release, and the /data released with it, would only work with 0.6.1.0. Conversely, the vegastrike engine could check to see if /data is the correct version and we could make this check dependent on the existence of such info, so it doesn't break mods at all.

[chuck_starchaser]

in unrelated news... http://ptex.us/overview.html probably not suited for realtime use, but since it's bsd licensed, anything they got going on can be knitpicked if there is anything useful to us in there.

Seems they use a separate texture for every polygon. I don't think that's a format that can help us in any way, unles NVidia and AMD decide to support it in future GPU, which I doubt.

edit: maybe i'll add the version check thing in /data tonight. Then windows users wrongly using the SVN of data with their 0.5.0 release binary will get a nice error message telling them to either revert to an earlier version or update their binary. This will allow us to at least avoid ambiguity in what versions of the engine and data people are using when they come to us with problems. The /data will always have a fixed version it is compatible with, and wont work with any other version.

Good! Any chance they could get a warning before they update, rather than a message to revert afterwards? Maybe temporarily renaming trunk to "non-windows" would send a clear message. SVN users will have to switch to "non-windows" to be able to use svn, and if they don't get the hint, they didn't deserve it.

for instance, /vegastrike engine may have a version of 0.6.0.1 the 4th subversion is for SVN revisions. not necessarily a release, though it could be. We increment this number when the engine changes how the /data works. It may not crash, it could just cause different functionality, in both cases, we increment it. Then /data is always set to a single version it is compatible with. This way we can keep them in sync at runtime and ensure that the /data being used is what is meant for the version of the engine being used. We could say, a released version is one which ends in .0, so like 0.6.1.0 would be a release, and the /data released with it, would only work with 0.6.1.0. Conversely, the vegastrike engine could check to see if /data is the correct version and we could make this check dependent on the existence of such info, so it doesn't break mods at all.

The problem with data checking engine checking data is that this is all after-the-fact stuff: The user is already in trouble when they get the message. Best would be some trick that prevents people from updating data to a version their engine doesn't support. That would be best done if the engine had svn built into it, as to do updates in-game; then the engine could check a compatibility xml file online; but we don't have this; so the next best thing IMO would be tags in data by engine version; so when users get a new engine, they switch their data svn path to a path that ends with engine version number.

[safemode]

I'd like to avoid the whole "i just downloaded a ton of data and found out my bins dont work with it) problem by keeping the user from using SVN directly at all.

Then in svn land, developers who know what is compatible to what when they make a commit can create tags to mark subversions, these can then be rolled up in incremental patches or used directly by an update program. The user then only has to ask the update program to check if anything new is available and it does all the work.



Basically, svn HEAD is for development. Tags can be made for releases. It doesn't work well going backwards, nor should your developers be forced to do so. Snip the misuse of svn HEAD in the butt now, or you'll be wishing you had later when an updater app does make it's way into existence.

[chuck_starchaser]

Sorry, I have a bad bug; can't seem to track it down. From some angles a surface looks smooth-shaded; from another angle it looks flat-shaded; and I've turned off all fancy features by now and the problem remains. Working hard at it, but can't promise to commit today as I was hoping to.

[safemode]

look into any sort of lighting hacks made to the game engine to deal with sphere maps and old texturing. Perhaps something is mucking up as a result to some hack to make things look nicer the old way, and there is nothing wrong with your shader code.

[charlieg]

Sorry, I have a bad bug; can't seem to track it down. From some angles a surface looks smooth-shaded; from another angle it looks flat-shaded; and I've turned off all fancy features by now and the problem remains. Working hard at it, but can't promise to commit today as I was hoping to.

You are allowed to commit stuff that is imperfect, as long as it is an improvement.

The commit rule is: if it works better than before, commit it. You can always commit additional improvements.

However [if you don't commit], when your hard disk or OS fails, or you make changes and it degrades and you forgot to make a backup, you can't go back to your original.

Perfect. The enemy of good.

[safemode]

Especially if it's not a default change. That is, it doesn't function by default and doesn't break the thing that does function by default.

In any case, you can always work with partially done stuff in your own branch and then merge your final product to HEAD.

[chuck_starchaser]

look into any sort of lighting hacks made to the game engine to deal with sphere maps and old texturing. Perhaps something is mucking up as a result to some hack to make things look nicer the old way, and there is nothing wrong with your shader code.

Nope; can't be, because the glass shader is working fine; it's only the highend shader that has the problem.

You are allowed to commit stuff that is imperfect, as long as it is an improvement.

Right now it's not an improvement; you can say it's "broken". Not only the smooth versus flat shading changing with angle, but that it's looking like crap generally. The hull of the station is supposed to look like stainless steel, but it looks like any material other than stainless steel. The only work I did on the high shader was "harmonization" with the other shaders, though, and I was testing them all at the time, last week; so I don't quite understand why it doesn't work now with the Diplomatic Center.
EDIT: No, thinking back again, I was testing only the highend_flat, glass and default shaders; NOT the highend.

In any case, you can always work with partially done stuff in your own branch and then merge your final product to HEAD.

I'm not that good at svn, yet, I just know the most basic commands; no idea how to branch or merge; --didn't know how to do it in TortoiseSVN under windows; much less now from the command line. But anyhow, this has got to be some very stupid thing causing all these troubles; it was working well a week ago.
But I'm taking a little break from shader debugging to fix up problems with the model first. I also noticed some paranormal phenomena when I meshered from obj to bfxm and back to xmesh that the textures mesher listed there were old texture names, ending with .jpg, even though my blender file is calling only .texture textures as far as I know. I fixed it, but I don't know how that happened; so maybe I'm all mixed up on the data side of things. That's why I want to finish fixing things with the model and redo the exports and mesherings, because if I'm not even sure what I've got data-wise then it makes it much harder to think about shader bugs; --not to say that a mesh could possibly look smooth or flat shaded depending on view angle, no matter where it comes from...

[safemode]

Might it be some logical issue in an attempt to make things realistic looking similar to how glass looks much more featureless when viewed straight on vs from an angle... Ie. You can see all the scratches and fingerprints and what not much more readily from various angles vs straight on.

also, I'm completely shooting in the dark here, but could the dielectric issue you had previously created to get the glass shader to look good have caused this bug in the high shader? From what i remember of you talking about how it treats dielectrics like painted surfaces and such, this upper layer is translucent and depending on the level of transparency, it can cause backscattering of the light (luster) vs straight reflection (clear coat/glossy ) etc ....perhaps on an angle, you see much more of this top layer vs the layer underneath and you end up losing the texture beneath this smooth upper layer.... again ...wild guesses. At work and bored.

[chuck_starchaser]

Shooting in the dark is perfectly legitimate, and sometimes it hits target. This time, no luck yet; the highend shader has no dielectric stuff whatsoever. I'm reserving dielectric distinctions for the highshaderplus family. No; it's got to have something to do with my computation of face-normal for gl normal interpolation corrections; but I turned all that off and it still does it.
Don't worry about it; I'm still working on the model, but tonight I'm going to go through the highshader code with a tooth-brush and find the bug, rain or shine.

[snow_Cat]

/me hopes that it is not overcast.

[chuck_starchaser]

The problem was in the vertex shader; I harmonized fragment shaders but forgot to harmonize vertex shaders.
What I changed in the glass vp was getting rid of light attenuation for each light source, which wasn't being used, and could be computed in the fp if needed, and using that fourth channel to pass light source size, as solid angle, measured in steradians, which is added to the material's shininess' equivalent filter size before computing phong. This is what achieved those round reflections of dynamic light sources on glass surfaces, for example. I transferred this new shininess algorithm to all shaders down to the default shader, but forgot to make the vp's pass light source size (0.00026 staradians --for the Sun, as seen from Earth, for now, as the engine doesn't yet calculate star size for the shaders' benefit). So, the fp's were getting light attenuation figures, instead, which are huge numbers compared to typical solid angles for stars.
Committing today for sure.

[charlieg]

Good work!

[chuck_starchaser]

Committed. Revision 12614.
WARNING: Windows users, do NOT update to this revision, unless you can compile the engine for windows with --enable-cubemaps... (and if you DO know how to compile for windows, we'd like to know how you did it).

If you are in Linux, you need to svn up your /vegastrike folder and compile a new engine:

Code: Select all

make clean
./bootstrap-sh
./configure --enable-cubemaps
make

Requirements: For the highend, highend_flat and fireglass shaders you need a ps3.0 gpu. 6600GT is the earliest gpu I know of, but it will be really slow; 8800GT or better is recommended.

Sorry it took an extra day, but the missed update of vertex shaders was only part of the problem; the other part was one place in the highend.fp where I wrote "fnormal" instead of "vnormal", but I missed it every time when comparing files visually; I finally found the problem with a diff tool called Meld.

What's new: Cubemaps were already there, in /textures/backgrounds, but were not being used, because the shaders were expecting spheremaps. Now all the shaders expect cubemaps and use them.
Glass objects now use a specialized shader/technique called "fireglass". Fly around and find an agricultural or a diplomatic base, and see the difference. Mining bases also have glass domes, but only the glass is using new shaders; the rest of the mining base is still its old, apathetic self. Not to mean that the agricultural and diplomatic bases aren't garbage; I've done the minimum necessary for them; --namely, fixed the unwraps to make it possible to bake ambient occlusions and added shininess in spec's alpha channel. But to fix the unwrap in the diplomatic center took so much work that there was no way to keep the original texture. So, the Diplomatic Center essentially has NO texture; only a baked bumpmap (a quick and dirty job) inside the docking. The rest of the station is just flat colors for diffuse and specular.
In the case of the Agricultural Station, it wasn't so hard to fix the unwrap to bake a marginal AO, so I was able to keep the original texture. However, the author of the original texture didn't submit the height map for storage in the masters repository; and there was no way to recover height data from the gimp bumpmap that was made from it, so I used ad-hoc curves and blendings in Gimp to try and come up with a normalmap, and it's garbage. Then again, the unwrap was garbage anyways. To do things right, the agri base needs to be re-unwrapped and then retextured from scratch.
So, very little to show, really, for all the suspense; but what I really wanted to avoid was having nothing to show with the new cubemaps. The real thing about this update IS cubemaps; --i.e. Klauss' work on the engine, rather than my work on shaders. New shaders and station updates are there just to demo the cubemaps.

What's the big deal with cubemaps?
* More detailed reflections...



* Blurred reflections off blurry (low shininess) materials. See stainless-like metal on the diplomatic center:



But don't most games use cubemaps?
Indeed, we're probably among the last moving to cubemaps... But at least not among the least: Many games and game engines have cubemaps but not all have DDS cubemaps with CubeMapGen-generated mipmaps, like we do now. No, we haven't leaped from last to first; but this is a pretty big leap for OSS gaming, anyways. What's important is the possibilities it opens up. From now on, if you want to represent "metal" (titanium or stainless), just specify technique="highend" in the xmesh, make the diffuse color black, specular white, or very light, and put a low value in the alpha channel (to make reflection blurry). Perfect "metal" look, and it's that easy.

Known issues:

• Cubemap seams are visible in the backgrounds. This is my fault, but Klauss said he could fix it programmatically, so, I'll just wait for his fix rather than change all 100 megs' worth of cubemaps.
• Some ships, notably the mule, as well as planets and stars, look yellow or white, or flicker. I haven't the foggiest idea why this is happening, but I think it's new since cubemaps, so I'll just email Klauss about it.
• In some backgrounds, there seem to be thick seams in the lesser LOD's. You can only see this on reflections off low shininess materials, so you probably won't even notice, but it's worrisome... CubeMapGen was supposed to prevent that... but it's probably me.
• And the worst known issue, and it was intentional: I've broken away from the old ad-hoc shininess computations. ALL ships will now look as glossy as shrink-wrapped unless or until you specify gloss in the alpha channel of the specular texture. No more lazy shortcuts. Just do it. It's easy: white for very glossy, very polished surfaces like car paint, glass or chromes. A dark color for very blurry reflectors, like teflon. Medium to dark gray for titanium or aluminium. Medium-light gray for stainless steel. Experiment.

What else is new?
Besides the updated highend shader and the brand new "fireglass" shader, there's a "highend_flat" shader. What's the "_flat" part for? This shader makes smooth-shaded geometry look flat-shaded. And, why not just flat-shade the mesh in Blender before exporting, instead? Because flat-shading meshes causes the vertex count to go through the roof... --you can expect a multiplication of vertices by a factor of about 5 or 6. Smooth-shading a mesh causes all facets to share vertices with neighboring facets, which is a lot more efficient. Now you can have your cake and eat it too: Work with smooth or flat shading in blender as per your preferences. When you're done, put all your flat-shaded stuff into a separate object, make it smooth-shaded before you export to .obj; then, in the .xmesh file for it specify technique="highend_flat".
Another thing that's new is the normalmap format. It has changed to UUUV (which is a short for dUdUdUdV), --but you might want to call it "the gray normalmap format", since all the others look blue. All shaders use it, instead of the the traditional dUdVZ or the previous VS standard of dUdVZdU (red channel repeated in alpha). All normalmaps in-game have been changed to use "gray format", including the Ancestor, Hyena, Schroedinger and LLama.

What's NOT new?

• No distinction between metals and dielectrics.
• No shadows.
• No specular occlusion.
• No detail textures.

These features will trickle in with the highendplus and ubershader families.

To Do (in a rather flexible order):

• Darken the central part of the diplomatic center; --looks ridiculous...
• Upload a generic normalmap format converter noodle for UUUV.
• Write a "Shininess and AO for Complete, Basket-Case Dummies" tutorial
• Figure out the problem of specifying incoming versus outgoing glow in glow maps (possibly by turning glow alpha to zero in xmesh to mean incoming glow (un-modulated by material)...(I will need unmodulated glow for some tricks to come, like a specular reflections off baked lights hack; but so I want future glow maps to be raw bakes, rather than premultiplied by material as they are now; yet current shaders need to be able to work with existing glow bakes, which are premultiplied, I believe...))
• Do the "fire" part of "fireglass" (lights, engine exhaust plumes, explosions (note that this will relieve the standard glow map from having to represent emitters, which ties into the conversion from outgoing light to incoming light glow maps))
• Remind Klauss about stuff I know of on his to-do list:
  • Seams problem
  • highend/default/lowend folders for shader families
  • separating bfxm headers to text files, to avoid having to use mesher when changing textures or shaders
  • mesher bugs
• Finish the Schroedinger (I was working on adding AO and shininess channels to it).
• Code a "highend_rock" shader for the rocky part of the Mining Base, as well as asteroids (basic idea is to simulate miniature (sub-pixel) reflectors at random orientations, so it looks like the rock contains tiny christals... That, as well as paralax).
• Do the new Mining Base I started long ago.
• Code "default_flat", "default_fireglass", and "default_rock" shader counterparts (lobotomized versions for compatibility with lower-end gpu's (right now all highend default to "default", but this is wrong, as the default shader can't do glass, can't do flat-shading, etceteras)).
• Finish the Llama.
• Hack and fix CubeMapGen, once and for all; and make real good, final cubemaps.
• Code "highendplus", etceteras...

[charlieg]

Please oh please make that post a dev blog update. That is just perfect for that.

Additional todo:
- go through all VS ships and mark which need new unwraps, which need new models, and which are ok
- start on the list produced by the above
- this needs somebody else (not chuck) because he can provide the instructions but he shouldn't be the only one capable of following them
- any takers? anybody want to help and knows how to create textures?

[charlieg]

• separating bfxm headers to text files, to avoid having to use mesher when changing textures or shaders
• mesher bugs

Total aside, but I have been thinking that Mesher would be better written in Python. It would make it easier to debug and easier for non-gurus to fix/contribute. So maybe a Vega Strike 'help us, python programmer required' ad would be good here as it is a fairly isolated part of the code.

[chuck_starchaser]

Please oh please make that post a dev blog update. That is just perfect for that.

Thanks. I tried to register for the blog entries, but same old problem... Said to check my mail for the password, but I got no mail because of sourceforge's draconian rules.

Additional todo:
- go through all VS ships and mark which need new unwraps, which need new models, and which are ok
- start on the list produced by the above
- this needs somebody else (not chuck) because he can provide the instructions but he shouldn't be the only one capable of following them
- any takers? anybody want to help and knows how to create textures?

If what I've seen is any indication, EVERY model in vegastrike probably needs to be re-unwrapped. The UV unwraps are disastrous. And I'm not just talking about overlapping islands, but about islands with gashes, folded corners, orphan polygons covering the whole uv map, severely deformed islands, inconsistent scaling... And most of the meshes need work to begin with: In the diplomatic center I must have deleted about a thousand polygons that were hidden inside, causing rampant overdraw, and taking up space in the texture for nothing... Not to mention orphan vertices and edges, split vertices where smooth shading was the obvious intent, and missing inner surfaces in areas that could be seen through the glass, which due to backface culling allowed seeing through hull parts. All of the above has been true or is true of the mining base, the agri station, the diplomatic center, the llama and the schroedinger. Huge amounts of cleanup work are needed, even to merely be able to bake an AO for them.
Worthy of a widely publicized Remodelathon campaign, or apply for google sponsorship

• separating bfxm headers to text files, to avoid having to use mesher when changing textures or shaders
• mesher bugs

Total aside, but I have been thinking that Mesher would be better written in Python. It would make it easier to debug and easier for non-gurus to fix/contribute. So maybe a Vega Strike 'help us, python programmer required' ad would be good here as it is a fairly isolated part of the code.

Problem is, you're assuming, as many do, that Python is a) Easier to learn than C++, and or b) There are more Python programmers than C++ programmers, and/or c) Python is easier to maintain than C++. I think all three assumptions are incorrect. I know many more C++ programmers, in RL as well as in the internet, than Python programmers. Python seems to be quick for prototyping, but it's harder (for me, anyways) to read than C++; and I had no problem learning C++, but I tried to learn Python and failed miserably. PU is stuck in the mud because of Python code much more so that because of C++ code. We've had many C++ programmers pass through our forum, but only two Python programmers that I remember. I'd in fact be very happy to see all the Python in the engine moved to C++, personally. At least I could then do something about it, perhaps... The problem with fixing mesher bugs is I don't know much about the formats, and therefore wouldn't probably be able to make out what the code should do. Though, one of the problems is rather mundane: it puts spurious ="" in xmesh, then chokes when reading them back; but mesher has also segfaulted on me, and sometimes it doesn't work if you make one mesh the main mesh, but works if you switch them around. I could try and look at the code, but I don't feel too confident about it. There's too many complications in it that IMO should never have been coded, like requiring a .mtl file to be there and using the Kd, Ks and whatnot info, which is bogus most of the time, and the xmesh files will have to be edited manually anyways, so what for was that "feature" added? But I'll have a look.

[charlieg]

Python is easier to learn than C++. Doing non-trivial things in C++ requires advanced knowledge. Python cuts out a lof of the guff and trivializes a lot of otherwise common problems. Then there's the build process (i.e. there is none).

The debugging is what makes it easier to work with. No more segfaults. Errors, meaningful errors. It's why writing C++ applications is an artform and most apps are buggy, crash ridden, problems to maintain.

Yes, python is easier to maintain. I don't think you'll find a sane C++ vet who knows python who will testify otherwise.

[chuck_starchaser]

Well, maybe you're right; but still, somebody would have to do the work of porting mesher, and it should be someone that knows Python AND C++ and who knows a thing or two about Wavefront .OBJ format, if not about bfxm and xmesh. And the way I look at it is there's probably just a couple of bugs that need to be ironed out; which might become 20 or 200 after a port. Other than the "couple of bugs", what I suggested to Klauss, and he liked the idea, was to make it so that mesher translates ONLY mesh data from xmesh to bfxm, but leaves the headers in xml format. So the engine would read the xml (xmesh) headers as text, then load the binary mesh data from the bfxm. This would allow us to edit the textures, scaling, cull mode, technique, front materials, etceteras, without having to use mesher at all. We'd only use mesher when there's a new mesh; but updating a texture name or technique would only require editing the xmesh headers; nothing more.
But now Klauss hasn't replied to an email of mine for like two weeks. I'm not sure if he's mad at me about something, or, heaven forbid, something about his dad --he had a bad operation recently.

[pheonixstorm]

Actually you just need a release binary... I have already compiled a debug win32 binary