Artwork/data overhauls

[safemode]

Right now, data4.x in the svn head is mostly made up of non-dds images. For those that are, we can refer to pre-dds revisions. So, with that in mind, I would make non-dds textures "Masters" and branch data4.x to a textures-master branch. Any new artwork by people would get committed to the texture-master branch in a master format. Then either that person or someone else would be directed to create a usable texture in the dataN.x directory.

What needs to be ratified :

The codec of master textures (new ones), and the depth of those masters.

The codec of usable textures (do we make it dds, or png).


I'm not sure nvcompress is available to non-developer-oriented win32 users, though it should have a build on the site, it's free and open source. So, all artwork authors should be able to generate both a master texture and a usable texture, but if that isn't the case for some reason, the artwork author should contact me or another developer to generate a dds if need be whenever a master is updated/added to the textures-master branch.


Hopefelly that's clear.

now, about the codecs of the master textures. They should be lossless, open/crossplatform formats. It would be nice for it to be in the native format of either gimp or photoshop (assuming photoshop's native format is actually open and not half-backwards engineered in gimp). But if that's not the case, then png is sufficient.

The depth of the image should be decided on a per-type basis, bump maps should be a given depth, etc etc. This needs to be decided on by people who are making them now, as i have no idea.


When a master texture is updated on the repository, a derived usable texture has to be generated. The DDS is automatically created in the "correct" depth and such, the type of DDS is decided on if it's alpha or non-alpha channel, which has to be specified. Either DXT1 or DXT5. mipmaps should be generated for all dds files unless you know for sure that the texture wont be seen at varying distances/sizes.

In the case that compression is not possible for a texture, it should be created as a png. PNG files are not compressed in game and are used in raw format. So care should be taken to not make too many png files.




So basically, we will have 3 repositories for data. With the main repository becoming data5.x (now currently data4.x).

Artwork changes first get committed to trunk/textures-master
Derived artwork then gets committed to branches/data5.x-test

Code/bin changes first get committed to branches/data5.x-test

After at least 1 other developer (hopefully on a sufficiently different operating condition) signs off on a given commit, that commit is then merged into trunk/data5.x from branches/data-5.x-test

data5.x should be 100% workable all the time (with only corner case bugs not found in testing phase)

so to summarize my plan :
: Nobody should be posting new code to data5.x.
: Nobody should be posting any artwork not png or dds to data5.x-test except the few places where bitmaps are used.
: All changes in data5.x should be merged from data5.x-test
: All merges into data5.x should be signed off by at least one other developer after testing
: Any new artwork should have a master copy in textures-master in an approved format/codec


I think doing this should eliminate any future issues like we've had recently and should eliminate all the mess we've got with the multitude of codecs and formats and basic disorganization with textures in VS.

[klauss]

I'd like to comment

In the case that compression is not possible for a texture, it should be created as a png. PNG files are not compressed in game and are used in raw format. So care should be taken to not make too many png files.

I'd change "possible" to "desirable".
For instance, compressing base art is more than possible, but not desirable. Basically, anything that won't apply inflight shouldn't be compressed IMO. Ie: base art, splash screens, etc.

From the implementation side, safemode, I'd love it if you could mod the dds loading procedure to automatically dump the mipmaps that are too big according to config/graphics/max_texture_dimension. Otherwise, since we have 1024 & 2048 art, it will be impossible for low-end systems to cope with. That way, the impossibility of runtime downsampling won't be a factor for dds usage - otherwise, I'd say anythin above 512x512 should be left uncompressed (so that runtime downsampling is possible).

[safemode]

making dds loading load mipmaps from the point of max texture size and lower is easy enough. Without recompressing or processing.


So that's not too bad.

I would be ok with not compressing things like splash screens and base art (since they dont have high numbers in the game to bloat memory usage) but things that are only used once like splash screens must get freed from gl. No in-game compression or mipmaps should be generated from anything not compressed on the disk.

And nothing non-compressed should be used in space flight that's bigger than a few hundred KB.



so yea, i'm all for not compressing one-time things like splash screens, provided they get freed from GL memory after use, and not compressing base art provided they dont get compressed in-game and they dont exist in large numbers, and I can make the hardware DDS code check the max texture size and use the correct mipmap that is appropriate if that size is smaller than the base image.

[safemode]

basically, the mod to the dds loading would be done right before calling the GL function. the entire dds file would be read in, and at the GL level, we'd be able to grep what the max texture size supported is. Then i can compute rather easily the offset in the file buffer i'd need to start from to begin at a mipmap that is below/at that max texture size and call the GL function with the new width/height/buffer offset.

This would have to be done in 3 places (as far as i've found so far), gl_texture.cpp, guitexture.cpp and glut_support.cpp.

[ace123]

Why in the world do we still have glut_support.cpp and guitexture.cpp?

I'm removing all the texture support in glut_support.cpp since it hasn't been referenced for the past five years.

for guitexture I"m just making it reuse texture code by making it store a Texture.

[charlieg]

Check out the changelog and readme files... there's a lot of unkept cruft in the VS codebase.

[ace123]

Looks like safemode got to those files first.

Anyway I just cleared out all the OpenGL texture calls... I made GuiTexture a simple texture class with two functions: read(string filename) and draw(Rect) that just wrap around Texture.

And I removed the GUITexture class altogether since it was not used anywhere.

[chuck_starchaser]

Safemode; what you're trying to do --standardizing the arts masters codecs and whatnot-- is laudable. And the idea of having a separate repo for art originals is something that should have been done from the start. However, there's a technical gottcha: You need to standardize the art methodology first. Trying to do a proper texturing job with Gimp, you end up with like a dozen xcf's...

• metalbase.xcf
• paint.xcf
• materials.xcf
• bump.xcf
• ambient_occlusion.xcf
• writings.xcf
• dirt.xcf
• scratches.xcf
• glows.xcf

...each having anywhere from 4 to 15 layers, and their outputs being used in yet another set of...

• diffuse.xcf
• specular.xcf
• shininess.xcf
• normal.xcf
• lightmap.xcf

...and when all is said and done you've got 900 megabytes for one ship.
Well, 300 compressed.
Sourceforge would not be impressed.
The best solution would be for all artists to move to using Blender nodes, instead, with which you can do any kind of blending imaginable and then some. A network can be made large enough to go from a set of original masks to final textures in one shot, without any intermediary textures. That brings 900 megs down to 9 megs worth of "originals".
At the PU forum we're working on retexturing PR's ships, and will soon have a Blender noodle to share with all VS mods. It exists already, though in a too non-final form to warrant revealing to the world yet. But just as an apetizer, it will take a set of masks:

• materials mask, where greyscale colors will encode metal finishes, and non-grayscale hues will denote paint.
• ambient occlusion bake (essential)
• frontal bake (essential for ships, used to modulate scratches/impacts)
• static lights bake (recommended)
• baked normalmap for mesh smoothing (optional)
• grooves map (negative part of bump map)
• protuberances map (positive part of bump map)
• logos and writing
• engine and window glows texture

And in one shot it will add impacts, scratches, dirt marks trailing from grooves, windmarks flanking protuberances, bit of rust on leading edges, well controlled diffuse and specular colors and shininess to make realistic looking materials, scratches on paint showing the material underneath, dents in the normal map where scratches begin, etceteras; and produce final png textures: diffuse, specular, glow, damage; with alpha channels encoding shininess, normalmap du and dv, and ambient occlusion.
So, if you can wait a week or two, we should have a pretty solid tool, set of shaders, and a standard art pipeline to go with it.

[safemode]

as far as data directories go it appears the latest plan is as follows:

data5.x-test will become data5.x, where development will continue as it does like now with data4.x.

The difference is data4.x will be stripped and renamed to something resembling "Masters" or such. Only texture data will be kept there and it'll be all the original forms that get used to create the dds files in data5.x.. Any texture updates will have to go into two categories.

If you're updating a dds with a dds, you can commit it to data5.x. Same goes for if the image in data5.x is the same type as the one you're commiting (for the few bitmap/jpg/png's left)

If you're updating a texture with a different type (dds in data5.x but you're updating the original texture which is a png or jpg) then you have to commit that file to the Masters repository, then either you or someone else will create DDS files from those new textures and commit them to data5.x.

[klauss]

I'd rather leave data4.x as it is and branch it into Masters - rationale being, if you wait for all devs to switch to working on data5.x, you'll have mayhem in your hands. It's easier to just branch a "Masters" from data4.x with svn copy.

[safemode]

That's what i said. Data4.x will be branched (svn copy) to Masters (with the DDS files reverted to thier originals). Data5.x-test will then replace data4.x in trunk and be named data5.x and that will be the new data dir for VS.

The reason is, data5.x-test has well over 1200 changed textures from data4.x, it makes no sense at all to merge that to data4.x and use data4.x just because people may be too lazy to rerun a svn co command instead of update. Either way you'd be redownloading everything. At least with the way i am mentioning, it'll make sense with the version and it's easier on sourceforge.

branches/data5.x-test -> trunk/data5.x
trunk/data4.x -> trunk/masters

data5.x-test wont exist anymore, data4.x wont exist anymore.

[klauss]

Since masters will then get stripped down to art only, I'd recommend:

data4.x cp-> trunk/masters
data4.x mv-> tags/pre-dds-whatever
data5.x-test mv-> data5.x

Just a suggestion.

[ace123]

We have to keep in mind that some developers have been modifying data4.x.
I would like to remove this confusion factor now if possible, since this is making it much harder to bughunt for release.

I'm going to start by doing the following.

Code: Select all

svn copy http://.../tags/before_dds http://.../branches/masters
svn move http://.../trunk/data4.x http://.../trunk/data5.x
svn update
cd data5.x
svn merge -r 11448:HEAD http://.../branches/data5.x-test
// Check for texture conflicts... make sure they are all valid
svn commit

Does this make sense?

If you are wondering why I'm doing it that way, it's to avoid an entire checkout if I already have the entire trunk directory checked out in one folder...

But if you prefer, I can do the opposite order:

Code: Select all

svn copy http://.../tags/before_dds http://.../branches/masters
svn move http://.../trunk/data4.x http://.../branches/data4.x
svn move http://.../branches/data5.x-test http://.../trunk/data5.x
svn update
cd data5.x
svn merge -r 11448:HEAD http://.../branches/data4.x
// Check for python/other conflicts... make sure they are all valid
svn commit

However, I see how the latter might require less transfer on commit... but more on update... so it might be better. What's your opinion on this?

I'm just want to make sure that I won't end up trashing the repository.

EDIT: Klauss made it before me...
anyway, data4.x is just simply broken as far as textures go... some of it is DDS, but apparently they might be corrupt.
I made a tag before any DDS conversions happened, in tags/before_dds...

[safemode]

you can do that if _you_ want to, uploading half a gig of textures in a commit is not something i want to do again. Nor is looking through them all and figuring out which conflicts (and they will all probably be conflicts) are valid is also not something i want to do.

It's much easier to merge in the few remaining changes in data4.x to data5.x-test and move that to /trunk/data5.x than the other way around. The end result is the same, you're just choosing to go the long way for no reason.


The other method Just make absolutely sure the merging of data4.x changes to data5.x are only non-texture updates. All texture changes (that aren't already handled in data5.x-test) need to be compressed to dds.

There should be few if any textures i haven't already merged into data5.x-test

[safemode]

Basically, verifying textures would consist of checking to see if a power of 2 fix missed getting put in data5.x-test (i think i got all of them, but maybe not) and making sure all the new normal/bump map textures made it in. Everything else i am 99.99% sure i've kept up to date, just in DDS form.


I just want to make it clear, the plan is not to continue using a repository called data4.x, but to use the textures already in data4.x to create a master repository. Then to copy the data5.x-test repository over to data5.x and use that.

After we make sure data5.x-test is synced up in all non-texture related areas.

all this idea that you're going to avoid redownloading all of the data directory is crap. You're going to download the entire contents, since textures make up most of the size of the data4.x dir, and i've converted 99% of them. There is no avoiding the basically, complete re-checkout of the data dir.

[safemode]

blah, new plan.

data4.x will get merged to data5.x-test (all relevant changes anyway).
data4.x (pre dds tag + power of two fixes + new bumpmaps and such) will get moved to a Masters repository.

data5.x-test then gets moved to data5.x and a redirect is placed in a data4.x repository to point to data5.x, for those not caring to change their local directory to point to the data5.x repository with the svn change command.

[ace123]

I'm sorry... you are right about the number of modifications required... data5.x has a huge number of differences from 4.x

I wasn't aware of a redirect option...

Anyway, it's not too important... I was just scared at the thought of everyone with their copies of SVN downloading another 500 megs.

[ace123]

Looks like the move is now complete.

Oh well... the problem's solved for me... at least in the sense that I have no choice but to re-download everything.

One thing SVN is really efficient at is instantly deleting a 1GB folder,

patrick@quad:~/vegastrike$ svn up
D data5.x
svn: Directory 'data5.x' is missing

As to the masters, it will be:
svn copy /tags/before_dds /branches/data_masters
Then, merge in any non-texture changes (modules, unit BFXM's)...
As to the new normal/bumpmaps, they don't have originals, do they?

I don't see the reason for having the masters in trunk/ since they are not really meant to be downloaded in most cases... the way I see it is downloading individual master textures off viewvc when they are needed, or else downloading a specific folder.

[safemode]

everyone is going to be downloading all the changes in the textures, and that's pretty much the entire size of data4.x as it is (only slightly bigger). That's just a reality of the sheer number of changes being made. People have to download changes, and there are a lot of them.



Now, For the love of god people, dont commit non-dds textures to data4.x unless you really want it to be uncompressed PNG. All other textures should be committed to the master repository, which is an exact copy of data4.x prior to being replaced by data5.x-test.



Basically there are two options in this testing phase.
1. You run svn switch "url to masters repository (copy of old data4.x)"
inside your data4.x dir.
This will allow you to continue using your data4.x dir as a master texture repository for contributing new art (non-dds). You should grab data4.x in a different directory though to actually play the game, since masters will eventually strip out all non-texture data.

2. You can run svn update in your data4.x and now have a much higher percentage of DDS files (should all be good and 2^n valid.). You will need to grab masters separately if you plan on doing any artwork.

[ace123]

I updated too early so I lost everything...

Anyway, it's fine... good thing I'm getting 3MB/s and not 200kB/s like at home

I thought masters was going to be before_dds, not the old broken data4.x directory.

[safemode]

i didn't want to change data4.x (now masters) before i determined that moving data5.x to data4.x would allow an update from an old data4.x to successfully run.

since i know it does now, i can go about modifying masters so it reflects pre-dds textures.

That's going ot take a bit of time, since i gotta go through the animations dir and determine what textures need to be reverted, since some were fixed for power of 2 issues, and we dont want to revert those (they're not dds either).

[safemode]

I'm probably not going to have the time to revert the dds files in the masters repository tonight.

for now, masters is exactly the same as the old data4.x dir. So it should work just as well as it did prior to this move, and commits can be made to it just as you were doing prior (though only textures should be committed, source code should be done to current data4.x).


i'll fix the masters tomorrow so that there are no dds files. unless someone beats me

[eddit]

You'll probably want to delete all the binaries and python from Masters as well, just leaving all the models and textures.

Just to check, we should be checking out data5.x or data4.x for latest goodness?

Edwin

[ace123]

safemode has renamed data5.x to data4.x to keep existing repositories.

Basically, you want to update your data4.x directory to get all the latest updates.

The "masters" directory is not yet an actual masters repository... in fact, at the moment, it is just the old data4.x renamed.
But safemode plans to put the good textures in it..

As to things like binaries, music, python... I don't see any good reason not to delete them... To have to be able to play VS from the masters directory without a copy of data4.x shouldn't be a problem.

[safemode]

if you dont care to add artwork, just update in your data4.x dir. basically, the usable data dir is still data4.x for artists looking to commit their work, they need to do so in the masters repository.