Units organization

Development directions, tasks, and features being actively implemented or pursued by the development team.
Post Reply
pyramid
Expert Mercenary
Expert Mercenary
Posts: 988
Joined: Thu Jun 15, 2006 1:02 am
Location: Somewhere in the vastness of space
Contact:

Units organization

Post by pyramid »

This proposal concerns the VS game only and does not affect the mods.

I was thinking about a reorganization of the units directory (/data/units). It has become pretty big already (102 subdirs).
In order to have and maintain a better overview for devs and artists, I'd like to stick vessels and installations into separate subdirectories with the following structure:
* /data/units/vessels/ for units of type Vessel
* /data/units/installations/ for units of type Installation

and similar for other unit types like upgrades, turrets, or subunits.

At the same time I'd like to apply the following naming convention:
* unit names (units.csv keys) and directory names are Title case, e.g. "Admonisher"
* all files within the unit directory are lower case, e.g. "admonisher.bfxm", "admonisher-hud.sprite"
loki1950
The Shepherd
Posts: 5841
Joined: Fri May 13, 2005 8:37 pm
Location: Ottawa
Contact:

Post by loki1950 »

Good idea and about time :wink: as you mentioned it is getting a bit confused as for sub-units should they not be associated with there primary unit.

Enjoy the Choice :)
my box::HP Envy i5-6400 @2Q70GHzx4 8 Gb ram/1 Tb(Win10 64)/3 Tb Mint 19.2/GTX745 4Gb acer S243HL K222HQL
Q8200/Asus P5QDLX/8 Gb ram/WD 2Tb 2-500 G HD/GF GT640 2Gb Mint 17.3 64 bit Win 10 32 bit acer and Lenovo ideapad 320-15ARB Win 10/Mint 19.2
jackS
Minister of Information
Minister of Information
Posts: 1895
Joined: Fri Jan 31, 2003 9:40 pm
Location: The land of tenure (and diaper changes)

Post by jackS »

Where are the factional variants going to fit into this hierarchy?

the units/factions, units/weapons, and units/subunits directories are already special-purposed in terms of file search order. We'll need to make some minor code tweaks to match whatever new organization we choose.

(also, before we get into trouble, the eject unit's key is hardcoded - so we'll need to double check any script-applied case changes, etc.)

@Loki: sub units should not (in general) be associated with the units that use them, because that is a many:1 relationship (many units will use the same subunit).
Deus Siddis
Elite
Elite
Posts: 1363
Joined: Sat Aug 04, 2007 3:42 pm

Post by Deus Siddis »

Could we call vessel 'vehicle' instead?
pyramid
Expert Mercenary
Expert Mercenary
Posts: 988
Joined: Thu Jun 15, 2006 1:02 am
Location: Somewhere in the vastness of space
Contact:

Post by pyramid »

I'd leave the units/factions, units/weapons, units/eject, units/navpoint, and and units/subunits as they are. There is no change required as they are already fairly well organized. My concern is units/vessels, and units/installations which will only require a change in units.csv to point to the right directory and correct capitalization in faction_ships.py.
Faction variants are represented by textures in the corresponding unit directory so I do not see what else would be required beyond the units.csv directory change. In the case of faction specific meshes, there would be a specific row entry in units.csv which also implies a directory change, not more.
Deus Siddis wrote:Could we call vessel 'vehicle' instead?
The unit class seems to be hardcoded somewhere. (and I do actually like 'vessel'.)
Nózmájner
Mercenary
Mercenary
Posts: 113
Joined: Fri Apr 01, 2005 1:36 pm
Location: Hungary

Post by Nózmájner »

It might be useful to further subcategorize those like:
* /data/units/vessels/class where class is like fighter, capital, freighter and so
or
* /data/units/vessels/faction

And it might be wise to create a naming scheme, and stick to it:
faction_name
shipclass_name.mesh
Respectively, and for directory naming and file naming.

or both faction and class in the file name, if subfoldering is not feasible.

It would be helpful when you need to skimm trough a lot of ships.

To be honest, the whole vs directory structure is a bit half baked, at least for me. I don't know how those stuffs are hard-coded or not, but the whole structure would deserve a reorganization, like for example:

Code: Select all

./vs/data/artwork/
./vs/data/artwork/music/
./vs/data/artwork/sfx
./vs/data/artwork/units/
.vs/data/scripts/
.vs/data/galaxy/
.vs/bin/
.vs/config
.vs/background (for manual and any other work, like any short story or description, tabletop rgn in the universe, watherver)
pyramid
Expert Mercenary
Expert Mercenary
Posts: 988
Joined: Thu Jun 15, 2006 1:02 am
Location: Somewhere in the vastness of space
Contact:

Post by pyramid »

Classifying crafts by faction could be troublesome especially where there is more than one manufacturing or utilizing faction. Classification by class is a possibility, however I am not sure how error-prone the editing might become with more detail added to the dir structure. Ideally we should avoid more than 3 directory levels (1=units, 2=class, 3=unitname).

I'm not so bad with the remaining directory structure, except would stick data/history into data/documentation to become data/documentation/history as that 2 files should not have a top-level directory on their own.

As for the naming scheme, I have proposed one in my first post:
* unit names (units.csv keys) and directory names > Title case, e.g. "Admonisher"
* all files within the specific unit directory > lower case, e.g. "admonisher.bfxm"
* all special files > unit name + dash "-" + identifier, e.g. , "admonisher-hud.sprite" "admonisher-shield.bfxm"
* textures already have a naming scheme > faction_tex_xyz.texture with x, y, z set by the UnitConverter (x=sub/unit number, y=currently unused, z=texture number) , e.g. "LIHW_tex_000.texture".
pyramid
Expert Mercenary
Expert Mercenary
Posts: 988
Joined: Thu Jun 15, 2006 1:02 am
Location: Somewhere in the vastness of space
Contact:

Post by pyramid »

In the first batch, the vessels were moved to data/units/vessels, directory names changed to title case, and all vessel keys too.
Now it's even easy to count the assets: 74 dirs means we have 74 different types of vessels available ingame.

Any problems, please report here.
chuck_starchaser
Elite
Elite
Posts: 8014
Joined: Fri Sep 05, 2003 4:03 am
Location: Montreal
Contact:

Post by chuck_starchaser »

Good stuff, Pyramid; I think I'll do the same with PU.

Another idea you might want to look into is having a separate repository for units.csv. We did that for PU and have been very happy since. In that repository, we have many small csv files for all kinds of categories. Small, easy to edit and deal with.
https://svn.wcjunction.com/csv/units.csv
username is username
password is password
When we commit a change to that repo, a script Wolphin wrote automatically puts together a new units.csv, from all the little files, and commits it to /trunk.
pyramid
Expert Mercenary
Expert Mercenary
Posts: 988
Joined: Thu Jun 15, 2006 1:02 am
Location: Somewhere in the vastness of space
Contact:

Post by pyramid »

Thanks for the info. I can see why it is handy to have the separation.

Actually, I do not dislike the editing of units.csv. It is right now fairly well sorted by unit groups and then (mostly) alphabetically within groups. Also UnitConverter inserts new units (currently vessels and installations) under the right group and in alphabetical order, too.
My concern is only if I should make the subdir structure default. Most probably I will make it a configurable option as there might be mods that would like to keep /data/units/ flat.
jackS
Minister of Information
Minister of Information
Posts: 1895
Joined: Fri Jan 31, 2003 9:40 pm
Location: The land of tenure (and diaper changes)

Post by jackS »

I think flat is a better default for the moment.

A) there can clearly be scenarios that involve unit types that are not cleanly semantically categorized into "vessels" and "installations"

B) this level of organizing is primarily for developer comfort -- I'm not convinced we won't be changing our minds again at some later point in the future. I think we can bear to revisit whether or not this is the manner of organization we're going to want to present to everyone who wants to create content for the engine at a later time.

To be honest, I was happier with the internal organization of units.csv when similar ship types were grouped (interceptor group, bomber group, etc), so that it was easy to compare between like vessels. However, that mostly just goes to show how far we have to go until we have a truly comfortable toolchain. The internal ordering of entries is something that should, in the end, be abstracted away from those doing the editing.
pyramid
Expert Mercenary
Expert Mercenary
Posts: 988
Joined: Thu Jun 15, 2006 1:02 am
Location: Somewhere in the vastness of space
Contact:

Post by pyramid »

Re Flat as default. I'm fine with that.

Re B) Yes, this is clearly for dev comfort and we can revisit this whenever deemed necessary.

Re Internal units.csv organization. The alphabetical sorting was more handy for me, but then I understand the need to have other sortings handy. Which brings me to the idea to actually have a tool that resorts units.csv rows.
jackS
Minister of Information
Minister of Information
Posts: 1895
Joined: Fri Jan 31, 2003 9:40 pm
Location: The land of tenure (and diaper changes)

Post by jackS »

So, I think the thing is that we don't want to confuse a view of the file with the actual internal organization of the file -- having a set of tools that can present alternate sorting orders, and display subsets of the units present. would be very useful, but having the actually saved file ping-pong between preferred views would just make the diffs atrocious. This is why I was commenting on (as much as I am absolutely enamored with your unit converter) we still have a ways to go before the tool users are fully insulated from the raw data :)
pyramid
Expert Mercenary
Expert Mercenary
Posts: 988
Joined: Thu Jun 15, 2006 1:02 am
Location: Somewhere in the vastness of space
Contact:

Post by pyramid »

Got you now. I'll revert the internal ordering of units.csv to like it was before when the opportunity arises, i.e. grouped by class and then within each class primary sorted by unit type, and secondary sorted by alpha. The classes themselves could be sorted by a predefined order, like vessels first, installations second, then whatever. I have no preference since I'll be primarily using the UnitConverter, even if only for testing purposes.
pyramid
Expert Mercenary
Expert Mercenary
Posts: 988
Joined: Thu Jun 15, 2006 1:02 am
Location: Somewhere in the vastness of space
Contact:

Post by pyramid »

The installations are now in their own subdir data/units/installations. At the same time the title case naming convention has been applied where each word separated by underscore "_" is title cased (e.g. "Commerce_Center").

We have 15 different installations integrated and one unused (Diplomatic Center). And it's much easier to navigate the directory structure.
Usually the /masters directory structure follows (or should follow) the /data directory structure. I will take this on during the next days.

The first drawback we have is that users will have probably to re-download lots of data since the move command seems to be recognized by svn as a delete and create on directory level (though the files are then moved) but many clients will just re-download everything.

The other disadvantage is that saved games would need a manual replace for all lower case named letters to title case, e.g. "hidalgo" to "Hidalgo". A small script might help here. If needed, I might provide the renaming pairs (from -> to). Also it is advisable to delete all ./vegastrike/sectors entries as some installations might be referenced by the old naming.

For installations I wasn't sure about the relation between keys for faction specific entries, e..g
* Starfortress,./installations/Starfortress
* starfortress__aera,./factions/aera/starfortress

1) Does the aera star fortress require to be title case, too?
2) Is there any relation between those keys? If so which?
3) How are this faction installations referenced by the engine? Is it by faction owning a system? Does this mean that when I have faction1 owning system A and then it is taken over by faction2, the model will be replaced by another one?
jackS
Minister of Information
Minister of Information
Posts: 1895
Joined: Fri Jan 31, 2003 9:40 pm
Location: The land of tenure (and diaper changes)

Post by jackS »

1&2) Yes, but the "aera" in "__aera" needs to match the case in factions.xml
The factional variants work by having all of the units declared with only their base key (say, "Starfortress"). Then, the engine will search for BaseName__factionName before it searches for BaseName. Thus, if the cases don't match, the first search will fail, and it will find BaseName, but not baseName__factionName.
3) Firstly, just to be (excessively) clear, vehicles can have factional model variants too -- installations are just the more common case because we wanted to be able to say "Mining base" and have that be whatever it happened to be for that group.
If you were to save out a copy of the contents of a system, and then manually change the ownership of that system, what you described would certainly happen. However, I'm not certain whether, during system takeover, the bases are regenerated or not - I'd have to look into that. System takeover, especially for hand-crafted systems is known to be in need of some additional testing and revisiting, so whatever the current answer is, it may change.
pyramid
Expert Mercenary
Expert Mercenary
Posts: 988
Joined: Thu Jun 15, 2006 1:02 am
Location: Somewhere in the vastness of space
Contact:

Post by pyramid »

Re: base naming with factions. It's clear to me now.

Re: system takeover. I can imagine the following scenarios happening (independently for each installation in a system):
a) installation is boarded and taken over by the invading faction
b) installation is destroyed (with large debris floating around)
c) a new installation is built (most probable after c) happeneing)

Ideally, in future developments, we should provide mechanisms for those scenarios to be moddable.
safemode
Developer
Developer
Posts: 2150
Joined: Mon Apr 23, 2007 1:17 am
Location: Pennsylvania
Contact:

Post by safemode »

With this change breaking old save-games and requiring a massive update of data, i believe we should update the roadmap, move any un-finished parts to the 0.5.2 area, and freeze 0.5.1 and just put it out when the unit re-organization is finished. Basically, finish up anything to beta quality levels that are half started that would otherwise cause the game to not play and get started on 0.5.2 stuff.
Ed Sweetman endorses this message.
pyramid
Expert Mercenary
Expert Mercenary
Posts: 988
Joined: Thu Jun 15, 2006 1:02 am
Location: Somewhere in the vastness of space
Contact:

Post by pyramid »

2 things I would like to see finished before freezing 0.5.1:
* CineMut completed. It's almost done. We are just waiting for chuck to complete and test the Hammer model.
* I still have a couple of installations (by Fendorin) that I'd like to get into the data set. They are basically done sitting on my hard drive and I am just waiting for jackS' direction on the dimensions of those installations.
charlieg
Elite Mercenary
Elite Mercenary
Posts: 1329
Joined: Thu Mar 27, 2003 11:51 pm
Location: Manchester, UK
Contact:

Post by charlieg »

Meh, I'm not a fan of the VS version numbering, not a fan at all.

0.x.y - the .y releases should never break save game compatability. When you do so, you should increment x instead.

0.6, 0.7... 0.23 if you have to before 1.0 - that way you can have more regular releases without worrying too much.
Free Gamer - free software games compendium and commentary!
FreeGameDev forum - open source game development community
chuck_starchaser
Elite
Elite
Posts: 8014
Joined: Fri Sep 05, 2003 4:03 am
Location: Montreal
Contact:

Post by chuck_starchaser »

One thing that CineMut needs, *badly*, is cubemap support. According to Klauss, the code is already there. He got them working in no time.
The catch was, and is, that to use LOD control of the cubemap for glossiness of materials, we need cubemaps generated by ATI's CubeMapGen.
But CubeMapGen exports only one kind of DDS file which has 60 layers for a 1024 cubemap (10 LOD's x 6 sides), and there's no routine currently in the engine that can read that format.
That's the only thing holding us back, really. Klauss was about to do it, and then his videocard died.
CineMut is not "finished" but it is sort of usable, good enough for experimenting with texturing for it, at least; and the Hammer could be released after; not that it should take much longer.
But it would be a pity to release CineMut without cubemaps. Its second-most-important feature (shininess/gloss representation) won't show with spheremaps; --or even with cubemaps made with anything but ATI's CubeMapGen.
safemode
Developer
Developer
Posts: 2150
Joined: Mon Apr 23, 2007 1:17 am
Location: Pennsylvania
Contact:

Post by safemode »

I can and will add dds cubemap reading support into the image reading code (vsimage) ...that's a non-issue, i just haven't gotten any time to get back to VS yet. moving sucks.


The only real issue is modifying the image requesting code to not request each side seperately, but to deal with backgrounds as if they come from 1 file (cubemap). It's not so much that it's complex, just that it's obfuscated a bit due to how ancient that code is.

It'll get done. soon with any luck.

Really, the debate about versions is as flame-worthy as politics. Some have suggested we do away with them and just use dates. Right now, all they really mean is a way for developers to differentiate certain milestones. I think things will split at 1.0.0, with a possible (old) kernel approach in creating a dev tree that will sync back to main every 1.x release, with non-breaking updates backported to minor releases 1.x.y, leaving the breaking updates to the 1.x sync.

It'd be nice if we could do that now, but we just dont have the man-power to force everyone to do it the "right" way. And things are moving so fast, we'd quickly run out of numbers before 1.0 and we'd still be far from 1.0 ready.
Ed Sweetman endorses this message.
charlieg
Elite Mercenary
Elite Mercenary
Posts: 1329
Joined: Thu Mar 27, 2003 11:51 pm
Location: Manchester, UK
Contact:

Post by charlieg »

It's not possible to run out of numbers before 1.0... 0.9 < 0.25 in version numbering. They're not fractions. ;)
Free Gamer - free software games compendium and commentary!
FreeGameDev forum - open source game development community
safemode
Developer
Developer
Posts: 2150
Joined: Mon Apr 23, 2007 1:17 am
Location: Pennsylvania
Contact:

Post by safemode »

if you dont want any N.X.y to break for all values of y, then you have to alter X when they do break. Now there are only 4 more numbers between 0.5 and 1.0 where then we would allow to break the game backwards compatibility. I believe that's an impossible restriction given both the rate of changes going into the game, and the amount of man-power that is available to work on the game. We can't effectively pool such breakages into one release, they happen over too large a span of time. And we can't always tell developers working on large changes to hold off or we risk not having the help at all when we need it.

I was under the impression you dont like 0.5.1 breaking 0.5.0 and thus no value of y can break within a value of X. So I'm not sure how you dont only get 4 allowances for a minor version not breaking a major-minor?

0.6 can break 0.5, 0.7 can break 0.6 etc. to 1.0 That's what you were saying as far as i understood it, 0.5.1 cannot break 0.5.0, and as such, all 0.5.x versions should be inter-compatible and not break save-game compatibility, or binary compatibilty even with the game-data.

that leaves us with only 4, and that's not enough for the amount of work that needs to be done before 1.0. After 1.0 is a different story altogether.

edit:

ok, i woke up and re-read your post again and see what you're saying. I always just did versioning like the second number goes from 0-9, with the first number and last number able to go from 0 to inf. I dont know why, but my brain just assumes that method.

I guess we could then increment the minor version, the only problem with that is sometimes this can occur without much of a big difference, so it comes up against the idea that version changes should reflect the amount of alteration. I guess in the end we're just butting up against the fact that we still have a development organizational problem. I just dont think we can fix the problem without being detrimental to getting work that we can get done, done, until we have a few more people involved in the project to help juggle the work.
Ed Sweetman endorses this message.
chuck_starchaser
Elite
Elite
Posts: 8014
Joined: Fri Sep 05, 2003 4:03 am
Location: Montreal
Contact:

Post by chuck_starchaser »

I'm not familiar with the current numbering system, but I would suggest, if not done so yet, to distinguish between game revisions and engine revisions.
Post Reply