Units organization

Development directions, tasks, and features being actively implemented or pursued by the development team.

Units organization

Postby pyramid » Tue Sep 30, 2008 5:33 pm

This proposal concerns the VS game only and does not affect the mods.

I was thinking about a reorganization of the units directory (/data/units). It has become pretty big already (102 subdirs).
In order to have and maintain a better overview for devs and artists, I'd like to stick vessels and installations into separate subdirectories with the following structure:
* /data/units/vessels/ for units of type Vessel
* /data/units/installations/ for units of type Installation

and similar for other unit types like upgrades, turrets, or subunits.

At the same time I'd like to apply the following naming convention:
* unit names (units.csv keys) and directory names are Title case, e.g. "Admonisher"
* all files within the unit directory are lower case, e.g. "admonisher.bfxm", "admonisher-hud.sprite"
User avatar
pyramid
Expert Mercenary
Expert Mercenary
 
Posts: 960
Topics: 44
Joined: Wed Jun 14, 2006 6:02 pm
Location: Somewhere in the vastness of space

Share On:

Share on Facebook Facebook Share on Twitter Twitter Share on Digg Digg

Postby loki1950 » Tue Sep 30, 2008 5:51 pm

Good idea and about time :wink: as you mentioned it is getting a bit confused as for sub-units should they not be associated with there primary unit.

Enjoy the Choice :)
my box::HP Envy i5-6400 @2Q70GHzx4 8 Gb ram/1 Tb(Win10 64)/3 Tb Mint 18/GTX745 4Gb acer S243HL K222HQL
Q8200/Asus P5QDLX/8 Gb ram/WD 2Tb 2-500 G HD/GF GT640 2Gb Mint 17.3 64 bit Win 10 32 bit acer and DELL E6400 4GB ram/100 GB HD Mint 17.3 6
User avatar
loki1950
The Shepherd
 
Posts: 5701
Topics: 51
Joined: Fri May 13, 2005 1:37 pm
Location: Ottawa

Postby jackS » Tue Sep 30, 2008 9:03 pm

Where are the factional variants going to fit into this hierarchy?

the units/factions, units/weapons, and units/subunits directories are already special-purposed in terms of file search order. We'll need to make some minor code tweaks to match whatever new organization we choose.

(also, before we get into trouble, the eject unit's key is hardcoded - so we'll need to double check any script-applied case changes, etc.)

@Loki: sub units should not (in general) be associated with the units that use them, because that is a many:1 relationship (many units will use the same subunit).
User avatar
jackS
Minister of Information
Minister of Information
 
Posts: 1895
Topics: 56
Joined: Fri Jan 31, 2003 2:40 pm
Location: The center of Centre county

Postby Deus Siddis » Tue Sep 30, 2008 9:42 pm

Could we call vessel 'vehicle' instead?
Deus Siddis
Elite
Elite
 
Posts: 1362
Topics: 13
Joined: Sat Aug 04, 2007 8:42 am

Postby pyramid » Wed Oct 01, 2008 2:26 am

I'd leave the units/factions, units/weapons, units/eject, units/navpoint, and and units/subunits as they are. There is no change required as they are already fairly well organized. My concern is units/vessels, and units/installations which will only require a change in units.csv to point to the right directory and correct capitalization in faction_ships.py.
Faction variants are represented by textures in the corresponding unit directory so I do not see what else would be required beyond the units.csv directory change. In the case of faction specific meshes, there would be a specific row entry in units.csv which also implies a directory change, not more.

Deus Siddis wrote:Could we call vessel 'vehicle' instead?

The unit class seems to be hardcoded somewhere. (and I do actually like 'vessel'.)
User avatar
pyramid
Expert Mercenary
Expert Mercenary
 
Posts: 960
Topics: 44
Joined: Wed Jun 14, 2006 6:02 pm
Location: Somewhere in the vastness of space

Postby Nózmájner » Wed Oct 01, 2008 2:33 am

It might be useful to further subcategorize those like:
* /data/units/vessels/class where class is like fighter, capital, freighter and so
or
* /data/units/vessels/faction

And it might be wise to create a naming scheme, and stick to it:
faction_name
shipclass_name.mesh
Respectively, and for directory naming and file naming.

or both faction and class in the file name, if subfoldering is not feasible.

It would be helpful when you need to skimm trough a lot of ships.

To be honest, the whole vs directory structure is a bit half baked, at least for me. I don't know how those stuffs are hard-coded or not, but the whole structure would deserve a reorganization, like for example:
Code: Select all
./vs/data/artwork/
./vs/data/artwork/music/
./vs/data/artwork/sfx
./vs/data/artwork/units/
.vs/data/scripts/
.vs/data/galaxy/
.vs/bin/
.vs/config
.vs/background (for manual and any other work, like any short story or description, tabletop rgn in the universe, watherver)
Nózmájner
Mercenary
Mercenary
 
Posts: 113
Topics: 8
Joined: Fri Apr 01, 2005 6:36 am
Location: Hungary

Postby pyramid » Wed Oct 01, 2008 5:01 am

Classifying crafts by faction could be troublesome especially where there is more than one manufacturing or utilizing faction. Classification by class is a possibility, however I am not sure how error-prone the editing might become with more detail added to the dir structure. Ideally we should avoid more than 3 directory levels (1=units, 2=class, 3=unitname).

I'm not so bad with the remaining directory structure, except would stick data/history into data/documentation to become data/documentation/history as that 2 files should not have a top-level directory on their own.

As for the naming scheme, I have proposed one in my first post:
* unit names (units.csv keys) and directory names > Title case, e.g. "Admonisher"
* all files within the specific unit directory > lower case, e.g. "admonisher.bfxm"
* all special files > unit name + dash "-" + identifier, e.g. , "admonisher-hud.sprite" "admonisher-shield.bfxm"
* textures already have a naming scheme > faction_tex_xyz.texture with x, y, z set by the UnitConverter (x=sub/unit number, y=currently unused, z=texture number) , e.g. "LIHW_tex_000.texture".
User avatar
pyramid
Expert Mercenary
Expert Mercenary
 
Posts: 960
Topics: 44
Joined: Wed Jun 14, 2006 6:02 pm
Location: Somewhere in the vastness of space

Postby pyramid » Wed Oct 01, 2008 3:48 pm

In the first batch, the vessels were moved to data/units/vessels, directory names changed to title case, and all vessel keys too.
Now it's even easy to count the assets: 74 dirs means we have 74 different types of vessels available ingame.

Any problems, please report here.
User avatar
pyramid
Expert Mercenary
Expert Mercenary
 
Posts: 960
Topics: 44
Joined: Wed Jun 14, 2006 6:02 pm
Location: Somewhere in the vastness of space

Postby chuck_starchaser » Wed Oct 01, 2008 4:11 pm

Good stuff, Pyramid; I think I'll do the same with PU.

Another idea you might want to look into is having a separate repository for units.csv. We did that for PU and have been very happy since. In that repository, we have many small csv files for all kinds of categories. Small, easy to edit and deal with.
https://svn.wcjunction.com/csv/units.csv
username is username
password is password
When we commit a change to that repo, a script Wolphin wrote automatically puts together a new units.csv, from all the little files, and commits it to /trunk.
User avatar
chuck_starchaser
Elite
Elite
 
Posts: 8014
Topics: 195
Joined: Thu Sep 04, 2003 9:03 pm
Location: Montreal

Postby pyramid » Thu Oct 02, 2008 1:40 am

Thanks for the info. I can see why it is handy to have the separation.

Actually, I do not dislike the editing of units.csv. It is right now fairly well sorted by unit groups and then (mostly) alphabetically within groups. Also UnitConverter inserts new units (currently vessels and installations) under the right group and in alphabetical order, too.
My concern is only if I should make the subdir structure default. Most probably I will make it a configurable option as there might be mods that would like to keep /data/units/ flat.
User avatar
pyramid
Expert Mercenary
Expert Mercenary
 
Posts: 960
Topics: 44
Joined: Wed Jun 14, 2006 6:02 pm
Location: Somewhere in the vastness of space

Postby jackS » Thu Oct 02, 2008 2:58 am

I think flat is a better default for the moment.

A) there can clearly be scenarios that involve unit types that are not cleanly semantically categorized into "vessels" and "installations"

B) this level of organizing is primarily for developer comfort -- I'm not convinced we won't be changing our minds again at some later point in the future. I think we can bear to revisit whether or not this is the manner of organization we're going to want to present to everyone who wants to create content for the engine at a later time.

To be honest, I was happier with the internal organization of units.csv when similar ship types were grouped (interceptor group, bomber group, etc), so that it was easy to compare between like vessels. However, that mostly just goes to show how far we have to go until we have a truly comfortable toolchain. The internal ordering of entries is something that should, in the end, be abstracted away from those doing the editing.
User avatar
jackS
Minister of Information
Minister of Information
 
Posts: 1895
Topics: 56
Joined: Fri Jan 31, 2003 2:40 pm
Location: The center of Centre county

Postby pyramid » Thu Oct 02, 2008 6:26 am

Re Flat as default. I'm fine with that.

Re B) Yes, this is clearly for dev comfort and we can revisit this whenever deemed necessary.

Re Internal units.csv organization. The alphabetical sorting was more handy for me, but then I understand the need to have other sortings handy. Which brings me to the idea to actually have a tool that resorts units.csv rows.
User avatar
pyramid
Expert Mercenary
Expert Mercenary
 
Posts: 960
Topics: 44
Joined: Wed Jun 14, 2006 6:02 pm
Location: Somewhere in the vastness of space

Postby jackS » Thu Oct 02, 2008 10:39 am

So, I think the thing is that we don't want to confuse a view of the file with the actual internal organization of the file -- having a set of tools that can present alternate sorting orders, and display subsets of the units present. would be very useful, but having the actually saved file ping-pong between preferred views would just make the diffs atrocious. This is why I was commenting on (as much as I am absolutely enamored with your unit converter) we still have a ways to go before the tool users are fully insulated from the raw data :)
User avatar
jackS
Minister of Information
Minister of Information
 
Posts: 1895
Topics: 56
Joined: Fri Jan 31, 2003 2:40 pm
Location: The center of Centre county

Postby pyramid » Thu Oct 02, 2008 3:30 pm

Got you now. I'll revert the internal ordering of units.csv to like it was before when the opportunity arises, i.e. grouped by class and then within each class primary sorted by unit type, and secondary sorted by alpha. The classes themselves could be sorted by a predefined order, like vessels first, installations second, then whatever. I have no preference since I'll be primarily using the UnitConverter, even if only for testing purposes.
User avatar
pyramid
Expert Mercenary
Expert Mercenary
 
Posts: 960
Topics: 44
Joined: Wed Jun 14, 2006 6:02 pm
Location: Somewhere in the vastness of space

Postby pyramid » Fri Oct 03, 2008 2:05 am

The installations are now in their own subdir data/units/installations. At the same time the title case naming convention has been applied where each word separated by underscore "_" is title cased (e.g. "Commerce_Center").

We have 15 different installations integrated and one unused (Diplomatic Center). And it's much easier to navigate the directory structure.
Usually the /masters directory structure follows (or should follow) the /data directory structure. I will take this on during the next days.

The first drawback we have is that users will have probably to re-download lots of data since the move command seems to be recognized by svn as a delete and create on directory level (though the files are then moved) but many clients will just re-download everything.

The other disadvantage is that saved games would need a manual replace for all lower case named letters to title case, e.g. "hidalgo" to "Hidalgo". A small script might help here. If needed, I might provide the renaming pairs (from -> to). Also it is advisable to delete all ./vegastrike/sectors entries as some installations might be referenced by the old naming.

For installations I wasn't sure about the relation between keys for faction specific entries, e..g
* Starfortress,./installations/Starfortress
* starfortress__aera,./factions/aera/starfortress

1) Does the aera star fortress require to be title case, too?
2) Is there any relation between those keys? If so which?
3) How are this faction installations referenced by the engine? Is it by faction owning a system? Does this mean that when I have faction1 owning system A and then it is taken over by faction2, the model will be replaced by another one?
User avatar
pyramid
Expert Mercenary
Expert Mercenary
 
Posts: 960
Topics: 44
Joined: Wed Jun 14, 2006 6:02 pm
Location: Somewhere in the vastness of space

Postby jackS » Fri Oct 03, 2008 10:57 am

1&2) Yes, but the "aera" in "__aera" needs to match the case in factions.xml
The factional variants work by having all of the units declared with only their base key (say, "Starfortress"). Then, the engine will search for BaseName__factionName before it searches for BaseName. Thus, if the cases don't match, the first search will fail, and it will find BaseName, but not baseName__factionName.
3) Firstly, just to be (excessively) clear, vehicles can have factional model variants too -- installations are just the more common case because we wanted to be able to say "Mining base" and have that be whatever it happened to be for that group.
If you were to save out a copy of the contents of a system, and then manually change the ownership of that system, what you described would certainly happen. However, I'm not certain whether, during system takeover, the bases are regenerated or not - I'd have to look into that. System takeover, especially for hand-crafted systems is known to be in need of some additional testing and revisiting, so whatever the current answer is, it may change.
User avatar
jackS
Minister of Information
Minister of Information
 
Posts: 1895
Topics: 56
Joined: Fri Jan 31, 2003 2:40 pm
Location: The center of Centre county

Postby pyramid » Fri Oct 03, 2008 1:11 pm

Re: base naming with factions. It's clear to me now.

Re: system takeover. I can imagine the following scenarios happening (independently for each installation in a system):
a) installation is boarded and taken over by the invading faction
b) installation is destroyed (with large debris floating around)
c) a new installation is built (most probable after c) happeneing)

Ideally, in future developments, we should provide mechanisms for those scenarios to be moddable.
User avatar
pyramid
Expert Mercenary
Expert Mercenary
 
Posts: 960
Topics: 44
Joined: Wed Jun 14, 2006 6:02 pm
Location: Somewhere in the vastness of space

Postby safemode » Wed Oct 08, 2008 2:18 pm

With this change breaking old save-games and requiring a massive update of data, i believe we should update the roadmap, move any un-finished parts to the 0.5.2 area, and freeze 0.5.1 and just put it out when the unit re-organization is finished. Basically, finish up anything to beta quality levels that are half started that would otherwise cause the game to not play and get started on 0.5.2 stuff.
Ed Sweetman endorses this message.
safemode
Developer
Developer
 
Posts: 2150
Topics: 84
Joined: Sun Apr 22, 2007 6:17 pm
Location: Pennsylvania

Postby pyramid » Wed Oct 08, 2008 4:07 pm

2 things I would like to see finished before freezing 0.5.1:
* CineMut completed. It's almost done. We are just waiting for chuck to complete and test the Hammer model.
* I still have a couple of installations (by Fendorin) that I'd like to get into the data set. They are basically done sitting on my hard drive and I am just waiting for jackS' direction on the dimensions of those installations.
User avatar
pyramid
Expert Mercenary
Expert Mercenary
 
Posts: 960
Topics: 44
Joined: Wed Jun 14, 2006 6:02 pm
Location: Somewhere in the vastness of space

Postby charlieg » Wed Oct 08, 2008 6:14 pm

Meh, I'm not a fan of the VS version numbering, not a fan at all.

0.x.y - the .y releases should never break save game compatability. When you do so, you should increment x instead.

0.6, 0.7... 0.23 if you have to before 1.0 - that way you can have more regular releases without worrying too much.
Free Gamer - free software games compendium and commentary!
FreeGameDev forum - open source game development community
User avatar
charlieg
Elite Mercenary
Elite Mercenary
 
Posts: 1328
Topics: 56
Joined: Thu Mar 27, 2003 4:51 pm
Location: Manchester, UK

Postby chuck_starchaser » Wed Oct 08, 2008 6:33 pm

One thing that CineMut needs, *badly*, is cubemap support. According to Klauss, the code is already there. He got them working in no time.
The catch was, and is, that to use LOD control of the cubemap for glossiness of materials, we need cubemaps generated by ATI's CubeMapGen.
But CubeMapGen exports only one kind of DDS file which has 60 layers for a 1024 cubemap (10 LOD's x 6 sides), and there's no routine currently in the engine that can read that format.
That's the only thing holding us back, really. Klauss was about to do it, and then his videocard died.
CineMut is not "finished" but it is sort of usable, good enough for experimenting with texturing for it, at least; and the Hammer could be released after; not that it should take much longer.
But it would be a pity to release CineMut without cubemaps. Its second-most-important feature (shininess/gloss representation) won't show with spheremaps; --or even with cubemaps made with anything but ATI's CubeMapGen.
User avatar
chuck_starchaser
Elite
Elite
 
Posts: 8014
Topics: 195
Joined: Thu Sep 04, 2003 9:03 pm
Location: Montreal

Postby safemode » Thu Oct 09, 2008 6:25 am

I can and will add dds cubemap reading support into the image reading code (vsimage) ...that's a non-issue, i just haven't gotten any time to get back to VS yet. moving sucks.


The only real issue is modifying the image requesting code to not request each side seperately, but to deal with backgrounds as if they come from 1 file (cubemap). It's not so much that it's complex, just that it's obfuscated a bit due to how ancient that code is.

It'll get done. soon with any luck.

Really, the debate about versions is as flame-worthy as politics. Some have suggested we do away with them and just use dates. Right now, all they really mean is a way for developers to differentiate certain milestones. I think things will split at 1.0.0, with a possible (old) kernel approach in creating a dev tree that will sync back to main every 1.x release, with non-breaking updates backported to minor releases 1.x.y, leaving the breaking updates to the 1.x sync.

It'd be nice if we could do that now, but we just dont have the man-power to force everyone to do it the "right" way. And things are moving so fast, we'd quickly run out of numbers before 1.0 and we'd still be far from 1.0 ready.
Ed Sweetman endorses this message.
safemode
Developer
Developer
 
Posts: 2150
Topics: 84
Joined: Sun Apr 22, 2007 6:17 pm
Location: Pennsylvania

Postby charlieg » Thu Oct 09, 2008 6:58 am

It's not possible to run out of numbers before 1.0... 0.9 < 0.25 in version numbering. They're not fractions. ;)
Free Gamer - free software games compendium and commentary!
FreeGameDev forum - open source game development community
User avatar
charlieg
Elite Mercenary
Elite Mercenary
 
Posts: 1328
Topics: 56
Joined: Thu Mar 27, 2003 4:51 pm
Location: Manchester, UK

Postby safemode » Thu Oct 09, 2008 7:40 am

if you dont want any N.X.y to break for all values of y, then you have to alter X when they do break. Now there are only 4 more numbers between 0.5 and 1.0 where then we would allow to break the game backwards compatibility. I believe that's an impossible restriction given both the rate of changes going into the game, and the amount of man-power that is available to work on the game. We can't effectively pool such breakages into one release, they happen over too large a span of time. And we can't always tell developers working on large changes to hold off or we risk not having the help at all when we need it.

I was under the impression you dont like 0.5.1 breaking 0.5.0 and thus no value of y can break within a value of X. So I'm not sure how you dont only get 4 allowances for a minor version not breaking a major-minor?

0.6 can break 0.5, 0.7 can break 0.6 etc. to 1.0 That's what you were saying as far as i understood it, 0.5.1 cannot break 0.5.0, and as such, all 0.5.x versions should be inter-compatible and not break save-game compatibility, or binary compatibilty even with the game-data.

that leaves us with only 4, and that's not enough for the amount of work that needs to be done before 1.0. After 1.0 is a different story altogether.

edit:

ok, i woke up and re-read your post again and see what you're saying. I always just did versioning like the second number goes from 0-9, with the first number and last number able to go from 0 to inf. I dont know why, but my brain just assumes that method.

I guess we could then increment the minor version, the only problem with that is sometimes this can occur without much of a big difference, so it comes up against the idea that version changes should reflect the amount of alteration. I guess in the end we're just butting up against the fact that we still have a development organizational problem. I just dont think we can fix the problem without being detrimental to getting work that we can get done, done, until we have a few more people involved in the project to help juggle the work.
Ed Sweetman endorses this message.
safemode
Developer
Developer
 
Posts: 2150
Topics: 84
Joined: Sun Apr 22, 2007 6:17 pm
Location: Pennsylvania

Postby chuck_starchaser » Thu Oct 09, 2008 8:08 am

I'm not familiar with the current numbering system, but I would suggest, if not done so yet, to distinguish between game revisions and engine revisions.
User avatar
chuck_starchaser
Elite
Elite
 
Posts: 8014
Topics: 195
Joined: Thu Sep 04, 2003 9:03 pm
Location: Montreal


Next

Return to Engine Development

Who is online

Users browsing this forum: No registered users and 2 guests

cron