Page 1 of 1

Multiple .csv files in /units [SOLVED]

Posted: Fri Jan 04, 2008 6:28 am
by chuck_starchaser
Problem:

units.csv is growing huge... Can't see the top row descriptions... Too many people working on it at the same time can cause commit conflicts...

Proposed solution:

Split it...
  • guns.csv
  • missiles.csv
  • mines.csv
  • fighters.csv
  • bombers.csv
  • corvettes.csv
  • destroyers.csv
  • frigates.csv
  • cruisers.csv
  • carriers.csv
Implementation:

If the code could just read all .csv files in the /units folder, then, which way
to split it could be up to the modders; --e.g.: by faction, or whatever.

Posted: Wed Jan 09, 2008 12:56 am
by Erk
Heartily agree. The proposed split looks really nice. This would make editing a lot easier to handle.

Posted: Sun Jan 13, 2008 3:32 pm
by Moe479
one .csv per ship,weapon,etc. would be also no biggi for uptodate hardisks/comps
this would be much better than these monsterios long files for modding, for final realeases they could be thrown together to gain the last bit of of loadingspeed for the engine

Posted: Sun Jan 13, 2008 3:53 pm
by loki1950
There is a minor issue with separating all of the CSV files the overhead in the loading each separately it takes time to access each and it adds up very fast.

Enjoy the Choice :)

Posted: Sun Jan 13, 2008 4:02 pm
by Moe479
as i wrote, for realese u should copy and paste them together into only a few files, but for testing an locating bugs its better to have the best possible overview to navigate quickly to the problem instead of 10+secs loading time for the editor and seeking trough thousends of lines.

Posted: Sun Jan 13, 2008 5:41 pm
by chuck_starchaser
I'm not sure the extra time of loading multiple csv files would be significant enough to be of any concern. The number of files loaded on boot is a pretty big number already. Another solution could be to keep them in one compressed file.

Posted: Sun Jan 13, 2008 8:53 pm
by jackS
I think modifying the engine to deal with multiple csv files may be overkill if the primary benefit is that the file is too big for pleasant developing. As such a change is unlikely to be heavily prioritized in the near future, even if one disagrees that the following solution is long-term viable, it would certainly work as a short-term solution

What would seem a simpler solution to me would be, for any mod that wants to, do two things:
1) Split the units.csv file into whatever partitioning makes sense for that mod (whether it be by ship type or content creator or alphabetical... doesn't matter) and check these partitioned .csv files into the repository. Remove units.csv
2) Write a (fairly trivial in perl, at least) perl/shell/etc. script that will create a units.csv file out of a set of smaller .csv files. If necessary, create a binary version for windows users without decent scripting support. Run this to create a local version of units.csv for the engine to consume.

Then, you can happily develop on smaller fractions of the units.csv without having to change anything in the engine at all.

Posted: Sun Jan 13, 2008 9:16 pm
by chuck_starchaser
Yeah, that sounds reasonable.
I wasn't aware there was an easy way to merge csv files. I'll run this idea by the pu team, see if someone knows perl or whatever else will do it.

Posted: Mon Jan 14, 2008 2:18 am
by rigelan
The concatenate function works just fine on csv files

Posted: Mon Jan 14, 2008 3:46 am
by jackS
rigelan wrote:The concatenate function works just fine on csv files
If they all have headers, then those will have to be stripped first, but otherwise, yes, that's about all there is to it.

Posted: Mon Jan 14, 2008 8:33 pm
by Link
Attached is a small program I've written to split units.csv into groups based on the type and role columns. The example below assumes you've saved it somewhere in your $PATH as splitunitscsv.py and that it's executable.

Simple example of usage:

Code: Select all

$ cd /path/to/data4.x/units
$ splitunitscsv.py -o /split/unit/file/dir units.csv
# You can now edit the split csv files in /split/unit/file/dir
# Once done, execute the following command to merge them again
$ splitunitscsv.py -j -i /split/unit/file/dir units.edited.csv
# You can then replace units.csv with units.edited.csv
For more details, use splitunitscsv.py --help.

Posted: Mon Jan 14, 2008 9:42 pm
by chuck_starchaser
Hahaha, Thanks Link; Damn! I just finished splitting the files manually.
http://deeplayer.com/pu/csv/units.7z
Doh!
:D
I'll pass your script to Nate, who was just about to program this.
Thanks!
;-)

Posted: Tue Jan 15, 2008 2:26 am
by chuck_starchaser
Update for the curious. We've committed the files to a separate svn repo.
Nate (CoffeeBot) will write software and set up svn hooks so that any commit of
any csv file to the csv repository triggers the assembly of a new units.csv, followed by
a commit of that units.csv to the pu repo.
So the concatenation, etc will be server-side and transparent.
That's the plan, anyways.
Thanks all.

Posted: Fri Jan 25, 2008 8:01 am
by Shark
chuck_starchaser wrote:I'm not sure the extra time of loading multiple csv files would be significant enough to be of any concern. The number of files loaded on boot is a pretty big number already. Another solution could be to keep them in one compressed file.
I agree. Also, this would facilitate keeping the models, textures and CSV files in one directory for each ship (e.g., "<root>/ships/plowshare/"). This way, if an editor wants to modify all aspects of a ship, all he/she needs to do is extract that single directory, and it will contain all the necessary files.

I don't think making adjustments for modders is always such a bad thing.

Posted: Fri Feb 08, 2008 2:11 am
by chuck_starchaser
Well, the system is in place, working like a charm. We have a /csv repository with many little csv files for the many categories of things. When we edit one of those files and commit it, a script wolphin wrote concatenates all the files into a new units.csv, and commits the latter to the /priv_pu repository.
Works beautifully, and it has already yielded fruits: With all ships in one file, sorted alphabetically, bogus values that were there for armor immediately stood out like a sore thumb and were corrected, for example.
I'd highly recommend to other mods to do the same.

As for making it easier for people to share units.csv mods, this is something we've also discussed, and what we might do one of these days is a) set up an ftp on the server where people can pool modded ships, and b) include a small executable in the /units folder that can append csv files to units.csv. Better not to mix this ship sharing extravaganza paradigm with the developers' paradigm. Developers need better organization and compartmentalization, more than they need free flow.