Multiple .csv files in /units [SOLVED]

For collaboration on developing the mod capabilities of VS; request new features, report bugs, or suggest improvements

Moderator: Mod Contributor

Post Reply
chuck_starchaser
Elite
Elite
Posts: 8014
Joined: Fri Sep 05, 2003 4:03 am
Location: Montreal
Contact:

Multiple .csv files in /units [SOLVED]

Post by chuck_starchaser »

Problem:

units.csv is growing huge... Can't see the top row descriptions... Too many people working on it at the same time can cause commit conflicts...

Proposed solution:

Split it...
  • guns.csv
  • missiles.csv
  • mines.csv
  • fighters.csv
  • bombers.csv
  • corvettes.csv
  • destroyers.csv
  • frigates.csv
  • cruisers.csv
  • carriers.csv
Implementation:

If the code could just read all .csv files in the /units folder, then, which way
to split it could be up to the modders; --e.g.: by faction, or whatever.
Last edited by chuck_starchaser on Sun Mar 09, 2008 12:20 am, edited 1 time in total.
Erk
Trader
Trader
Posts: 26
Joined: Thu Dec 20, 2007 12:33 am
Location: Japan

Post by Erk »

Heartily agree. The proposed split looks really nice. This would make editing a lot easier to handle.
Moe479
Merchant
Merchant
Posts: 57
Joined: Wed Jan 24, 2007 7:13 am

Post by Moe479 »

one .csv per ship,weapon,etc. would be also no biggi for uptodate hardisks/comps
this would be much better than these monsterios long files for modding, for final realeases they could be thrown together to gain the last bit of of loadingspeed for the engine
loki1950
The Shepherd
Posts: 5841
Joined: Fri May 13, 2005 8:37 pm
Location: Ottawa
Contact:

Post by loki1950 »

There is a minor issue with separating all of the CSV files the overhead in the loading each separately it takes time to access each and it adds up very fast.

Enjoy the Choice :)
my box::HP Envy i5-6400 @2Q70GHzx4 8 Gb ram/1 Tb(Win10 64)/3 Tb Mint 19.2/GTX745 4Gb acer S243HL K222HQL
Q8200/Asus P5QDLX/8 Gb ram/WD 2Tb 2-500 G HD/GF GT640 2Gb Mint 17.3 64 bit Win 10 32 bit acer and Lenovo ideapad 320-15ARB Win 10/Mint 19.2
Moe479
Merchant
Merchant
Posts: 57
Joined: Wed Jan 24, 2007 7:13 am

Post by Moe479 »

as i wrote, for realese u should copy and paste them together into only a few files, but for testing an locating bugs its better to have the best possible overview to navigate quickly to the problem instead of 10+secs loading time for the editor and seeking trough thousends of lines.
chuck_starchaser
Elite
Elite
Posts: 8014
Joined: Fri Sep 05, 2003 4:03 am
Location: Montreal
Contact:

Post by chuck_starchaser »

I'm not sure the extra time of loading multiple csv files would be significant enough to be of any concern. The number of files loaded on boot is a pretty big number already. Another solution could be to keep them in one compressed file.
jackS
Minister of Information
Minister of Information
Posts: 1895
Joined: Fri Jan 31, 2003 9:40 pm
Location: The land of tenure (and diaper changes)

Post by jackS »

I think modifying the engine to deal with multiple csv files may be overkill if the primary benefit is that the file is too big for pleasant developing. As such a change is unlikely to be heavily prioritized in the near future, even if one disagrees that the following solution is long-term viable, it would certainly work as a short-term solution

What would seem a simpler solution to me would be, for any mod that wants to, do two things:
1) Split the units.csv file into whatever partitioning makes sense for that mod (whether it be by ship type or content creator or alphabetical... doesn't matter) and check these partitioned .csv files into the repository. Remove units.csv
2) Write a (fairly trivial in perl, at least) perl/shell/etc. script that will create a units.csv file out of a set of smaller .csv files. If necessary, create a binary version for windows users without decent scripting support. Run this to create a local version of units.csv for the engine to consume.

Then, you can happily develop on smaller fractions of the units.csv without having to change anything in the engine at all.
chuck_starchaser
Elite
Elite
Posts: 8014
Joined: Fri Sep 05, 2003 4:03 am
Location: Montreal
Contact:

Post by chuck_starchaser »

Yeah, that sounds reasonable.
I wasn't aware there was an easy way to merge csv files. I'll run this idea by the pu team, see if someone knows perl or whatever else will do it.
rigelan
Confed Special Operative
Confed Special Operative
Posts: 291
Joined: Sat Jan 28, 2006 2:58 am
Location: Des Moines, Iowa

Post by rigelan »

The concatenate function works just fine on csv files
AMD 4800 - NVIDIA 7600 - Slackware Linux 12.2
jackS
Minister of Information
Minister of Information
Posts: 1895
Joined: Fri Jan 31, 2003 9:40 pm
Location: The land of tenure (and diaper changes)

Post by jackS »

rigelan wrote:The concatenate function works just fine on csv files
If they all have headers, then those will have to be stripped first, but otherwise, yes, that's about all there is to it.
Link
Merchant
Merchant
Posts: 41
Joined: Wed Jul 06, 2005 2:23 pm
Location: In the pilot seat of my Areus.
Contact:

Post by Link »

Attached is a small program I've written to split units.csv into groups based on the type and role columns. The example below assumes you've saved it somewhere in your $PATH as splitunitscsv.py and that it's executable.

Simple example of usage:

Code: Select all

$ cd /path/to/data4.x/units
$ splitunitscsv.py -o /split/unit/file/dir units.csv
# You can now edit the split csv files in /split/unit/file/dir
# Once done, execute the following command to merge them again
$ splitunitscsv.py -j -i /split/unit/file/dir units.edited.csv
# You can then replace units.csv with units.edited.csv
For more details, use splitunitscsv.py --help.
You do not have the required permissions to view the files attached to this post.
chuck_starchaser
Elite
Elite
Posts: 8014
Joined: Fri Sep 05, 2003 4:03 am
Location: Montreal
Contact:

Post by chuck_starchaser »

Hahaha, Thanks Link; Damn! I just finished splitting the files manually.
http://deeplayer.com/pu/csv/units.7z
Doh!
:D
I'll pass your script to Nate, who was just about to program this.
Thanks!
;-)
chuck_starchaser
Elite
Elite
Posts: 8014
Joined: Fri Sep 05, 2003 4:03 am
Location: Montreal
Contact:

Post by chuck_starchaser »

Update for the curious. We've committed the files to a separate svn repo.
Nate (CoffeeBot) will write software and set up svn hooks so that any commit of
any csv file to the csv repository triggers the assembly of a new units.csv, followed by
a commit of that units.csv to the pu repo.
So the concatenation, etc will be server-side and transparent.
That's the plan, anyways.
Thanks all.
Shark
Confed Special Operative
Confed Special Operative
Posts: 360
Joined: Tue Mar 02, 2004 9:34 am
Contact:

Post by Shark »

chuck_starchaser wrote:I'm not sure the extra time of loading multiple csv files would be significant enough to be of any concern. The number of files loaded on boot is a pretty big number already. Another solution could be to keep them in one compressed file.
I agree. Also, this would facilitate keeping the models, textures and CSV files in one directory for each ship (e.g., "<root>/ships/plowshare/"). This way, if an editor wants to modify all aspects of a ship, all he/she needs to do is extract that single directory, and it will contain all the necessary files.

I don't think making adjustments for modders is always such a bad thing.
chuck_starchaser
Elite
Elite
Posts: 8014
Joined: Fri Sep 05, 2003 4:03 am
Location: Montreal
Contact:

Post by chuck_starchaser »

Well, the system is in place, working like a charm. We have a /csv repository with many little csv files for the many categories of things. When we edit one of those files and commit it, a script wolphin wrote concatenates all the files into a new units.csv, and commits the latter to the /priv_pu repository.
Works beautifully, and it has already yielded fruits: With all ships in one file, sorted alphabetically, bogus values that were there for armor immediately stood out like a sore thumb and were corrected, for example.
I'd highly recommend to other mods to do the same.

As for making it easier for people to share units.csv mods, this is something we've also discussed, and what we might do one of these days is a) set up an ftp on the server where people can pool modded ships, and b) include a small executable in the /units folder that can append csv files to units.csv. Better not to mix this ship sharing extravaganza paradigm with the developers' paradigm. Developers need better organization and compartmentalization, more than they need free flow.
Post Reply