Multiple .csv files in /units [SOLVED]

For collaboration on developing the mod capabilities of VS; request new features, report bugs, or suggest improvements

Moderator: Mod Contributor

Multiple .csv files in /units [SOLVED]

Postby chuck_starchaser » Thu Jan 03, 2008 11:28 pm

Problem:

units.csv is growing huge... Can't see the top row descriptions... Too many people working on it at the same time can cause commit conflicts...

Proposed solution:

Split it...
  • guns.csv
  • missiles.csv
  • mines.csv
  • fighters.csv
  • bombers.csv
  • corvettes.csv
  • destroyers.csv
  • frigates.csv
  • cruisers.csv
  • carriers.csv

Implementation:

If the code could just read all .csv files in the /units folder, then, which way
to split it could be up to the modders; --e.g.: by faction, or whatever.
Last edited by chuck_starchaser on Sat Mar 08, 2008 5:20 pm, edited 1 time in total.
User avatar
chuck_starchaser
Elite
Elite
 
Posts: 8014
Topics: 195
Joined: Thu Sep 04, 2003 9:03 pm
Location: Montreal

Share On:

Share on Facebook Facebook Share on Twitter Twitter Share on Digg Digg

Postby Erk » Tue Jan 08, 2008 5:56 pm

Heartily agree. The proposed split looks really nice. This would make editing a lot easier to handle.
Erk
Trader
Trader
 
Posts: 26
Topics: 9
Joined: Wed Dec 19, 2007 5:33 pm
Location: Japan

Postby Moe479 » Sun Jan 13, 2008 8:32 am

one .csv per ship,weapon,etc. would be also no biggi for uptodate hardisks/comps
this would be much better than these monsterios long files for modding, for final realeases they could be thrown together to gain the last bit of of loadingspeed for the engine
Moe479
Merchant
Merchant
 
Posts: 57
Topics: 7
Joined: Wed Jan 24, 2007 12:13 am

Postby loki1950 » Sun Jan 13, 2008 8:53 am

There is a minor issue with separating all of the CSV files the overhead in the loading each separately it takes time to access each and it adds up very fast.

Enjoy the Choice :)
my box::HP Envy i5-6400 @2Q70GHzx4 8 Gb ram/1 Tb(Win10 64)/3 Tb Mint 18/GTX745 4Gb acer S243HL K222HQL
Q8200/Asus P5QDLX/8 Gb ram/WD 2Tb 2-500 G HD/GF GT640 2Gb Mint 17.3 64 bit Win 10 32 bit acer and DELL E6400 4GB ram/100 GB HD Mint 17.3 6
User avatar
loki1950
The Shepherd
 
Posts: 5707
Topics: 51
Joined: Fri May 13, 2005 1:37 pm
Location: Ottawa

Postby Moe479 » Sun Jan 13, 2008 9:02 am

as i wrote, for realese u should copy and paste them together into only a few files, but for testing an locating bugs its better to have the best possible overview to navigate quickly to the problem instead of 10+secs loading time for the editor and seeking trough thousends of lines.
Moe479
Merchant
Merchant
 
Posts: 57
Topics: 7
Joined: Wed Jan 24, 2007 12:13 am

Postby chuck_starchaser » Sun Jan 13, 2008 10:41 am

I'm not sure the extra time of loading multiple csv files would be significant enough to be of any concern. The number of files loaded on boot is a pretty big number already. Another solution could be to keep them in one compressed file.
User avatar
chuck_starchaser
Elite
Elite
 
Posts: 8014
Topics: 195
Joined: Thu Sep 04, 2003 9:03 pm
Location: Montreal

Postby jackS » Sun Jan 13, 2008 1:53 pm

I think modifying the engine to deal with multiple csv files may be overkill if the primary benefit is that the file is too big for pleasant developing. As such a change is unlikely to be heavily prioritized in the near future, even if one disagrees that the following solution is long-term viable, it would certainly work as a short-term solution

What would seem a simpler solution to me would be, for any mod that wants to, do two things:
1) Split the units.csv file into whatever partitioning makes sense for that mod (whether it be by ship type or content creator or alphabetical... doesn't matter) and check these partitioned .csv files into the repository. Remove units.csv
2) Write a (fairly trivial in perl, at least) perl/shell/etc. script that will create a units.csv file out of a set of smaller .csv files. If necessary, create a binary version for windows users without decent scripting support. Run this to create a local version of units.csv for the engine to consume.

Then, you can happily develop on smaller fractions of the units.csv without having to change anything in the engine at all.
User avatar
jackS
Minister of Information
Minister of Information
 
Posts: 1895
Topics: 56
Joined: Fri Jan 31, 2003 2:40 pm
Location: The center of Centre county

Postby chuck_starchaser » Sun Jan 13, 2008 2:16 pm

Yeah, that sounds reasonable.
I wasn't aware there was an easy way to merge csv files. I'll run this idea by the pu team, see if someone knows perl or whatever else will do it.
User avatar
chuck_starchaser
Elite
Elite
 
Posts: 8014
Topics: 195
Joined: Thu Sep 04, 2003 9:03 pm
Location: Montreal

Postby rigelan » Sun Jan 13, 2008 7:18 pm

The concatenate function works just fine on csv files
AMD 4800 - NVIDIA 7600 - Slackware Linux 12.2
User avatar
rigelan
Confed Special Operative
Confed Special Operative
 
Posts: 291
Topics: 33
Joined: Fri Jan 27, 2006 7:58 pm
Location: Des Moines, Iowa

Postby jackS » Sun Jan 13, 2008 8:46 pm

rigelan wrote:The concatenate function works just fine on csv files

If they all have headers, then those will have to be stripped first, but otherwise, yes, that's about all there is to it.
User avatar
jackS
Minister of Information
Minister of Information
 
Posts: 1895
Topics: 56
Joined: Fri Jan 31, 2003 2:40 pm
Location: The center of Centre county

Postby Link » Mon Jan 14, 2008 1:33 pm

Attached is a small program I've written to split units.csv into groups based on the type and role columns. The example below assumes you've saved it somewhere in your $PATH as splitunitscsv.py and that it's executable.

Simple example of usage:
Code: Select all
$ cd /path/to/data4.x/units
$ splitunitscsv.py -o /split/unit/file/dir units.csv
# You can now edit the split csv files in /split/unit/file/dir
# Once done, execute the following command to merge them again
$ splitunitscsv.py -j -i /split/unit/file/dir units.edited.csv
# You can then replace units.csv with units.edited.csv


For more details, use splitunitscsv.py --help.
Attachments
splitunitscsv.py
Python program to split units.csv
(11.43 KiB) Downloaded 101 times
Link
Merchant
Merchant
 
Posts: 41
Topics: 7
Joined: Wed Jul 06, 2005 7:23 am
Location: In the pilot seat of my Areus.

Postby chuck_starchaser » Mon Jan 14, 2008 2:42 pm

Hahaha, Thanks Link; Damn! I just finished splitting the files manually.
http://deeplayer.com/pu/csv/units.7z
Doh!
:D
I'll pass your script to Nate, who was just about to program this.
Thanks!
;-)
User avatar
chuck_starchaser
Elite
Elite
 
Posts: 8014
Topics: 195
Joined: Thu Sep 04, 2003 9:03 pm
Location: Montreal

Postby chuck_starchaser » Mon Jan 14, 2008 7:26 pm

Update for the curious. We've committed the files to a separate svn repo.
Nate (CoffeeBot) will write software and set up svn hooks so that any commit of
any csv file to the csv repository triggers the assembly of a new units.csv, followed by
a commit of that units.csv to the pu repo.
So the concatenation, etc will be server-side and transparent.
That's the plan, anyways.
Thanks all.
User avatar
chuck_starchaser
Elite
Elite
 
Posts: 8014
Topics: 195
Joined: Thu Sep 04, 2003 9:03 pm
Location: Montreal

Postby Shark » Fri Jan 25, 2008 1:01 am

chuck_starchaser wrote:I'm not sure the extra time of loading multiple csv files would be significant enough to be of any concern. The number of files loaded on boot is a pretty big number already. Another solution could be to keep them in one compressed file.


I agree. Also, this would facilitate keeping the models, textures and CSV files in one directory for each ship (e.g., "<root>/ships/plowshare/"). This way, if an editor wants to modify all aspects of a ship, all he/she needs to do is extract that single directory, and it will contain all the necessary files.

I don't think making adjustments for modders is always such a bad thing.
User avatar
Shark
Confed Special Operative
Confed Special Operative
 
Posts: 360
Topics: 33
Joined: Tue Mar 02, 2004 2:34 am

Postby chuck_starchaser » Thu Feb 07, 2008 7:11 pm

Well, the system is in place, working like a charm. We have a /csv repository with many little csv files for the many categories of things. When we edit one of those files and commit it, a script wolphin wrote concatenates all the files into a new units.csv, and commits the latter to the /priv_pu repository.
Works beautifully, and it has already yielded fruits: With all ships in one file, sorted alphabetically, bogus values that were there for armor immediately stood out like a sore thumb and were corrected, for example.
I'd highly recommend to other mods to do the same.

As for making it easier for people to share units.csv mods, this is something we've also discussed, and what we might do one of these days is a) set up an ftp on the server where people can pool modded ships, and b) include a small executable in the /units folder that can append csv files to units.csv. Better not to mix this ship sharing extravaganza paradigm with the developers' paradigm. Developers need better organization and compartmentalization, more than they need free flow.
User avatar
chuck_starchaser
Elite
Elite
 
Posts: 8014
Topics: 195
Joined: Thu Sep 04, 2003 9:03 pm
Location: Montreal



Return to Modding Engine Room

Who is online

Users browsing this forum: No registered users and 1 guest

cron