Game data storage

hsoj · Post by **hsoj** » Mon Nov 26, 2007 5:33 pm

I was taking a look at the new methods that vegastrike uses in order to store its game data, and was wondering if other options where looked at as appose to csv files. From my experience, flat files like this take a lot of resources when manipulating them. Have options like sqlite, and cdb been explored to store this data?

Post by **loki1950** » Mon Nov 26, 2007 10:08 pm

Welcome hsoj you missed lots of XML as well

the problem with the db backends you mention is query time tie will it fit into the the physics frame for a playable frame rate.and are they properly supported in the three platforms we run on.The cvs files are not new really they have been part of the engine since the start.they are a pain to edit though but a dedicated editor has been developed which make it a doodle

Enjoy the Choice

hsoj · Post by **hsoj** » Mon Nov 26, 2007 11:44 pm

It seems that using this format of files would actually take more time to search for the data that is contained. Of course, this depends on the amount of times these resources are used and how they're stored. For example, you may not need to load the data that contains a ship's price when you view one in the universe; but in the logic, you will need to know important information on the ship (e.g, armor, speed, etc.).

I have benchmarked using data stored in this manner, against using something with a DB back end (i.e, sqlite). My findings are that using flat files like csv or xml, actually take longer time to manipulate. I previously used the CDB example, as a pointer to projects that are capable of running in the same manner of a flatfile, but add to lowering the cost of accessing them.

Also, it may turn out that it really does not provide enough of an improved performance to implement it into the client. But, it may prove beneficial to the server daemon if it is designed to handle a multitude of NPC/players.

Post by **ace123** » Tue Nov 27, 2007 9:39 am

Actually CSV support is a relatively new addition compared to what we had before.

It used to be done with XML files everywhere, which took forever to load (XML is really slow to read). CSV is much faster since it's all array indices and associative maps compared to a complex tree structure and parser in XML.

Post by **jackS** » Tue Nov 27, 2007 6:10 pm

I'd be tempted to believe the file format isn't actually of key importance to performance, given that we read the entire file and transform it into in-memory structures. Since all of the entries are read only, this task is further simplified.

Post by **ace123** » Wed Nov 28, 2007 4:33 am

True... the main problem I remember were the XMeshes before we converted to binary formats...

GoldenGnu · Post by **GoldenGnu** » Thu Nov 29, 2007 2:05 pm

I like the CSV files.. It makes editing/reading the files by hand (notepad, openoffice etc.) and help programs (Like VSTrade) a lot easier.... If the data was kept in a database, editing by hand wouldn't be an options, It would require a dedicated editor...

That's just my 2 cent...

hsoj · Post by **hsoj** » Thu Nov 29, 2007 5:01 pm

I agree that the CSV format is great for the ease of manipulation, but that is only to the human eye. It seems that the way that this data is maintained from within the client could be changed to lower overhead (Possibly?). Mind you, I am strictly speaking from a standpoint of programming in the UNIX environment.

I really do not see this subject being something that would arise any type of importance at this state, but I do feel that it might be something that would want to be visited down the road when more data is being implemented into the game.