Unified DB Resource

Development directions, tasks, and features being actively implemented or pursued by the development team.

Unified DB Resource

Postby pheonixstorm » Sun Feb 28, 2010 3:50 am

By DB I am refering to our friendly flat file database also know as the great ol csv files. As it stands we have one huge units.csv that holds far more data than it should. At some point we should split this file into more manageable chunks. ie ships, weapons, planets, bases... This (yet again) would make it easier on content developers to add new stuff to the current game or mods based off the VS engine. It doesn't look like it would create too much of a problem to change the surrent file function to call these extra files though I do not know if the units class has a particular order it has to follow or if it might be easier to split the main units class up into smaller chunks for certain objects such as splitting planets/bases and actual units apart or planets and units/bases or some other combination.

I figure splittig the file apart would be the easier of the two.. any thoughts?

*EDIT* While on the subject of databases... has anyone looked at having vegaserver use some type of REAL database such as SQL or a built in sql style database (nwn uses a built-in sql style database though it is slower than using something like mysql or postgre)
Because of YOU Arbiter, MY kids? can't get enough gas. OR NIPPLE! How does that mkae you feeeel? ~ Halo
User avatar
pheonixstorm
Elite
Elite
 
Posts: 1567
Topics: 113
Joined: Mon Jan 25, 2010 7:03 pm

Share On:

Share on Facebook Facebook Share on Twitter Twitter Share on Digg Digg

Re: Unified DB Resource

Postby chuck_starchaser » Sun Feb 28, 2010 4:08 am

pheonixstorm wrote:By DB I am refering to our friendly flat file database also know as the great ol csv files. As it stands we have one huge units.csv that holds far more data than it should. At some point we should split this file into more manageable chunks. ie ships, weapons, planets, bases... This (yet again) would make it easier on content developers to add new stuff to the current game or mods based off the VS engine. It doesn't look like it would create too much of a problem to change the surrent file function to call these extra files though I do not know if the units class has a particular order it has to follow or if it might be easier to split the main units class up into smaller chunks for certain objects such as splitting planets/bases and actual units apart or planets and units/bases or some other combination.

I figure splittig the file apart would be the easier of the two.. any thoughts?

Have you looked at the way we do units.csv in PU?
We have a separate repository for csv files, where units.csv is split into like 20 or 30 small, manageable csv files.
When we change something, we change it into one of these files and commit to the csv repo.
When SVN finishes that commit, a script takes all those small files and puts them into a single, big units.csv file
and commits it in turn to the priv_pu repository.

*EDIT* While on the subject of databases... has anyone looked at having vegaserver use some type of REAL database such as SQL or a built in sql style database (nwn uses a built-in sql style database though it is slower than using something like mysql or postgre)

Yes, someone did, but the problem is that we were all going to have to learn SQL to use it. A csv file is as simple to understand as a text file; a database is not. We already have too steep a learning curve for modders. There hasn't been a new vs mod in years, because modders need to learn Python, need to learn to use mesher, nvcompress, need to learn the intricacies of xml universe and system files, the intricacies of xmesh. Add SQL to that and 90% of those of us left will go away.
Now, if you're thinking about a nice gui program that helps you enter/edit the data in a more intuitive way, etceteras, that's a different story, and what's under the hood won't matter. (As long as it works PERFECTLY.)
User avatar
chuck_starchaser
Elite
Elite
 
Posts: 8014
Topics: 195
Joined: Thu Sep 04, 2003 9:03 pm
Location: Montreal

Re: Unified DB Resource

Postby charlieg » Sun Feb 28, 2010 6:01 am

I think the real solution here is to have Vega Strike support multiple csv files rather than an svn hack to combine them (something not convenient or easy to understand for newcomers) or yet another complication (SQL).
Free Gamer - free software games compendium and commentary!
FreeGameDev forum - open source game development community
User avatar
charlieg
Elite Mercenary
Elite Mercenary
 
Posts: 1328
Topics: 56
Joined: Thu Mar 27, 2003 4:51 pm
Location: Manchester, UK

Re: Unified DB Resource

Postby chuck_starchaser » Sun Feb 28, 2010 7:02 am

That was my original suggestion, in fact; but JackS didn't want to complicate the engine for modders' convenience.
He argued that off-line tools could achieve the same thing. So, we did so.
One could argue that for the engine it would be easier to read all systems from one file, or all python code from
one file; so this wouldn't be a fist ever that user convenience was put ahead of engine simplicity...
User avatar
chuck_starchaser
Elite
Elite
 
Posts: 8014
Topics: 195
Joined: Thu Sep 04, 2003 9:03 pm
Location: Montreal

Re: Unified DB Resource

Postby charlieg » Sun Feb 28, 2010 7:20 am

That's a bit of daftness though. It's a handful of lines of code to read input from multiple files instead of one. It's just a case of doing it elegantly (i.e. making it easy to understand for both modders and the engine) and it's a cop out to go 'this will complicate the engine'. How? I think jackS was being disingenuous and pushing the responsibility elsewhere because he didn't want to do it and didn't think any of the other active coders would get around to it.
Free Gamer - free software games compendium and commentary!
FreeGameDev forum - open source game development community
User avatar
charlieg
Elite Mercenary
Elite Mercenary
 
Posts: 1328
Topics: 56
Joined: Thu Mar 27, 2003 4:51 pm
Location: Manchester, UK

Re: Unified DB Resource

Postby klauss » Sun Feb 28, 2010 10:22 am

@charlieg: most likely so ;)

@chuck: want to add a ticket for that? it should be rather easy.
Oíd mortales, el grito sagrado...
Call me "Menes, lord of Cats"
Wing Commander Universe
User avatar
klauss
Elite
Elite
 
Posts: 7243
Topics: 55
Joined: Mon Apr 18, 2005 7:40 am
Location: LS87, Buenos Aires, República Argentina

Re: Unified DB Resource

Postby pheonixstorm » Sun Feb 28, 2010 11:40 am

chuck_starchaser wrote:Have you looked at the way we do units.csv in PU?
We have a separate repository for csv files, where units.csv is split into like 20 or 30 small, manageable csv files.
When we change something, we change it into one of these files and commit to the csv repo.
When SVN finishes that commit, a script takes all those small files and puts them into a single, big units.csv file
and commits it in turn to the priv_pu repository.
Didn't know this, but it shouldnt be done this way. See notes towards the end

chuck_starchaser wrote:Yes, someone did, but the problem is that we were all going to have to learn SQL to use it. A csv file is as simple to understand as a text file; a database is not. We already have too steep a learning curve for modders. There hasn't been a new vs mod in years, because modders need to learn Python, need to learn to use mesher, nvcompress, need to learn the intricacies of xml universe and system files, the intricacies of xmesh. Add SQL to that and 90% of those of us left will go away.
Now, if you're thinking about a nice gui program that helps you enter/edit the data in a more intuitive way, etceteras, that's a different story, and what's under the hood won't matter. (As long as it works PERFECTLY.)
SQL isn't hard, and there are a lot of tools to make this even easier that are already available anyway. reading a csv or tab seperate flat file can be far more difficult that reading or writing an sql file. Besides... its all about the tools which the original devs completely did NOT take into account to make this game modder friendly.

Wife is calling.. will have to review this when shes done with me... stupid birthday... :wink:
charlieg wrote:I think the real solution here is to have Vega Strike support multiple csv files rather than an svn hack to combine them (something not convenient or easy to understand for newcomers) or yet another complication (SQL).
With the exception of the bit on sql this was my entire point. As far as sql... has anyone looked at an sql file? Between the function code in C/C++ and the actually sql file its not difficult. Hell to make an sql file you dont even NEED to learn it, just copy n paste or use one of the free apps.

chuck_starchaser wrote:That was my original suggestion, in fact; but JackS didn't want to complicate the engine for modders' convenience.
He argued that off-line tools could achieve the same thing. So, we did so.
One could argue that for the engine it would be easier to read all systems from one file, or all python code from
one file; so this wouldn't be a fist ever that user convenience was put ahead of engine simplicity...
This has got to be the biggest crock of shit I have heard from a developer... complicate the engine? How does it complicate the engine if it reads multiple files into the same object?!?! :roll: Granted you can use off-line tools but did he or anyone else who designed VS CREATE THE TOOLS FOR MODDERS. Uh no.

Anyway, does the idea of leaving the files split and not using the joiner to merge them sound good? Also, for vegaserver does adding SQL sound good? And don't give me a fit over sql.. if you run a server for a game you generally have to learn sql to some degree. At minimum you have to learn to use commonly available tools at most.. the syntax of adding and removing stuff. May need to split this up.. thread for server side and this thread for client side.
Because of YOU Arbiter, MY kids? can't get enough gas. OR NIPPLE! How does that mkae you feeeel? ~ Halo
User avatar
pheonixstorm
Elite
Elite
 
Posts: 1567
Topics: 113
Joined: Mon Jan 25, 2010 7:03 pm

Re: Unified DB Resource

Postby klauss » Sun Feb 28, 2010 12:11 pm

The biggest point against a real database backend is that it is non-versionable.

sqlite has csv and plaintext backing stores (ie, it stores its data in csv and plaintext files), but that doesn't make it versionable, since when you start manipulating rows the changes are widespread and don't reflect the intention very well.

Ie, if I want to change the size of a unit, I go to units.csv and change the value in scale. Easy. If I do that with sqlite, sqlite may reorder the rows and a lot of other stuff that it does to keep the file manageable. SVN will show a lot of changes, and all for nothing.

Also, if we use the csv backend of sqlite, I don't see any real benefit from the ad-hoc csv parsing VS already has. SQLite isn't any more performant without indices or a binary-format store, and people won't go about editing the file with SQL statements. Plus, you can hopelessly break the csv store if you're not careful, because that sqlite stores its data in a csv doesn't mean it will be forgiving of editing mistakes. VS is really forgiving, we had to do that a long time ago to allow people to use excel. And still it borks from time to time.

Usually, games don't really need a relational database. Not unless their database becomes really big and you start needing indices and all the complex stuff you really don't want to reinvent. That's usually the case only (and not always) with MMOs, which VS isn't yet. VS loads the whole table into memory - there's no more SQL than "select * from units". Using an SQL backend, in that case, is simply overkill.
Oíd mortales, el grito sagrado...
Call me "Menes, lord of Cats"
Wing Commander Universe
User avatar
klauss
Elite
Elite
 
Posts: 7243
Topics: 55
Joined: Mon Apr 18, 2005 7:40 am
Location: LS87, Buenos Aires, República Argentina

Re: Unified DB Resource

Postby charlieg » Sun Feb 28, 2010 12:58 pm

Just to help nip this SQLness in the bud...

SQL is useful if the data set is 1) highly relational and 2) large; needs optimized querying. I don't think either #1 or #2 apply here. We're talking about a few hundred of any particular item, at most (usually less).

Suppose there could be a meaningful gain to using SQL (other than phoenixstorm knowing it), then it needs implementing - and integrating at all points where data is manipulated/read (since SQL is a storage mechanism and not just a serializsation tool). That's a lot of work. For a project that has traditionally suffered from developers being light on time, is it a worthwhile addition to the development burden? The reason for integrating something like SQLite had better be a damned good one, because it's just going to distract from one of the hundreds of more pressing matters.

What is that reason, phoenixstorm?
Free Gamer - free software games compendium and commentary!
FreeGameDev forum - open source game development community
User avatar
charlieg
Elite Mercenary
Elite Mercenary
 
Posts: 1328
Topics: 56
Joined: Thu Mar 27, 2003 4:51 pm
Location: Manchester, UK


Re: Unified DB Resource

Postby pheonixstorm » Sun Feb 28, 2010 10:56 pm

klauss wrote:The biggest point against a real database backend is that it is non-versionable.
I'm not sure if I understand you on this.. Are you refering to the actual sql files you would use to create the db and the tables or am I missing something?

klauss wrote:sqlite has csv and plaintext backing stores (ie, it stores its data in csv and plaintext files), but that doesn't make it versionable, since when you start manipulating rows the changes are widespread and don't reflect the intention very well.
This sounds close to your first quote, clarify please.

klauss wrote:Ie, if I want to change the size of a unit, I go to units.csv and change the value in scale. Easy. If I do that with sqlite, sqlite may reorder the rows and a lot of other stuff that it does to keep the file manageable. SVN will show a lot of changes, and all for nothing.
Reorder the rows? Guess I need to play with sqllite.. Then again, I wouldn't want to use something to manipulate flat files anyway. Not if i'm understanding you right...

klauss wrote:Also, if we use the csv backend of sqlite, I don't see any real benefit from the ad-hoc csv parsing VS already has. SQLite isn't any more performant without indices or a binary-format store, and people won't go about editing the file with SQL statements. Plus, you can hopelessly break the csv store if you're not careful, because that sqlite stores its data in a csv doesn't mean it will be forgiving of editing mistakes. VS is really forgiving, we had to do that a long time ago to allow people to use excel. And still it borks from time to time.
Excel is horrible for any type of flat file work.. it only works well for spreadsheets written and used with excel... It especially hated flat files in moo3 and warzone to some extent. This is why if you want people to mod your game you HAVE to have tools for them to work with. Best example I have see was the aurora toolset for neverwinter nights (don't play many fps games so i'm limited to sims, rpgs, and wargames). Side note, I would think you wouldn't want VS to be very forgiving. That can lead to bugs. Rather the user learn how to edit the file properly or use a dev created tool instead of hacking about with excel or even calc (openoffice spreadsheet is calc correct?)

klauss wrote:Usually, games don't really need a relational database. Not unless their database becomes really big and you start needing indices and all the complex stuff you really don't want to reinvent. That's usually the case only (and not always) with MMOs, which VS isn't yet. VS loads the whole table into memory - there's no more SQL than "select * from units". Using an SQL backend, in that case, is simply overkill.
Its not about needing complexity at all, its about data manipulation and data storage. I have worked on php projects that could have used a flat file or sql. SQL normally wins out though if you feel that you need a much larger storage or future growth that what a flat file would limit you to.

Back when I worked on modding warzone the developers sent us a cd that had a wealth of information. One such nugget was an access database that contained all the information for units, techs, weapons, etc. This access database was not part of the game as all the information was broken up into comma seperated .txt files. The db file though made our job so much easier to edit and change those files. It was a tool and it did its job well. Edit, export, done. For VS I don't think the client really needs sql but I do have another idea... I'm glad you brought up sqllite. You gave me an idea for making the client a little more persistent. Correct me if it already does this. Basics: Economy, its rather flat and doesnt flow well for protracted games. This IS supposed to be a space trading game based on Elite, Privateer, etc correct? Why not take it a step further and get some semblence of an actual economy up and running? Supply and demand on a more permanent basis (key the db to a particular save file). Delete it when the player deletes his save game. This may be something for chuck to ticket as a minor enhancement/feature. Or use this to backup the players save or even information that should have a better solution than in the savegame.


charlieg wrote:SQL is useful if the data set is 1) highly relational and 2) large; needs optimized querying. I don't think either #1 or #2 apply here. We're talking about a few hundred of any particular item, at most (usually less).
Not true on either account. I wrote a download manager for a website that used a SQL backend. It didnt have to keep track of a lot and wasnt higly relational or in need of a lot of optimizing. It DID need to track a growing list of downloaded files, version, locations (to keep leechers away) and an easy to use interface to add/edit/delete the list. I could have used flat files, but SQL was the better choice.

Let me make something clear: SQL does not mean complex monolithic databases. You can design a small database able to handle units.csv or something the size of the SQL database that handles world of warcraft (check out MaNGOS and Unified DB)

Now, on a smaller scale neverwinter nights actually comes to mind. It uses a small internal sql style database that can be interfaced using the auroa toolset. The toolset is used for AI scripting, Map creation, placing objects, sounds, (if you have a copy of nwn try installing aurora under wine. I don't remember if it works on not.) as well as the scripting for objects. You create a store and a shopkeeper. Say you want the shopkeeper to have a certain inventory and a certain amount he/she can spend. For this you use the database to store these values. This is NOT for the nwn server either, this is for created single player and multiplayer content modules.

charlieg wrote:Suppose there could be a meaningful gain to using SQL (other than phoenixstorm knowing it), then it needs implementing - and integrating at all points where data is manipulated/read (since SQL is a storage mechanism and not just a serializsation tool). That's a lot of work. For a project that has traditionally suffered from developers being light on time, is it a worthwhile addition to the development burden? The reason for integrating something like SQLite had better be a damned good one, because it's just going to distract from one of the hundreds of more pressing matters.
No... it wont. The focus should remain where it is and when everyone grows tired of it moe along to a side project. This or something else minor that has already been discussed and beaten and discussed some more. This is not a "drop what you're doing and work on this" addition. Getting a win32 build and fixing shader glitches are far more important. I feel this is more of a lets play around with it for now and see where we are after next release then decide on what the next HIGHEST priority.

chuck_starchaser wrote:Ticket #13 added.
I was going to comment on this but I can't access your site chuck. I think it was about the original idea of splitting the units csv back into its smaller parts. Somethig that WOULD be quick to do and would make modding easier.

At any rate, adding sql into the game goes kinda like this

VS client - doubtful w/o stronger reasons for it (more persistence(sp))
VS Server - needed for player data, accounts, etc

The client can for the most part do w/o unless you want to try and make the players interaction with the universe more dynamic. The server though.. it depends. If we go MMO then we need to go ahead and outline the path we need to take for the server code now. Thats one reason I wanted to split this thread into Unified DB Resource - Client and Unified DB Resource - Server so we can keep the ideas seperate for the most part. Server has far different needs than client. Networking code server side has to be extremely optimized to handle multiple connections. Thats were MaNGOS gets killed last i read (2 years ago i think) as they max out at about 1k connections iirc

Some of this is speculation while some of this is setting a roadmap for future POSSIBLE development.
Food for thought.
Because of YOU Arbiter, MY kids? can't get enough gas. OR NIPPLE! How does that mkae you feeeel? ~ Halo
User avatar
pheonixstorm
Elite
Elite
 
Posts: 1567
Topics: 113
Joined: Mon Jan 25, 2010 7:03 pm

Re: Unified DB Resource

Postby pheonixstorm » Sun Feb 28, 2010 11:16 pm

Fora short sql snippet
Code: Select all
CREATE TABLE `account` (
   `id` bigint(20) unsigned NOT NULL auto_increment
   `username`      varchar(32) NOT NULL default '',
   `sha_pass_hash` varchar(40) NOT NULL default '',
   `gmlevel` tinyint(3) unsigned NOT NULL default '0',
   `sessionkey` longtext,
   `email` text,
   `joindate` timestamp NOT NULL default CURRENT_TIMESTAMP,
   `last_ip` varchar(30) NOT NULL default '127.0.0.1',
   `failed_logins` int(11) unsigned NOT NULL default '0',
   `locked` tinyint(3) unsigned NOT NULL default '0',
   `last_login` timestamp NOT NULL default '0000-00-00 00:00:00',
   `online` tinyint(4) NOT NULL default '0',
   PRIMARY KEY  (`id`),
   UNIQUE KEY `idx_username` (`username`),
 ) ENGINE=MyISAM


SQL isn't hard. Most of the commands we need can be looked up on the sql site and it doesn't take a lot of time to build the tables. Doing the inserts is fairly straightforward as well.

Code: Select all
INSERT INTO account VALUES (blah, blah, blah)

or
Code: Select all
INSERT INTO account (what rows) VALUES (your values)


I had to cut n paste the first code section.. but it gives you the general idea. This is whats in the sql files used to make the database.. once its running its not as difficult to work with.

Mostly it all boils down to having the right tools for the job. Something i'm working on. If you create the proper tools for the job then it won't matter if the modder knows how to create a csv, read a csv or work with sql files.
Because of YOU Arbiter, MY kids? can't get enough gas. OR NIPPLE! How does that mkae you feeeel? ~ Halo
User avatar
pheonixstorm
Elite
Elite
 
Posts: 1567
Topics: 113
Joined: Mon Jan 25, 2010 7:03 pm

Re: Unified DB Resource

Postby chuck_starchaser » Sun Feb 28, 2010 11:49 pm

You can't access wcjunction because my room-mate is upgrading the distro on it as we speak (er.. type).

Anyways, what Klauss says about mysql files not being versionable is that you can't look at a meaningful
diff between two revisions on svn.

Then again, we could perhaps solve the problem by having sql write data as comma-delimited text files.

You're mixing things, though: When you speak about keeping track of the economy with mysql, you're
talking about game state, which is a different kettle of fish than mod data. And that's "server side"
even if we're talking single player; because single player should be coded in client-server fashion, even
if there's just one client and one server and both are local. It's just the correct way of coding it.
But what this means is that every time you save a game you're writing a new, probably huge sql file.
I have a plan to address the current hugeness of the savegame, by getting rid of so many strings and
variables, and saving small, efficient bit arrays. IOW, I'm looking for a compressed and fast system,
rather than something bigger and slower than what we got.

But anyhow, we don't have even the simplest multiplayer paradigm working, so speaking about mmo's
is pie in the sky. And what this whole thread sounds to me is like "I got a solution; let me show you,
you have a problem, and it fits my solution." Take the economy, for example: This topic has been
discussed probably more than any other in these forums. It is NOT EASY to simulate an economy.
Data representation is a huge and delicate thing. In any case, the first thing to do is to formalize
the problem, and work from the top down, and eventually find out what's needed at the bottom.
You, on the other hand, seem to think that you know well what's at the bottom... mysql, of course.
Well, excuse me while I laugh. You got a hammer, so everything is a nail.
User avatar
chuck_starchaser
Elite
Elite
 
Posts: 8014
Topics: 195
Joined: Thu Sep 04, 2003 9:03 pm
Location: Montreal

Re: Unified DB Resource

Postby charlieg » Mon Mar 01, 2010 7:54 am

phoenixstorm: Nobody is suggesting SQL is hard or unsuitable for managing data. (You are preaching to the choir in some regards.)

What I asked (and you didn't answer) was: what reason is there to implement SQL storage/management? You haven't yet given one.

Of course, I understood you weren't advocating do this as a highest priority; but why add to the task list something that has no obvious or significant benefit? You have failed to really give a benefit other than your own familiarisation and preference.
Free Gamer - free software games compendium and commentary!
FreeGameDev forum - open source game development community
User avatar
charlieg
Elite Mercenary
Elite Mercenary
 
Posts: 1328
Topics: 56
Joined: Thu Mar 27, 2003 4:51 pm
Location: Manchester, UK

Re: Unified DB Resource

Postby Mr. Fahrenheit » Mon Mar 01, 2010 6:54 pm

I don't know if this is a useful suggestion, I do not pretend to know anything about the codebase, How Things Are Done, or any of that. But I have been around flat files and whatnot a long time.

A fun concept for really large flat files which hold all sorts of information is a simple 'record type' indicator. Maybe you're already using it; maybe you'd want to consider it in some far off future time.

You simply state that the first field of every record will state what sort of record this line of the file is. So '1' means ship info follows, or '2' means planet info. It lets you have different numbers of fields for each record type, etc., thus eliminating some records having many blank fields. A simple trick of 'documentation' allows you to specify what each record type consists of, and what each field is used for. You can also fiddle around with having sub-records, continuation records, etc. Heady stuff from the '70s, I know.

The modern, standard, non-sql alternative might be to run it all the records through XML encoders / decoders. Thus spake the Wikipedia: "Although XML’s design focuses on documents, it is widely used for the representation of arbitrary data structures..."
Mr. Fahrenheit
Trader
Trader
 
Posts: 26
Topics: 6
Joined: Mon Jan 15, 2007 9:06 am

Re: Unified DB Resource

Postby chuck_starchaser » Mon Mar 01, 2010 7:26 pm

Fahrenheit, what you're describing is ancient history in vegastrike. Units.csv used to be XML. And everybody hated it,
with a passion; and finally one day the developers vowed to popular demand and put the unit data into a csv. Happiness.
There's no question that XML gives you more flexibility. What the real question is what's the main purpose of units.csv?

Is it to store data (for a machine to read)?
Is it to express data (for programmers to review)?
Is it to show data (for non-programmers to look at or edit)?

It's all of the above; but what's more important? The more important aspect of it is being an interface; for programmers
and non-programmers alike.

So, when we speak of space efficiency, as in your numbered headers; that's ONLY good for a machine reading it; so it's
a feature that helps machine representation (perhaps) at the expense of the more important purposes it serves.

When we speak of flexibility, that's rhubarb for programmers, but shit for artistic types.

And so, here comes the other great idea, storm's: His idea was basically, "let's split the interface and data storage aspects
by putting an app between them: mysql. Sounds groovey on the surface. But there's one more thing a flat file is good for,
and that's what Klauss mentioned many posts ago: With a CSV you can look at a diff and tell what changed between one
revision and another. That's what he calls "versionable". A diff is understandable when the data is in text, AND when it
changes the least amount it needs to change for a given change. If the data is binary, and the tool you use may decide
to rearrange it from time to time to suit its internal purposes, then it's totally "un-versionable".

And this is what programmer types often don't get. Units.csv is NOT primarily a data storage; nor is it even an interface;
it is a document.
User avatar
chuck_starchaser
Elite
Elite
 
Posts: 8014
Topics: 195
Joined: Thu Sep 04, 2003 9:03 pm
Location: Montreal

Re: Unified DB Resource

Postby klauss » Mon Mar 01, 2010 7:57 pm

The storage format would matter absolutely nothing, however, for artist types, if there was indeed a program between the engine, the programmer, and the artist. Notice the threefold interface.

A database is NOT such a thing. Those who used an access database probably don't realize the amount of programming behind those things - access is a whole IDE, with a basic, lousy, but to-the-point RDBMS (like sqlite I'd say), VBA scripting, forms, and wizards. Access (the application) serves thus a triple purpose as IDE (for developing those database applications), RDBMS (the database engine within it), and execution environment (where the VBA - that's Visual Basic - scripts run in conjunction with the RDBMS for users to do their thing).

The usual access database people use is the database + lots and lots of VBA scripting implementing a simple CRUD system (create, remove, update, display). That's when there isn't even more complex scripting going on.

Plain SQL (as in sqlite) is the least user friendly thing you'll find in your whole existence. XML was barney in comparison.

Programmers don't realize just how obscure SQL seems to the average joe.

But a real interface for artists would be wonderful, and would allow programmers to pick the backing store they wish and the one that better serves the machine's purpose, because the artists' interface would hide all the nitty gritty details.
Oíd mortales, el grito sagrado...
Call me "Menes, lord of Cats"
Wing Commander Universe
User avatar
klauss
Elite
Elite
 
Posts: 7243
Topics: 55
Joined: Mon Apr 18, 2005 7:40 am
Location: LS87, Buenos Aires, República Argentina

Re: Unified DB Resource

Postby chuck_starchaser » Tue Mar 02, 2010 8:22 am

But even then, we'd have to have a plain text or csv document that's read into my sql when the game boots;
unless we'd use mysql itself to implement version control of the data it manages; which would result in a dual system
for version control; --svn diffs for most things; mysql for databases; --in which case we'd probably want to put the
mysql file outside of svn version control, so as not to have an exploding repo size.

((OTOH, the wisdom of having data under version control is suspect, in the first place, admittedly. Textures being
a prime example of unversionable stuff versioned, which puts a big damper on the freedom of artists. I'd like to,
for instance, improve the textures for the Diplomatic Station; but I know the cost in repo size of doing incremental
changes, so I shy away from doing it; like I feel I should be sure the changes are final before I commit them. This
is NOT good...))
User avatar
chuck_starchaser
Elite
Elite
 
Posts: 8014
Topics: 195
Joined: Thu Sep 04, 2003 9:03 pm
Location: Montreal

Re: Unified DB Resource

Postby klauss » Tue Mar 02, 2010 8:38 am

Yep, versioning data is a problem.

But with textures, at least you have a reasonable history - the cost server-side may be high, but the benefit is there, you'll see, at revision x, they changed this texture.

I've thought about versioning a database, and the best idea I had (and that's why I mentioned a threefold interface) is that there could be a versionable representation of the data in SVN (say xml or whatever) that is handled by content authoring tools, and a script that transforms that into an appropriate format for the engine. That could be sqlite - although I'm still skeptical about the benefits of a relational database in the engine.

The most usual way commercial games achieve that is by at least zipping all assets for a release. Access from a zip archive is substantially faster: zips don't have clusters so data is tightly packed if you have tons of small files. Zips compress S3TC textures even further (noticeably, mind you, S3TC is highly compressible by algorithms like deflate), decreasing I/O during batch loads, and zips are easier on the filesystem and the installer. There are plenty libraries for managing zips, and unlike tar.gz, you don't have to unzip the whole thing, you can get a single file, fast.

Furthermore, and this only matters to commercial projects, zips usually get encrypted, to protect the assets.
Oíd mortales, el grito sagrado...
Call me "Menes, lord of Cats"
Wing Commander Universe
User avatar
klauss
Elite
Elite
 
Posts: 7243
Topics: 55
Joined: Mon Apr 18, 2005 7:40 am
Location: LS87, Buenos Aires, República Argentina

Re: Unified DB Resource

Postby chuck_starchaser » Tue Mar 02, 2010 8:58 am

[Off-topic tangent:]
I suggested we zip textures on a per-unit basis at least, long time ago.
I've heard bzip is the best.
Definitely, textures are taking up too much server bandwidth, in plain dds.
[/Off-topic tangent:]

But, getting back to the versioning off-topic tangent, I could do without versioning of textures.
I hate having to live with the worry of a repo size that increases linearly with time as textures
are modified, let alone added. What for? If a texture didn't look nicer I wouldn't commit it in
the first place.
The problem is that svn mixes up two concepts: vertical and horizontal controls. Horizontal
controls is why we want it for data (collision detection and resolution). Vertical controls are
good in terms of keeping a version number; but one should be able to tag assets for NOT
keeping backups to the beginning of time. One backup for textures would be plenty.
Then again, svn was designed for versioning code, not materials.
Someone should write "msvn".
User avatar
chuck_starchaser
Elite
Elite
 
Posts: 8014
Topics: 195
Joined: Thu Sep 04, 2003 9:03 pm
Location: Montreal

Re: Unified DB Resource

Postby safemode » Tue Mar 02, 2010 11:05 am

the argument for sql'ing the csv files depends on an application interface to abstract the sql from non-developers. Well, if you need an application to abstract the database then why do we need the database? Have the application abstract the csv better than the ones we have (pretty sure we have one) and be done with it. No need to change the csv files.

The problem with svn'ing data is that we can't set an arbitrary revision to no longer track. We have 10 years of version history and we dont need it. Nobody is likely to care about revision < a year or so ago. So why is it not possible to set aside history up to a given revision, and delete it (with the option to either archive and pull it or flat delete)

Or can we?

I'm pretty sure there is a method that would allow us to periodically (maybe every couple years) purge the data repo, using nothing but svn and retaining history up to any given point we choose.

svnadmin dump can take a revision range and svnload can take that range and create a new repo out of it. we can then completely remove the old repo and then copy back the new one (or renaming it if possible ...or perhaps we can set naming scheme).
Ed Sweetman endorses this message.
safemode
Developer
Developer
 
Posts: 2150
Topics: 84
Joined: Sun Apr 22, 2007 6:17 pm
Location: Pennsylvania

Re: Unified DB Resource

Postby chuck_starchaser » Tue Mar 02, 2010 11:31 am

I was browsing threads at the svn mailing list a while ago, and the standard way of purging history
is precisely that: make a raw copy of the tree, make it a new repo, tell people to switch, wait a
week or two, and delete the old, IIRC.
User avatar
chuck_starchaser
Elite
Elite
 
Posts: 8014
Topics: 195
Joined: Thu Sep 04, 2003 9:03 pm
Location: Montreal

Re: Unified DB Resource

Postby klauss » Tue Mar 02, 2010 12:01 pm

Yep, purging history is very painful.

And in SF a bit more painful even. You have to log in to their ssh servers (if they still have those), prepare a backup, prepare a script that will do the purge flawlessly (because they'll run it blindly), and then post a support request for them to run the script.

It's horrible.
Oíd mortales, el grito sagrado...
Call me "Menes, lord of Cats"
Wing Commander Universe
User avatar
klauss
Elite
Elite
 
Posts: 7243
Topics: 55
Joined: Mon Apr 18, 2005 7:40 am
Location: LS87, Buenos Aires, República Argentina

Re: Unified DB Resource

Postby safemode » Tue Mar 02, 2010 12:15 pm

difficult as it may be, it would only be something we would have to worry about doing once every other year or so. not a big deal. The script would only need to be written once and probably consist of 3 lines and we'd just have to set the correct revisions.

It would _massively_ reduce our footprint, and probably speed up svn responsiveness.
Ed Sweetman endorses this message.
safemode
Developer
Developer
 
Posts: 2150
Topics: 84
Joined: Sun Apr 22, 2007 6:17 pm
Location: Pennsylvania

Re: Unified DB Resource

Postby chuck_starchaser » Tue Mar 02, 2010 12:45 pm

Not sure if this would help... You could make a raw copy and upload it to wcjunction; tell people to switch to it, and once it
works you could simply delete the one in source forge, then create a new one. It would save you having to mess with
scripts and whatnot.

EDIT:
Never mind; I'm starting to get the problem: vegastrike and data are just folders in a common repo...
Who did that?, and why?
User avatar
chuck_starchaser
Elite
Elite
 
Posts: 8014
Topics: 195
Joined: Thu Sep 04, 2003 9:03 pm
Location: Montreal


Next

Return to Engine Development

Who is online

Users browsing this forum: No registered users and 1 guest

cron