Unified DB Resource

Development directions, tasks, and features being actively implemented or pursued by the development team.
klauss
Elite
Elite
Posts: 7243
Joined: Mon Apr 18, 2005 2:40 pm
Location: LS87, Buenos Aires, República Argentina

Re: Unified DB Resource

Post by klauss »

@chuck: besides, that would remove all history. AFAIK, the idea is to only remove OLD history. That can be done with svndump and such, but SF brings a whole layer of administrative overhead to that, because you can't access their repositories like that at whim.

However, it is true, it only needs to be done once every year at most. The first time is the hardest.
Oíd mortales, el grito sagrado...
Call me "Menes, lord of Cats"
Wing Commander Universe
pheonixstorm
Elite
Elite
Posts: 1567
Joined: Tue Jan 26, 2010 2:03 am

Re: Unified DB Resource

Post by pheonixstorm »

chuck_starchaser wrote: And so, here comes the other great idea, storm's: His idea was basically, "let's split the interface and data storage aspects
by putting an app between them: mysql. Sounds groovey on the surface. But there's one more thing a flat file is good for,
and that's what Klauss mentioned many posts ago: With a CSV you can look at a diff and tell what changed between one
revision and another. That's what he calls "versionable". A diff is understandable when the data is in text, AND when it
changes the least amount it needs to change for a given change. If the data is binary, and the tool you use may decide
to rearrange it from time to time to suit its internal purposes, then it's totally "un-versionable".

And this is what programmer types often don't get. Units.csv is NOT primarily a data storage; nor is it even an interface;
it is a document.
The problem here is Klauss is in part WRONG. The database itself is binary but the files needed to populate or create that database are as text based as the csv and ARE versionable. You wouldn't want to keep a populated DB in svn just the files to create it. DB creation would be during install.

No matter how you go about doing data storage there is one thing that will eventually need it: VegaServer. The concept around VServer looks and acts like a MMOs retarded cousin or maybe a matchmaking service.. but based on the typed of game this is either way VServer needs a rewrite. If you have to log in to play your one of the two mentioned before.

Vegastrike should have two modes. Peer-peer for lan play and client/server for MMO style play. peer-peer should be client only though you could build it around a mini-server as was mentioned in another post.
Because of YOU Arbiter, MY kids? can't get enough gas. OR NIPPLE! How does that mkae you feeeel? ~ Halo
chuck_starchaser
Elite
Elite
Posts: 8014
Joined: Fri Sep 05, 2003 4:03 am
Location: Montreal
Contact:

Re: Unified DB Resource

Post by chuck_starchaser »

pheonixstorm wrote:
chuck_starchaser wrote:And so, here comes the other great idea, storm's: His idea was basically, "let's split the interface and data storage aspects
by putting an app between them: mysql. Sounds groovey on the surface. But there's one more thing a flat file is good for,
and that's what Klauss mentioned many posts ago: With a CSV you can look at a diff and tell what changed between one
revision and another. That's what he calls "versionable". A diff is understandable when the data is in text, AND when it
changes the least amount it needs to change for a given change. If the data is binary, and the tool you use may decide
to rearrange it from time to time to suit its internal purposes, then it's totally "un-versionable".
And this is what programmer types often don't get. Units.csv is NOT primarily a data storage; nor is it even an interface;
it is a document.
The problem here is Klauss is in part WRONG. The database itself is binary but the files needed to populate or create that database are as text based as the csv and ARE versionable. You wouldn't want to keep a populated DB in svn just the files to create it. DB creation would be during install.

No matter how you go about doing data storage there is one thing that will eventually need it: VegaServer.
Right. So the question about MySQL is whether or not it's a better choice for data representation in terms of speed, efficiency, robustness, etceteras; NOT whether it's a better human interface for data, and therefore not whether it offers new flexibilities.

On the technical merits I agree and I was pitching MySQL a long time ago; reason being that I foresaw a time when the universe might become persistent AND changeable, and the only way servers could deal safely with asynchronously changing universes would be to use industry strength databases with built-in journaling safeties; and if the server used databases, the clients might as well use databases too.

The reason I stopped pitching it was political: The other guy that started pitching MySQL before me, he wanted to make MySQL the interface; and he wanted to move EVERYTHING to MySQL, including real time data that needed to be fast; so I didn't want to appear to be supporting his pitch.
The concept around VServer looks and acts like a MMOs retarded cousin or maybe a matchmaking service.. but based on the typed of game this is either way VServer needs a rewrite. If you have to log in to play your one of the two mentioned before.
I agree that vegaserver needs a rewrite... If for no other reason, the mere fact that we know nothing about it and therefore can't work much with it. My fear is that sooner or later, Hellcat and Ace will suddenly come back and start replacing vegaserver with Sirikata, and I don't know what they are using vis a vis data representations for the persistent universe in Sirikata, but I suspect it's not MySQL, since I'm subscribed to their mailing list and have never seen MySQL mentioned, and so I fear they may have reinvented databases but coded them in Javascript, or some such horror. So I agree that there's a role for MySQL; just not that it's the solution for *everything*. Specially not real-time stuff like dynamic economics simulation; and specially not in any role where its presence would become apparent to data-side modders. That is, switching from containers and serialization hacks to a database should be a transparent transition that modders would never realize it happened --unless informed so.
Vegastrike should have two modes. Peer-peer for lan play and client/server for MMO style play. peer-peer should be client only though you could build it around a mini-server as was mentioned in another post.
I don't know; this is a multiplayer subject.
klauss
Elite
Elite
Posts: 7243
Joined: Mon Apr 18, 2005 2:40 pm
Location: LS87, Buenos Aires, República Argentina

Re: Unified DB Resource

Post by klauss »

@phoenixstorm & @chuck: I've had to maintain data scripts in SQL at work. It's worse, worse much worse than csv, xml or anything else.

Versioning an SQL script isn't fun.

And it's a mere copy of your database, so when you're developing you have to modify the real database you're using. Meaning whatever change you make to your working copy doesn't get applied to the versioned copy of the database, you have to move the changes manually. You'll forget. You'll make mistakes.

No... it's not an option. Take it from someone that works every day with that.

EDIT: if we're ever forced, technologically, to adopt a relational databse, well, we'll cope with the bad things then. Right now, VS doesn't need a relational database - all data in VS is rather flat. I'd be more tempted to try NoSQL solutions, but then again comes the problem of versioning the data stored in those solutions.

EDIT2: I remember mkruer's plan to move everything into a database. But remember he was already planning on exploiting that, he would have the database in a public server, work it as a means of having instantaneous client updates, the universe gained a lot of dynamicity because the sql structure had built-in timelines (so you could fetch the current tarsus model or the 5 year old model, and they would be different), etc... so he wanted to exploit the relational nature. For him, sql was about adding functionality to the data store, not merely changing backend. It would also involve a lot of work, and it turned out that noone wanted to do that work.
Oíd mortales, el grito sagrado...
Call me "Menes, lord of Cats"
Wing Commander Universe
chuck_starchaser
Elite
Elite
Posts: 8014
Joined: Fri Sep 05, 2003 4:03 am
Location: Montreal
Contact:

Re: Unified DB Resource

Post by chuck_starchaser »

Yeah, I certainly wasn't suggesting versioning databases, and I think storm has seen the light too. I think what he was talking about now was keeping units.csv as is, under version control, and have the engine make a database from it when the game boots. I.e.: Totally a back-end thing.
As far as a persistent multiplayer universe, what I had in mind was a server using sql but NO version control on that whatsoever.
EDIT:
My perception of the biggest advantage of using sql in a server-based, persistent universe, is the robustness aspect: The fact that database code has all the bells and whistles to keep the data consistent, atomic operations and all that.
pheonixstorm
Elite
Elite
Posts: 1567
Joined: Tue Jan 26, 2010 2:03 am

Re: Unified DB Resource

Post by pheonixstorm »

@Klauss
If you make changes to your working copy how was it done? Generally if you start changing anything it should be done by creating new update scripts. The only time I have ever had to update the database directly is when the db is used as the app backend to export new flat files (warzone2100, moo3) or work done with mangos using a php db tool in which case you can also export your database anyway.

Keeping a versioned db isn't difficult if you use the right tools and do it the right way. Its like any other aspect of development. You wouldn't want a basic programmer trying to change a pascal program or a perl scripter hacking your python. To me the csv is messy, xml is messy.. but thats mostly because (in the case of xml) I don't know it or use it. I like C/C++, SQL, PHP, and ASM (though I really don't understand a lot of asm yet but I can follow it).
EDIT2: I remember mkruer's plan to move everything into a database. But remember he was already planning on exploiting that, he would have the database in a public server, work it as a means of having instantaneous client updates, the universe gained a lot of dynamicity because the sql structure had built-in timelines (so you could fetch the current tarsus model or the 5 year old model, and they would be different), etc... so he wanted to exploit the relational nature. For him, sql was about adding functionality to the data store, not merely changing backend. It would also involve a lot of work, and it turned out that noone wanted to do that work.
Is there a post on this you can link? The idea sounds interesting though the amount of work involved may be too much.

@chuck

My initial idea was mostly for vserver anyway with a few uses for vstrike. units.csv serves the client fine atm but later we may want to think of other ways to handle the data storage for a more robust econ. Mainly its getting units.csv cleaned up and ungunked.
So I agree that there's a role for MySQL; just not that it's the solution for *everything*. Specially not real-time stuff like dynamic economics simulation;
This is where I don't agree. While doing *everything* in SQL is rather dumb keeping a dynamic persistent econ fits a database very well. Speed is key so MySQL or Postgree fits the bill. A dynamic econ would not however fit with a flat file. Speed and size of the flat file db is horrid for the amount of data used in a symulated econ. Again, this is server side we are talking about and not client. Flat files client side would not be an issue as there would never be enough data to need access to more than one planets data at a time.

This is the biggest reason why I said to keep your thoughts on the client and server seperate. Server is a must (eventually) for SQl while client side can happily chug along with its normal mode of operation w/o any serious pitfalls.

@All

As I stated somewhere above I think eventually we should decide on how miltiplayer should really be handled. Do we go with Vegaserver and Vegastrike as a solo/mmo or do we handle Vegastrike as a peer server for a limited number of players (possibly no campaign just comp generated missions or create scenarios)

These last multiplayer ideas (this one and the ones above) should be in a new thread so please wait for that thread to post thoughts or opinions. Last idea, split vegaserver off the main line for mmo only. Vegaserver has its own VS client (90-99% shared code with peer) while for those who just want dogfights with cap ships can use the normal VS that run peer-peer IP connections. I don't know how much code would need to be changed if any to get this working but it makes for one hell of a game idea. Family calls so i'll post more in a few.
Because of YOU Arbiter, MY kids? can't get enough gas. OR NIPPLE! How does that mkae you feeeel? ~ Halo
klauss
Elite
Elite
Posts: 7243
Joined: Mon Apr 18, 2005 2:40 pm
Location: LS87, Buenos Aires, República Argentina

Re: Unified DB Resource

Post by klauss »

pheonixstorm wrote:@Klauss
If you make changes to your working copy how was it done? Generally if you start changing anything it should be done by creating new update scripts.
We usually commit changes that require schema and/or "support data" changes with an SQL script attached, either in the commit comment, or as a new sql script inside a dedicated folder for that.
When we deploy we know we have to run all those scripts in succession.
Maintaining them, though, is double work: since whenever you're working on your working copy, you HAVE to run those scripts in your development database. Usually there's trial and error, so the script to be committed is generated a posteriori, when you know what changes need committing.
Well, actually, I write the script, run it initially, and if I see changes I alter the database and the script at the same time. But that's discipline, and not everybody has that.
pheonixstorm wrote:The only time I have ever had to update the database directly is when the db is used as the app backend to export new flat files (warzone2100, moo3)
I don't get it. If you're developing, you have to change your database to test your work. You can't commit a new sql script without testing it, which means running it. Testing. Rolling back if there's a problem (and that's not a transactional rollback, it's manual, since you had to commit).
There's another downside to database: goodby svn revert.
pheonixstorm wrote:Keeping a versioned db isn't difficult if you use the right tools and do it the right way.
It's not the case in VS... but... does that "isn't difficult" apply to a 50GB database? Do you "just export" it? What about a 1TB databse?
What if 99.999% of that 1TB is raw data you seldom manipulate, and the rest is the one that needs versioning?
Don't minimize the problem... it's small only if you never worked with real databases under real use.

Imagine a vegaserver. Running. Getting online content (ie: player history, a dynamic universe, the players modifying the database with their every action). How do you version that database? Because... you know... it's a schizoid database. You have "versionable" content, that is the content the engines needs and players don't change - like ship/cargo/upgrade catalogues and whatnot. And there's dynamic content, the content you have to maintain across upgrades, the content that has to be versioned only in schema but not its data.

Is that a simple problem for you?
pheonixstorm wrote:To me the csv is messy, xml is messy.. but thats mostly because (in the case of xml) I don't know it or use it. I like C/C++, SQL, PHP, and ASM (though I really don't understand a lot of asm yet but I can follow it).
That sounds to me like "I have MySql so everything's a database to me".
(analogue to "I have a hammer so everything's a nail to me")

So let me challenge you:

pheonixstorm wrote:Speed is key so MySQL or Postgree fits the bill.
What bill?
pheonixstorm wrote:A dynamic econ would not however fit with a flat file.
Why?
pheonixstorm wrote:Speed and size of the flat file db is horrid for the amount of data used in a symulated econ.
What's that amount?
pheonixstorm wrote:Server is a must (eventually) for SQl
Why?
pheonixstorm wrote:As I stated somewhere above I think eventually we should decide on how miltiplayer should really be handled. Do we go with Vegaserver and Vegastrike as a solo/mmo or do we handle Vegastrike as a peer server for a limited number of players (possibly no campaign just comp generated missions or create scenarios)
There's an important question there:
Will vegastrike multiplayer be a cooperative peer-to-peer system (as in, we trust in the client's good intentions and we don't actively prevent cheating?)
Or will vegastrike go the mmo route and implement a client-server system where the server strictly enforces anti-cheating rules?

It's a serious design decision with serious different routes and objectives.

For one, peer-to-peer systems fare much better in very big worlds, and are decentralized so easier to set up with comunity-driven nodes. But there's cheating... you can't prevent it, and people will and do cheat.

For two, client-server systems are expensive to maintain (someone has to pay for the server, and it has to be a powerful server, or if things get big enough, even a server farm). If VS devs can't afford the expense, we might end up with 0 (zero - none) servers, and thus NO multiplayer whatsoever.

Client-server systems also need a lot of effort to get anti-cheating right.
pheonixstorm wrote:These last multiplayer ideas (this one and the ones above) should be in a new thread so please wait for that thread to post thoughts or opinions. Last idea, split vegaserver off the main line for mmo only. Vegaserver has its own VS client (90-99% shared code with peer) while for those who just want dogfights with cap ships can use the normal VS that run peer-peer IP connections. I don't know how much code would need to be changed if any to get this working but it makes for one hell of a game idea. Family calls so i'll post more in a few.
If you're proposing both at the same time (peer-to-peer and server-client), you're dreaming pipes.
The team is struggling to get singleplayer ahead, getting that, plus peer-to-peer multiplayer, plus server-client, and maintaining it all, well...
pheonixstorm wrote:
EDIT2: I remember mkruer's plan to move everything into a database. But remember he was already planning on exploiting that, he would have the database in a public server, work it as a means of having instantaneous client updates, the universe gained a lot of dynamicity because the sql structure had built-in timelines (so you could fetch the current tarsus model or the 5 year old model, and they would be different), etc... so he wanted to exploit the relational nature. For him, sql was about adding functionality to the data store, not merely changing backend. It would also involve a lot of work, and it turned out that noone wanted to do that work.
Is there a post on this you can link? The idea sounds interesting though the amount of work involved may be too much.
No link at hand - you can search the forums though.
Oíd mortales, el grito sagrado...
Call me "Menes, lord of Cats"
Wing Commander Universe
pheonixstorm
Elite
Elite
Posts: 1567
Joined: Tue Jan 26, 2010 2:03 am

Re: Unified DB Resource

Post by pheonixstorm »

klauss wrote: We usually commit changes that require schema and/or "support data" changes with an SQL script attached, either in the commit comment, or as a new sql script inside a dedicated folder for that.
When we deploy we know we have to run all those scripts in succession.
Maintaining them, though, is double work: since whenever you're working on your working copy, you HAVE to run those scripts in your development database. Usually there's trial and error, so the script to be committed is generated a posteriori, when you know what changes need committing.
Well, actually, I write the script, run it initially, and if I see changes I alter the database and the script at the same time. But that's discipline, and not everybody has that.
I'm not sure I follow. What kind of db app are you refering to anyway. There shouldn't be a lot of updates from this end. For the most part unit.csv changes how often?? The only thing we would really need to worry about is how the tables work then populate those tables with what the system needs as a base to start. Its not like we are trying to populate a db with 1000 different ships with thousands of different ship systems or weapons. I will discuss this more below answering the mmo cost issue.
I don't get it. If you're developing, you have to change your database to test your work. You can't commit a new sql script without testing it, which means running it. Testing. Rolling back if there's a problem (and that's not a transactional rollback, it's manual, since you had to commit).
There's another downside to database: goodby svn revert.
Part of this can be seen above. To test it though and reverting changes... I've only had to work with game size databases and most of those were for NWN or Mangos. Either one never got larger than about 100-150mb everything included. We have to track two things. Creating the database and populating. Database creation will change the least and will be the easiest to version. The data we populate the db with will be tricky when doing the versioning as sql updates in the script are a bit more difficult to follow the more you change it. Not impossible though, and why would we hand of something like this to someone who can handle the discipline. Its like fixing all the bugs and polishing VS to a commercial level project and then handing it back over to jackS to add new featres using a .10 rev mindset

It's not the case in VS... but... does that "isn't difficult" apply to a 50GB database? Do you "just export" it? What about a 1TB databse?
What if 99.999% of that 1TB is raw data you seldom manipulate, and the rest is the one that needs versioning?
Don't minimize the problem... it's small only if you never worked with real databases under real use.
I'm not minimizing the problem just the scale I think you are trying to put this in. Different databases require different means. A db the size of one used by VS i would version and delete/recreate all day long to find/fix/polish the scripts. A database like what you are talking about.. well thats just reaching. For one a database as large as either of those is going to be a fully populated high use development that doesnt allow for mistakes. So, the tools used would be a nice cluster with at least one backup system and a seperate developmental server/db to test w/e changes are being made in a stable eviornment. Not to mention that the 1% you have to manipulate will probably NOT share the same table as the other 99%. So, if all else fails, export only the tables you need to version. How often does a database actually change when its first put in use anyway. How much raw data is normally installed when you create a fresh database. Again, see below on cost.
Imagine a vegaserver. Running. Getting online content (ie: player history, a dynamic universe, the players modifying the database with their every action). How do you version that database? Because... you know... it's a schizoid database. You have "versionable" content, that is the content the engines needs and players don't change - like ship/cargo/upgrade catalogues and whatnot. And there's dynamic content, the content you have to maintain across upgrades, the content that has to be versioned only in schema but not its data.
Is that a simple problem for you?
Once the database is online YOU DONT VERSION IT. You only version the the changes made to the db design not the content. You have a clean install that has all starter content ie ships/upgrades and all the tables: history, accounts, ships, planets, etc. The content may never change but the design might. You modify the account table to add in for bans or key matching (used in nwn mini-mmo server to disallow multi-accounting) and take out ip record keeping. This you version, this changes. Content doesnt matter after you use the database unless you completely rewrite the database.
That sounds to me like "I have MySql so everything's a database to me".
(analogue to "I have a hammer so everything's a nail to me")
I could also say for you its "I hate MySQL and refuse to use it" It might be true or could just be that someone you work with doesnt know how to properly care and feed a SQL database and its driving you batty. Who knows... You do great work so I can't bash you too much :P
Klauss wrote:
pheonixstorm wrote:As I stated somewhere above I think eventually we should decide on how miltiplayer should really be handled. Do we go with Vegaserver and Vegastrike as a solo/mmo or do we handle Vegastrike as a peer server for a limited number of players (possibly no campaign just comp generated missions or create scenarios)
There's an important question there:
Will vegastrike multiplayer be a cooperative peer-to-peer system (as in, we trust in the client's good intentions and we don't actively prevent cheating?)
Or will vegastrike go the mmo route and implement a client-server system where the server strictly enforces anti-cheating rules?

It's a serious design decision with serious different routes and objectives.

For one, peer-to-peer systems fare much better in very big worlds, and are decentralized so easier to set up with comunity-driven nodes. But there's cheating... you can't prevent it, and people will and do cheat.

For two, client-server systems are expensive to maintain (someone has to pay for the server, and it has to be a powerful server, or if things get big enough, even a server farm). If VS devs can't afford the expense, we might end up with 0 (zero - none) servers, and thus NO multiplayer whatsoever.

Client-server systems also need a lot of effort to get anti-cheating right.
It doesn't matter if you go peer or server, you can make anti-cheating in either.

Yes it is, but its only as difficult as you want it to be. Thats why discussion are good. We get ideas (good or bad) and use them later or not at all. Plus you might be able to get a new feature idea in the process (hey it happens!)

Peer-peer is more lan play like a nice quake fest. It starts, it ends.. there isn't much to it in terms or real adventure. Not for a trading game like this.

Server expensive? Only if you make it that way. Take a page from the wow clone community or the nwn community. Both have dedicated servers (nwn small scale wow huge player base). You don't have to run a server farm to have a few thousand players. We are desinging Eve here are we? And if we are we better charge for it too. Just because VS is under GPL doesnt mean we can't make money off of it. Yes we could create a single Eve style server or we can create a server for the masses to setup and play (wow clones). I can setup a nwn or wow clone server here at home and run it off my dsl and still have several hundred players or more. The limit is going to be in the code as well as the size of the data sent between the client and server. Nwn data packets for server games is very small. I don't know about wow data packets so can't comment there.

People cheat no matter how difficult you make it. In either peer or server adding code to nullify cheating is not that difficult. Server, you run data entirely server side.. peer, you check hash or crc to make sure everyone is using the same files. Thats just basic for starters. Memory hacks are another problem.. but you can always add in anti-debugging code for some of those. At the more extreme (for a free game) you could require anyone who wants to play mp games register the game to get a unique key or some such and if they are caught cheating ban that key.
Because of YOU Arbiter, MY kids? can't get enough gas. OR NIPPLE! How does that mkae you feeeel? ~ Halo
Neskiairti
Confed Special Operative
Confed Special Operative
Posts: 334
Joined: Thu Jan 11, 2007 4:10 am

Re: Unified DB Resource

Post by Neskiairti »

yes, they are designing 'eve' equivalent.

Every tool has its purpose, units.csv is hardly the place to use mySQL. If something needs a relational database in the future, that is another story, and no one here would force vegastrike to use the wrong tool for the job :p
klauss
Elite
Elite
Posts: 7243
Joined: Mon Apr 18, 2005 2:40 pm
Location: LS87, Buenos Aires, República Argentina

Re: Unified DB Resource

Post by klauss »

I don't know how WOW servers work, but I imagine they're not as CPU intensive to the server as VS, which has to handle the dynamic universe, python scripting, physics, collisions, and all that.

In fact, all that is consuming a healthy amount of CPU time in single player already. So a server for 100+ players would most likely need many cores, and an engine capable of using them.

I think you're underestimating the CPU needs of VS. Or overestimating its efficiency.

About the database... I'll have to reply later, they're demanding the computer back here at home ;)
Oíd mortales, el grito sagrado...
Call me "Menes, lord of Cats"
Wing Commander Universe
pheonixstorm
Elite
Elite
Posts: 1567
Joined: Tue Jan 26, 2010 2:03 am

Re: Unified DB Resource

Post by pheonixstorm »

klauss wrote:About the database... I'll have to reply later, they're demanding the computer back here at home ;)
I know how that goes... makes me glab I have 3 up and running around here.

As far as wow goes.. I have to put my little one to sleep so I shall post in about 30 minutes. And im not overestimating VS, im trying not to overestimate the work that is being done NOW to clean up and optimize VS.
Because of YOU Arbiter, MY kids? can't get enough gas. OR NIPPLE! How does that mkae you feeeel? ~ Halo
Neskiairti
Confed Special Operative
Confed Special Operative
Posts: 334
Joined: Thu Jan 11, 2007 4:10 am

Re: Unified DB Resource

Post by Neskiairti »

perhaps you are underestimating how much work needs to be done? :p
seriously though, what are you asking for? using mySQL for what reason?

when it was said that forcing people to use mySQL instead of CSV would certainly drive people away. changing them, would give no benifit, a better choice would be to add a little user friendly editor.
pheonixstorm
Elite
Elite
Posts: 1567
Joined: Tue Jan 26, 2010 2:03 am

Re: Unified DB Resource

Post by pheonixstorm »

Did you read the thread at all?? The reason is in the thread. Also, if this was supposed to be like EVE then there would be no single player element.
Because of YOU Arbiter, MY kids? can't get enough gas. OR NIPPLE! How does that mkae you feeeel? ~ Halo
charlieg
Elite Mercenary
Elite Mercenary
Posts: 1329
Joined: Thu Mar 27, 2003 11:51 pm
Location: Manchester, UK
Contact:

Re: Unified DB Resource

Post by charlieg »

pheonixstorm wrote:Did you read the thread at all?? The reason is in the thread.
The reason is still not clear to me - other than [your] familiarity with it (note: I believe developer familiarity is a valid reason).
Free Gamer - free software games compendium and commentary!
FreeGameDev forum - open source game development community
chuck_starchaser
Elite
Elite
Posts: 8014
Joined: Fri Sep 05, 2003 4:03 am
Location: Montreal
Contact:

Re: Unified DB Resource

Post by chuck_starchaser »

pheonixstorm wrote:
chuck_starchaser wrote:So I agree that there's a role for MySQL; just not that it's the solution for *everything*. Specially not real-time stuff like dynamic economics simulation;
This is where I don't agree. While doing *everything* in SQL is rather dumb keeping a dynamic persistent econ fits a database very well. Speed is key so MySQL or Postgree fits the bill. A dynamic econ would not however fit with a flat file.
This is the kind of thing that make me want to ignore you from now on, and I think I will. You have NO IDEA what the requirements are for simulating a dynamic economy, yet you speak as if you knew all about it... "(sql) fits the bill"... "(a) flat file (would not)"... Klauss challenged you to prove it would "fit the bill" for a dynamic economy. The trick is you'd have to explain how a dynamic economy works, in the first place. And of course you ignored the challenge.
You gained respect pretty fast in these forums, with the windows build project; but, at least as far as I'm concerned, you seem to have become more like a troll, just wanting to waste the developers' time for the hell of it... pushing, and pushing, and pushing mysql... Which, even if you were right, it would be a far less than urgent concern. I'm sick of this; --won't even check this thread anymore.
charlieg
Elite Mercenary
Elite Mercenary
Posts: 1329
Joined: Thu Mar 27, 2003 11:51 pm
Location: Manchester, UK
Contact:

Re: Unified DB Resource

Post by charlieg »

That's an overreaction... he's put a lot of effort in (not as much as you, of course) and you'll "make an enemy" because he has an idea he's trying to discuss?

I mean, you might have a point if he was injecting this topic into other threads, but you can just ignore the topic... phoenixstorm is not forcing this on anybody, so your charge of wasting developer time is nigh unfounded (unless seeing an idea that you disagree with counts as wasting your time). What happened to agreeing to disagree? You can exit a thread any time you like, no need to castigate the OP.
Free Gamer - free software games compendium and commentary!
FreeGameDev forum - open source game development community
chuck_starchaser
Elite
Elite
Posts: 8014
Joined: Fri Sep 05, 2003 4:03 am
Location: Montreal
Contact:

Re: Unified DB Resource

Post by chuck_starchaser »

True.
I guess I'm personally angry. I've put some thought into the problem of simulating economies, but he treats the subject so lightly, like he knows all about it, and exactly what is required for it. I'd like to see him shut the F@*# up and code one.
klauss
Elite
Elite
Posts: 7243
Joined: Mon Apr 18, 2005 2:40 pm
Location: LS87, Buenos Aires, República Argentina

Re: Unified DB Resource

Post by klauss »

chuck_starchaser wrote:You have NO IDEA what the requirements are for simulating a dynamic economy, yet you speak as if you knew all about it... "(sql) fits the bill"... "(a) flat file (would not)"... Klauss challenged you to prove it would "fit the bill" for a dynamic economy. The trick is you'd have to explain how a dynamic economy works, in the first place. And of course you ignored the challenge.
That's exactly why I challenged.
I want him to explain what he thinks the dynamic economy is, and inspect all his preconceptions.
No answer would seem to suggest that he's got no clear idea of the subject, and was talking from pure prejudice (as in judging early), meaning his suggestion for Mysql as a solution to a problem he didn't know is in fact an unfounded one, and dismissable.

Mysql might be a solution - but that has to be decided after analyzing the problem, something he did not do. Unless he answers the challenge.

Even if he does not, we'll have to answer that same challenge ourselves at some point... so the decision is only postponed.

I do believe SQL solutions have a strong point in their favor: their atomicity, persistence, fault tolerance, are all features we really would need in a multiplayer server, and using SQL as a database backend would certainly be one easy way to accomplish that. Only, using SQL might make other problems harder to solve - so it's not a no-brainer. There are other solutions that provide both those attributes, and they all deserve consideration.
Oíd mortales, el grito sagrado...
Call me "Menes, lord of Cats"
Wing Commander Universe
pheonixstorm
Elite
Elite
Posts: 1567
Joined: Tue Jan 26, 2010 2:03 am

Re: Unified DB Resource

Post by pheonixstorm »

First off, my idea for sql inclusion was not solely based on its use for a dynamic or persistent econ. Overall for a persistent universe as its current role VServer does not fufill is generally not fit with a flat file database. There is too much trouble constantly opening adding deleting and editing the same files over and over again that you are bound to screw something up no matter how carefully coded the functions are. Eventually (and probably quickly) you will have two edits on the same file and only one will get posted correctly. This doesnt include the size that flat files get after you add soo much to them. What happens if we have a server with 500 users.. 800 custom ship types and any number of other additions.

Ease of updates: If you mod the units.csv and there is an update for it (and you don't compile yourself or know svn) what happens to those changes youve made? With a csv you can cut n paste if you havent overwritten your original file. With sql, just run the updated script and you don't loose anything you added yourself.
Load sizes: How many times do you really want to open that 10mb flat file for 300bytes worth of info. What about seek times to get that 300 bytes? What about a 50mb or 100mb flat file? Do you really want to load and store that much data in memory when you really don't need it?
# of Files: 1 Database or multiple csv files?

@chuck
Ignore? If I had gone a week w/o answering klauss's challenge then yes, it would be an ignored challenge.

Now that thats out of the way.. can we all get back to the ideas at hand??
Because of YOU Arbiter, MY kids? can't get enough gas. OR NIPPLE! How does that mkae you feeeel? ~ Halo
pheonixstorm
Elite
Elite
Posts: 1567
Joined: Tue Jan 26, 2010 2:03 am

Re: Unified DB Resource - Challenge

Post by pheonixstorm »

klauss wrote:So let me challenge you:
pheonixstorm wrote:Speed is key so MySQL or Postgree fits the bill.
What bill?
pheonixstorm wrote:A dynamic econ would not however fit with a flat file.
Why?
pheonixstorm wrote:Speed and size of the flat file db is horrid for the amount of data used in a symulated econ.
What's that amount?
pheonixstorm wrote:Server is a must (eventually) for SQl
Why?
For a persistent server you don't want to keep huge amount of data in memory for an extended period of time. There is far more important data that should always remain in memory, plus you need the memory to hold data on whoever is currently playing that also constantly changes every few seconds (speed, rotation, weapons fire). Why load and unload the same file multiple times (using up a given amount of ram each time) and risk an overlap. Granted you can code in file locks, but do they always work? And how long does each of these locks last? If player A is doing the same thing as player b, but b has the lock and is taking say 10 minutes to do soemthing that is going to create problems for player a who could have been done in less than a minute. Back to files.. how long does each file access take to load and unload the file from memory? How much memory does it take? The larger the file the more time and memory gets used up. So file access times are key. Speed wins. On to file locks.. If we want to keep one person from hogging up a lock you can again code a wa to deal with it.. but it goes back to reloading the file to catch the updated changes and to modify for new changes. So we come back to speed of access. On a not so modern hard drive the problem gets worse with a flat file system unless you can get really really efficient with your coding.

With modern systems I don't know how large a flat file needs to be before you starting loosing effficiency, I know awhile back you start getting slammed about 25mb or more.. by now that may be closer to 250mb I don't know. The problem here is its not just one file you are reading/writing. You have to have data for player accounts, planets, ships, fleets (flightgroups), cargo types, cargo size, cargo values from one planet to another... So do we append the files or sort them and write them out.. the list of what should we do we or can we dos gets longer and longer. How do we handle all these files. Let them settle somewhere in the data directory, do we pack them to keep from having so much clutter? That opens up another problem. Now we have to open the .pak (or w/e zip type file we use) so we need to add in the code for it as well.

Another issue.. how persistent and how dynamic do we really want this to be? Do we want a mini eve that everyone can setup and run on there own? Do we just want a mp server that comes and goes as the owner sees fit.

This should answer some of your challenge I think.. if not, pick at it and i'll find a counter (probably)
Because of YOU Arbiter, MY kids? can't get enough gas. OR NIPPLE! How does that mkae you feeeel? ~ Halo
pheonixstorm
Elite
Elite
Posts: 1567
Joined: Tue Jan 26, 2010 2:03 am

Re: Unified DB Resource - Econ

Post by pheonixstorm »

So what kind of econ do you want chuck? Player defined or vs generated or both?

Player defined is simplest. You only really have to find how much the prices will flucuate at any given moment based on the cargo getting hauled the most by players. Player goes from point a to point b selling grain. Eventually point b will get satuarated and the price drops. How does this get decided? Do we store everything that gets bought and sold? Do we store JUST the amount that a planet sells and ignore what it buys? Do we start each planet with an initial capital fund and hence a planet can go broke if it only buys?

What about a dynamic universe driven econ? Does each merchant ship carry and sell a set cargo from one planet to the next? Do we need to map trade routes between planets? What about planet classification... Agri, mine, refinery, manufature, pleasure... Do we also then limit how much each planet produces or can consume at any given moment?

So, planet A is an agricultural planet. How much of its raw prodcut does it sell and how much can it refine itself to create processed foods? In turn how much raw material does it buy such as ore, oil, etc. Does it have any capcity to refine these raw goods or does it primarily import processed goods? Spare parts, plastics, prefabs...

On planet A do we also need to track how much raw material it produces at any given time? Say 5000 metric tons a day? How do we do a day? Current 24hr real time day? Simulate a 24hr game time day?

By dynamic econ I would like to see trade routes between major hubs with smaller spoke routes to lesser developed or type specific worlds. Each planet has a given amount it can produce in a given time period. As such you can only buy or sell a certain amount of goods before your profits flatten out.

DATA: Planet
Name: Vega
Type: Agricultural
Product1: Grain
Produced: 5000 metric tons per 24/hr
Max rate: 500/ton
Min rate: 100/ton
current rate: 300/ton
demand: +-50 (based on supply/demand of given product at each level not overall)

This isn't very detailed.. but should give a picture of what could be possible.

Along the above.. player a goes to vega, picks up 500mt of grain and goes to the local mining base. They are desperate for foodstuffs so they buy up all 500 at a high price. Good profit, so player a goes back to vega and picks up another 500mt. This time the mining base isn't in need of as much food. 3 other traders already came by so the price has dropped dramaticly.. what was originally a 400 credit profit per ton has dropped to maybe 50 credit per ton. Now this just tracks player activity.

Back to the planet data.. each planet in addition to a production amount will also hold a max amount that it will buy as well as how much capital it has to spend to buy these goods (generally players should never see that a planet is broke). Some planets will not want certain items. If you try to sell raw goods to a pleasure planet they either may not buy or buy at the lowest possible price. What respectable pleasure planet has any industry to refine materials anyway??

So, you have player activety to track or not, and computer controller cargo haulers that should be tracked. Or at least randomly generate trade routes based on the planet it connects too. Heavy industrial world or refinery planet would need the most from raw ore and materials while population centers would want mostly refined goods and some raw goods. Population centers would be a jack of all trades buying and selling everything legal. manufactoring centers would buy up all refined raw goods and sell off w/e it may or may not specialize in.

So, we classify planets as best we can either be as specific as possible or just generalizing.
We decide of in we want to setup and deal with trade routes and create merchant traffice to go from point to point (established routes) or do we randomly generate what gets bought and sold? This gives us more pirates on lesser routes and fewer on known heavy trafficed routes were protection is higher.
Do we cap the amount planets will buy and sell or do we leave off the buy cap since planets or systems might get blockaded at any given point, driving prices higher on some goods?
How much detail do we want to put into this? How much data can be run in memory and how much must be saved elsewhere.

It boils down to the level of detail we want for the game economy. Simplify or go all out balls to the wall real-time as realistic as we can get.
Me, if we could handle it I would rather go for something real-time. Merchants fly their routes and sell/buy goods traveling from point to point in the game. Opens up long term convoy missions, pirate attacks (using larger ships/fleets), and even raiding merchies by hostile factions. If we have the code for it we could watch entire convoys get boarded and hauled off. That would put the squeeze on middle of nowhere worlds and drive up prices if we set how much material a planet needs per day. Could also hurt what its output is depending on the world..

And that leads me to something I forgot.. in the quest for detail do we also have refining/manufacturing output tied to the amount of raw material avaliable to a given planet? And for that matter what about the import of proccessed goods for say an agricultural world.. they need spare parts for the machinery used to harvest the crops. Does a loss of material in one area affect everything else or does that level of detail create its own problems...

I don't take this lightly w/e you may think. I think of these things because I looooooove to micromanage. Thats probably why I like 4x games, simcity games, roller coaster tycoon, civ3, civ4, etc.

The information above might be out of order or hard to follow, but they are some of the ideas I have had. Do you want to take into consideration population migration conditions? Factor in weather? Natural disasters?

Hell.. I spent a fair amount of time in my old Battletech days writing out entirely planetary econs. Basiclly I wanted to control a small kingdom and raid my neighbors. But there was a lot of econ data I needed to have so i could calculate how much I could spend building up military forces.. upkeep, pay, etc.. The people I played with thought I was nuts doing this but I had a lot of fun. Did the same thing when I got into GURPS space. That was a lot more fun though as I was not bound by a pregenerated universe. This doesn't include what i've done to try and redesign and clone moo3 or even my own 4x game.

So, what else do you wan to analyze and discuss? A more detailed analysis of hows and whys from a data perspective or a code perspective? Is this enough information or shall I reorder and expand upon what I have already typed up?

Klauss, I have ideas.. plenty of ideas but I can't say its a clear idea w/o defining a few uncertainties on how it should or will work. How much work gets put into it. Will it be complex? Simple? Somewhere inbetween??

This is part of the problem, and i'm debating a POSSIBLE solution. Probably the best solution.. but thats still up in the air until we can figure out exactly what we want vegaserver to become.
Because of YOU Arbiter, MY kids? can't get enough gas. OR NIPPLE! How does that mkae you feeeel? ~ Halo
Neskiairti
Confed Special Operative
Confed Special Operative
Posts: 334
Joined: Thu Jan 11, 2007 4:10 am

Re: Unified DB Resource

Post by Neskiairti »

you are probably going to have to drop down to a mathematical model. there were others before who poked at economies and found trouble. in code they really are some of the hardest things to model properly simply because every node on the model you add produces N more complexity. Usually the very simplistic models work best with some form of script forcing them not to get too far out of wack.
chuck_starchaser
Elite
Elite
Posts: 8014
Joined: Fri Sep 05, 2003 4:03 am
Location: Montreal
Contact:

Re: Unified DB Resource

Post by chuck_starchaser »

I'm not going to read all the posts; just injecting some thought here. Simplistic economics would not do the trick. It is NOT easy to fool players by injecting random numbers and whatnot. There's a yawning gulf of complexity going from NO economics simulation, to a level where it can be really believable. In a believable simulated economy you'd perceive *EVERYTHING* making sense, consistent, coherent.
That implies, for example, that you see most cargo ships going one way, A, but you think there's more money along route B, but then you find out that you neglected to take certain details into consideration, and that the merchant shippers were right, and you were wrong. Which is not to say that you shouldn't eventually get better at trading than the AI's, but that they should be at least fairly smart.
Local stocks and prices are another area that needs a lot of sophistication to make sense. If a base is out of food, the price should go up much faster than if the item that's run out is pleasure bots. That is, demand elasticity needs to be taken into account. And it has to be weighted against price elasticity; otherwise demand elasticity by itself tells you nothing about price without hacks and magical numbers.
And price elasticity depends on the % of disposable income, and cultural factors, which should be part of the AI profile of factions.
But disposable income also depends on whether an economy is exanding or recessing, in the longer term; and whether the money supply is expanding or shrinking in the shorter term, which depend on policies, which should also be part of the AI.
Simulating economies, you'll soon find out, if you do your homework, can't be done on a static universe. If you do, it is likely to be so rigid that economies either exand factorially or collapse to zero; and then you have to hack stability into them. You need to model bankruptcies and upstarts, at least statistically, and this ties into modeling banking, lending, and venture capital; --i.e.: "follow the money".
Which raises such questions as "should we model government bonds?"
Which may sound hilarious at first, but when you think about it, the bond markets around the world are many times bigger than the stock markets, both in terms of total capital and volume of trading. If you don't model bond trading, it makes no sense to model stock markets; and if you don't model stock markets, you don't model venture capital.
In other words, you can talk about modeling the economy as if everybody knew what you're talking about, but I certainly don't; and I think that what most people mean when they say "model the economy" is just a few simple hacks here and there to make prices vary a bit in response to wars and catastrophes, but that's NOT "modeling the economy", it's just a hack, and players will recognize it for what it is. And what it is is "decorative"; --i.e.: it will NOT increase believability.
pheonixstorm
Elite
Elite
Posts: 1567
Joined: Tue Jan 26, 2010 2:03 am

Re: Unified DB Resource

Post by pheonixstorm »

@Chuck
Do you think it better to build the econ in stages or all at once? May be easier to stage each part and test, your thoughts?

Thats one reason I skimmed over the econ in most of my posts. The complexity of the issue is soemthing that will by itself probably take up a 15 page discussion once the time comes to really come to terms on how much effort should really be put into it. Although injecting a simplistic econ model just based on trade would be easiest to code its not the most realistic. As we are talking about a game though the matter gets more complex just trying to balance how realistic we want it to run. As far as the player is concerned how much detail will they actually see.

In this thread I wanted to focus on the data storage side, not complex issues we don't have the time to spend properly fleshing out items such as
A. Player account data
B. Econ models
C. Large scale warfare (system/sector wide conclicts)

I may mention them, but I really didn't want to stray too far off the main topic. If you disagree fine, if you agree good.


A question for everyone
How many of you (doesnt matter for or against SQL) think that vegaserver should be an mmo? Don't think about cost, don't think about how hard it might be to code. Think Online persistent privateer, elite, or whateer other flight sim trading game you have played. Think fun!
Because of YOU Arbiter, MY kids? can't get enough gas. OR NIPPLE! How does that mkae you feeeel? ~ Halo
klauss
Elite
Elite
Posts: 7243
Joined: Mon Apr 18, 2005 2:40 pm
Location: LS87, Buenos Aires, República Argentina

Re: Unified DB Resource

Post by klauss »

pheonixstorm wrote: How many of you (doesnt matter for or against SQL) think that vegaserver should be an mmo? Don't think about cost, don't think about how hard it might be to code. Think Online persistent privateer, elite, or whateer other flight sim trading game you have played. Think fun!
That's pointless... however much anyone would like an mmo, saying so without caring for how much work actually producing an mmo takes is a pointless exercise.

MMOs are the hardest games ever produced. They have to go against a pile of crap to merely get a few parties communicating - let alone being fun. They have to battle against scalability issues on the server side (or struggle with the complexities of distributed simulation in peer-to-peer environments). They have to struggle with cheaters, handle cheating, discourage cheating, preventing cheating, booting cheaters (because some WILL find a way past any preventive measure), and in general, struggle with users themselves: because users want to cheat, and cheating ruins the experience of most everybody, so they're sick games to be developing these days.

Failing to consider all that is like dreaming in technicolor.
Oíd mortales, el grito sagrado...
Call me "Menes, lord of Cats"
Wing Commander Universe
Post Reply