ram usage

[safemode]

It seems a little odd that simply running VS and hitting the main menu, we're already using ~240MB I'm pretty sure I can't read 240MB of textures in the 5 seconds it takes to load VS to the main menu.

I'm hoping to spur a little interest in memory cleaning for post 0.5

We have a crapload of static variables everywhere for config options. Why dont we have a config class that just reads _all_ the config variables in sets the appropriate value to a intuitive variable and gets executed on loading a game. This way we dont have multiple instances of static variables (different variables) all referencing the same config options.

Perhaps a class "GameOptions". It can have public data members for every game option and an init() method. Vegastrike could execute the init() method when you start it. No need for static anything for game options.

I'm not sure what else is using such large amounts of ram, as no systems or large numbers of textures/units are loaded at this time.

[Dyskord]

Don#t know, if music is loaded at that point (playing without music since some time now), but i found the music eats up a lot of RAM, ultimatively leading to crashes / Hangups, when having played for some time, switching songs and stuff.

[safemode]

when music/sound is disabled. the music thread should be loading nothing, and doing nothing. it should sit there, take requests from the main game to play a sound / music, and immediately trash it.

if that's not how it works, then it should work that way.

[spiffariffic]

Is there an easy (relatively) way to check how much memory each thread is using on linux? Also, how does one check for memory leaks? Same or similar method?

[safemode]

you could try valgrind... but valgrinding vegastrike is a guaranteed way of wasting about 2 hours of your life.


the other way would be to use a library that wraps calls to malloc and new so that it keeps track of freeing them... but that wouldn't show you if we're holding multiple copies of an object ...just leaks

The only real way of checking for wastes of memory is following the code.

for instance, textures get loaded by vsimage. They are allocated in vsimage, but the pointer is returned to the caller and they are expected to free that memory when they're done with it. So we'd check to make sure that happens.

In other cases it's not so simple, as in assignments by value. We have plenty of cases where we copy a set of ints or floats or structs into a vector or list and keep that list, then do deep copies to another list/vector elsewhere. When the values are never changed, this is bad practice. This is what currently happens with loading meshes. Rather than store them in a shared format for various other subsystems to use, we have to convert and copy to those other subsystems because VS has a lot of code written and pulled in from different projects.

That kind of inefficiency wont show up on any mem-checking program.

Then there's python which we need to look at carefully too. We have to make sure we're not unecessarily holding instances of objects we no longer need, not keeping copies of lists around etc etc.

[klauss]

Two things.

First, music eats hundreds of MBs of RAM and can build up to GBs. Really. It's "ugly", to say the least.

I'm already getting dirty there, rewriting music to do streaming, so this should improve sometime soon.

Then there's python. Python NEVER frees memory. Maybe python2.5 does - in very rare cases (really, I've used 2.5 for some time and it seldom frees memory, though it should do so).

Python code handles lots of objects at times, so we should be careful with that. Maybe port some of it to C/C++ (I've already mentioned pyrex somewhere - let me mention it again).

Oh - and do you really think those static variable impact memory consumption by any noticeable amount? I've noticed VS binaries tend to be huge when using debug mode (in linux at least they become over 100MB).

[safemode]

I dont ever run with music or sound enabled. I dont know if it still loads the files, if so that's a big bug.


Python I can understand having an issue. Like i said in your other thread about pyrex, we have a lot less people proficient in python to make really good python code than we have in C++, so the evolution of it goes slower. So i can definitely see how we have a lot of waste in there.


as for the static vars. I dont think they use a lot of space, but it's a two fold purpose that i'm getting rid of them. One, we want options consolidated and in one place, for maintainability. Two, it's a start at cleaning up the excesses of memory consumption. We start by culling the easy stuff, then work on harder stuff, not usually the other way around. Especially not right before a release either.

[klauss]

Don't get me wrong, I'm all for "options refactoring" stuff. It just won't help with memory consumption, that's all.

Sad about pyrex
Sad sad

But, truly, economy modelling handles too many objects to let python handle it efficiently. Some kind of porting to C++ would be necessary if that turned out to be the problem.

I'd take care of textures first though.

[Miramor]

A thought: could Python be replaced as the game's scripting language with something faster and less bloaty, e.g. Lua (which is used in a lot of games)?

[ace123]

I love the idea of being able to script things like this.

You see where we have gotten with so much stuff in C++. The Privateer Gold folks rewrote the entire base computer in Python.

It was actually easy to write the python code to work in multiplayer because it supports lexical scope, so you don't need to save any state or worry about writing a new function. You just define a function nested inside of the scope and python will keep running it once you get the response.

True, the economy doesn't need a lot of these benefits.

Perhaps it would help to explore another scripting language that supports all the important features (lexically scoped or good class support, a C++ library) but is also fast unlike python?
Then we could support both languages for a while, and phase out Python, while the time-critical pieces get moved to the new language.

Edit: Lua coild work for this... though I don't know how much faster/less memory intensive Lua is than Python.
What about Lisp?

Maybe something like PyPy or some other fast python library (stackless python?), but they can only improve it so much...

I suppose a dynamically loaded library could work, but it is very, very difficult to modify something like that (you still need a compiler, and you still need to know how to hack C++) and at that point, you might as well compile a new executable.

[safemode]

I really dont think python is all that slow. VS uses 60% cpu on my machine with maximum settings on everything (why it only uses 60% when vsync is disabled is a mystery, i would think it would use 100%, i can only assume we wait on the draw to GL). What we have a problem with is ram usage and possibly inefficient code due to less eyes working on making the python better, as opposed to the C++ code.

I think the python code needs some serious looks into memory management. I dont think speed should be a very high concern right now, as that is not where we are hurting.

That and I dont think we have the man power to move to another scripting frontend.

[klauss]

I don't think moving to another scripting language is wise. There's already too much invested in python to do the switch.

I repeat: python's memory management is weird, it never frees memory (although it reuses "freed" memory). So memory used once by python is unavailable to C++, and increases swapping likelihood. So we'd have to make sure python code never instantiates too many objects at once, so that it never reserves (at any point in time) too much memory (that will never be freed even if the objects/collections are later destroyed)

[Miramor]

You sure there's no way of making Python's memory management compatible with C++? IANAP but what you're proposing sounds a little bit... hackish.

[safemode]

It's a limitation in the language, not implementation. python just can't reclaim memory.

2.5 has a garbage collector, but it can only reclaim very limited things.

Basically it comes down to python has to be coded very well, or it's very easy to eat up ram unecessarily.

This really means we need to do as much fumbling with objects as possible in C++. We dont want to create objects in loops in python at all.

I think it'd be best to keep python scripts to "instructional" code that the C++ code uses to decide how to behave (mission directives, quests ), but all the function of VS should occur in C++. This means moving dynamic universe to C++, economy should be kept C++, etc.

Basically we should use python as a more complicated version of xml for telling C++ what to do, not take the place and function of C++


note: i'm not saying C++ reads the python. Basically what i'd like is less functionality in python, and better C++ so that we can have the decreased functionality in python without sacrificing modability ease by only having to write python.

[klauss]

Safemode: we're in 100% agreement