On the road to Python 3.x +

Development directions, tasks, and features being actively implemented or pursued by the development team.
Post Reply
wirser
Bounty Hunter
Bounty Hunter
Posts: 201
Joined: Wed Dec 01, 2004 8:06 pm
Location: not where i want to be

On the road to Python 3.x +

Post by wirser »

With compilable/runable VS (taose) the next project needs to be updating the python (used for the data) to version 3.x complient.

While still a rookie in python, I've decided to see if I can figure it out.

after cd into VS_root dir in terminal, I used this

Code: Select all

find ./ -name "*.py" >python.lst
to get a complete list of all python code, which will be used for project and other logging.

now learning that the "import" command is simular to #include in C/C++
using grep I did this next

Code: Select all

grep -R 'import ' */*.py  > ../py_imports_data
in the data folder to be able to make a sort of dependancy tree
I already noted one curious thing ,some modules being imported more than once in some files

next will be to use IDLE for python to verify all python modules needing checked
after that I may try to import older mods data
if current_situation=fact and if current_situation=faith
then current_forcast = excitement
loki1950
The Shepherd
Posts: 5841
Joined: Fri May 13, 2005 8:37 pm
Location: Ottawa
Contact:

Re: On the road to Python 3.x +

Post by loki1950 »

For those extra import calls you might check the logical flow of those instances might be that they are in fact only used once :wink:

Enjoy the Choice :)
my box::HP Envy i5-6400 @2Q70GHzx4 8 Gb ram/1 Tb(Win10 64)/3 Tb Mint 19.2/GTX745 4Gb acer S243HL K222HQL
Q8200/Asus P5QDLX/8 Gb ram/WD 2Tb 2-500 G HD/GF GT640 2Gb Mint 17.3 64 bit Win 10 32 bit acer and Lenovo ideapad 320-15ARB Win 10/Mint 19.2
TBeholder
Elite Venturer
Elite Venturer
Posts: 753
Joined: Sat Apr 15, 2006 2:40 am
Location: chthonic safety

Re: On the road to Python 3.x +

Post by TBeholder »

If only. Inside function definitions.

Code: Select all

def MakeMiningBar(concourse,concoursetext,background,bartext):
    import bar
    return bar.MakeBar(concourse,
            concoursetext,
            bartext,
            background,
            True,
            '0',
            False,
            '',
            (.87735,-.018),
            [],
            [
            (-0.8025, -1.055 , 0.5, .8),
            (-0.1725, -0.4558125, 0.1758125, 0.5385),
            #(-0.0325, -0.3558125, 0.1758125, 0.5385),
            ])
"Two Eyes Good, Eleven Eyes Better." -Michele Carter
wirser
Bounty Hunter
Bounty Hunter
Posts: 201
Joined: Wed Dec 01, 2004 8:06 pm
Location: not where i want to be

Re: On the road to Python 3.x +

Post by wirser »

Like I said before I was at first looking a way map the python flow
I am not too worried about the imports just yet, still it looks wierd.
speaking of imports I need to find "VS" or some way to meet the requirement

doing a topic search for differences between python 2.x and 3.x point to 3+ changes
so far I've used search terms "xrange,print '"

the "print" seems simple enough to fix, just make
sure to enclose in () .if I understand the problem

the "xrange" I still have to study,it is used alot

There is a divison problem which may cause trouble , in short it causes wrong answers. so all of the math using division needs to be
checked. but the code should run just with bad math 8^0
if current_situation=fact and if current_situation=faith
then current_forcast = excitement
wirser
Bounty Hunter
Bounty Hunter
Posts: 201
Joined: Wed Dec 01, 2004 8:06 pm
Location: not where i want to be

Re: On the road to Python 3.x +

Post by wirser »

Ok I now have all print statements converted to print functions for python 3.x
I am still testing it but I have found out some interesting facts
there are 2 pdb.py and one bdb.py (supplied with python)
being I wasn't doing any debuging and have current versions of these I am going to have them deleted in the diff file I'll make.

I even found on a module comment about being an update for python 1.6 :o
Isuspect most if not all network modules can be updated

some comments said they wern't used anymore so I'm going to look into this before I create diff

BUGS
When trading Ive adopted a statagy of looking at all prices wether I needed them,giving me a better how to buy and/sell.
I have locked up more than once doing this :twisted: .
units.csv while present gives error about either missing or corrupt.
shader prob not python related.

Oh Loki I'd like to start a clean bug report/seperate tree for VS-taose bugs, but don't know if it should remain at sf or move to github or whereever.
if current_situation=fact and if current_situation=faith
then current_forcast = excitement
loki1950
The Shepherd
Posts: 5841
Joined: Fri May 13, 2005 8:37 pm
Location: Ottawa
Contact:

Re: On the road to Python 3.x +

Post by loki1950 »

Might be easier to work on GitHub as that is where the taose source is and they have a bug tracking interface/template.

Enjoy the Choice :)
my box::HP Envy i5-6400 @2Q70GHzx4 8 Gb ram/1 Tb(Win10 64)/3 Tb Mint 19.2/GTX745 4Gb acer S243HL K222HQL
Q8200/Asus P5QDLX/8 Gb ram/WD 2Tb 2-500 G HD/GF GT640 2Gb Mint 17.3 64 bit Win 10 32 bit acer and Lenovo ideapad 320-15ARB Win 10/Mint 19.2
wirser
Bounty Hunter
Bounty Hunter
Posts: 201
Joined: Wed Dec 01, 2004 8:06 pm
Location: not where i want to be

Re: On the road to Python 3.x +

Post by wirser »

on bug tracker.
I am holding off on getting github accnr for now but will keep it in mind

About creating print diff's
I have created two diffs related to printing, one for data ond one for engine/objconv using "diff -ur"
However Due to changes in BB I can no longre upload/include them here :?
I used to be able to attach a file but not today

both use top folder as patch/diff root.(for the newbies, this is the directory that the diff and patch are issued from)

In addition I found that there is a problem with at least on python module (file) dealing with inconsistant indentation.
I figure more if not most of python code suffers this problem.
while python 2 gives warnings, python 3 gives errors.
I'm going to try to use "Idle" to try to correct these problems (I don't know if I can automate the process any, if not this will take some time). anyhow my next patch(es) will be for this then I'll attempt the xrange issue

if the code passes Idle for python 3 checks the final patch should be ready
loki1950
The Shepherd
Posts: 5841
Joined: Fri May 13, 2005 8:37 pm
Location: Ottawa
Contact:

Re: On the road to Python 3.x +

Post by loki1950 »

My preferred Python IDE is Ninja not Idle but there maybe some scripts available on the official Python migration web sites.

Enjoy the Choice :)
my box::HP Envy i5-6400 @2Q70GHzx4 8 Gb ram/1 Tb(Win10 64)/3 Tb Mint 19.2/GTX745 4Gb acer S243HL K222HQL
Q8200/Asus P5QDLX/8 Gb ram/WD 2Tb 2-500 G HD/GF GT640 2Gb Mint 17.3 64 bit Win 10 32 bit acer and Lenovo ideapad 320-15ARB Win 10/Mint 19.2
wirser
Bounty Hunter
Bounty Hunter
Posts: 201
Joined: Wed Dec 01, 2004 8:06 pm
Location: not where i want to be

Re: On the road to Python 3.x +

Post by wirser »

HMM Ninja IDE is not in Debian Repos

But I did find the website
http://ninja-ide.org/
which has Debian/ubuntu pkgs and others, and has a seperate repository too

When I get some time I'll look at it, Idle isn't special, I just got intrduced to it working with my Raspberry pi.
To me IDE's do not mean much just yet ( I started using cp/m ed for a word processor back in the day).
Just do a search for python IDE (and get lost) :P

Thanks for the info btw
if current_situation=fact and if current_situation=faith
then current_forcast = excitement
wirser
Bounty Hunter
Bounty Hunter
Posts: 201
Joined: Wed Dec 01, 2004 8:06 pm
Location: not where i want to be

Re: On the road to Python 3.x +

Post by wirser »

Update:
I spent some time and checked all .py files in VS-taose tree with Idle for python 3.
aside from the print statements/functions I missed , most of the errors were simular (changing enclosing brackets and such)

indentations were next cause of errors (not properly set)
also declaring octal number with "0" allowed in python2 instead "0o" python3 demands, was common
.
I found out that the data/modules/builtin directory contains module supplied from python < ver 2.7, which got to "pass python 3 checks-, with usual errors

I tried running VS with "updated .py files and it behaves with differences eve3n though I am still using python 2.7 to execute.
what changes I made to get the python code to pass are supposed to work both on 2.7 and 3.5 versions so maybe I "fixed" some unknown problems. maybe Idle is tougher than the python interupter

I did not run into any "xrange" errors but maybe this is remidied by "_from-future__" module I found

Finally It seem to be fortunate I didn't post my print patches as I found mistakes with Idle

more to come
if current_situation=fact and if current_situation=faith
then current_forcast = excitement
wirser
Bounty Hunter
Bounty Hunter
Posts: 201
Joined: Wed Dec 01, 2004 8:06 pm
Location: not where i want to be

Re: On the road to Python 3.x +

Post by wirser »

today I got VS-taose to compile with python 3, and it even started

****KLUDGE ALERT ******
I had ran into a "Vega-py3" tree some time ago and tried using the c code from this to get a compile
but not even this did not work
searching for the source of the errors I can across a referrence to libboost_python.
In debian (I assume ubuntu) if you link to libboost_python you get linked to libboost_python2.7 and not the desired libboost_python-py3


to fix this after running cmake and ccmake edit /engine/build/CMakeCache.txt
change libboost_python.so to libboost_python-py35.so or whatever the correct libboost_python turns out to be .
Another cmake FIXME
it started but went straight to space and not sybase so needs more work

It looks like i'll see if I can get taose python /c interface working myself
if current_situation=fact and if current_situation=faith
then current_forcast = excitement
wirser
Bounty Hunter
Bounty Hunter
Posts: 201
Joined: Wed Dec 01, 2004 8:06 pm
Location: not where i want to be

UPDATE On the road to Python 3.x +

Post by wirser »

Up until now I have been making changes that work with python 2.7 and 3.5 .

Now I have gotten to the point where to compile/run with python 3.5, changes must be made to about a dozen c++ files , which are not yet known to be backward compatable.
Of course the needed code deals with creating the "c++" modules needed by Python (3.5)
This has the effect of putting a player in a special "empty" universe :shock: with no way out ("Esc" then "Q" does exit cleanly)

When(if) I get a running python3 version I'll sttempt to code in some python version specific code

CMake side note
during my recompiling I run into a situation where cmake would not produce any file in my engine/build directory.
It turns out I started cmake/ccmake in the wrong directory and all of the output files were always put there afterwards .
in order to get cmake to work again like it is supposed< iha to delete CMakeFiles folder, CMakeCache.txt , cmake_install.cmake wherever they apper,. make sure you are in the seperate "build" and run cmake/ccmake again if you succeed you will have a proper Make file

Second for reasons not yet known to me, the :boost/python" lib path doesn' appear in ccmake until after successfully generating a Makefile (don't know why, but it needs to be changed for python3.x linking)
if current_situation=fact and if current_situation=faith
then current_forcast = excitement
TBeholder
Elite Venturer
Elite Venturer
Posts: 753
Joined: Sat Apr 15, 2006 2:40 am
Location: chthonic safety

Re: On the road to Python 3.x +

Post by TBeholder »

wirser wrote:In debian (I assume ubuntu) if you link to libboost_python you get linked to libboost_python2.7 and not the desired libboost_python-py3
This should depend on what boost libraries are installed, no?
"Two Eyes Good, Eleven Eyes Better." -Michele Carter
wirser
Bounty Hunter
Bounty Hunter
Posts: 201
Joined: Wed Dec 01, 2004 8:06 pm
Location: not where i want to be

Re: On the road to Python 3.x +

Post by wirser »

TBeholder wrote

>> wirser wrote:
>> In debian (I assume ubuntu) if you link to libboost_python you get linked to libboost_python2.7 and not the ?>>desired libboost_python-py3

>This should depend on what boost libraries are installed, no?

No, It depends on what python version boost is compiled with. on Debian 9.9 there are two versions- one compiled against ver 2.7 and the other against version 3.5. The VS code doesn't know the difference (yet). Bottom line Vegastrike (taose tree) compiles and runs with python 2.7, libboost_python2.7. (libboost_pythonpy27) the Python version must be the same as libboost_python was compiled with. It is not a problem having both installed at the same time - You just have to link to the correct version.
Even though the boost version is the same there can be two (or more) incompatable libboost_python versions, due to Python
if current_situation=fact and if current_situation=faith
then current_forcast = excitement
wirser
Bounty Hunter
Bounty Hunter
Posts: 201
Joined: Wed Dec 01, 2004 8:06 pm
Location: not where i want to be

Re: On the road to Python 3.x +

Post by wirser »

Sorry for being gone so long.

UPDATE

After attemptimg to get VS to compile/run with Python - I failed!
I got An error free compile but it doesn't properly run. The main issue is that the module "VS" is not being built during startup as it should. Secondly I found out changing "print": statements to functions is not as simple as most articles on the subject claim. Basically I need to research further (at least I am learning something).

Unless there is some yet unseen method the python 3.x version will NOT be compatible with the old VS. I am still hoping for perhaps an "either / or" option to compile VS/engine/src/python directory (or even a "python2 and python3 directories with all else remaining the same).
I have been trying to learn python, but have ran into other problems related to how debian packaged python (known issue in ubuntu). In short it affects how "modules" such as pip and bjam are called. I do not yet know how this affects VS-py3. The stated fix seems simple enough just change some "path" statements
Another issue is that "python" code is embedded in C++ data statements, thus avoiding compiler error checking until run time. And last (for now) is that there are many "patches" some of which are from python 2.4 and perhaps earlier which makes understanding source code functions difficult.

It is a good thing I do not depend on this working, Still it is a puzzle I'd like to solve.
if current_situation=fact and if current_situation=faith
then current_forcast = excitement
loki1950
The Shepherd
Posts: 5841
Joined: Fri May 13, 2005 8:37 pm
Location: Ottawa
Contact:

Re: On the road to Python 3.x +

Post by loki1950 »

Re those "Python" code is embedded in C++ data statements those are not data statements that is the python that VS uses to actually run all the python scripts it does not repeat dose not use the system installed Python at all they comprise a Python interpreter within the VS binary one of the reasons that klauss was always sceptical about porting to Python 3.

Enjoy the Choice :)
my box::HP Envy i5-6400 @2Q70GHzx4 8 Gb ram/1 Tb(Win10 64)/3 Tb Mint 19.2/GTX745 4Gb acer S243HL K222HQL
Q8200/Asus P5QDLX/8 Gb ram/WD 2Tb 2-500 G HD/GF GT640 2Gb Mint 17.3 64 bit Win 10 32 bit acer and Lenovo ideapad 320-15ARB Win 10/Mint 19.2
wirser
Bounty Hunter
Bounty Hunter
Posts: 201
Joined: Wed Dec 01, 2004 8:06 pm
Location: not where i want to be

Re: On the road to Python 3.x +

Post by wirser »

HMM...
This bit of information explains some odd aspects with the VS source i.e having a python bebugger in the sourtce tree (I thought it was just a mistake). It also comfirms some private fears concerning the scope of the project.
Rabbit holes within rabbit holes -deal with that Neo
It also tends to shed light on why boost codes was done the way it was before. So if I understand this by changing the python code to python 3 I am probably breaking the VS-python functions. Which is beyond my understanding at present

Good thing I have a warped sense of humor
if current_situation=fact and if current_situation=faith
then current_forcast = excitement
loki1950
The Shepherd
Posts: 5841
Joined: Fri May 13, 2005 8:37 pm
Location: Ottawa
Contact:

Re: On the road to Python 3.x +

Post by loki1950 »

So it's a warp hole not rabbit hole :wink:

Enjoy the Choice :)
my box::HP Envy i5-6400 @2Q70GHzx4 8 Gb ram/1 Tb(Win10 64)/3 Tb Mint 19.2/GTX745 4Gb acer S243HL K222HQL
Q8200/Asus P5QDLX/8 Gb ram/WD 2Tb 2-500 G HD/GF GT640 2Gb Mint 17.3 64 bit Win 10 32 bit acer and Lenovo ideapad 320-15ARB Win 10/Mint 19.2
wirser
Bounty Hunter
Bounty Hunter
Posts: 201
Joined: Wed Dec 01, 2004 8:06 pm
Location: not where i want to be

Re: On the road to Python 3.x +

Post by wirser »

loki1950 wrote:So it's a warp hole not rabbit hole :wink:
I was thinking nore about quoting the movie"zathura", but to keep this family freindly I'll pass.

Anyways I found out four files have no current use :?

in VS/engine/src are four files starting with "cs_", which at one time "quientened some error messages for python 2.4. while just wrapping other python includes.
long story short is I tried removing these and correcting the #includes , the compiler result does run and debugging is made easier.

Here is a copy of the source diff I made.

Code: Select all

diff -r ./orig-engine/src/cmd/base_init.cpp ./engine/src/cmd/base_init.cpp
2c2
< #include "cs_python.h"
---
> #include <Python.h>
diff -r ./orig-engine/src/cmd/base_util.h ./engine/src/cmd/base_util.h
9c9
< #include "cs_boostpythonobj.h"
---
> #include <boost/python/object.hpp>
diff -r ./orig-engine/src/command.cpp ./engine/src/command.cpp
2c2
< #include "cs_python.h"
---
> #include <Python.h>
Only in ./orig-engine/src: cs_boostpythonclass.h
Only in ./orig-engine/src: cs_boostpython.h
Only in ./orig-engine/src: cs_boostpythonobj.h
Only in ./orig-engine/src: cs_python.h
diff -r ./orig-engine/src/gfx/vec.cpp ./engine/src/gfx/vec.cpp
31c31
< #include "cs_boostpythonclass.h"
---
> #include <boost/python/class.hpp>
36c36
< #include "cs_boostpython.h"
---
> #include <boost/python.hpp>
diff -r ./orig-engine/src/main.cpp ./engine/src/main.cpp
20c20
< #include "cs_python.h"
---
> #include <Python.h>
diff -r ./orig-engine/src/python/init.h ./engine/src/python/init.h
13c13
< #include "cs_boostpython.h"
---
> #include <boost/python.hpp>
diff -r ./orig-engine/src/python/python_class.h ./engine/src/python/python_class.h
11c11
< #include "cs_python.h"
---
> #include <Python.h>
28c28
< #include "cs_boostpython.h"///module.hpp>
---
> #include <boost/python.hpp>///module.hpp>
diff -r ./orig-engine/src/python/python_compile.h ./engine/src/python/python_compile.h
9c9
< #include "cs_python.h"
---
> #include <Python.h>
diff -r ./orig-engine/src/savegame.cpp ./engine/src/savegame.cpp
1c1
< #include "cs_python.h"
---
> #include <Python.h>
diff -r ./orig-engine/src/star_system_generic.cpp ./engine/src/star_system_generic.cpp
42c42
< #include "cs_boostpython.h"
---
> #include <boost/python.hpp>
diff -r ./orig-engine/src/universe_util_generic.cpp ./engine/src/universe_util_generic.cpp
30c30
< #include "cs_python.h"
---
> #include <Python.h>
I also noticed the "VS/engine/boost/" folder is not needed if cmake option " use system boost" is on.
Maybe an added option to d/l source if needed in cmake :shock:
if current_situation=fact and if current_situation=faith
then current_forcast = excitement
loki1950
The Shepherd
Posts: 5841
Joined: Fri May 13, 2005 8:37 pm
Location: Ottawa
Contact:

Re: On the road to Python 3.x +

Post by loki1950 »

You may not have noted that the svn repo includes the boost libs for that non system boost option it meant that the relevant source was already on hand :?

Enjoy the Choice :)
my box::HP Envy i5-6400 @2Q70GHzx4 8 Gb ram/1 Tb(Win10 64)/3 Tb Mint 19.2/GTX745 4Gb acer S243HL K222HQL
Q8200/Asus P5QDLX/8 Gb ram/WD 2Tb 2-500 G HD/GF GT640 2Gb Mint 17.3 64 bit Win 10 32 bit acer and Lenovo ideapad 320-15ARB Win 10/Mint 19.2
wirser
Bounty Hunter
Bounty Hunter
Posts: 201
Joined: Wed Dec 01, 2004 8:06 pm
Location: not where i want to be

Re: On the road to Python 3.x +

Post by wirser »

I understand the convience of including it but, who maintains the boost source (keeping in sync with boost source)? Looking at Python code, I keep tripping over libboost, It's not an issue yet- but when Python 3 arrives , libboost-python needs to either recompiled linked to python 3 or needs tu be installed concurrently (for dynamically linked VS).

I am a fan of having outside source (ie boost) staying outside :wink:
if current_situation=fact and if current_situation=faith
then current_forcast = excitement
nabaco
Explorer
Explorer
Posts: 8
Joined: Sun Mar 22, 2020 5:19 pm

Re: On the road to Python 3.x +

Post by nabaco »

Hi,
Any progress with this issue? With my basic knowledge of C++/Python, I'd like to help if possible.

Regarding boost/, if we have to maintain a loca copy, can we keep it as a git submodule? https://github.com/boostorg/python

Wirser, any chance you can share your work somewhere (maybe a branch on github?), so we'll be able to cooperate on this?

What are the tasks that are left to be done now?
wirser
Bounty Hunter
Bounty Hunter
Posts: 201
Joined: Wed Dec 01, 2004 8:06 pm
Location: not where i want to be

Re: On the road to Python 3.x +

Post by wirser »

:evil: NOTE this would have been better if the "BBS" had not refreshed during my reply and wiped out everything :evil:
nabaco wrote:Hi,
Any progress with this issue? With my basic knowledge of C++/Python, I'd like to help if possible
.

Yes and no, I had to reevaluate my work on this issue due to the new release which is vastly different from the "Taose" tree.
For now at least until I complete some prerequisite task which don't divide up well, I would recommend studying the following
1) gdb operation -how where to set break points
2) valgrind operation -memory leak detection
3) doxygen usage/ configuration
4) Python differences between versions 2.4 an 3.5
5) ogre implementation
6) Compiling with gcc , visual C and others (Linux, mac, windows) flags linking etc
7) studying the Vegastrike file structures both data(assets) and code
8 ) OpenGL implementation

As you see there are many areas needing attention, currently I am still identifying issues so hang tight
nabaco wrote:Regarding boost/, if we have to maintain a local copy, can we keep it as a git submodule? https://github.com/boostorg/python
Boost code or any other "externals" should not be included in the Vegastrike repository tree, in my opinion. That being said I am also considering other issues with the current repository structure in general. For now I am not in a hurry to make any drastic changes. To let you know, I have thought of making a "clean" repository in the near future which will handle this and other issues.
BTW it looks like it is best to compile with "system boost" (cmake option)if available, having two versions of boost seems to cause problems.

nabaco wrote:Wirser, any chance you can share your work somewhere (maybe a branch on github?), so we'll be able to cooperate on this?
What are the tasks that are left to be done now?
I have been posting updates either or at the GitHub page https://github.com/vegastrike/Vega-Strike-Engine-Source in the issues tab

Work is in progress in areas such as code housekeeping and standard development- which will be used for future work.

There is an interest in the project again, but some foundational work must first be completed. be patient and soon lots of tasks will be available
if current_situation=fact and if current_situation=faith
then current_forcast = excitement
Post Reply