new tool: vs_validator

A forum for the discussion and development of programs to assist working on or playing with the Vegastrike engine and data sets.
JasonStiletto
Merchant
Merchant
Posts: 34
Joined: Sun Sep 24, 2006 8:17 pm
Contact:

new tool: vs_validator

Post by JasonStiletto »

updated 12-6-07
vs_validator-win32.zip
vs_validator-linux32.zip
vs_validator-source.zip (GPLicious and Pascal)


validates contents of units.csv against my understanding of them. Which isn't horrible.

current output:

Code: Select all

Finding data directory.
testing /home/stiletto/data4.x/units/units.csv
Using /home/stiletto/data4.x/
Parser Begins
Processing /home/stiletto/data4.x/master_part_list.csv no error checks - Done.
Reading /home/stiletto/data4.x/units/units.csv
Header Read
Unknown fields: 
 .. None.
Header Processed.
row 657 column 8 Hud_image "planet-ocean-hud.spr" file not found
row 697 column 8 Hud_image "Shipyard-hud.spr" file not found
row 713 column 2 Directory ./contraband directory does not exist.
row 714 column 2 Directory ./contraband directory does not exist.
row 715 column 2 Directory ./contraband directory does not exist.
row 718 cargo_import Cat Contraband/Luddite not found in master part list
row 727 column 8 Hud_image "base_locationmarker.spr" file not found
row 728 column 8 Hud_image "base_locationmarker.spr" file not found
row 729 column 8 Hud_image "base_locationmarker.spr" file not found
row 800 column 2 Directory ./h496 directory does not exist.
Database processed.
Max Fieldlengths (Str only)
---------------------------
cargo.Category: 45
cargo.description: 50
cargo.filename: 28
cargo_import.Cat: 49
light.meshfile: 18
mounts.SIZE: 50
mounts.type: 28
subunit.filename: 35
units.BSP_Mesh: 21
units.Cockpit: 18
units.Combat_Role: 12
units.Directory: 47
units.Explosion: 19
units.Hud_Functionality: 66
units.Hud_image: 36
units.Index: 39
units.Max_Hud_Functionality: 65
units.Mesh: 40
units.Name: 36
units.Object_Type: 19
units.STATUS: 33
units.Shield_Mesh: 23
upgrades.filename: 26
Last edited by JasonStiletto on Fri Dec 07, 2007 3:47 am, edited 1 time in total.
jackS
Minister of Information
Minister of Information
Posts: 1895
Joined: Fri Jan 31, 2003 9:40 pm
Location: The land of tenure (and diaper changes)

Post by jackS »

ooooh, nice!

I am most appreciative.
JasonStiletto
Merchant
Merchant
Posts: 34
Joined: Sun Sep 24, 2006 8:17 pm
Contact:

Post by JasonStiletto »

My data viewer doesn't like the borked records much. This way everyone wins. If, you know, the corrections are actually made. ^^
Zeog
ISO Party Member
ISO Party Member
Posts: 453
Joined: Fri Jun 03, 2005 10:30 am
Location: Europe

Post by Zeog »

Does the validator check if the categories which are entered into the field for buyable cargo are actually present in the master parts list? It should display many errors for that as I haven't updated them yet. We had quite a lot of spelling errors there as well, so a find and replace strategy doesn't always work.

Thank you so much!
JasonStiletto
Merchant
Merchant
Posts: 34
Joined: Sun Sep 24, 2006 8:17 pm
Contact:

Post by JasonStiletto »

not as yet, it's no trouble to add it to the code base though. I'll release an expanded version today or tomorrow that checks the MPL and filenames are actually associated with files and the like, but for now the initial release already lists the most harmful problems of which there are several.
JasonStiletto
Merchant
Merchant
Posts: 34
Joined: Sun Sep 24, 2006 8:17 pm
Contact:

Post by JasonStiletto »

The version I'm working on now verifies cargo and cargo_import, cargo seems to pass okay but cargo import throws hundreds of errors. Most are, from what I can tell, valid errors, but

row 553 cargo_import Cat missions/Attack_Confed not found in master part list
row 553 cargo_import Cat missions/Attack_Luddite not found in master part list
row 554 cargo_import Cat missions/Rlaan_Milspec not found in master part list
row 555 cargo_import Cat missions/Rlaan not found in master part list
row 555 cargo_import Cat missions/Attack_Aera not found in master part list
row 555 cargo_import Cat missions/Attack_Luddite not found in master part list

Missions.. are.. cargo? This apparently deals with part of the program I don't understand yet.

After everything else I'm working a minor bug in my field size reporter, are there any other fields I should check specifically against data that isn't hard to get to? Suggestions welcome, precise suggestions more welcome.
Zeog
ISO Party Member
ISO Party Member
Posts: 453
Joined: Fri Jun 03, 2005 10:30 am
Location: Europe

Post by Zeog »

The valid errors are due to my let's say laziness. We reorganized the cargo items and for testing purposes (pictures etc) I made all the categories available at the starting planet which is an ocean planet. The other sites still show the old category names which should therefore be detected. Good! :-)

As for the missions. I'm not sure but I think that missions were introduced as a kind of cargo to rather "own" the mission, a hack once and now the standard.
JasonStiletto
Merchant
Merchant
Posts: 34
Joined: Sun Sep 24, 2006 8:17 pm
Contact:

Post by JasonStiletto »

if they're supposed to be there, so be it, but I don't know how to say they're valid, so.. erm.. so do you suppose I should just filter those results?

attached is the results from the latest build. I've got a plan whereby vsdataboy could flag everything invalid but implementation would probably take me about two weeks. For now I'm just going to worry about increasing core functionality and keep error reporting a separate package.

I'm going to try to get it to verify file existence for the endless meshes and what before I release the next verification tool. That should be able to do about 90% of what is needed. Range warnings could be put on individual fields too, but at the moment I don't see that as all that important. A correctly working tool should precede one that has flame stripes.
You do not have the required permissions to view the files attached to this post.
jackS
Minister of Information
Minister of Information
Posts: 1895
Joined: Fri Jan 31, 2003 9:40 pm
Location: The land of tenure (and diaper changes)

Post by jackS »

Finally got some time allocated to play with this - but I'm encountering something odd. The magic number doesn't seem to be matching up; the executable within the zip in the posted link isn't being accepted as being a compatible Win binary. I DL'd it again on the odd chance of bit rot, but the problem persists. What zip utility did you use to make the zip (just to sanity check)?.

If you have the time, could you directly mail my ministerofinformation account at gmail a copy of the windows binary for the validator?
JasonStiletto
Merchant
Merchant
Posts: 34
Joined: Sun Sep 24, 2006 8:17 pm
Contact:

Post by JasonStiletto »

apparently Info-ZIP from http://www.info-zip.org/ ..

Copyright (c) 1990-2006 Info-ZIP. All rights reserved.

probably version 2.32, or at least, that's what I've got installed today.

It was Cross compiled from Linux, and worked under Wine and the other puter. I'll verify that it runs under windows again tonight and e-mail you out a copy of the latest working version.
jackS
Minister of Information
Minister of Information
Posts: 1895
Joined: Fri Jan 31, 2003 9:40 pm
Location: The land of tenure (and diaper changes)

Post by jackS »

thanks much.

Looked over the initial output from your validator.
Made what I believe to be fixes (albeit, with typo correction, there's always the chance of a .... typo) to most of the ones flagged in the first of the two attachments. That units.csv is checked in. (MPL compliance enforcement pass has NOT been done yet, so it'll still belch errors)

Two notes:

row 570 column 69 - Radar_Range 3.00E+038<> Integer?
That's not actually really an error, as the radar range is a float.

I haven't fixed the cargo import wierdness on what I presume to be the mineable asteroid chunks (lines 599-606). I want to double-check as to whether something "special" was done for that, or if those are just typos before I munge anything.

Looking good to me so far :)
jackS
Minister of Information
Minister of Information
Posts: 1895
Joined: Fri Jan 31, 2003 9:40 pm
Location: The land of tenure (and diaper changes)

Post by jackS »

okidoke, one more update - the asteroid cargo bug should now be fixed in the units.csv in svn.

It was in the wrong category (should have been cargo entries, not cargo_import)

mystery solved (probably :) )
Zeog
ISO Party Member
ISO Party Member
Posts: 453
Joined: Fri Jun 03, 2005 10:30 am
Location: Europe

Post by Zeog »

Strange...
I still get the asteroid chunks errors (lines 599--606) but the rest is gone which still should be there! SVN Revision 10783. I'm issuing the command for the vs_validator from the data directory itself. However, I have to supply the "./" argument otherwise it doesn't find the data directory... I have the feeling that it is not working the way it was intented to. Any ideas?

Code: Select all

Finding data directory.
Using ./
Parser Begins
Header Read
Unknown fields:
Index
Header Processed.
row 574 column 69 - Radar_Range 30000000000.0<> Integer?
row 577 column 69 - Radar_Range 30000000000.0<> Integer?
row 599 cargo missionCargo falsenot known boolean value.
row 599 cargo missionCargo falsenot known boolean value.
row 600 cargo missionCargo falsenot known boolean value.
row 600 cargo missionCargo falsenot known boolean value.
row 601 cargo missionCargo falsenot known boolean value.
row 601 cargo missionCargo falsenot known boolean value.
row 602 cargo missionCargo falsenot known boolean value.
row 602 cargo missionCargo falsenot known boolean value.
row 603 cargo missionCargo falsenot known boolean value.
row 603 cargo missionCargo falsenot known boolean value.
row 604 cargo missionCargo falsenot known boolean value.
row 604 cargo missionCargo falsenot known boolean value.
row 605 cargo missionCargo falsenot known boolean value.
row 605 cargo missionCargo falsenot known boolean value.
row 606 cargo missionCargo falsenot known boolean value.
row 606 cargo missionCargo falsenot known boolean value.
Database processed.
JasonStiletto
Merchant
Merchant
Posts: 34
Joined: Sun Sep 24, 2006 8:17 pm
Contact:

Post by JasonStiletto »

jackS wrote:thanks much.

row 570 column 69 - Radar_Range 3.00E+038<> Integer?
That's not actually really an error, as the radar range is a float.
Changed units.inc to define that value as a float, so that line should validate properly now.
JasonStiletto
Merchant
Merchant
Posts: 34
Joined: Sun Sep 24, 2006 8:17 pm
Contact:

Post by JasonStiletto »

Zeog wrote:Strange...
I still get the asteroid chunks errors (lines 599--606) but the rest is gone which still should be there! SVN Revision 10783. I'm issuing the command for the vs_validator from the data directory itself. However, I have to supply the "./" argument otherwise it doesn't find the data directory... I have the feeling that it is not working the way it was intended to. Any ideas?
it should find the data directory when run from the data directory. the latest version that i uploaded didn't have MPL compliance checking.

here's the newest code

vs_validator_linux.rar
vs_validator_win32.rar
vs_validator_source.rar (in pascal)

Edit: Version bump so that linux version walks search correctly
JasonStiletto
Merchant
Merchant
Posts: 34
Joined: Sun Sep 24, 2006 8:17 pm
Contact:

Post by JasonStiletto »

updated x2, now does first generation checking to make sure meshes, directories, and what not are actually there.


vs_validator_linux.rar
vs_validator_win32.rar
vs_validator_source.rar (in pascal)
jackS
Minister of Information
Minister of Information
Posts: 1895
Joined: Fri Jan 31, 2003 9:40 pm
Location: The land of tenure (and diaper changes)

Post by jackS »

This version seems to have some problems finding the shield meshes (as in, it appears convinced that it can't find any of them)

Also, some of the primitive meshes didn't look like they were being found - does the tool search the shared "<datadir>/meshes" directory?
JasonStiletto
Merchant
Merchant
Posts: 34
Joined: Sun Sep 24, 2006 8:17 pm
Contact:

Post by JasonStiletto »

The only things with "Mesh" that aren't showing up for me are these:

Code: Select all

row 578 column 14 Mesh "landscape.bfxm" file not found
row 578 column 16 Rapid_Mesh "landscape_BSP.bfxm" file not found
row 578 column 17 BSP_Mesh "landscape_BSP.bfxm" file not found
row 579 column 14 Mesh "landscape.bfxm" file not found
row 579 column 16 Rapid_Mesh "landscape_BSP.bfxm" file not found
row 579 column 17 BSP_Mesh "landscape_BSP.bfxm" file not found
row 590 column 14 Mesh "" file not found
It may be a windows specific problem though. will test it on my windows box.

and as for searching the shared dir, err, no. I will see what I can do though.

EDIT: I freshened the code. I was trying to parse the other _Mesh like Mesh is parsed, so they were being read as empty strings that also should be files. The error was showing up under windows and not linux because adding nothing to a directory string gives a directory string, and a test if a file exists works for directories under linux but not windows - for directories.

anyway, fixed.

Edit2: Made it also check meshes subdirectory
JasonStiletto
Merchant
Merchant
Posts: 34
Joined: Sun Sep 24, 2006 8:17 pm
Contact:

Post by JasonStiletto »

if you find errors in it's evaluations, let me know and I'll try to fix it.
jackS
Minister of Information
Minister of Information
Posts: 1895
Joined: Fri Jan 31, 2003 9:40 pm
Location: The land of tenure (and diaper changes)

Post by jackS »

It finds all the meshes I expect it to find now :)

Great work! :)
Zeog
ISO Party Member
ISO Party Member
Posts: 453
Joined: Fri Jun 03, 2005 10:30 am
Location: Europe

Post by Zeog »

I started to replace the failing cargo categories in units.csv. With the validator which is now working very well by the way I noticed that the cargo category validation needs to be a little relaxed.
An example: There exist items the following cargo categories in the master_part_list.csv:
Natural_Products/Food/Area
Natural_Products/Food/Confed
but in units.csv Natural_Products/Food is allowed as well is however currently reported as an error.
JasonStiletto
Merchant
Merchant
Posts: 34
Joined: Sun Sep 24, 2006 8:17 pm
Contact:

Post by JasonStiletto »

Ah, I didn't realize. I'll modify it tonight, should be a new version by tomorrow sometime.
Paul
Merchant
Merchant
Posts: 34
Joined: Mon Jan 15, 2007 6:55 pm

Re: new tool: vs_validator

Post by Paul »

Hey folks, I'm having some trouble with this validator. When I try to run it, I get an error stating that it is not a valid win32 application. Perhaps the archive is corrupted. Could you update it please?
JasonStiletto
Merchant
Merchant
Posts: 34
Joined: Sun Sep 24, 2006 8:17 pm
Contact:

Post by JasonStiletto »

Sorry I disappeared, life exploded unexpectedly. Here is the current output and the freepascal source code for the validator. I'll see if I can make the data editor run again next week, coat it in a headly layer of GPL and upload it. I don't have a cross compiler going yet as I've just set pascal up on this computer.

Code: Select all

Finding data directory.
testing /home/stiletto/data4.x/units/units.csv
Using /home/stiletto/data4.x/
Parser Begins
Processing /home/stiletto/data4.x/master_part_list.csv no error checks - Done.
Reading /home/stiletto/data4.x/units/units.csv
Header Read
Unknown fields: 
 .. None.
Header Processed.
row 203 column 14 Mesh "dodo.bfxm" file not found
row 269 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 270 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 278 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 279 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 282 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 283 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 284 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 286 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 287 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 291 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 297 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 298 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 649 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 658 column 8 Hud_image "planet-ocean-hud.spr" file not found
row 669 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 670 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 671 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 672 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 673 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 674 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 676 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 677 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 681 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 682 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 683 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 685 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 686 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 687 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 689 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 690 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 691 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 692 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 693 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 698 column 8 Hud_image "Shipyard-hud.spr" file not found
row 701 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 702 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 703 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 704 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 705 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 713 column 2 Directory ./factions/upgrades/aera_mil_missions directory does not exist.
row 713 cargo_import Cat missions/Aera_Milspec not found in master part list
row 714 column 2 Directory ./factions/upgrades/aera_missions directory does not exist.
row 714 cargo_import Cat missions/Aera not found in master part list
row 714 cargo_import Cat missions/Attack_Confed not found in master part list
row 714 cargo_import Cat missions/Attack_Rlaan not found in master part list
row 715 column 2 Directory ./factions/upgrades/confed_mil_missions directory does not exist.
row 715 cargo_import Cat missions/Confed_Milspec not found in master part list
row 716 column 2 Directory ./factions/upgrades/confed_missions directory does not exist.
row 716 cargo_import Cat missions/Attack_Aera not found in master part list
row 716 cargo_import Cat missions/Attack_ISO not found in master part list
row 716 cargo_import Cat missions/Attack_Luddite not found in master part list
row 716 cargo_import Cat missions/Attack_Rlaan not found in master part list
row 716 cargo_import Cat missions/Confed not found in master part list
row 717 column 2 Directory ./factions/upgrades/iso_missions directory does not exist.
row 717 cargo_import Cat missions/Attack_Luddite not found in master part list
row 717 cargo_import Cat missions/ISO not found in master part list
row 718 column 2 Directory ./factions/upgrades/pirate_missions directory does not exist.
row 718 cargo_import Cat missions/Attack_Confed not found in master part list
row 718 cargo_import Cat missions/Attack_Luddite not found in master part list
row 718 cargo_import Cat missions/Pirate not found in master part list
row 719 column 2 Directory ./factions/upgrades/rlaan_mil_missions directory does not exist.
row 719 cargo_import Cat missions/Rlaan_Milspec not found in master part list
row 720 column 2 Directory ./factions/upgrades/rlaan_missions directory does not exist.
row 720 cargo_import Cat missions/Attack_Aera not found in master part list
row 720 cargo_import Cat missions/Attack_Luddite not found in master part list
row 720 cargo_import Cat missions/Rlaan not found in master part list
row 721 column 2 Directory ./factions/upgrades/standard_missions directory does not exist.
row 721 cargo_import Cat missions/Attack_Pirate not found in master part list
row 721 cargo_import Cat missions/Merchant not found in master part list
row 721 cargo_import Cat missions/Neutral not found in master part list
row 723 column 2 Directory ./contraband directory does not exist.
row 724 column 2 Directory ./contraband directory does not exist.
row 725 column 2 Directory ./contraband directory does not exist.
row 728 cargo_import Cat Contraband/Luddite not found in master part list
row 737 column 8 Hud_image "base_locationmarker.spr" file not found
row 738 column 8 Hud_image "base_locationmarker.spr" file not found
row 739 column 8 Hud_image "base_locationmarker.spr" file not found
row 803 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 807 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 808 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 810 column 2 Directory ./h496 directory does not exist.
Database processed.
Max Fieldlengths (Str only)
---------------------------
cargo.Category: 45
cargo.description: 50
cargo.filename: 28
cargo_import.Cat: 49
light.meshfile: 18
mounts.SIZE: 50
mounts.type: 28
subunit.filename: 35
units.BSP_Mesh: 21
units.Cockpit: 18
units.Combat_Role: 12
units.Directory: 47
units.Explosion: 19
units.Hud_Functionality: 66
units.Hud_image: 36
units.Index: 39
units.Max_Hud_Functionality: 65
units.Mesh: 40
units.Name: 36
units.Object_Type: 19
units.STATUS: 33
units.Shield_Mesh: 23
upgrades.filename: 26
You do not have the required permissions to view the files attached to this post.
loki1950
The Shepherd
Posts: 5841
Joined: Fri May 13, 2005 8:37 pm
Location: Ottawa
Contact:

Post by loki1950 »

Thx Jason should make someone happy when we start combing the data tree prior to release.

Enjoy the Choice :)
my box::HP Envy i5-6400 @2Q70GHzx4 8 Gb ram/1 Tb(Win10 64)/3 Tb Mint 19.2/GTX745 4Gb acer S243HL K222HQL
Q8200/Asus P5QDLX/8 Gb ram/WD 2Tb 2-500 G HD/GF GT640 2Gb Mint 17.3 64 bit Win 10 32 bit acer and Lenovo ideapad 320-15ARB Win 10/Mint 19.2
Post Reply