new tool: vs_validator

A forum for the discussion and development of programs to assist working on or playing with the Vegastrike engine and data sets.

new tool: vs_validator

Postby JasonStiletto » Sun Oct 29, 2006 12:44 pm

updated 12-6-07
vs_validator-win32.zip
vs_validator-linux32.zip
vs_validator-source.zip (GPLicious and Pascal)


validates contents of units.csv against my understanding of them. Which isn't horrible.

current output:
Code: Select all
Finding data directory.
testing /home/stiletto/data4.x/units/units.csv
Using /home/stiletto/data4.x/
Parser Begins
Processing /home/stiletto/data4.x/master_part_list.csv no error checks - Done.
Reading /home/stiletto/data4.x/units/units.csv
Header Read
Unknown fields:
 .. None.
Header Processed.
row 657 column 8 Hud_image "planet-ocean-hud.spr" file not found
row 697 column 8 Hud_image "Shipyard-hud.spr" file not found
row 713 column 2 Directory ./contraband directory does not exist.
row 714 column 2 Directory ./contraband directory does not exist.
row 715 column 2 Directory ./contraband directory does not exist.
row 718 cargo_import Cat Contraband/Luddite not found in master part list
row 727 column 8 Hud_image "base_locationmarker.spr" file not found
row 728 column 8 Hud_image "base_locationmarker.spr" file not found
row 729 column 8 Hud_image "base_locationmarker.spr" file not found
row 800 column 2 Directory ./h496 directory does not exist.
Database processed.
Max Fieldlengths (Str only)
---------------------------
cargo.Category: 45
cargo.description: 50
cargo.filename: 28
cargo_import.Cat: 49
light.meshfile: 18
mounts.SIZE: 50
mounts.type: 28
subunit.filename: 35
units.BSP_Mesh: 21
units.Cockpit: 18
units.Combat_Role: 12
units.Directory: 47
units.Explosion: 19
units.Hud_Functionality: 66
units.Hud_image: 36
units.Index: 39
units.Max_Hud_Functionality: 65
units.Mesh: 40
units.Name: 36
units.Object_Type: 19
units.STATUS: 33
units.Shield_Mesh: 23
upgrades.filename: 26
Last edited by JasonStiletto on Thu Dec 06, 2007 8:47 pm, edited 1 time in total.
JasonStiletto
Merchant
Merchant
 
Posts: 34
Topics: 3
Joined: Sun Sep 24, 2006 1:17 pm

Share On:

Share on Facebook Facebook Share on Twitter Twitter Share on Digg Digg

Postby jackS » Mon Oct 30, 2006 6:02 pm

ooooh, nice!

I am most appreciative.
User avatar
jackS
Minister of Information
Minister of Information
 
Posts: 1895
Topics: 56
Joined: Fri Jan 31, 2003 2:40 pm
Location: The center of Centre county

Postby JasonStiletto » Mon Oct 30, 2006 7:11 pm

My data viewer doesn't like the borked records much. This way everyone wins. If, you know, the corrections are actually made. ^^
JasonStiletto
Merchant
Merchant
 
Posts: 34
Topics: 3
Joined: Sun Sep 24, 2006 1:17 pm

Postby Zeog » Tue Oct 31, 2006 5:08 am

Does the validator check if the categories which are entered into the field for buyable cargo are actually present in the master parts list? It should display many errors for that as I haven't updated them yet. We had quite a lot of spelling errors there as well, so a find and replace strategy doesn't always work.

Thank you so much!
Zeog
ISO Party Member
ISO Party Member
 
Posts: 453
Topics: 13
Joined: Fri Jun 03, 2005 3:30 am
Location: Europe

Postby JasonStiletto » Tue Oct 31, 2006 10:00 am

not as yet, it's no trouble to add it to the code base though. I'll release an expanded version today or tomorrow that checks the MPL and filenames are actually associated with files and the like, but for now the initial release already lists the most harmful problems of which there are several.
JasonStiletto
Merchant
Merchant
 
Posts: 34
Topics: 3
Joined: Sun Sep 24, 2006 1:17 pm

Postby JasonStiletto » Fri Nov 03, 2006 9:51 am

The version I'm working on now verifies cargo and cargo_import, cargo seems to pass okay but cargo import throws hundreds of errors. Most are, from what I can tell, valid errors, but

row 553 cargo_import Cat missions/Attack_Confed not found in master part list
row 553 cargo_import Cat missions/Attack_Luddite not found in master part list
row 554 cargo_import Cat missions/Rlaan_Milspec not found in master part list
row 555 cargo_import Cat missions/Rlaan not found in master part list
row 555 cargo_import Cat missions/Attack_Aera not found in master part list
row 555 cargo_import Cat missions/Attack_Luddite not found in master part list

Missions.. are.. cargo? This apparently deals with part of the program I don't understand yet.

After everything else I'm working a minor bug in my field size reporter, are there any other fields I should check specifically against data that isn't hard to get to? Suggestions welcome, precise suggestions more welcome.
JasonStiletto
Merchant
Merchant
 
Posts: 34
Topics: 3
Joined: Sun Sep 24, 2006 1:17 pm

Postby Zeog » Fri Nov 03, 2006 12:14 pm

The valid errors are due to my let's say laziness. We reorganized the cargo items and for testing purposes (pictures etc) I made all the categories available at the starting planet which is an ocean planet. The other sites still show the old category names which should therefore be detected. Good! :-)

As for the missions. I'm not sure but I think that missions were introduced as a kind of cargo to rather "own" the mission, a hack once and now the standard.
Zeog
ISO Party Member
ISO Party Member
 
Posts: 453
Topics: 13
Joined: Fri Jun 03, 2005 3:30 am
Location: Europe

Postby JasonStiletto » Fri Nov 03, 2006 12:34 pm

if they're supposed to be there, so be it, but I don't know how to say they're valid, so.. erm.. so do you suppose I should just filter those results?

attached is the results from the latest build. I've got a plan whereby vsdataboy could flag everything invalid but implementation would probably take me about two weeks. For now I'm just going to worry about increasing core functionality and keep error reporting a separate package.

I'm going to try to get it to verify file existence for the endless meshes and what before I release the next verification tool. That should be able to do about 90% of what is needed. Range warnings could be put on individual fields too, but at the moment I don't see that as all that important. A correctly working tool should precede one that has flame stripes.
Attachments
checked.txt
(217.4 KiB) Downloaded 112 times
JasonStiletto
Merchant
Merchant
 
Posts: 34
Topics: 3
Joined: Sun Sep 24, 2006 1:17 pm

Postby jackS » Thu Nov 16, 2006 11:50 pm

Finally got some time allocated to play with this - but I'm encountering something odd. The magic number doesn't seem to be matching up; the executable within the zip in the posted link isn't being accepted as being a compatible Win binary. I DL'd it again on the odd chance of bit rot, but the problem persists. What zip utility did you use to make the zip (just to sanity check)?.

If you have the time, could you directly mail my ministerofinformation account at gmail a copy of the windows binary for the validator?
User avatar
jackS
Minister of Information
Minister of Information
 
Posts: 1895
Topics: 56
Joined: Fri Jan 31, 2003 2:40 pm
Location: The center of Centre county

Postby JasonStiletto » Fri Nov 17, 2006 12:12 am

apparently Info-ZIP from http://www.info-zip.org/ ..

Copyright (c) 1990-2006 Info-ZIP. All rights reserved.

probably version 2.32, or at least, that's what I've got installed today.

It was Cross compiled from Linux, and worked under Wine and the other puter. I'll verify that it runs under windows again tonight and e-mail you out a copy of the latest working version.
JasonStiletto
Merchant
Merchant
 
Posts: 34
Topics: 3
Joined: Sun Sep 24, 2006 1:17 pm

Postby jackS » Fri Nov 17, 2006 12:42 am

thanks much.

Looked over the initial output from your validator.
Made what I believe to be fixes (albeit, with typo correction, there's always the chance of a .... typo) to most of the ones flagged in the first of the two attachments. That units.csv is checked in. (MPL compliance enforcement pass has NOT been done yet, so it'll still belch errors)

Two notes:

row 570 column 69 - Radar_Range 3.00E+038<> Integer?
That's not actually really an error, as the radar range is a float.

I haven't fixed the cargo import wierdness on what I presume to be the mineable asteroid chunks (lines 599-606). I want to double-check as to whether something "special" was done for that, or if those are just typos before I munge anything.

Looking good to me so far :)
User avatar
jackS
Minister of Information
Minister of Information
 
Posts: 1895
Topics: 56
Joined: Fri Jan 31, 2003 2:40 pm
Location: The center of Centre county

Postby jackS » Fri Nov 17, 2006 1:00 am

okidoke, one more update - the asteroid cargo bug should now be fixed in the units.csv in svn.

It was in the wrong category (should have been cargo entries, not cargo_import)

mystery solved (probably :) )
User avatar
jackS
Minister of Information
Minister of Information
 
Posts: 1895
Topics: 56
Joined: Fri Jan 31, 2003 2:40 pm
Location: The center of Centre county

Postby Zeog » Fri Nov 17, 2006 5:04 pm

Strange...
I still get the asteroid chunks errors (lines 599--606) but the rest is gone which still should be there! SVN Revision 10783. I'm issuing the command for the vs_validator from the data directory itself. However, I have to supply the "./" argument otherwise it doesn't find the data directory... I have the feeling that it is not working the way it was intented to. Any ideas?
Code: Select all
Finding data directory.
Using ./
Parser Begins
Header Read
Unknown fields:
Index
Header Processed.
row 574 column 69 - Radar_Range 30000000000.0<> Integer?
row 577 column 69 - Radar_Range 30000000000.0<> Integer?
row 599 cargo missionCargo falsenot known boolean value.
row 599 cargo missionCargo falsenot known boolean value.
row 600 cargo missionCargo falsenot known boolean value.
row 600 cargo missionCargo falsenot known boolean value.
row 601 cargo missionCargo falsenot known boolean value.
row 601 cargo missionCargo falsenot known boolean value.
row 602 cargo missionCargo falsenot known boolean value.
row 602 cargo missionCargo falsenot known boolean value.
row 603 cargo missionCargo falsenot known boolean value.
row 603 cargo missionCargo falsenot known boolean value.
row 604 cargo missionCargo falsenot known boolean value.
row 604 cargo missionCargo falsenot known boolean value.
row 605 cargo missionCargo falsenot known boolean value.
row 605 cargo missionCargo falsenot known boolean value.
row 606 cargo missionCargo falsenot known boolean value.
row 606 cargo missionCargo falsenot known boolean value.
Database processed.
Zeog
ISO Party Member
ISO Party Member
 
Posts: 453
Topics: 13
Joined: Fri Jun 03, 2005 3:30 am
Location: Europe

Postby JasonStiletto » Fri Nov 17, 2006 6:34 pm

jackS wrote:thanks much.

row 570 column 69 - Radar_Range 3.00E+038<> Integer?
That's not actually really an error, as the radar range is a float.


Changed units.inc to define that value as a float, so that line should validate properly now.
JasonStiletto
Merchant
Merchant
 
Posts: 34
Topics: 3
Joined: Sun Sep 24, 2006 1:17 pm

Postby JasonStiletto » Fri Nov 17, 2006 7:17 pm

Zeog wrote:Strange...
I still get the asteroid chunks errors (lines 599--606) but the rest is gone which still should be there! SVN Revision 10783. I'm issuing the command for the vs_validator from the data directory itself. However, I have to supply the "./" argument otherwise it doesn't find the data directory... I have the feeling that it is not working the way it was intended to. Any ideas?


it should find the data directory when run from the data directory. the latest version that i uploaded didn't have MPL compliance checking.

here's the newest code

vs_validator_linux.rar
vs_validator_win32.rar
vs_validator_source.rar (in pascal)

Edit: Version bump so that linux version walks search correctly
JasonStiletto
Merchant
Merchant
 
Posts: 34
Topics: 3
Joined: Sun Sep 24, 2006 1:17 pm

Postby JasonStiletto » Sat Nov 18, 2006 6:48 pm

updated x2, now does first generation checking to make sure meshes, directories, and what not are actually there.


vs_validator_linux.rar
vs_validator_win32.rar
vs_validator_source.rar (in pascal)
JasonStiletto
Merchant
Merchant
 
Posts: 34
Topics: 3
Joined: Sun Sep 24, 2006 1:17 pm

Postby jackS » Sat Nov 18, 2006 7:21 pm

This version seems to have some problems finding the shield meshes (as in, it appears convinced that it can't find any of them)

Also, some of the primitive meshes didn't look like they were being found - does the tool search the shared "<datadir>/meshes" directory?
User avatar
jackS
Minister of Information
Minister of Information
 
Posts: 1895
Topics: 56
Joined: Fri Jan 31, 2003 2:40 pm
Location: The center of Centre county

Postby JasonStiletto » Sat Nov 18, 2006 8:09 pm

The only things with "Mesh" that aren't showing up for me are these:

Code: Select all
row 578 column 14 Mesh "landscape.bfxm" file not found
row 578 column 16 Rapid_Mesh "landscape_BSP.bfxm" file not found
row 578 column 17 BSP_Mesh "landscape_BSP.bfxm" file not found
row 579 column 14 Mesh "landscape.bfxm" file not found
row 579 column 16 Rapid_Mesh "landscape_BSP.bfxm" file not found
row 579 column 17 BSP_Mesh "landscape_BSP.bfxm" file not found
row 590 column 14 Mesh "" file not found


It may be a windows specific problem though. will test it on my windows box.

and as for searching the shared dir, err, no. I will see what I can do though.

EDIT: I freshened the code. I was trying to parse the other _Mesh like Mesh is parsed, so they were being read as empty strings that also should be files. The error was showing up under windows and not linux because adding nothing to a directory string gives a directory string, and a test if a file exists works for directories under linux but not windows - for directories.

anyway, fixed.

Edit2: Made it also check meshes subdirectory
JasonStiletto
Merchant
Merchant
 
Posts: 34
Topics: 3
Joined: Sun Sep 24, 2006 1:17 pm

Postby JasonStiletto » Sat Nov 18, 2006 11:51 pm

if you find errors in it's evaluations, let me know and I'll try to fix it.
JasonStiletto
Merchant
Merchant
 
Posts: 34
Topics: 3
Joined: Sun Sep 24, 2006 1:17 pm

Postby jackS » Sun Nov 19, 2006 12:18 am

It finds all the meshes I expect it to find now :)

Great work! :)
User avatar
jackS
Minister of Information
Minister of Information
 
Posts: 1895
Topics: 56
Joined: Fri Jan 31, 2003 2:40 pm
Location: The center of Centre county

Postby Zeog » Tue Nov 28, 2006 2:01 pm

I started to replace the failing cargo categories in units.csv. With the validator which is now working very well by the way I noticed that the cargo category validation needs to be a little relaxed.
An example: There exist items the following cargo categories in the master_part_list.csv:
Natural_Products/Food/Area
Natural_Products/Food/Confed
but in units.csv Natural_Products/Food is allowed as well is however currently reported as an error.
Zeog
ISO Party Member
ISO Party Member
 
Posts: 453
Topics: 13
Joined: Fri Jun 03, 2005 3:30 am
Location: Europe

Postby JasonStiletto » Tue Nov 28, 2006 5:04 pm

Ah, I didn't realize. I'll modify it tonight, should be a new version by tomorrow sometime.
JasonStiletto
Merchant
Merchant
 
Posts: 34
Topics: 3
Joined: Sun Sep 24, 2006 1:17 pm

Re: new tool: vs_validator

Postby Paul » Fri Feb 02, 2007 5:52 pm

Hey folks, I'm having some trouble with this validator. When I try to run it, I get an error stating that it is not a valid win32 application. Perhaps the archive is corrupted. Could you update it please?
Paul
Merchant
Merchant
 
Posts: 34
Topics: 5
Joined: Mon Jan 15, 2007 11:55 am

Postby JasonStiletto » Sun Nov 18, 2007 12:15 am

Sorry I disappeared, life exploded unexpectedly. Here is the current output and the freepascal source code for the validator. I'll see if I can make the data editor run again next week, coat it in a headly layer of GPL and upload it. I don't have a cross compiler going yet as I've just set pascal up on this computer.

Code: Select all
Finding data directory.
testing /home/stiletto/data4.x/units/units.csv
Using /home/stiletto/data4.x/
Parser Begins
Processing /home/stiletto/data4.x/master_part_list.csv no error checks - Done.
Reading /home/stiletto/data4.x/units/units.csv
Header Read
Unknown fields:
 .. None.
Header Processed.
row 203 column 14 Mesh "dodo.bfxm" file not found
row 269 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 270 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 278 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 279 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 282 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 283 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 284 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 286 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 287 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 291 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 297 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 298 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 649 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 658 column 8 Hud_image "planet-ocean-hud.spr" file not found
row 669 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 670 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 671 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 672 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 673 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 674 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 676 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 677 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 681 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 682 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 683 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 685 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 686 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 687 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 689 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 690 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 691 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 692 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 693 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 698 column 8 Hud_image "Shipyard-hud.spr" file not found
row 701 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 702 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 703 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 704 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 705 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 713 column 2 Directory ./factions/upgrades/aera_mil_missions directory does not exist.
row 713 cargo_import Cat missions/Aera_Milspec not found in master part list
row 714 column 2 Directory ./factions/upgrades/aera_missions directory does not exist.
row 714 cargo_import Cat missions/Aera not found in master part list
row 714 cargo_import Cat missions/Attack_Confed not found in master part list
row 714 cargo_import Cat missions/Attack_Rlaan not found in master part list
row 715 column 2 Directory ./factions/upgrades/confed_mil_missions directory does not exist.
row 715 cargo_import Cat missions/Confed_Milspec not found in master part list
row 716 column 2 Directory ./factions/upgrades/confed_missions directory does not exist.
row 716 cargo_import Cat missions/Attack_Aera not found in master part list
row 716 cargo_import Cat missions/Attack_ISO not found in master part list
row 716 cargo_import Cat missions/Attack_Luddite not found in master part list
row 716 cargo_import Cat missions/Attack_Rlaan not found in master part list
row 716 cargo_import Cat missions/Confed not found in master part list
row 717 column 2 Directory ./factions/upgrades/iso_missions directory does not exist.
row 717 cargo_import Cat missions/Attack_Luddite not found in master part list
row 717 cargo_import Cat missions/ISO not found in master part list
row 718 column 2 Directory ./factions/upgrades/pirate_missions directory does not exist.
row 718 cargo_import Cat missions/Attack_Confed not found in master part list
row 718 cargo_import Cat missions/Attack_Luddite not found in master part list
row 718 cargo_import Cat missions/Pirate not found in master part list
row 719 column 2 Directory ./factions/upgrades/rlaan_mil_missions directory does not exist.
row 719 cargo_import Cat missions/Rlaan_Milspec not found in master part list
row 720 column 2 Directory ./factions/upgrades/rlaan_missions directory does not exist.
row 720 cargo_import Cat missions/Attack_Aera not found in master part list
row 720 cargo_import Cat missions/Attack_Luddite not found in master part list
row 720 cargo_import Cat missions/Rlaan not found in master part list
row 721 column 2 Directory ./factions/upgrades/standard_missions directory does not exist.
row 721 cargo_import Cat missions/Attack_Pirate not found in master part list
row 721 cargo_import Cat missions/Merchant not found in master part list
row 721 cargo_import Cat missions/Neutral not found in master part list
row 723 column 2 Directory ./contraband directory does not exist.
row 724 column 2 Directory ./contraband directory does not exist.
row 725 column 2 Directory ./contraband directory does not exist.
row 728 cargo_import Cat Contraband/Luddite not found in master part list
row 737 column 8 Hud_image "base_locationmarker.spr" file not found
row 738 column 8 Hud_image "base_locationmarker.spr" file not found
row 739 column 8 Hud_image "base_locationmarker.spr" file not found
row 803 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 807 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 808 cargo_import Cat upgrades/Hull_Upgrades not found in master part list
row 810 column 2 Directory ./h496 directory does not exist.
Database processed.
Max Fieldlengths (Str only)
---------------------------
cargo.Category: 45
cargo.description: 50
cargo.filename: 28
cargo_import.Cat: 49
light.meshfile: 18
mounts.SIZE: 50
mounts.type: 28
subunit.filename: 35
units.BSP_Mesh: 21
units.Cockpit: 18
units.Combat_Role: 12
units.Directory: 47
units.Explosion: 19
units.Hud_Functionality: 66
units.Hud_image: 36
units.Index: 39
units.Max_Hud_Functionality: 65
units.Mesh: 40
units.Name: 36
units.Object_Type: 19
units.STATUS: 33
units.Shield_Mesh: 23
upgrades.filename: 26
Attachments
validator.tar.bz2
(27.33 KiB) Downloaded 101 times
JasonStiletto
Merchant
Merchant
 
Posts: 34
Topics: 3
Joined: Sun Sep 24, 2006 1:17 pm

Postby loki1950 » Sun Nov 18, 2007 8:20 am

Thx Jason should make someone happy when we start combing the data tree prior to release.

Enjoy the Choice :)
my box::HP Envy i5-6400 @2Q70GHzx4 8 Gb ram/1 Tb(Win10 64)/3 Tb Mint 18/GTX745 4Gb acer S243HL K222HQL
Q8200/Asus P5QDLX/8 Gb ram/WD 2Tb 2-500 G HD/GF GT640 2Gb Mint 17.3 64 bit Win 10 32 bit acer and DELL E6400 4GB ram/100 GB HD Mint 17.3 6
User avatar
loki1950
The Shepherd
 
Posts: 5727
Topics: 51
Joined: Fri May 13, 2005 1:37 pm
Location: Ottawa


Next

Return to Tools & Utilities

Who is online

Users browsing this forum: No registered users and 1 guest