Yet another CSV viewer and editor for linux/windows/mac

ismeagain · Post by **ismeagain** » Mon Mar 26, 2007 6:53 pm

http://developer.berlios.de/projects/csvomatic/
I made a berlios project page for my CSV editor. It's written in wxpython with the DSV addon so it's completely free software for any platform. Imagine it as a fusion between notepad and microsoft excel - except without the irritating bugs. I used this program for a few years under linux and windows, so there's probably a few minor bugs, but no show stoppers. Please read the documentation as the program will be a bit confusing for the average user. I used to load database tables in this program and it will load a file as big as the memory in your computer with a 1:1 ratio of memory used to file loaded. 4:1 when you are displaying, so don't edit/view more than 100000 rows at a time, if you are dealing with gigabytes of data. I used this the other day to edit my serialized xml and add an upgrade to my ship that I couldn't find to purchase anywhere. Blah blah built in find functions etc Regex pattern matching and so on. An entirely over-engineered piece of software.

Post by **loki1950** » Mon Mar 26, 2007 7:09 pm

Welcome ismeagain the download link at http://developer.berlios.de/projects/csvomatic/ seems to be broken i get a "404"

from both severs offered.

Enjoy the Choice

ismeagain · Post by **ismeagain** » Mon Mar 26, 2007 7:33 pm

*sigh* yes, you are right loki.....grrr I will try and fix that crud. It worked for me last month

ismeagain · Post by **ismeagain** » Mon Mar 26, 2007 7:48 pm

ok I uhh, well, I fooled around on berlios to try and get my script back up there somehow. I failed. But! I did post a new copy of my script hah hah under a different project release. Perhaps that's how I am supposed to do these things? Well, try this out for size!
http://prdownload.berlios.de/csvomatic/CSVOMATIC.py
I tested it and I could download the script fine, now.

Post by **loki1950** » Mon Mar 26, 2007 8:13 pm

Bingo thx will try it out and comment

Enjoy the Choice

ismeagain · Post by **ismeagain** » Mon Mar 26, 2007 9:08 pm

Here's the document I made on how to use it. The program only seems complicated, there's not much to it, in reality. This is a description of every pull down menu command.
Menu->command
description
Thanks for your download. It gives my program a purpose in life.

file->open
opens a file to edit
file->save
saves the file
file->exit
exits.
row->drop
deletes a specified range of rows from top to bottom
row->rows
views a specified range of data rows in a grid
row->row filter
deletes the rows of data that have been matched in a find or you
have selected with a mouse click in the grid
column->drop
deletes a specified range of columns from left to right
column->columns
views a specified range of columns in a grid
column->column filter
deletes the columns of data that have been matched in a find or you
have selected with a mouse click in the grid
data->find data
Matches and displays the data found in the current file using a
regular expression. These matches can be deleted using the row or column
filter.
data->import data
This loads the current text in the main notepad window, into memory
for display or searching purposes.
data->export data
This displays all the current data contents in the notepad window so
it can be edited or copy and pasted.
data->data options
Use this to set options like what your delimiter is(default tab) and
which row in the file is going to be the column headers(default 0).

ismeagain · Post by **ismeagain** » Mon Mar 26, 2007 9:13 pm

The mouse cursor performs almost all tasks the notepad window does, if you select a row with the right mouse button then bring up a context menu with the left mouse button. Click the mouse on the column headers or row numbering.

ismeagain · Post by **ismeagain** » Wed Mar 28, 2007 10:30 pm

here's a quick screenshot of the data find grid/data display. I resized the image, as it's original size was 2880x900(my native screen resolution). As you can see, no big fancy bullshit program here, only viewing and editing delimited data. The DSV addon is called "python-DSV" in case anyone was having trouble finding it.

targ collective · Post by **targ collective** » Thu May 10, 2007 2:41 pm

So... How would you go about running this gem on, say, Windows XP? There's no executable or file of similar function. Do I need another piece of software to use this?

Short answer is you can't. Long answer is you really really can't, not in a million years. Then I found this link - http://www.goldb.org/goldblog/2007/02/2 ... ables.aspx - which someone generous and python literate might just click on to expand the user base a bit.

Please? I don't have Python on my machine, nor do I have any idea how to get it.

Post by **ace123** » Thu May 10, 2007 11:07 pm

It should be really simple...

Go to the Python site: http://www.python.org/download/
and Install the latest version (2.5.x)

Then, save the ".py" file, and you should be able to just run it once you have python installed.

Python is pretty cross platform, so you can likely get it to work.

wowsher · Post by **wowsher** » Sat May 12, 2007 2:40 pm

wxpython with the DSV addon is also required according to a post earlier not just the python install.

wxpython can be found here and is an easy install. www.wxpython.org
DSV addon can be found here http://python-dsv.sourceforge.net/

I can compile this py to an exe for windows if that is ok with ismegain.

thanks
wowsher

targ collective · Post by **targ collective** » Mon May 14, 2007 9:57 am

First, my apologies. I should have read this thread properly rather than just skimmed it.

Now I've installed Python 2.5 and WXpython, after which I installed the DSV addon. But it seems some files your code relies on simply are not present - it says x lines loaded okay but the widjets for the cells and the data in them refuses to load. I also get a message saying my wxPython version is obsolete or something (update wxsomething as soon as possible) which leaves me in a fix.

Compiling to an .exe without asking the DSV and wxPython team first would likely be somewhat rude. I suggested that without realising the program's dependencies...

Sorry I'm so green in this; I've never worked with Python before (unless you can count twiddling a couple of values in Modules).

wowsher · Post by **wowsher** » Mon May 14, 2007 2:44 pm

Actually wxpython and DSV should not mind how they are used as they are both open source libraries which are used to build programs for any use. The only person who really may or may not care is ismeagain.

I did try to run this and get the deprication warning, load the file, it lists x rows loaded but no further error or grid with data shows up....

No worries we all started somewhere and I for one am always glad to see someone try!

thanks
wowsher

targ collective · Post by **targ collective** » Tue May 15, 2007 1:54 pm

Oh, the python file works alright. The problem's our end and with the lack of a proper dependency list.

It tells me I need to use the wx.dataset or some such. I'm certain that, when installed, this dataset will fix the problems.

Potential solutions: Develop a wxpython version.

Include a link to this new wx.dataset that has apparently trumped wxpython. The official wxwidgets page poits to wxpython; wxpython points to wx.dataset; wx.dataset is unlinked.

Is it Windows-compatible? Could ismeagain do a round-robin find/replace in the code to replace the new, shiny commands with the old, traditional syntax now covered in dust? Windows users stay tuned...

ismeagain · Post by **ismeagain** » Fri Jun 08, 2007 3:53 am

oh hey guys and girls! I didn't even realise there were people struggling over here with the software. I run it in fedora 6 with the latest software installed and I originally developed it in windows xp with wxwidgets. I used to make binary builds all the time in windows and I can probably muster up the courage to do it again. Unless someone else has already done it, in which case I won't bother. The code is all based on gpl and python lisence and I personally give anyone permission to fiddle with the code provided in my berlios project(Just don't claim you wrote it). It's a program I wrote in my spare time to deal with large delimited data sets. I am working on a windows build as we speak, it'll take a couple days to figure it out again.

targ collective · Post by **targ collective** » Fri Jun 08, 2007 10:50 am

That's excellent news! Could you do us all a favour and link directly to all the dependencies?

ismeagain · Post by **ismeagain** » Sat Jun 09, 2007 2:13 am

oh man...there's a whirlwind of hell coming my way with this new namespace change in wxpython. The windows users really tipped me off, I didn't even have to worry about that in linux...yet. Anyways, this windows build requires nothing besides what is inside the zip file. I tested it on windows 2000 and built it on windows xp with py2exe.
http://prdownload.berlios.de/csvomatic/ ... -win32.zip
----unzip to a directory then run CSVOMATIC.exe
For anyone working with the source, I will make a collection of links to the requirements that make it work.

python: www.python.org
wxpython: www.wxpython.org
python-DSV: http://python-dsv.sourceforge.net/

These are the holy trinity of CSV-O-MATIC.
Yes, I realise there's a bug or two left in there but I can work on it, now that there are users once again giving me purpose. Yes...yes...I know python-dsv uses wx namespaces incorrectly still that's why I put the change on hold, until python-DSV changes theirs too. Of course, I could always contact the author and ask about it. Feedback welcome.

targ collective · Post by **targ collective** » Sun Jun 10, 2007 4:20 pm

...

Fix the bugs!

Seriously, it's great to have something to replace Excel (which truncates Master_Part_List). Things to consider: Optional prefix-suffix term replacement on find-replace would be absolutely fantastic, as would some form of skin support. IE a term might need to be replaced/appended and in all places in the file it might read FooA FooB except in one place where it reads FooB FooC. Don't ask me why, it's too complicated, it's to do with terms of reference.

Other than that the ability to fill out my tax return, wash the car, cook, entertain guests and maintain the holiday home always pleases!

ismeagain · Post by **ismeagain** » Sun Jun 10, 2007 6:25 pm

I do believe you are talking about regular expression pattern matching and replacement. I do have rudimentary regex matching, so it shouldn't be too hard to extend that into search and replace. I will work on killing the bug I can remember. If you click delete on the popup menu it changes the column headers to A B C D E even though the data doesn't change. It's not a show stopper, just wierd. I know why it happens...I forgot to perform a header column refresh after removing header 1. The only reason I never made regex search and replace is because I always used visual studio in windows to perform that task.
My personal pet peeve was microsoft excel's inability to display over 500 columns of data. Also microsoft excel could never load a 1 gigabyte file. It would die, I've tried. Ohhh I just remembered another irritating bug! If the column you are trying to edit is wider than the screen it just keeps scrolling back and forth. OK 2 bugs to kill, I'll be on that.

targ collective · Post by **targ collective** » Sun Jun 10, 2007 8:00 pm

That's great! Your tool requires one .dll which is not in the zip - I've attached it here; add it to your zip package - and apart from that I've had relatively few problems getting it running.

Things it needs (Interface):

It needs to load all columns automatically in the selected file, and default to comma separated format, at least for the Vegastrike community. The first box - the configurator - needs combining into the main window under a submenu. The load dialogue needs to include .csv files under text files, at least. Intelligent format recognition - defaulting to comma seperated values for .csv files, for example - would be a plus.

The main window has no Maximise/Restore button. The main window defaults windowed. Both of these items need fixing.

The program should display the Load... box immediately on load, preferably with the format selection within the same Load... box (defaulting to comma). The program should have some provision for adopting either of the two behaviours - load from a set default directory (with me this would be Privateer Universe; for the Vegastrike devs most likely Vegastrike) or remember last used directory.

Things it needs (Program)

First and highest priority, suppress that deprecation warning. No-one likes file bloat from program logs.

(This is lowest priority, but nice when you get the rest done.) Some form of basic formula support, ala MS Excel, would be appreciated. If the user types in a formula prefix in a cell (let us say =, as in Excel) and then clicks out of the cell, the cell's original contents should be restored. If the formula's structure is invalid, the cell should display #ERROR, or similar, under no circumstances trapping the cursor. Ability to copy, paste and drag values and formulas, both absolute and non-absolute (as well as having enough intelligence to see a number pattern, for example in 1 and 2 creating 3, 4, 5 on drag-replace) would all be nice, going on essential save the last item which will see little use if implemented.

EDIT: You may want to find out if another open-source developer has created routines for formula calcs; almost certainly someone has, somewhere, and you might be able to 'borrow' them, with permission of course. /EDIT

That should be enough to keep you busy for the next couple of months. Plus bugfixes. Do us a favour please, and post an update every 2 - 5 days or so; can't speak for the others but I'd all but given up hope of hearing from you again. Can't have that. At least participate in the forums, okay?

Oh, and thanks - this has the makings of a great tool!

ismeagain · Post by **ismeagain** » Tue Jun 12, 2007 6:39 pm

ok! I've taken your requests into consideration. I've already implemented the *.csv and the delimiter changing.
The delimiter is now detected when you load a file, and if the delimiter is found to be ; TAB or , then the program changes delimiters and tells you(in case that's not what you wanted).
Ok so windows users have a mass advantage in using this program of mine. What you need to do is associate the file *.csv with csvomatic.exe then every time you open "my computer" or whatever they call it now, you go to the directory containing the csv files then you can launch csvomatic AND load the file by double clicking on it. Windows will put that little icon I drew on all your csv files, indicating you're ready to rock. It's a blue grid that says DSV in front of it. Yah...I drew that myself! Crud, well I guess linux users can use my dsv icon too, so I should post it on berlios. Linux doesn't complain the icon is missing like windows does. I will look in to the windows wierdness that has cropped up, it probably has different window defaults than linux. I'll make an official post when I've completed the windows version changes as well. I think I am going to have to upgrade my linux as well and start cranking on the depreciation warning changes.

targ collective · Post by **targ collective** » Wed Jun 13, 2007 10:19 am

Phew, I was worried I'd scared you off.

I'm going to be pretty inactive over the next week or two, so don't worry if you get little feedback from me, but there's one last really trivial item on the wishlist.

Units.csv uses a header row, then a description of the type of data that goes into a given column directly below that header row. Some configuration in header control to select the number of and types of headers, as well as some sort of referential tooltip, would be great. (The ideal would be the ability to highlight a portion of the datatype tooltip depending on where you are coding/editing in the cell with feedback if you go outside the defined syntax, but that would probably be asking too much.)

ismeagain · Post by **ismeagain** » Thu Jun 14, 2007 1:00 am

hah hah it's hard to scare me off! I have fully converted csvomatic to the new namespaces in wxpython. The downside is....DSV still uses the old wxpython namespace..so you still get the warning message. Of course, I could fix this if I wanted, but the author says his CSV code is going to be in the main library of routines for python. I'm stoked over that one! I should wheedle and bother him to finish that. So anyways... now the linux and windows versions will be both for wx 2.8 on python 2.6. There are definitely things...wrong with my program now that I've upgraded to the latest wx. At least now my windows problems and linux problems are the same

. Nothing super critical it's all to do with the header row in the grid display. Seems the old grid header code routines are failing to work as they used to. Ugh oh yah I might have to make my DSV icon inline, as the new wx tries to load the ico file from the CWD instead of the directory you launched CSVOMATIC from. I'm probably going to wait a bit before I make a new build, so I can explore the differences between this new wx and the old one. So you all know now, I've broken out of wx namespace hell!!!!

ismeagain · Post by **ismeagain** » Thu Jun 14, 2007 8:28 am

ok boys and girls!
http://prdownload.berlios.de/csvomatic/ ... -win32.zip
windows build...of course. DSV will still complain I think and make the log...but I'm not so sure anymore because it didn't when I ran it in XP. If you think the FIND GRID display having no min/max/restore and only a close button is a problem I can change it.... it's just I use that as a MODAL window(can't click on the one behind it until you close that one) so you have to close it, before you can change things in the main window.
http://prdownload.berlios.de/csvomatic/CSVOMATIC.py
the raw script I run in fedora 7. You will need the icon too. Put it in the directory with CSVOMATIC.py
http://prdownload.berlios.de/csvomatic/DSV.ico

targ collective · Post by **targ collective** » Thu Jun 14, 2007 6:26 pm

Excellent! Marked as my favourite csv editor.

Vega Strike Forums

Yet another CSV viewer and editor for linux/windows/mac

Yet another CSV viewer and editor for linux/windows/mac

Actually you need a couple more things...

windows build