forum.vdsworld.com

Dr. Dread · Posted: Tue Feb 04, 2003 11:46 pm Post subject: VDS lists - bulk problems

I could use a little hint now. Here's the deal:

I process some flatfile databases in VDS. The program works flawless with files containing
perhaps 30,000 or 50,000 records. Sometimes a database will contain up to 150,000 lines, however,
and then VDS chokes on it.

For now I use an old DOS utility (and when I say old, I mean so old that even Mac will find it
ancient - it's dating back to '86 Laughing

) to split the input files into manageable chunks,
then I process the each chunk consecutively and assemble them back together by DOS
COPY. It works, but I don't think that it is a satisfactory solution.

So any ideas, guys?

Greetz
Dread

PS: Tried the VDSLIST.dll. I can bring it to work (in spite of a little bug), but it brings the program
to a crawl. Speed is perhaps reduced by 80% Sad

Mac · Posted: Tue Feb 04, 2003 11:58 pm Post subject:

The problem is that VDS is loading (or trying to load) the whole
file in memory for lists. About the only way around this is to open
the file and work straight from the drive. Not sure, but I think
the VDS4 @binfile() should be capable of doing this (or maybe
VDSIO.DLL), it'd just take quite a bit more code.

BTW, I didn't even have a computer in '86 ya old geezer. Razz

I did write my own DOS file splitter/splicer though. Wink

Cheers, Mac Smile

Mac · Posted: Wed Feb 05, 2003 12:09 am Post subject:

One more idea Dread,

Visible lists (Windows 95 only) have a limit of 32,767 items, but
the LIST CREATE lists do NOT. This can be used to work around
the Win95 limit, and ya might try this same method to speed up
your visible lists.

Load/process the file from a hidden list, and only display chunks
of it in the visible one as needed. Wink

Cheers, Mac Smile

Dr. Dread · Posted: Wed Feb 05, 2003 7:07 am Post subject:

MarkTrubo · Posted: Wed Feb 05, 2003 10:17 am Post subject:

I would be very interested in the solution as I too work with a number of lists that have problems just past 100K records. Do a search back for an old post called "Are big lists bad?"

Dr. Dread · Posted: Wed Feb 05, 2003 11:24 am Post subject:

Hey Mark!

Very interesting. Seems like we struck the same problem. I experience it on my WinXP box,
haven't tried in another OS.

I've been playing around a bit with VDSLIST.DLL for this. It is actually quite interesting
because it works quite differently from VDS lists:

* Big lists are loaded very fast and with very little use of RAM (big lists load slowly in
VDS and a 8 Meg file takes about 100 Meg RAM on my system)

* If you cycle through a loaded list step by step, VDS lists are much faster than
calling the DLL.

So for start-to-end processing a regular VDS list will be faster (unless you run into the 100,000 limit).
But if you want to just count the lines in the file or access a limited number of specific lines
the DLL will outspeed a plain VDS list.

Greetz
Dread

BTW: There seems to be a couple of bugs in VDSLIST.DLL:
http://www.vdsworld.com/forum/viewtopic.php?t=1164

Garrett · Posted: Wed Feb 05, 2003 7:30 pm Post subject:

My solution to large files was (if starting from scratch) to limit my files
10,000 lines and then start another file.

I've been doing it this way with web site stats. I've got several client
sites that I made the cgi programs for and I keep each month's statistics
in seperate flat files. When the client needs to search for a specific entry,
I search through all the files. It doesn't seem to be too slow using
seperate files like this.

Of course, this is fine if you're starting from scratch and no other
programs require access to these files. But if you're dealing with files
that already exists and other programs require only the single file, well,
not sure about that.

-Garrett

moke · Contributor Joined: 02 Jan 2002 Posts: 162

Hey Dread,

If you're still working on this problem, I use VDSINOUT.DLL in 3.51 and it does a great job. You can R/W directly from the file or load it into memory. It took a while to get used to using it but now it works great. You can also use it to do quick appending of data if you need to write a strind a file.

moke

Dr. Dread · Posted: Tue Feb 11, 2003 7:51 am Post subject:

Thankx Moke, I'll try it out even though I must admit that I'm not the greatest wizard
of binaries Shocked

Hmmm, perhaps you've got a piece of code that I could have a look at ? Could make
life quite a bit easier.

Greetz
Dread

moke · Contributor Joined: 02 Jan 2002 Posts: 162

Dread,
I don't know any one who looks forward to working w/ binaries but
sometimes you have no choice. This should get you started.

moke

Dr. Dread · Posted: Wed Feb 12, 2003 6:46 am Post subject:

Great moke!

That was a good jumpstart - I had something betaish up and running in just a few minutes. Now
it'll be a lot easier to do some experiments.

Greetz
Dread