forum.vdsworld.com Forum Index forum.vdsworld.com
Visit VDSWORLD.com
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 


long lists
Goto page 1, 2  Next
 
Post new topic   Reply to topic    forum.vdsworld.com Forum Index -> Knowledge Base
View previous topic :: View next topic  
Author Message
Serge
Professional Member
Professional Member


Joined: 04 Mar 2002
Posts: 1480
Location: Australia

PostPosted: Mon Jan 03, 2005 8:38 am    Post subject: long lists Reply with quote

hi,

just thought i'd share what i have found recently concerning long lists

by using the api LB_INITSTORAGE found in the API help/resource file, i have been able to list more than 250,000 items in a list box making up more than 5 Mb in file size

i am wanting to test for 10Mb and hope to do so soon

there has been some discussion in earlier times about list sizes - may be this post will help

serge

_________________
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
Skit3000
Admin Team


Joined: 11 May 2002
Posts: 2166
Location: The Netherlands

PostPosted: Mon Jan 03, 2005 10:07 am    Post subject: Reply with quote

Woh, well done! Do big lists also load quicker then previously with this API? Smile
_________________
[ Add autocomplete functionality to your VDS IDE windows! ]
Voor Nederlandse beginners met VDS: bekijk ook eens deze tutorial!
Back to top
View user's profile Send private message
Boo
Valued Contributor
Valued Contributor


Joined: 31 Oct 2003
Posts: 599
Location: Gulf Breeze, Florida USA

PostPosted: Mon Jan 03, 2005 12:26 pm    Post subject: Reply with quote

Cool, Serge! Would it be possible for you to post an example?

Thanks,

- Boo
Back to top
View user's profile Send private message
jules
Professional Member
Professional Member


Joined: 14 Sep 2001
Posts: 1043
Location: Cumbria, UK

PostPosted: Mon Jan 03, 2005 12:30 pm    Post subject: Reply with quote

I think the issues of speed, and also of the number of elements that can be loaded into string lists (as opposed to list boxes, which are all that this message will affect) are caused by limitations of the Delphi 2 memory manager used by the current version of VDS.
_________________
The Tech Pro
www.tech-pro.net
Back to top
View user's profile Send private message Visit poster's website
Serge
Professional Member
Professional Member


Joined: 04 Mar 2002
Posts: 1480
Location: Australia

PostPosted: Mon Jan 03, 2005 2:10 pm    Post subject: Reply with quote

skit,

i have not tried to load lists as the programs i am working on simply requires me to generate potentially huge lists and to save them

boo,

i'll see what i can do as simply copying and pasting samples from my program will not help

jules,

you are right as my findings only apply to list boxes

serge

_________________
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
Serge
Professional Member
Professional Member


Joined: 04 Mar 2002
Posts: 1480
Location: Australia

PostPosted: Tue Jan 04, 2005 7:55 am    Post subject: Reply with quote

hi boo,

here you go - i wrote the code on vd 4.5 and the testing was done under windows xp

Code:
Title Long Lists
  DIALOG CREATE,New Dialog,-1,0,240,461
REM *** Modified by Dialog Designer on 4/01/2005 - 14:59 ***
  DIALOG ADD,LIST,LIST1,10,10,220,356
  DIALOG ADD,BUTTON,BUTTON1,410,90,64,24,Process
  DIALOG ADD,STATUS,STATUS1,
  DIALOG ADD,PROGRESS,PROGRESS1,375,10,220,24,0,,SMOOTH
  DIALOG ADD,BUTTON,BUTTON2,410,20,64,24,Save
  DIALOG ADD,BUTTON,BUTTON3,410,165,64,24,Open
  DIALOG SHOW
 
  rem === the 1000000 is for the upper limit for the number of lines ===
  rem === the 5000000 is for the number of bytes to be allocated to the size of the list ===
  %A = @sendmsg(@win(~LIST1),$01A8,1000000,5000000)

:Evloop
  wait event
  goto @event()
 
:BUTTON1BUTTON
  %%number_lines = @input(Enter number of lines to generate:)
 
  if %%number_lines
      if @not(@numeric(%%number_lines))
         warn You need to enter a valid number.
      goto evloop
      end
  else
      goto evloop   
  end
 
  dialog disable, button1
  dialog disable, button2
  dialog disable, button3
  %%file_size = 0
  list clear, list1
  %i = 0
  repeat
      %%new_entry = This is line number @fadd(%i,1) of %%number_lines
      list add, list1, %%new_entry
    rem === you need to add 2 for the end of line carriage return that equals 2 bytes ===
    %%file_size = @fadd(%%file_size,@fadd(@len(%%new_entry),2))
    dialog set, progress1, @name(@fdiv(@fmul(@fadd(%i,1),100),%%number_lines))
    dialog set, status1, # Entries: @fadd(%i,1)@tab()@tab()File Size: %%file_size Bytes
    %i = @succ(%i)
  until @equal(%i,%%number_lines)
  dialog enable, button1
  dialog enable, button2
  dialog enable, button3
  warn Generation Complete!
  goto evloop

:button2BUTTON
  dialog disable, button1
  dialog disable, button2
  dialog disable, button3

  %%dummy = @datetime()
  list savefile, list1, @path(%0)trial.txt
  %%time = @datetime(nn:ss,@fsub(@datetime(),%%dummy))

  dialog enable, button1
  dialog enable, button2
  dialog enable, button3
 
  if @ok()
      warn File Saved in %%time
  else
      warn There was a problem with saving the file.   
  end

  goto evloop 
 
:button3BUTTON
  dialog disable, button1
  dialog disable, button2
  dialog disable, button3
 
  list clear, list1
 
  %%dummy = @datetime()
  list loadfile, list1, @path(%0)trial.txt 
  %%time = @datetime(nn:ss,@fsub(@datetime(),%%dummy))
 
  dialog enable, button1
  dialog enable, button2
  dialog enable, button3
 
  warn File Loaded in %%time
 
  goto evloop
 
:Close
  exit


some interesting things i have found

- i have been able to generate lists up to 200,000 items of around 7.5 Mb which i have saved successfully and reloaded successfully

- the code above will show you how long in minutes: seconds it took to save or load the data

- for the 200,000 items and 7.5Mb list, it took a few seconds (on my laptop) to save but 1mn and 10 seconds to load

- BUT if i load the same 200,000 entries files the first time i run the code, it took around 37 seconds to do so - loading a large file seems a lot quicker the first time the program is run, no idea why!

- as another example, i generated a list of 100,000 items and 3.7Mb - this file took 1 second to save and 17 seconds to load - BUT the first time i run the program, it loaded in 1 second!!!

i just did a quick run under windows 98se and can't seem to get past around 1167390 bytes and 32736 items in list - is there a difference between 98 and xp that would account for the difference?

serge

_________________


Last edited by Serge on Tue Jan 04, 2005 7:59 am; edited 1 time in total
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
Serge
Professional Member
Professional Member


Joined: 04 Mar 2002
Posts: 1480
Location: Australia

PostPosted: Tue Jan 04, 2005 7:57 am    Post subject: Reply with quote

the problem under 98 seems to be the number of lines - it seems to put a limit on it in spite of the api call i used and in spite of no such limit existing under xp

i ran the script under vds 4.5 in both cases (wind 98se and xp)

serge

_________________
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
jules
Professional Member
Professional Member


Joined: 14 Sep 2001
Posts: 1043
Location: Cumbria, UK

PostPosted: Tue Jan 04, 2005 9:53 am    Post subject: Reply with quote

32736 does not seem like a surprising limitation in Windows 98 which is a 16-bit OS. There are a lot of 64K limitations in it, and 64K can hold 32768 16-bit pointers, or 32736 + 32 bytes of other stuff.
_________________
The Tech Pro
www.tech-pro.net
Back to top
View user's profile Send private message Visit poster's website
Boo
Valued Contributor
Valued Contributor


Joined: 31 Oct 2003
Posts: 599
Location: Gulf Breeze, Florida USA

PostPosted: Tue Jan 04, 2005 3:08 pm    Post subject: Reply with quote

Thanks for the info. Serge! Cool
Back to top
View user's profile Send private message
Serge
Professional Member
Professional Member


Joined: 04 Mar 2002
Posts: 1480
Location: Australia

PostPosted: Tue Jan 04, 2005 10:55 pm    Post subject: Reply with quote

jules,

that win98se is a 16 bit o/s, does that mean that the list boxes are allocated 1 unit of memory, if you know what i mean

but then, what of the use of th api call to increase the amount of memory set aside for the list box?

what would be the equivalent limit under xp?

boo,

you are welcome and it was fun to do Smile

serge

_________________
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
FreezingFire
Admin Team


Joined: 23 Jun 2002
Posts: 3508

PostPosted: Wed Jan 05, 2005 2:29 am    Post subject: Reply with quote

@Jules: Windows 98 is a 16-bit OS? Question Confused

@Serge: Nice info. Very Happy

_________________
FreezingFire
VDSWORLD.com
Site Admin Team
Back to top
View user's profile Send private message Visit poster's website
PGWARE
Web Host


Joined: 29 Dec 2001
Posts: 1537

PostPosted: Wed Jan 05, 2005 6:41 am    Post subject: Reply with quote

A visual listbox will be much slower to load because Windows handles all the api PAINT messages. Anytime items are added to a VISIBLE listbox the scrollbar has to be painted and handled, the items, position and selection of a listbox have be maintained as well. This takes considerable memory and will cause your program to become slower.

When using visual lists, its always best to use a listbox that is eitehr hidden or DISABLED - hidden being the better option as it doesnt ahve to deal with teh numerous on screen repaint messages and redrawing. This will speed up your app greatly.

The INITSTORAGE api is generally used to speed up the usage of large lists, it really I do not think was intended to up the max amount a list box can hold. Its used to set aside a portion of memory to handle listbox storage and manipulation.
Back to top
View user's profile Send private message
Serge
Professional Member
Professional Member


Joined: 04 Mar 2002
Posts: 1480
Location: Australia

PostPosted: Wed Jan 05, 2005 8:31 pm    Post subject: Reply with quote

Quote:
The INITSTORAGE api is generally used to speed up the usage of large lists, it really I do not think was intended to up the max amount a list box can hold. Its used to set aside a portion of memory to handle listbox storage and manipulation.


interesting, pg ... however, i do find that without it, i cannot create very long entries in my list boxes (but only under xp)

also, the api help file says

Quote:
Allocates memory for storing list box items. An application sends this message before adding a large number of items to a list box.


it also says

Quote:
If you overestimate, some extra memory is allocated;


which explains why when i set aside 5Mb, i could generate a list of a bit more than 5 Mb

sorry pg!

serge

_________________
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
jules
Professional Member
Professional Member


Joined: 14 Sep 2001
Posts: 1043
Location: Cumbria, UK

PostPosted: Thu Jan 06, 2005 9:04 am    Post subject: Reply with quote

Serge wrote:
jules,

that win98se is a 16 bit o/s, does that mean that the list boxes are allocated 1 unit of memory, if you know what i mean

but then, what of the use of th api call to increase the amount of memory set aside for the list box?

what would be the equivalent limit under xp?

boo,

you are welcome and it was fun to do Smile

serge


I don't really know what you mean, but all memory allocations to the GUI subsystem, known as "resources", come from fixed size tables, which is why you have the problem of running out of resources in Win9x. Plus, many other things are kept track of using 16-bit pointers, as I said.

Win NT family systems are totally 32-bit, so the equivalent limit would be more than 2 billion.

_________________
The Tech Pro
www.tech-pro.net
Back to top
View user's profile Send private message Visit poster's website
jules
Professional Member
Professional Member


Joined: 14 Sep 2001
Posts: 1043
Location: Cumbria, UK

PostPosted: Thu Jan 06, 2005 9:12 am    Post subject: Reply with quote

FreezingFire wrote:
@Jules: Windows 98 is a 16-bit OS? Question Confused


Yes, 'fraid so. It supports 32-bit programs, of course, but most of the core OS was just an update of Windows 3, which was 16-bit only. That's why the best thing to do with Windows 9x is consign it to the dustbin of history where it belongs.

Remember, at the time Win 95 came out, Microsoft was already developing a 32-bit Windows - NT. It was only when it became clear that NT would need more powerful computers than were commonly used at that time, that they realised they would need to extend the life of Win 3 by a few years. So 9x is not a 32-bit OS - although Microsoft tried to pretend it was - Windows 3 with some 32-bit extensions, a new desktop, and the ability to run 32-bit software.

_________________
The Tech Pro
www.tech-pro.net
Back to top
View user's profile Send private message Visit poster's website
Display posts from previous:   
Post new topic   Reply to topic    forum.vdsworld.com Forum Index -> Knowledge Base All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum




Twitter@vdsworld       RSS

Powered by phpBB © 2001, 2005 phpBB Group