forum.vdsworld.com Forum Index forum.vdsworld.com
Visit VDSWORLD.com
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 


Extract ip-addresses from strings and ip-validity-check

 
Post new topic   Reply to topic    forum.vdsworld.com Forum Index -> Visual DialogScript 6 Source Code
View previous topic :: View next topic  
Author Message
bornsoft
Contributor
Contributor


Joined: 19 Feb 2009
Posts: 113
Location: Germany

PostPosted: Wed Oct 13, 2010 3:13 am    Post subject: Extract ip-addresses from strings and ip-validity-check Reply with quote

Hi,

here is a function i wrote to parse out ip-addresses from mail-headers for spam-checking.

I think it's a good parser example you can easy transpose to parse out other informatins from strings.

Feel free to modify.


Code:

#                                                                                         
#    Function bsIP - by bornSoft                    ( VDS6 required )                     
#    ---------------------------                    -----------------                     
#                                                                                         
#    Usage:    %x = @bsIP( <String> )                                                     
#                                                                                         
#              %x = @bsIP( Extract , <String> [ , <Number>/ALL/UNIQUE ] )                 
#                                                                                         
#    This function returns "1" if <String> is an IP-formatted number or NULL if not.     
#                                                                                         
#    If argument EXTRACT is given, the function retuns the first ip-formatted number     
#     found in <Sring> or Null.                                                           
#                                                                                         
#    The third argument is optional.                                                     
#                                                                                         
#    If it is an integer number, for example 2, the function returns the second found     
#    ip-formatted number. If <Number> is greater then the count, NULL is returned.       
#                                                                                         
#    If the third argument is set to ALL, the function returns all found ip-formatted     
#    numbers separated by the current field separator. Doubles are also returned.         
#                                                                                         
#    If the third argument is set to UNIQUE, the function returns all unique found       
#    ip-formatted numbers in ascending order separated by the current field separator.   
#                                                                                         
#                                                                                         

#DEFINE Function,bsIP

#    Subfunction         
#DEFINE Function,bsIPCheck


  #    A string that contains some ip-addresses           
  %%String = "Received: from 213.198.55.118  (HELO mail08c.verio.de) (213.198.55.118) and another IP is 127.0.0.1"
  #    Extract all ip-addresses from the string           
  info Now we are going to extract all ip-adresses from this string: @tab()@cr()@cr()@tab()@chr(34)%%String@chr(34) @tab()
  info Extracted: @cr()@cr()@tab()@bsIP(Extract,%%String,All) @tab()
  #    Extract the first found ip-address from the string 
  info And now the first found ... @tab()
  info Extracted: @cr()@cr()@tab()@bsIP(Extract,%%String) @tab()
  #    Extract the third ip-address from the string       
  info The third one ... @tab()
  info Extracted: @cr()@cr()@tab()@bsIP(Extract,%%String,3) @tab()
  #    Extract only unique ip-addresses from the string   
  info And now all unique ip-adresses in ascending order ... @tab()
  info Extracted: @cr()@cr()@tab()@bsIP(Extract,%%String,UNIQUE) @tab()
  #    Check ip-address for validity                       
  info "And at least we check if  [ 192.168.0,100 ]  is a valid ip-address  ( Notice there is a comma! )"@tab()
  %%IP = "192.168.0,100"
  if @bsIP(%%IP)
      info %%IP  is a valid ip-address. @tab()
   else
      info %%IP  is not a valid ip-address. @tab()
  end
  exit


#    Begin Function bsIP                                                                 

:bsIP
  %1 = @trim(%1)
  if %1
      if @equal(%1,Extract)
          goto bsIPExtract
        else
          goto bsIPCheck
      end
  end
  exit

:bsIPExtract
  %P = @pos(".",%2)
  if @greater(%P,0)
      %L = @new(list)
      #    First break the string on spaces   
      list assign,%L,@strrep(%2," ",@cr(),all)
      #    Mark all unwanted lines           
      %C = @count(%L)
      %x = 0
      repeat
          %i = @item(%L,%x)
          %P = @pos(".",%i)
          if @zero(%P)
              list put,%L,#
          end
          %x = @succ(%x)
      until @equal(%x,%c)
      #    Delete all unwanted lines         
      list assign,%L,@strrep(@text(%L),#@cr()@lf(),,all)
      #    Remove unwanted characters         
      %N = "0123456789."
      %c = @count(%L)
      repeat
          %x =
          %i = @item(%L)
          while @not(@zero(@len(%i)))
              %S = @substr(%i,1)
              if @greater(@pos(%S,%N),0)
                  %x = %x%S
              end
              %i = @strdel(%i,1)
          wend
          if @bsIPCheck(%x)
              list put,%L,%x
            else
              list put,%L,#
          end
          %i = @next(%L)
      until @equal(@index(%L),%c)
      #    Delete all unwanted lines         
      list assign,%L,@strrep(@text(%L),#@cr()@lf(),,all)
      if @text(%L)
          if @not(%3)
              %R = @item(%L,0)
            elsif @numeric(%3)
              if @greater(%3,@count(%L))
                  %R =
                else
                  %R = @item(%L,@pred(%3))
              end
            elsif @equal(%3,ALL)
              %R = @strrep(@trim(@text(%L)),@cr()@lf(),@fsep(),all)
            elsif @equal(%3,UNIQUE)
              %N = @new(list,sorted)
              list assign,%N,%L
              %R = @strrep(@trim(@text(%N)),@cr()@lf(),@fsep(),all)
              list close,%L
              list close,%N
          end
      end
  end
  exit %R

:bsIPCheck
  if @numeric(@strrep(%1,".","",all))
      %F = @fsep()
      option Fieldsep,.
      parse "%A;%B;%C;%D",%1
      option Fieldsep,%F
      if @greater(%A,255) @greater(%B,255) @greater(%C,255) @greater(%D,255)
          %R =
        elsif @greater(0,%A) @greater(0,%B) @greater(0,%C) @greater(0,%D)
          %R =
        elsif @greater(@len(%A),3) @greater(@len(%B),3) @greater(@len(%C),3) @greater(@len(%D),3)
          %R =
        else
          %R = 1
      end
  end
  exit %R



.
Back to top
View user's profile Send private message Visit poster's website
bornsoft
Contributor
Contributor


Joined: 19 Feb 2009
Posts: 113
Location: Germany

PostPosted: Thu Oct 14, 2010 8:11 pm    Post subject: Reply with quote

.
Here is another version for the one who is interested.

It's completely rewritten and works much faster and more reliable now.
An 80kB html file from the Wikipedia site with IP numbers in it is completely parsed out in 500 milliseconds on my WinXP/SP3 machine.

The code looks a bit more complicated, thats primal for speed-improvement.
On my testings I noticed, that it is very important for speed, in which order the things are done.
My first try took 6,5 seconds for the same file compared to half a second now.

I also noticed, that the built in @STRREP( ,ALL) function in VDS6 is significant slow compared to doing it with REPEAT/UNTIL and @pos() which then again is faster then WHILE/WEND.

Code:

#                                                                                         
#    Function bsIP - by bornSoft - Version 2         ( VDS6 required )                   
#    ---------------------------------------         -----------------                   
#                                                                                         
#                                                                                         
#    Usage:    %x = @bsIP( <String> )                                                     
#                                                                                         
#              %x = @bsIP( Extract , <String> [ , <Number>/ALL/UNIQUE ] )                 
#                                                                                         
#    This function returns "1" if <String> is an IP-formatted number or NULL if not.     
#                                                                                         
#    If argument EXTRACT is given, the function retuns the first ip-formatted number     
#    found in <Sring> or Null.                                                           
#                                                                                         
#    The third argument is optional:                                                     
#                                                                                         
#    If it is an integer number, for example 2, the function returns the second found     
#    ip-formatted number. If <Number> is greater then the count, NULL is returned.       
#                                                                                         
#    If the third argument is set to ALL, the function returns all found ip-formatted     
#    numbers separated by the current field separator. Doubles are also returned.         
#                                                                                         
#    If the third argument is set to UNIQUE, the function returns all unique found       
#    ip-formatted numbers in ascending order separated by the current field separator.   
#                                                                                         
#                                                                                         

Title Function bsIP Example

#DEFINE Function,bsIP

#    Subfunctions           
#DEFINE Function,bsIP_Check
#DEFINE Function,bsIP_NumTrim
#    SubCommand             
#DEFINE Command,bsIP_Replace


  #    Get a file with some ip-addresses in it         
  %%Text = At first we need a text with a couple of ip-addresses in it. @tab()@cr()For this we download the Wikipedia article about ip addresses. @tab()@cr()@cr()Press OK to start or CANCEL to exit.@cr()
  %%URL = "http://en.wikipedia.org/wiki/IP_address"
  %%File = @path(%0)wikipedia.txt
  if @not(@file(@path(%0)wikipedia.txt))
      if @not(@query(%%Text))
          exit
      end
      DIALOG CREATE,bsIP Example by bornSoft,-1,0,300,133,COLOR FFFFBB,NOTITLE,ONTOP
      DIALOG ADD,STYLE,STYLE1,Arial,24,B,,GRAY
      DIALOG ADD,TEXT,TEXT1,47,36,,,Downloading ...,,STYLE1
      DIALOG SHOW
      file copy,%%URL,%%File
      if @not(@file(%%File))
          Dialog hide
          info ERROR: File could not be downloaded. Check internet connection. @tab()
          exit
        else
          Dialog hide
          info File successfully downloaded. @tab()
      end
  end
  %L = @new(list)
  list loadfile,%L,%%File
  %%Text = @text(%L)
  list close,%L

  #    Extract all ip-addresses from the string           
  info Now we extract all ip-addresses and measure processing time. @tab()
  %%StartTime = @datetime()
  %%IP = @bsIP(Extract,%%Text,ALL)
  %%Duration = @datetime("nn:ss:z",@fsub(@datetime(),%%StartTime))
  %%IP = @strrep(%%IP,@fsep(),@cr(),all)
  info All IP adresses from %%Url @tab()@cr()@cr()@tab()Filesize:@tab()@tab()@file(%%File,Z) bytes@cr()@tab()Duration:@tab()@tab()%%Duration @tab()@cr()@cr()%%IP

  #    Extract the first found ip-address                 
  info And now the first found ... @tab()
  info The first found is:@tab()@bsIP(Extract,%%Text) @tab()

  #    Extract the third ip-address from the string       
  info The fourth one ... @tab()
  info The fourth one is:@tab()@bsIP(Extract,%%Text,4) @tab()

  #    Extract only unique ip-addresses from the string   
  info And now all unique ip-adresses in ascending order ... @tab()
  %%IP = @bsIP(Extract,%%Text,UNIQUE)
  info All unique IP adresses from %%Url @tab()@cr()@cr()@strrep(%%IP,@fsep(),@cr(),all)

  #    Check ip-address for validity                     
  info "And at least we check if  [ 192.168.0,100 ]  is a valid ip-address  ( Notice there is a comma! )"@tab()
  %%IP = "192.168.0,100"
  if @bsIP(%%IP)
      info %%IP  is a valid ip-address. @tab()
   else
      info %%IP  is not a valid ip-address. @tab()
  end
  exit

#    Begin Function bsIP                                                                 
:bsIP
  %1 = @trim(%1)
  if %1
      if @equal(%1,Extract)
          goto bsIP_Extract
        else
          goto bsIP_Check
      end
  end
  exit

:bsIP_Extract
  if @greater(@pos(".",%2),0)
      %L = @new(list)
      %2 = @strrep(%2,@chr(160),@chr(32),ALL)
      list assign,%L,%2
      #    Delete all lines with no dot                   
      gosub bsIP_NoDot
      #    Delete all lines with no numerics             
      gosub bsIP_NoNum
      #    Break on commas                               
      bsIP_Replace %L,",",@cr()@lf()
      gosub bsIP_NoDot
      gosub bsIP_NoNum
      #    Break on equals                               
      bsIP_Replace %L,"=",@cr()@lf()
      gosub bsIP_NoDot
      #    Break on <                                     
      bsIP_Replace %L,"<",@cr()@lf()
      gosub bsIP_NoDot
      #    Break on >                                     
      bsIP_Replace %L,">",@cr()@lf()
      gosub bsIP_NoDot
      gosub bsIP_NoNum
      #    Trim items to numbers in the beginning and the end 
      %c = @count(%L)
      %x = 0
      %N = "0123456789"
      repeat
          %i = @item(%L,%x)
          %S = @substr(%i,1)
          %W = @len(%i)
          repeat
              %P = @pos(%S,%N)
              if @zero(%P)
                  %i = @strdel(%i,1)
                  %S = @substr(%i,1)
              end
          until @greater(%P,0) @equal(%P,%W)
          #    From the back     
          if @greater(@len(%i),0)
              %S = @strdel(%i,1,-1)
              while @zero(@pos(%S,%N))
                  %i = @substr(%i,1,-1)
                  %S = @strdel(%i,1,-1)
              wend
          end
          list put,%L,%i
          %x = @succ(%x)
      until @equal(%x,%c)
      #    Delete all lines with no dot                   
      gosub bsIP_NoDot
      #    Delete all lines shorter then 7 chars         
      %x = 0
      %c = @count(%L)
      while @less(%x,%c)
          %i = @item(%L,%x)
          if @less(@len(%i),7)
              list put,%L,"#"
          end
          %x = @succ(%x)
      wend
      gosub bsIP_Delete
      #    Break on (                                     
      bsIP_Replace %L,"(",@cr()@lf()
      gosub bsIP_NoDot
      #    Break on )                                     
      bsIP_Replace %L,")",@cr()@lf()
      gosub bsIP_NoDot
      gosub bsIP_NoNum
      #    Break on [                                     
      bsIP_Replace %L,"[",@cr()@lf()
      gosub bsIP_NoDot
      #    Break on ]                                     
      bsIP_Replace %L,"]",@cr()@lf()
      gosub bsIP_NoDot
      gosub bsIP_NoNum
      #    Break on slashes                               
      bsIP_Replace %L,"/",@cr()@lf()
      gosub bsIP_NoDot
      #    Break the string on all spaces                 
      bsIP_Replace %L," ",@cr()@lf()
      gosub bsIP_NoDot
      #    Delete all lines with no numeric chars         
      gosub bsIP_NoNum
      #    Process line by line                           
      %%L2 = @new(list)
      list assign,%%L2,%L
      %x = 0
      repeat
          %i = @item(%%L2,%x)
          list assign,%L,@strrep(%i,".",@cr()@lf(),ALL)
          #    Trim items to only numeric Characters     
          #    at the beginning and at the end           
          list assign,%L,@bsIP_NumTrim(@text(%L))
          #    Restore all Dots                           
          %T = @trim(@text(%L))
          %T = @strrep(%T,@cr()@lf(),".",ALL)
          list assign,%L,%T@cr()@lf()
          #    Put this item back to %%L2                 
          list put,%%L2,@item(%L)
          %x = @succ(%x)
      until @equal(%x,@count(%%L2))
      list assign,%L,%%L2
      list close,%%L2
      #    Check validity and mark invalids               
      %x = 0
      repeat
          %i = @item(%L,%x)
          if @not(@bsIP_Check(%i))
              list put,%L,"#"
          end
          %x = @succ(%x)
      until @equal(%x,@count(%L))
      #    Delete invalid lines                           
      gosub bsIP_Delete
      #    Check which output was wanted                 
      if @text(%L)
          if @not(%3)
              %R = @item(%L,0)
            elsif @numeric(%3)
              if @greater(%3,@count(%L))
                  %R =
                else
                  %R = @item(%L,@pred(%3))
              end
            elsif @equal(%3,ALL)
              %R = @strrep(@trim(@text(%L)),@cr()@lf(),@fsep(),all)
            elsif @equal(%3,UNIQUE)
              %N = @new(list,sorted)
              list assign,%N,%L
              %R = @strrep(@trim(@text(%N)),@cr()@lf(),@fsep(),all)
              list close,%L
              list close,%N
          end
      end
  end
  exit %R

#    SubCommand bsIP_Replace  ( Faster then @STRREP() )       
#    %1 = List / %2 = SearchCharacter / %3 = RepaceCharacter 
:bsIP_Replace
  %x = 0
  %c = @count(%1)
  while @less(%x,%c)
      %i = @item(%1,%x)
      %P = @pos(%2,%i)
      while @greater(%P,0)
          %i = @strdel(%i,%P)
          %i = @strins(%i,%P,%3)
          %P = @pos(%2,%i)
      wend
      list put,%1,%i
      %x = @succ(%x)
  wend
  list assign,%1,@text(%1)
  exit

#    Gosub bsIP_Delete 
:bsIP_Delete
  %x = 0
  repeat
      %i = @item(%L,%x)
      if @equal(%i,"#")
          list delete,%L
        else
          %x = @succ(%x)
      end
  until @equal(%x,@count(%L))
  exit

#    Gosub bsIP_NoDot   
:bsIP_NoDot
  %C = @count(%L)
  %x = 0
  repeat
      %i = @item(%L,%x)
      if @zero(@pos(".",%i))
          list put,%L,#
      end
      %x = @succ(%x)
  until @equal(%x,%C)
  list seek,%L,0
  gosub bsIP_Delete
  exit

#    Gosub bsIP_NoNum 
:bsIP_NoNum
  %C = @count(%L)
  %x = 0
  repeat
      %i = @item(%L,%x)
      %N = 0
      %O =
      repeat
          if @greater(@pos(%N,%i),0)
              %O = 1
          end
          %N = @succ(%N)
      until %O @greater(%N,9)
      if @not(%O)
          list put,%L,#
      end
      %x = @succ(%x)
  until @equal(%x,%C)
  list seek,%L,0
  gosub bsIP_Delete
  exit

#    Subfunction bsIP_NumTrim  ( %1 = String ) 
:bsIP_NumTrim
  %L = @new(list)
  list assign,%L,%1
  %C = @count(%L)
  %x = 0
  %N = "0123456789"
  repeat
      %i = @item(%L,%x)
      %S = @substr(%i,1)
      %W = @len(%i)
      %P = 1
      while @both(@zero(@pos(%S,%N)),@not(@greater(%P,%W)))
          %i = @strdel(%i,1)
          %S = @substr(%i,1)
          %P = @succ(%P)
      wend
      %P = 1
      %S = @substr(%i,%P)
      %R =
      while @not(@zero(@pos(%S,%N)))
          %R = @substr(%i,1,%P)
          %P = @succ(%P)
          %S = @substr(%i,%P)
      wend
      if %R
          %E = %E%R#
      end
      %x = @succ(%x)   
  until @equal(%x,@count(%L))
  list close,%L
  %R = @strrep(%E,"#",@cr()@lf(),ALL)
  exit %R

#    Subfunction bsIP_Check 
:bsIP_Check
  if @numeric(@strrep(%1,".","",all))
      %F = @fsep()
      option Fieldsep,.
      parse "%A;%B;%C;%D;%E",%1
      option Fieldsep,%F
      if @not(%E)
          if @greater(%A,255) @greater(%B,255) @greater(%C,255) @greater(%D,255)
              %R =
            elsif @greater(0,%A) @greater(0,%B) @greater(0,%C) @greater(0,%D)
              %R =
            elsif @greater(@len(%A),3) @greater(@len(%B),3) @greater(@len(%C),3) @greater(@len(%D),3)
              %R =
            else
              %R = 1
          end
      end
  end
  exit %R


.
Back to top
View user's profile Send private message Visit poster's website
DaveR
Valued Contributor
Valued Contributor


Joined: 03 Sep 2005
Posts: 413
Location: Australia

PostPosted: Sun Nov 21, 2010 6:52 am    Post subject: Reply with quote

bornsoft wrote:
.
Here is another version for the one who is interested.

It's completely rewritten and works much faster and more reliable now.
An 80kB html file from the Wikipedia site with IP numbers in it is completely parsed out in 500 milliseconds on my WinXP/SP3 machine.

The code looks a bit more complicated, thats primal for speed-improvement.
On my testings I noticed, that it is very important for speed, in which order the things are done.
My first try took 6,5 seconds for the same file compared to half a second now.

I also noticed, that the built in @STRREP( ,ALL) function in VDS6 is significant slow compared to doing it with REPEAT/UNTIL and @pos() which then again is faster then WHILE/WEND.

Hi bornsoft,

Thanks for the speed tips.

_________________
cheers

Dave
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    forum.vdsworld.com Forum Index -> Visual DialogScript 6 Source Code All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum

Twitter@vdsworld       RSS

Powered by phpBB © 2001, 2005 phpBB Group