• RSS
  • Twitter
  • FaceBook

Welcome to ISAserver.org

Forums | Register | Login | My Profile | Inbox | RSS RSS icon | My Subscription | My Forums | Address Book | Member List | Search | FAQ | Ticket List | Log Out

RE: Discussion for article on using scripts for Domain Name and URL Sets

Users viewing this topic: none

Logged in as: Guest
  Printable Version
All Forums >> [ISA Server 2004 Firewall] >> Access Policies >> RE: Discussion for article on using scripts for Domain Name and URL Sets Page: <<   < prev  1 [2]
Login
Message << Older Topic   Newer Topic >>
RE: Discussion for article on using scripts for Domain ... - 9.Mar.2006 6:42:30 PM   
fastcuda

 

Posts: 15
Joined: 28.Mar.2003
From: FL, USA
Status: offline
I know this is a little late, but who knows who will need this.

To fix the "Cannot create a file when that file already exist" error in the ImportURLs.vbs script.

Add "On Error Resume Next" at the top of the script. This adds error handling to the script.
Before when you received that message ,the script would quit. But with this statement in it, it will continue
reading the file to the end.

I've also updated this script with error handling, to not add lines that begin with '#' (Squidguard files) and
append the entries with "http://*." when they are entered into the URL set.

I will post it if there is enough interest in it.

< Message edited by fastcuda -- 9.Mar.2006 6:44:13 PM >

(in reply to tshinder)
Post #: 21
RE: Discussion for article on using scripts for Domain ... - 17.Apr.2006 10:21:54 PM   
robbosch

 

Posts: 75
Joined: 21.Sep.2004
From: Denver, CO
Status: offline
I've followed this script and have had trouble loading the 580K domains provided by Squidguard.  The script operates flawlessly but uses 95% of the CPU plus and has run for over 12 hours without completion.  In testing, it appears adding an item to the list using "BlockedURLs.Add "http://" & NewLine" where newline is the domain I'm adding gets longer and longer as the number of domains grow.  About every 10,000 I see a doubling in the amount of time to complete the add.

Has anyone tested this from the XML file method?  My next test was to use the ExportToFile method to get an XML structure, then recreate in my vbscript and import from file (ImportFromFile method).

Just hoping to get some commentary before going through this coding excercise. 

Rob

(in reply to fastcuda)
Post #: 22
RE: Discussion for article on using scripts for Domain ... - 23.Sep.2006 4:01:24 AM   
moisesandre

 

Posts: 4
Joined: 18.Sep.2006
Status: offline
Dear friends,
I want to share with you my experience importing squidguard like blacklists to ISA server.
1- Sources of blacklists:
- http://www.squidguard.org/blacklist/
- http://urlblacklist.com/ <-- this one is veeerryyy big and allways up to date.

2- You will use the "url" and "domain" files. You will edit them with MS Excel.

3- Domain files:
3.1. Sort data
3.2. Remove IP addresses (like 25.165.23.2).
3.3. Remove duplicates (tip: http://support.microsoft.com/?kbid=262277)
3.4. Use "concatenate" function to generate a list starting with "*." (like *.abcdef.com)
3.5. Copy and paste this list in a text file (i.e. domain.txt)

4- URL files:
3.1. Sort data
3.2. Remove duplicates (tip: http://support.microsoft.com/?kbid=262277)
3.4. Use "concatenate" function to generate a list starting with "http://" (like http://www.abcdef.com)
3.5. Copy and paste this list in a text file and save it(i.e. url.txt)

5. Import to ISA server using vbs scripts:
5.1. http://www.isaserver.org/articles/2004domainseturlset.html <-- very simple and very good. I used this one.
5.2. http://isatools.org/dest2004.zip

6- Tips:
6.1. To generate the big porn and adult files (600,000 lines!) I used MS Access and SQL queries.
6.2. If you don't want to throw away IP domains, you can transfer them to url files (http://25.165.23.2 like urls are accepted)
6.3. Be patient. The script takes time to run. Specialy for big lists. But it works! As an example, my adult list takes 12 hous to run.
6.4. When you have the lists ready, export them to XML files.
6.5. Big lists makes ISA server sssllllooowww to "Apply" changes. So configure small lists first.
6.6. Use whitelists instead of "External" destination.

I dont know if we can upload files to this forum. If anybody want, I can share my XML files.

Anybody can help me with some Excel or Access macro to do this job atomaticaly? Specific to UPDATE files.

Thanks!

Moisés André Nisenbaum.
Brazil.



(in reply to fastcuda)
Post #: 23
RE: Discussion for article on using scripts for Domain ... - 23.Sep.2006 7:07:20 AM   
robbosch

 

Posts: 75
Joined: 21.Sep.2004
From: Denver, CO
Status: offline
I don't think writing the script is the difficult part...and would be happy to post a script to process as you do there.  I've given up on putting the blocks in place because of the load times.  How long did it take to import the file using XML?  I was wondering if the 12 hours you indicated was the load time or just processing the squid blacklist.

Rob

(in reply to moisesandre)
Post #: 24
RE: Discussion for article on using scripts for Domain ... - 25.Sep.2006 3:36:20 AM   
moisesandre

 

Posts: 4
Joined: 18.Sep.2006
Status: offline
Hi, Rob.

It took 12 hours to import a text file with 600,000 lines into an ISA URL Set or Domain Name Set.
After having the lists loaded, the ISA "apply" process takes more time.
I didnt see any difference in clients navigation.
My server is a Sempron 1.8 with 1Gb RAM.

The good news is: Import from XML takes only few minutes.
But someone has once to spend hours to generate the ISA lists to export to XML.
We can build this XML lists together and share them.
I can share my files, I only need some place to store the XML files.
Another thing important to think about is how to upgrade the lists. I think we can monthly compare the old and the new list and generate a difference list do add to Domain Name/URL set.

Regards,

Moisés André.


(in reply to robbosch)
Post #: 25
RE: Discussion for article on using scripts for Domain ... - 25.Sep.2006 3:44:49 AM   
robbosch

 

Posts: 75
Joined: 21.Sep.2004
From: Denver, CO
Status: offline
I guess my thought was to just use the script to create the XML in the first place rather than going through the URL set pain.  I'm thinking that a simple vbscript to process the flatfiles and eliminate all the dups and such is not that problematic and can be pretty fast.  The issue is the creation of the URL set in ISA...so I'm thinking just create the XML. 

I haven't looked at this in a while but I had exported another URL set into XML and the XML structure was pretty straightforward. 

(in reply to moisesandre)
Post #: 26
RE: Discussion for article on using scripts for Domain ... - 25.Sep.2006 2:22:41 PM   
moisesandre

 

Posts: 4
Joined: 18.Sep.2006
Status: offline
It would be excelent if we could insert lines directly in XML file.
I did try using notepad but it didnt work :-(

Is there any XML "editor" ?

regards,

Moisés André.

(in reply to robbosch)
Post #: 27
RE: Discussion for article on using scripts for Domain ... - 25.Sep.2006 3:40:00 PM   
robbosch

 

Posts: 75
Joined: 21.Sep.2004
From: Denver, CO
Status: offline
Scripting this to create the XML object seems like the way to move forward.  I won't be able to take a look at that for a while but should be able to at some point.  Putting this block in has moved down on our priority list.


(in reply to moisesandre)
Post #: 28
RE: Discussion for article on using scripts for Domain ... - 27.Sep.2006 12:30:39 PM   
moisesandre

 

Posts: 4
Joined: 18.Sep.2006
Status: offline
Ok, Rob.
By the way, do you know a good way to learn scripts?

Regards.
Moises.

(in reply to robbosch)
Post #: 29
RE: Discussion for article on using scripts for Domain ... - 18.Jan.2007 4:17:02 PM   
mleos

 

Posts: 1
Joined: 18.Jan.2007
Status: offline
http://urlblacklist.com/?sec=download

Here you may find some very extensive blacklists but there's some duplicate records

(in reply to armani007)
Post #: 30

Page:   <<   < prev  1 [2] << Older Topic    Newer Topic >>
All Forums >> [ISA Server 2004 Firewall] >> Access Policies >> RE: Discussion for article on using scripts for Domain Name and URL Sets Page: <<   < prev  1 [2]
Jump to:

New Messages No New Messages
Hot Topic w/ New Messages Hot Topic w/o New Messages
Locked w/ New Messages Locked w/o New Messages
 Post New Thread
 Reply to Message
 Post New Poll
 Submit Vote
 Delete My Own Post
 Delete My Own Thread
 Rate Posts