• RSS
  • Twitter
  • FaceBook

Welcome to ISAserver.org

Forums | Register | Login | My Profile | Inbox | RSS RSS icon | My Subscription | My Forums | Address Book | Member List | Search | FAQ | Ticket List | Log Out

Discussion for article on using scripts for Domain Name and URL Sets

Users viewing this topic: none

Logged in as: Guest
  Printable Version
All Forums >> [ISA Server 2004 Firewall] >> Access Policies >> Discussion for article on using scripts for Domain Name and URL Sets Page: [1] 2   next >   >>
Login
Message << Older Topic   Newer Topic >>
Discussion for article on using scripts for Domain Name... - 7.Sep.2004 11:52:00 PM   
tshinder

 

Posts: 50013
Joined: 10.Jan.2001
From: Texas
Status: offline
This thread is for discussing the article on Domain Name and URL Sets at http://www.isaserver.org/articles/2004domainseturlset.html.

Answers to the questions:

1. Is the ISA firewall a stateful firewall? What types of stateful firewalling does the ISA firewall perform?
Yes. The ISA firewall is a stateful firewall. The ISA firewall is both a stateful filtering firewall, like all other enterprise level firewalls and a stateful application layer inspection firewall. Unlike second generation 'hardware' firewalls which only perform stateful filtering, the ISA firewall is a best of breed third generation firewall that is able to perform stateful application layer inspection.

2. What elements must you first create in the Microsoft Internet Security and Acceleration Server 2004 management console before running the scripts?
If you want to use the script to import URLs into a URL Set, then you must first create the URL Set in the ISA Management console before importing the entries into the set. If you want to use the script to import entries into a Domain Name Set, then you must first create the Domain Name Set before importing the entries into the set.


3. If the new entries donĂt appear in the URL Set or the Domain Name Set, what should you do to fix the problem?
The most likely reason for the new entries not appearing in the URL Set or Domain Name set is that you haven't refreshed the view in the ISA Management console. You can either close and reopen the console, or you can click the Refresh button in ISA console's button bar.

Thanks!
Tom

[ September 08, 2004, 01:30 AM: Message edited by: tshinder ]
Post #: 1
RE: Discussion for article on using scripts for Domain ... - 8.Sep.2004 6:10:00 AM   
armani007

 

Posts: 5
Joined: 13.Jan.2004
From: Vancouver, BC
Status: offline
Can anyone else recommend sites that have domain and url blocking files that I can use with this example?

(in reply to tshinder)
Post #: 2
RE: Discussion for article on using scripts for Domain ... - 8.Sep.2004 9:07:00 AM   
tuanevnit

 

Posts: 5
Joined: 28.Nov.2003
Status: offline
I also need list of Adult, Sexualy site, can anyone help me to built that list of site?

(in reply to tshinder)
Post #: 3
RE: Discussion for article on using scripts for Domain ... - 10.Sep.2004 6:39:00 PM   
Guest
I followed the instructions but when I try to run I get the message "Cannot create a file when that file already exist"
Line: URLSet.Add UrlsFile.ReadLine
Code: 800700B7

Why this occurs??

(in reply to tshinder)
  Post #: 4
RE: Discussion for article on using scripts for Domain ... - 10.Sep.2004 7:08:00 PM   
kakkabomb

 

Posts: 1
Joined: 7.Jun.2004
Status: offline
Hi there

I also got the same error. I used blacklists that i downloaded and tried to extract the list's into excel and then into my .txt file.

There was repeated url's in my lists and some of them were not valid URL's they had underscores.

i.e. http://www.happy_days.com

The big problem was the repeated URL's

I used excel to find the duplicates, I selected my whole column,re-arranged everything in alphabetical order (just cos i like it that way)

Then I went to top menubar, DATA-> filter-> advanced filter-> (a popup message pops up...I clicked ok) then I selected "filter the list,in-place", my list range was the relevant url's, then i selected "unique records only" the clicked ok

I copied the resulting column and pasted it into a new coloum to compare the two. After I copied and pasted the resulting column i cleared the filter by repeating this:

I went to top menubar, DATA-> filter-> show all

I then copied my new column and pasted it over the one with the doubles.

I then copied the new list into my .txt file and it worked like a charm. I imported over 9000 porn sites

I hope this helps

I wish there was a place where I could get uptodate blacklists by catorgory

Like porn,ads,gambling,warez etc. It would make admin much easier.

i also find it easier to keep all the .txt url's in a relevant excel spreadsheet so that if one of the url's are giving a user a problem, i can quickly search through the spreadsheet and go directly to the URL set in ISA 2004 and remove the problem url

[ September 10, 2004, 07:11 PM: Message edited by: kakkabomb ]

(in reply to tshinder)
Post #: 5
RE: Discussion for article on using scripts for Domain ... - 10.Sep.2004 8:10:00 PM   
Guest
Dear friends,
Someone have a blacklist with porn sites... hackers site... to mail-me ?

thanks
gbiel2001@yahoo.com.br

(in reply to tshinder)
  Post #: 6
RE: Discussion for article on using scripts for Domain ... - 11.Sep.2004 12:26:00 PM   
Guest
I've imported the URL's at http://www.mvps.org/winhelp2002/hosts.htm in a Domain Name Set on my ISA Server 2004.

No problem so far, I've I type in the URL *exactly* like specified in the Domain Name Set, ISA Server blocks it. I.e. 'x0.nl' is one of the domains that is blocked. However, If I type in 'www.x0.nl' ISA Server does not block the URL.

Of course I can use the * wildcard in the domainname (*.x0.nl), that way www.x0.nl is blocked but x0.nl is not.

How can I solve this other than putting both URLs(*.x0.nl and x0.nl) in my Domain Name Set?

(in reply to tshinder)
  Post #: 7
RE: Discussion for article on using scripts for Domain ... - 14.Sep.2004 12:28:00 PM   
Guest
Nobody with a solution for this?

(in reply to tshinder)
  Post #: 8
RE: Discussion for article on using scripts for Domain ... - 14.Sep.2004 10:08:00 PM   
Guest
Ramon,

Use "*x0.nl" without the quotes.

(in reply to tshinder)
  Post #: 9
RE: Discussion for article on using scripts for Domain ... - 16.Sep.2004 8:00:00 PM   
Guest
Will give that a try.. thanks! [Smile]

(in reply to tshinder)
  Post #: 10
RE: Discussion for article on using scripts for Domain ... - 27.Sep.2004 12:35:00 AM   
lwilliams

 

Posts: 6
Joined: 24.Oct.2003
Status: offline
I too, followed the instructions for using the vbscript to add URL and Domain sets but when the script is executed, I get the message "Cannot create a file when that file already exist"
Line: URLSet.Add UrlsFile.ReadLine
Code: 800700B7

Anybody know whey this maybe happening? Thanks!

(in reply to tshinder)
Post #: 11
RE: Discussion for article on using scripts for Domain ... - 25.Jan.2005 12:56:00 AM   
Dolken

 

Posts: 1
Joined: 25.Jan.2005
Status: offline
For those of you who needed help with creating a list of blacklisted servers, you can download a free spyware product such as AdAware or Spybot and use their txt files.

(in reply to tshinder)
Post #: 12
RE: Discussion for article on using scripts for Domain ... - 25.Jan.2005 10:30:00 AM   
myxiplx

 

Posts: 136
Joined: 16.Mar.2001
Status: offline
The error "Cannot create a file when that file already exist" happens if you have a duplicate domain in your list.

I'm working on an update to the script that can check for this.

Ross

(in reply to tshinder)
Post #: 13
RE: Discussion for article on using scripts for Domain ... - 25.Jan.2005 10:46:00 AM   
myxiplx

 

Posts: 136
Joined: 16.Mar.2001
Status: offline
Ok, I've a slightly updated script [Smile]

This will look for any .CSV text files in the current directory, will create new Domain Name Sets for each text file, and will import all domains found in the file.

The script warns you of any errors / duplicates, but you can easily turn off this warning if you have a lot of duplicates. You can also customise the script to add domains in the form *domain.name.

I've used the CSV file extension because it's easy to export lists from Excel in CSV form, and TXT files are a little too common (I didn't want to risk of importing someone's README.TXT file into my server [Wink]

Save this into a file with a VBS extension and have fun.

Ross

code:
'ISA Domain Name Set Import Utility
'Written by Ross Smith, 25-Jan-2005

'Imports all CSV files in current directory into ISA server as Domain Name Sets
'Uses the name of the CSV file to name the Domain Name Set
'A single domain name should be listed on each line of the CSV file.
'WARNING! ** This script will the content of any existing sets with the contents of the text files **

'Usage notes:
'Script must be run locally from ISA Server
'With a little modification, the script can also import URL sets.
'My understanding of the difference is that
'URL Sets can only apply to HTTP protocol, but can include IP addresses
'Domain sets can block all protocols but cannot include IP addresses

On Error Resume Next

'Find current directory
Call ScriptPath(strFolder,strCommand,strProgramName)
If Err.number <> 0 then
Wscript.Echo "Error finding program path"
Wscript.Quit
End If
'wscript.echo "Called from folder : -> " & strFolder
'wscript.echo "Calling Program Line : -> " & strCommand
'wscript.echo "Calling Program Name : -> " & strProgramName

'Connect to ISA server
Set Isa = CreateObject("FPC.Root")
Set CurArray = Isa.GetContainingArray
Set RuleElements = CurArray.RuleElements
Set DomainNameSets = RuleElements.DomainNameSets
'Set URLSets = RuleElements.URLSets
If Err.number <> 0 then
Wscript.Echo "Error connecting to ISA server objects"
Wscript.Quit
End If
Set FileSys = CreateObject("Scripting.FileSystemObject")
If Err.number <> 0 then
Wscript.Echo "Error connecting to file system object"
Wscript.Quit
End If

'List CSV files in current folder
'Update directory string to correct format for WMI query
strFolder = replace(strFolder,"\","\\") 'Backslash needs escaping for WMI
strFolder = right(strFolder,len(strFolder)-2) 'Remove drive letter
strComputer = "."
Set objWMIService = GetObject("winmgmts:{impersonationLevel=impersonate}!\\" & strComputer & "\root\cimv2")
Set FileList = objWMIService.ExecQuery("SELECT * FROM CIM_DataFile WHERE Path = '" & strFolder & "' AND Extension = 'csv'")
If Err.number <> 0 then
Wscript.Echo "Error creating WMI Connection and listing current files"
Wscript.Quit
End If

For Each objFile In FileList
'Uncomment the following line to display the name of the current set if needed
'wscript.echo("Domain Set Name: " & objFile.FileName)

'See if Domain Name Set exists, create if it does not
if NOT Item_Exists(DomainNameSets,objFile.FileName) then
set DomainNameSet = DomainNameSets.Add(objFile.FileName)
else
Set DomainNameSet = DomainNameSets.Item(objFile.FileName)
'The following line is an example of how this could be modified for URL Sets
'Set URLSet = URLSets.Item(objFile.FileName)
end if
If Err.number <> 0 then
Wscript.Echo "Error adding Domain Name Set: " & objFile.FileName
Wscript.Quit
End If

Set DomainsFile = FileSys.OpenTextFile(objFile.FileName & ".csv", 1)
If Err.number <> 0 then
Wscript.Echo "Error opening text file: " & objFile.FileName & ".csv"
Wscript.Quit
End If

For i = 1 to DomainNameSet.Count
DomainNameSet.Remove 1
Next
If Err.number <> 0 then
Wscript.Echo "Error removing domains from set: " & objFile.FileName
Wscript.Quit
End If

Do While DomainsFile.AtEndOfStream <> True
strDomain = DomainsFile.ReadLine
'Add the domain to the list, exactly as given in the text file.
'If you prefer, you can modify this line to include all sub-domains by using the line:
'DomainNameSet.Add * & "strDomain"
DomainNameSet.Add strDomain
If Err.number <> 0 then
'Display a warning if domain cannot be added.
'This commonly happens if you have duplicate domains.
'Comment out the line below if you do not want to be warned every time this happens.
Wscript.Echo "Error adding domain '" & strDomain & "'. This may be a duplicate domain."
End If
Loop

'Uncomment the two lines below to prompt the user as each file is imported.
'WScript.Echo "Saving '" & objFile.FileName & "'..."
CurArray.Save
If Err.number <> 0 then
Wscript.Echo "Error saving changes."
End If
'WScript.Echo "Done"
Next

WScript.Echo "Done"


Function ScriptPath(callingfolder, callingcommandline, callingprogramname)
ScriptPath=Left(Wscript.scriptfullname,Instr(1,WScript.ScriptFullName,wscript.scriptname,1)-1)
callingfolder=scriptpath
callingcommandline=wscript.scriptfullname
callingprogramname=left(wscript.scriptname,len(wscript.scriptname)-4)
End Function


Private Function Item_Exists(oCollection , sItemName)
'This function returns True if the item exists in the given collection, False if it does not exist
Dim oItem 'Object
Item_Exists = False
For Each oItem In oCollection
If oItem.Name = sItemName Then
Item_Exists = True
Exit Function
End If
Next
End Function


(in reply to tshinder)
Post #: 14
RE: Discussion for article on using scripts for Domain ... - 27.Jan.2005 7:18:00 PM   
isa-msd

 

Posts: 1
Joined: 27.Jan.2005
Status: offline
how do i import the URls in ISA 2004?

quote:
Originally posted by <Ramon>:
I've imported the URL's at http://www.mvps.org/winhelp2002/hosts.htm in a Domain Name Set on my ISA Server 2004.

No problem so far, I've I type in the URL *exactly* like specified in the Domain Name Set, ISA Server blocks it. I.e. 'x0.nl' is one of the domains that is blocked. However, If I type in 'www.x0.nl' ISA Server does not block the URL.

Of course I can use the * wildcard in the domainname (*.x0.nl), that way www.x0.nl is blocked but x0.nl is not.

How can I solve this other than putting both URLs(*.x0.nl and x0.nl) in my Domain Name Set?


(in reply to tshinder)
Post #: 15
RE: Discussion for article on using scripts for Domain ... - 18.Mar.2005 5:01:00 PM   
Guest
Hi

your script is very nice but how do i have to change it to make use of a blocklist containing malicious IPs? My intention is to create a computerset containing every ip of the blocklist instead of a URLset containing http URLs.

THX Philoman

(in reply to tshinder)
  Post #: 16
RE: Discussion for article on using scripts for Domain ... - 21.Apr.2005 11:01:00 PM   
adisegna

 

Posts: 49
Joined: 26.Mar.2005
From: Palm Beach Gardens Florida
Status: offline
Those of you that are looking for sites to block and have installed some type of spyware remover should check out your registry key
[HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings\P3P\History]

Export the key
Open the reg file from excel to parse the data using the Text Import Wizard. Concatenate *. with the domain name and there's your file.

I have a list of 1500 domains compiled if anyone needs a starting point.

(in reply to tshinder)
Post #: 17
RE: Discussion for article on using scripts for Domain ... - 11.Jul.2005 7:11:00 AM   
pmuiruri

 

Posts: 4
Joined: 28.Jan.2005
From: Nairobi
Status: offline
For those getting the error message:

"Cannot create a file when that file already exists" follow these steps:

1) Download Rich KrolĂs blocklists (URL and Domains) from http://www.tacteam.net/isaserverorg/download/blocklists.zip
2) Extract the zip file to a disk folder on the ISA 2004 server
3) From ISA Server box, drill down to Array Name=> Firewall Policy => Open Task Pane
4) Click Network Objects
5) Right Click URL Sets and select New URL Set
6) Supply a name for the URL set e.g. Forbidden URLS. Refresh the URL sets
7) Right click the new URL set and select Properties
8) Key in a new URL you wish to block e.g. http://sex.com
9) Once done, save (apply) the settings
10) Right click the URL set and select Export all
11) Select the option to Export confidential information and enter a password
12) Supply a name for the export file and click Open
13) Click Next => Finish

You now have two sets of files; one from your own server and one from http://www.tacteam.net/isaserverorg/download/blocklists.zip

14) Using Notepad, open the .xml generated from your own server. Copy everything till the line above the URL you manually entered. The last line should be similar to ˘<fpc4:URLStrings>÷
15) Using Notepad, open the .xml file you download and paste till the first URL entry and save the file
16) Using Notepad, open the .xml from your own server. Copy everything below the last URL entry you manually entered. The line should be similar to ˘</fpc4:URLStrings>÷
17) Using Notepad, open the .xml file you download, scroll to the line below the last URL entry and paste. Save the file.

At this point you have introduced your own server settings to the downloaded file. You are ready to import this file to your server.

18) In your ISA server, right click the URL Set you made earlier and select Import All
19) Select the file you downloaded and later modified using the preceding steps. Select to import Server-specific information and supply the same password you used earlier.
20) Congrats! You just populated your ISA Server with a list of forbidden URLS.
21) Crete a new Access Rule that denies all HTTP, HTTPS, FTP traffic to the destination = your URL set
22) Apply the changes and test

You can use the same procedure above to create a forbidden domains set.

Good Luck.
Patrick Muiruri.

(in reply to tshinder)
Post #: 18
RE: Discussion for article on using scripts for Domain ... - 23.Sep.2005 9:05:00 AM   
hwilkins

 

Posts: 37
Joined: 30.Jul.2004
From: Memphis
Status: offline
Hmm.. I replied to this and swear I saw my reply posting, but it is missing now... strange. [mystery solved -- I must have hit the 'next' button and posted the reply on another thread... duh]

In any event, I've answered my own question by now. It was basically a request for help finding block-lists. The file refefenced in the chain above (http://www.tacteam.net/isaserverorg/download/blocklists.zip) has a date of 4/2005, which is not updated to the level that I'd need to make this useful.

I found, among other sites, squidGuard (http://www.squidguard.org/blacklist/) and URLblacklist (http://urlblacklist.com/?sec=download). I'm also looking at MVPS (http://www.mvps.org/winhelp2002/hosts.txt) which I think is referenced above as well.

I intend to write a script to unify various of these lists and import them. Of course, I'll have to do some sanitizing on them -- get rid of all-IPs for domain entries, sort to remove duplicates, etc. No big, and I'll post here if requested...

My first quandry was whether I could do Tom's import silently -- it appears the answer is 'yes' by simply removing the echo statements.

Now my problem is this: the porn list from URLblacklist.com is ~450,000 lines, and since a domain deny-rule only blocks the exact FQDN in the list, I have to duplicate that list with "*." pre-pended onto it. This raises an immediate issue and a question:

Issue: I started Tom's import script ~8 hours ago on this ~900,000-line file, and it has read ~16MB of the 18MB file so far. This is on a relatively recent test machine, with no other activity...

Question: what is the performance penalty for long lists of domains or URLs? I.e., on a 3.2GHz system with 4GB RAM, will a list of 900,000 domains in a block-list noticeably impact performance, or does ISA create a state-table to quickly hit/miss these lists?

Thanks,
Bobby

[ September 23, 2005, 09:18 AM: Message edited by: Bobby ]

(in reply to tshinder)
Post #: 19
RE: Discussion for article on using scripts for Domain ... - 24.Sep.2005 12:55:00 PM   
hwilkins

 

Posts: 37
Joined: 30.Jul.2004
From: Memphis
Status: offline
Does anyone know of a way to script the import of an exported XML file? Also, is it any faster than the import script Tom provided?

For very large lists, the import script provided at the top gets very slow and allocates ~10x the size of the list in RAM (i.e., an 18MB, 900,000 entry list took 12 hours and allocated 278MB RAM in the process).

I'm hoping that a scripted import of a well-formattred XML file will use less resources...

(in reply to tshinder)
Post #: 20

Page:   [1] 2   next >   >> << Older Topic    Newer Topic >>
All Forums >> [ISA Server 2004 Firewall] >> Access Policies >> Discussion for article on using scripts for Domain Name and URL Sets Page: [1] 2   next >   >>
Jump to:

New Messages No New Messages
Hot Topic w/ New Messages Hot Topic w/o New Messages
Locked w/ New Messages Locked w/o New Messages
 Post New Thread
 Reply to Message
 Post New Poll
 Submit Vote
 Delete My Own Post
 Delete My Own Thread
 Rate Posts