Welcome to ISAserver.org

Forums | Register | Login | My Profile | Inbox | RSS RSS icon | My Subscription | My Forums | Address Book | Member List | Search | FAQ | Ticket List | Log Out

ISA SQL Instance Using a Lot of Memory

Users viewing this topic: none

Logged in as: Guest
  Printable Version
All Forums >> [ISA Server 2004 General ] >> ISA 2004 SBS >> ISA SQL Instance Using a Lot of Memory Page: [1]
Login
Message << Older Topic   Newer Topic >>
ISA SQL Instance Using a Lot of Memory - 24.Oct.2005 10:32:00 AM   
HandyAndy

 

Posts: 29
Joined: 1.Aug.2005
From: Kernersville, NC USA
Status: offline
I have a number of SBS installs running ISA 2004 on Full SQL, they use very little memory (around 7,500k). I recently installed a new system and this one is using a lot of memory. It is a plain vanella install, and the only tweak to ISA was to allow ftp and the default rule WSUS installs for itself. Here is the memory useage.
""

So what would cause ISA's SQL Instance to use so much memory?

Thank You,
HA
Post #: 1
RE: ISA SQL Instance Using a Lot of Memory - 27.Oct.2005 9:22:00 PM   
Guest
MSDE is a hog. I had to throttle that sucker on my box as well.

Did you see the throttle commands on the www.smallbizserver.net web site for this?

(in reply to HandyAndy)
  Post #: 2
RE: ISA SQL Instance Using a Lot of Memory - 3.Nov.2005 11:10:00 AM   
Guest
Hello:

There's an easy-to-use script to limit the amount of memory MSDE consumes at www.ISAscripts.org, it's named ISA_MSDE_Max_Memory.vbs.

Another one there can gracefully detach a database so it can be deleted or moved; it's named ISA_MSDE_Detach_Database.vbs.

Cheers.

(in reply to HandyAndy)
  Post #: 3
RE: ISA SQL Instance Using a Lot of Memory - 11.Nov.2005 3:25:40 AM   
MRIS

 

Posts: 58
Joined: 4.Aug.2005
Status: offline
The answer is that there is a memory leak in the SQL server (MSDE) that even if you update the MSDE with Service Pack 4 for SQL Server MSDE, doesn't get resolved. The memory is consumed by a single client connection to the MSDE, which just grows over time and never shrinks. The connection is from the firewall service to the MSDE's master table, and is used exclusively to repeatedly query the database for INANE information such as the full path names of the actual database files it has. It's essentially the firewall service inventorying the database about it's files, quering their size, and how many there are etc etc. However, left unchecked, this client connection's memory allocation just grows and never shrinks. But here is the kicker, simply right-clicking on the connection using enterprise manager and saying "kill connection", does just that. The firewall service doesn't miss a beat and merely reconnects and continues asking it's INANE questions. However all the memory previously occupied by the just-killed connection is released, and the process of the new client connection growing it's RAM use starts again.

So how do you fix it? Easy, there are a few choices:

a. don't use the MSDE for logging, rather just log to text files. - disadvantages is that logging is much poorer under this config, with loss of some reporting features.
b. don't log at all. - disadvantages speak for themselves, but it's a valid option.
c. use a full copy of SQL server instead - disadvantages include that the firewall can't automatically manage the size of the database, this has to be managed manually.
d. Do what I did: Create a SQLAgent scheduled task using enterprise manager that queries the database for any connections that are all of the following:
1. to the master table
2. from the firewall service
3. consuming more than 40MB of RAM.
(it will only ever find one that matches all of these conditions)
- kill it.

That's exactly what's happening here, and it does it on all instances of ISA2004 with SP1 that log to the included MSDE. The people who haven't noticed it yet are happily ignorant of WHY their MSDE sqlservr.exe process is consuming 1.7GB of RAM, merely assuming it's using all that RAM as part of its normal functioning (eg buffers, index tables etc), not realising the subtelty of what's really going on; one buggy firewall services' simple, single connection to the master table repeatedly quering simple stuff that shouldn't need hundreds of megabytes or even GBs of RAM to efficiently process. Also most people don't "limit" the amount of RAM the sql process can consume, allowing it to grow unbound, thus the actual stopping of the firewall service bug doesn't occur within the lifetime of the session and they don't see it (eg they reboot their box every few weeks for other reasons). I only noticed it myself after I had limited the memory allowed for the SQL process to 23MB, causing the memory exhaustion effect to occur within about 3 days on a small 5-user LAN. Once the memory is exhausted, you can't even connect to the MSDE using enterprise manager, and neither can the firewall. So I wrote a script that is automatically run once a day that tracks down this rogue memory-soaked connection from the firewall service to the master table and simply kills it.

(in reply to Guest)
Post #: 4
RE: ISA SQL Instance Using a Lot of Memory - 12.Nov.2005 10:03:55 PM   
sam_hunter

 

Posts: 46
Joined: 12.Nov.2005
Status: offline
Any word on if this will be fixed?

(in reply to MRIS)
Post #: 5
RE: ISA SQL Instance Using a Lot of Memory - 13.Nov.2005 12:36:57 AM   
Jim Harrison

 

Posts: 231
Joined: 5.May2001
From: Redmond, WA
Status: offline
Actually, the answer is:

http://support.microsoft.com/?id=909636


_____________________________

Jim Harrison
MCP(NT4, W2K), A+, Network+, PCG
My ISAServer.org Stuff
My Site

(in reply to sam_hunter)
Post #: 6
RE: ISA SQL Instance Using a Lot of Memory - 13.Nov.2005 2:53:28 AM   
MRIS

 

Posts: 58
Joined: 4.Aug.2005
Status: offline
Thanks Jim, but that's not the answer.  Its a method of limiting the memory the MSDE instance of SQL for the firewall can use.  If you do limit the memory, then the memory leak I described above will consume it all much quicker, and then the MSDE actually fails, rejecting any futher connections.  The firewall can't connect to the MSDE and then the Alert job that checks for database logging failure stops the firewall service altogether, resulting in that ISA server going completely offline.

(in reply to Jim Harrison)
Post #: 7
RE: ISA SQL Instance Using a Lot of Memory - 13.Nov.2005 8:09:04 AM   
Jim Harrison

 

Posts: 231
Joined: 5.May2001
From: Redmond, WA
Status: offline
In fact, this does work.
It's been proven many times over.

_____________________________

Jim Harrison
MCP(NT4, W2K), A+, Network+, PCG
My ISAServer.org Stuff
My Site

(in reply to MRIS)
Post #: 8
RE: ISA SQL Instance Using a Lot of Memory - 13.Nov.2005 9:39:37 AM   
MRIS

 

Posts: 58
Joined: 4.Aug.2005
Status: offline
Jim, we're talking about subtley different issues.  I agree that what you are saying is correct.  What you are saying is that using the script to limit the amount of RAM the MSDE SQL instance is allowed to consume does work.  It's also possible to use SQL Enterprise Manager to limit it in the same way.
However, what I am saying, is that even if you do set the limit, say, to 100MB, yes, the SQL MSDE is limited to that memory usage, BUT I am also saying tthat there is a connection, from the firewall service, to the master table in that MSDE, that leaks memory, and eventually consumes the entire 100MB allowed for SQL, causing the MSDE to fail at that point.

(in reply to Jim Harrison)
Post #: 9
RE: ISA SQL Instance Using a Lot of Memory - 13.Nov.2005 10:18:11 AM   
sam_hunter

 

Posts: 46
Joined: 12.Nov.2005
Status: offline
I've adjusted the MSDE ram in my server and ISA or MSDE does not fail.

When you say it's failing can you describe more fully?  I have seen underpowered servers and midnight maintenance cause the firewall service to fail.

(in reply to HandyAndy)
Post #: 10
RE: ISA SQL Instance Using a Lot of Memory - 13.Nov.2005 11:57:02 AM   
MRIS

 

Posts: 58
Joined: 4.Aug.2005
Status: offline
Hi Sam_Hunter,

It doesn't fail right away, it's something that happens over a period of time.  Here's how to precipitate it:  Set the memory limit of the MSDE SQL instance to something small, like 30MB.  The MSDE failure should happen within about 6 days of operation, assuming no reboots occur during that time.
Windows Server 2003 SP1
ISA Server 2004 SP1
SQL MSDE SP4

It is possible to view what's happening using Enterprise Manager.  Look down into the MSDE SQL Server: Management --> Current Activity --> Process Info
Then look at the connections from the client application "Microsoft Internet Security and Acceleration Server 2004" to the master table.  There should be 2 connections.  ONE of these connections will be using a LOT MORE memory, as seen in the "Memory Usage" column.  This figure will only climb.  Once the memory allocated to SQL server is all consumed, the MSDE fails, the firewall service is unable to (re)connect to the MSDE, and experiences a logging failure.  There's a predefined alert in ISA server that automatically STOPS the firewall service if a logging failure occurs.  With the firewall service stopped, the server falls OFF the network.

< Message edited by MRIS -- 13.Nov.2005 12:00:19 PM >

(in reply to sam_hunter)
Post #: 11
RE: ISA SQL Instance Using a Lot of Memory - 13.Nov.2005 10:10:27 PM   
sam_hunter

 

Posts: 46
Joined: 12.Nov.2005
Status: offline
SBS 2003 sp1
ISA sp1
MSDE SP4

Memory throttle set at 100

No failures here.

(in reply to HandyAndy)
Post #: 12
RE: ISA SQL Instance Using a Lot of Memory - 14.Nov.2005 5:05:32 AM   
MRIS

 

Posts: 58
Joined: 4.Aug.2005
Status: offline
but 6 days haven't gone by yet (for 25-30MB) for 100MB, you might have to wait for over a month.

(in reply to sam_hunter)
Post #: 13
RE: ISA SQL Instance Using a Lot of Memory - 14.Nov.2005 8:52:50 AM   
sam_hunter

 

Posts: 46
Joined: 12.Nov.2005
Status: offline
My SBS box has been with ISA 2004/msde since May.

Granted I patch each month, but it's not fallen over.  When we see ISA fall over it's because the hardware is underpowered not that the MSDE is the cause.

(in reply to HandyAndy)
Post #: 14
RE: ISA SQL Instance Using a Lot of Memory - 14.Nov.2005 8:53:38 AM   
sam_hunter

 

Posts: 46
Joined: 12.Nov.2005
Status: offline
BTW the recommendation for SBS boxes is 100.  One month we had no patches and it stayed up 60 days with no runs, no drips, no errors.

(in reply to sam_hunter)
Post #: 15
RE: ISA SQL Instance Using a Lot of Memory - 14.Nov.2005 11:44:18 AM   
MRIS

 

Posts: 58
Joined: 4.Aug.2005
Status: offline
ok, I'm willing to accept that.  It's entirely possible that the issue only affects specific users of ISA2004 due to some as yet undiscovered reason.  However I have heard a few reports and it's even happened to me that the MSDE becomes unresponsive, and researching this has in my case always yielded the result that the connection from the firewall service to the master table was the one consuming 99% of the available RAM allocated to the MSDE.

(in reply to sam_hunter)
Post #: 16
RE: ISA SQL Instance Using a Lot of Memory - 9.Jan.2006 10:33:43 AM   
matjaz

 

Posts: 5
Joined: 26.Jan.2005
Status: offline
Well quite funny that MSDE. After a while I saw these in event log:

A shortage of available memory caused the Firewall service to fail.
The Event Viewer Data window displays the number of active connections.

I turned off SQL logging and turn on text logging for 7 days period.
However, even after restart of server I try to stop MSDE and of course it won't allow me (without stopping other ISA
related services).

Q: How to remove SQL completely from ISA 2004 and provide working ISA services.

Thanks.

(in reply to MRIS)
Post #: 17
RE: ISA SQL Instance Using a Lot of Memory - 9.Jan.2006 3:18:59 PM   
matjaz

 

Posts: 5
Joined: 26.Jan.2005
Status: offline
Don't bother. I find out how.

(in reply to matjaz)
Post #: 18
RE: ISA SQL Instance Using a Lot of Memory - 31.Jan.2006 8:47:37 PM   
RedSunshine

 

Posts: 35
Joined: 14.Apr.2003
From: Dallas, TX
Status: offline
This is very interesting...

I have a 2003 Standard Server (not SBS) with ISA loaded.  It is running ISA SP1.  Also I have the full-blown SQL Standard on this box and it is patched to SP3.  I have used Enterprise Manager to set the MSDE (MSFW Instance) to use a fixed amount of 80MB of RAM.  This box has been running strong for 210 days.  The MSDE instance is currently using 64MB or RAM.

Any thoughts are appreciated.  I'm just wondering why some would have ISA failures while obviously I do not.

(in reply to matjaz)
Post #: 19
RE: ISA SQL Instance Using a Lot of Memory - 9.Aug.2007 9:16:57 AM   
almo

 

Posts: 1
Joined: 9.Aug.2007
Status: offline
So, what's strange to me is that I have text logging (MSDE shouldn't even need to start up, although it does), have the max mem configured to 384MB, and I still see rapid growth in the SQL ERRORLOG file (starting database BLAH, milliseconds later starting database FOOBAR) and memory swell.

COME ON Microsoft, this issue has been known for a while; I am running ISA 2006(!) and have MSDE logging turned *off* but it still is a dependent service, I'd like to use the memory for more productive stuff like firewalling maybe?!

(in reply to MRIS)
Post #: 20

Page:   [1] << Older Topic    Newer Topic >>
All Forums >> [ISA Server 2004 General ] >> ISA 2004 SBS >> ISA SQL Instance Using a Lot of Memory Page: [1]
Jump to:

New Messages No New Messages
Hot Topic w/ New Messages Hot Topic w/o New Messages
Locked w/ New Messages Locked w/o New Messages
 Post New Thread
 Reply to Message
 Post New Poll
 Submit Vote
 Delete My Own Post
 Delete My Own Thread
 Rate Posts