-------- Original-Nachricht -------- Betreff: Re: [Xotcl] memory leak analysis Datum: Fri, 30 Sep 2005 19:25:52 +0200 Von: Gustaf Neumann neumann@wu-wien.ac.at An: Ben Thomasson ben.thomasson@gmail.com Referenzen: 44eb229c0509300852m10e879a7k@mail.gmail.com
Ben Thomasson schrieb:
Hi,
I am writing a simulation engine using xotcl. I have noticed as I create and destroy objects over time this process will use all available memory. I have tracked down and eliminated old objects.
what do you mean by "old objects"?
With the object count staying roughly constant the memory continues to increase at about 1k per second. What tools or tricks are useful for tracking down memory leaks in Tcl or XOTcl?
are you using theads?
If something is leaking memory it is most likely more complicated than just object creation and destruction. At least on my system the following script runs without any leaks.
for {set i 1} {$i < 10000000} {incr i} { set o [Object create obj-$i] $o destroy }
can it be that you are somewhere building a table (assoc. array) with object names?
xotcl has a bunch of compile flags to report memory usage, one can compile tcl with a flag that enables a command for memory checking.
-gustaf
Thanks,
Ben
-- You know you have reached perfection of design not when you have nothing more to add, but when you have nothing more to take away. --Antoine de Saint Exupéry
Xotcl mailing list Xotcl@alice.wu-wien.ac.at http://alice.wu-wien.ac.at/mailman/listinfo/xotcl
Am 30.09.2005 um 19:28 schrieb Gustaf Neumann:
What tools or tricks are useful for tracking down memory leaks in Tcl or XOTcl?
Tricks? No tricks. Hard work and lots of money to spend on good memory analyzer (Purify), heh...
And, watching "top" output for hours while doing "time {....} 100000" on some spots you *suspect* may be raining on your party!
Honestly, this is one of the most ugly time consuming boring unproductive difficult stupid (... you name it ...)
tasks you can imagine. Have been doing this for quite a few years already, so I know what I'm saying.
Couple of hints...
Is your Tcl or your C-code leaking? (this is not as dumb question as it may appear on the first glace...)
If you are using threads, try in a non-threaded env.
Try isolating parts of your code by divide/conquer method (start from top and work yourself down) and use "time" to repeat sections of code. At the same time keep an eye on the "top" utility... (Poor Man's Purify)
If you are running in an event mode, do you clean event loop fast enough?
If possible, try compilng *ALL* the C-code you use with -DTCL_MEM_DEBUG and read "memory" man-page. This is somehow difficult to use (gives A LOT of info) but in some cases it saved my skin.
As of my experience, Tcl/XOTcl are leak-free. If not I'd already scream aloud. If you are using any other C-level extensions, well...
If not, then most probably you do not care enough about destroying the objects. What I do is to (mostly) instantiate objects like this:
Object new -volatile
which guarantees to garbade-collect them when the scope (i.e. proc) were I new'ed them is gone. This won't work for "globaly" needed objects, though...
Memory "leaks" can be quite "hidden", for example arrays getting larger, lists getting longer, etc. etc. Those are not "leaks" per-se, hence never caught by any C-level memory debugger. For that you'd need to employ the poor-man's purify, like described above.
Eh... good luck!
Zoran
Zoran Vasiljevic wrote:
Am 30.09.2005 um 19:28 schrieb Gustaf Neumann:
What tools or tricks are useful for tracking down memory leaks in Tcl or XOTcl?
Tricks? No tricks. Hard work and lots of money to spend on good memory analyzer (Purify), heh...
And, watching "top" output for hours while doing "time {....} 100000" on some spots you *suspect* may be raining on your party!
Honestly, this is one of the most ugly time consuming boring unproductive difficult stupid (... you name it ...)
tasks you can imagine. Have been doing this for quite a few years already, so I know what I'm saying.
While I agree with difficult and sometimes boring, I find the best way to not make it an unproductive and time consuming task is to get a good memory analyzer, as stated. Purify really is worth the price, when such problems arise. We have a copy that I have used many times. Valgrind comes close, finds different things, but I still prefer Purify. Valgrind is free though, so there's not excuse not to try that.
Jeff Hobbs, The Tcl Guy http://www.ActiveState.com/, a division of Sophos
Am 30.09.2005 um 20:15 schrieb Jeff Hobbs:
We have a copy that I have used many times. Valgrind comes close, finds different things, but I still prefer Purify. Valgrind is free though, so there's not excuse not to try that.
Unfortunately (as you already know) ... Those will not report regular memory usage (C-wise) when you just expand an array, extend the list, flood the event loop, forget to destoy objects, etc. pp., indefinitetly. Hence, I found the Poor Man's Purify ([time] + glancing at the top display) equaly useful.
What I believe, (under assumption that Ben is using only Tcl/XOTcl code) is that he's hitting something like the above. I had those problems en mass until I persuaded Gustaf to make the "-volatile" mechanism in XOTcl...
Zoran
I recompiled tcl8.4.10 and xotcl1.3.6 for -DTCL_MEM_DEBUG=1 -g and -00 and then run valgrind on the attached scripts. Unfortunately I couldnt find anything in the debug outputs to pin point the problem. However I did find that the scripts memLeakMixin and memLeakMultiMixin do increase memory over time. memLeakClass does not increase memory over time.
It seems that the memory leak is related to the number of instmixins on the class A. memLeakMulitMixin uses far more memory than does memLeakMixin.
Any suggestions where to go next?
Ben
On 30/09/05, Zoran Vasiljevic zv@archiware.com wrote:
Am 30.09.2005 um 20:15 schrieb Jeff Hobbs:
We have a copy that I have used many times. Valgrind comes close, finds different things, but I still prefer Purify. Valgrind is free though, so there's not excuse not to try that.
Unfortunately (as you already know) ... Those will not report regular memory usage (C-wise) when you just expand an array, extend the list, flood the event loop, forget to destoy objects, etc. pp., indefinitetly. Hence, I found the Poor Man's Purify ([time] + glancing at the top display) equaly useful.
What I believe, (under assumption that Ben is using only Tcl/XOTcl code) is that he's hitting something like the above. I had those problems en mass until I persuaded Gustaf to make the "-volatile" mechanism in XOTcl...
Zoran
-- You know you have reached perfection of design not when you have nothing more to add, but when you have nothing more to take away. --Antoine de Saint Exupéry
Thanks for the advice. I have made sure to garbage collect my unneeded objects. I need to look into those other sources of memory usage.
My application is soley Tcl/XOTcl and does not use threads. The majority of the code is XOTcl. There are no C extensions.
I am using introspection to find objects and values of the variables in those objects. This has been very helpful in identifying "lost objects."
I will recompile the tcl and xotcl binaries to enable memory usage commands. I still suspect a list or array out of control. I guess I am in for a few more nights of staring at top.
Thanks for the time { ... } 10000... idea. That will be useful in my unit tests.
Thanks,
Ben
xotclsh
% package require XOTcl 1.3.6 % package require Tcl 8.4 %
On 30/09/05, Zoran Vasiljevic zv@archiware.com wrote:
Am 30.09.2005 um 19:28 schrieb Gustaf Neumann:
What tools or tricks are useful for tracking down memory leaks in Tcl or XOTcl?
Tricks? No tricks. Hard work and lots of money to spend on good memory analyzer (Purify), heh...
And, watching "top" output for hours while doing "time {....} 100000" on some spots you *suspect* may be raining on your party!
Honestly, this is one of the most ugly time consuming boring unproductive difficult stupid (... you name it ...)
tasks you can imagine. Have been doing this for quite a few years already, so I know what I'm saying.
Couple of hints...
Is your Tcl or your C-code leaking? (this is not as dumb question as it may appear on the first glace...)
If you are using threads, try in a non-threaded env.
Try isolating parts of your code by divide/conquer method (start from top and work yourself down) and use "time" to repeat sections of code. At the same time keep an eye on the "top" utility... (Poor Man's Purify)
If you are running in an event mode, do you clean event loop fast enough?
If possible, try compilng *ALL* the C-code you use with -DTCL_MEM_DEBUG and read "memory" man-page. This is somehow difficult to use (gives A LOT of info) but in some cases it saved my skin.
As of my experience, Tcl/XOTcl are leak-free. If not I'd already scream aloud. If you are using any other C-level extensions, well...
If not, then most probably you do not care enough about destroying the objects. What I do is to (mostly) instantiate objects like this:
Object new -volatile
which guarantees to garbade-collect them when the scope (i.e. proc) were I new'ed them is gone. This won't work for "globaly" needed objects, though...
Memory "leaks" can be quite "hidden", for example arrays getting larger, lists getting longer, etc. etc. Those are not "leaks" per-se, hence never caught by any C-level memory debugger. For that you'd need to employ the poor-man's purify, like described above.
Eh... good luck!
Zoran
-- You know you have reached perfection of design not when you have nothing more to add, but when you have nothing more to take away. --Antoine de Saint Exupéry
-
Ben Thomasson -
Gustaf Neumann -
Jeff Hobbs -
Zoran Vasiljevic