Re: memory usage

[Greg Harvey <Greg dot Harvey at thezone dot net>]

Miroslav Silovic <silovic@zesoi.fer.hr> writes:

> Christian Lynbech <chl@tbit.dk> writes:
> 
> > >>>>> "Lynn" == Lynn Winebarger <owinebar@free-expression.org> writes:
> > 
> > Lynn> ... grow to 183 megs! 
> > 
> > It sounds very much like you have been bitten by the heap growth
> > algorithm.
> > 
> > The default algorithm is (when the need to expand the heap has been
> > identified) to allocate a new segment twice as big as the previous
> > segment, which isn't so hot when you have a medium sized requirement
> > on heap size.
> 
> The issue is that it shouldn't -need- to grow. Heap growth only
> happens if GC hasn't freed enough memory, and that should only happen
> if the memory gets allocated too much.
> 
> In my experience this happens if SMOB free routine doesn't cleanup
> correctly, or if it returns the wrong value for the ammount of memory
> that got freed (the latter thing can really bite you).

Actually, in this case it's almost definately expmem causing a problem
it shouldn't. If after a collection you have less than a quarter of
the heap space available, twice the heap is allocated; this isn't so
bad in the small case, but if you have a 32 megabyte heap, another 32
megabytes is a really big (and not at all justified) leap. A quick and
pathetic example:

(current cvs guile)
guile> (define x (make-list 1000000))
guile> (gc)
guile> (gc-stats)
((gc-time-taken . 1195) (cells-allocated . 1049409) (cell-heap-size . 2654208) (bytes-malloced . 162318) (gc-malloc-threshold . 241264) (cell-heap-segments (1075367944 . 1075105800) (1075896328 . 1075372040) (1077473288 . 1075900424) (1082195976 . 1077477384) (1096355848 . 1082200072)))
guile> (* 2654208 8)
21233664
guile> (do ((i x (cdr i))) ((null? i) #t) (set-car! i (cons 'a 'b)))
#t
guile> (gc)
guile> (gc-stats)
((gc-time-taken . 2971) (cells-allocated . 2049458) (cell-heap-size . 7962624) (bytes-malloced . 162344) (gc-malloc-threshold . 241264) (cell-heap-segments (1075367944 . 1075105800) (1075896328 . 1075372040) (1077473288 . 1075900424) (1082195976 . 1077477384) (1096355848 . 1082200072) (1138827272 . 1096359944)))
guile> (* 8 7962624)
63700992

Here, guile has (really incorrectly) grown the heap to almost 64 megs,
when there isn't nearly that much being used. Even worse, every single
bit of that memory will be touched before the next collection, so
performance is going to go south if you don't have that much
memory. Just to contrast with what I think is correct behaviour (or at
least much closer to correct behavior), this is a run with my current
gc patch.

[greg /usr/src/guile-gc-opt]$  GUILEGC_INIT_SEG_SIZE=8192 GUILEGC_SEG_SIZE=524288 guile
guile> (gc-set! relative-heap-reclaimed 3)
(relative-heap-reclaimed 3)
guile> (define x (make-list 1000000))
guile> (gc-stats)
((gc-time-taken . 1115) (gc-max-mark-stack-size . 256) (gc-n-collections . 35) (cells-allocated-1 . 1445986) (heap-size-1 . 1581056) (usable-heap-size-1 . 1577968) (gc-min-gc-yield-1 . 525989) (cells-allocated-2 . 0) (heap-size-2 . 0) (usable-heap-size-2 . 0) (gc-min-gc-yield-2 . 0) (bytes-malloced . 163638) (gc-malloc-threshold . 181788) (cell-heap-segments (134586368 . 134520832) (1079349248 . 1075154944) (1083551744 . 1079357440) (1087754240 . 1083559936)))
guile> (do ((i x (cdr i))) ((null? i) #t) (set-car! i (cons 'a 'b)))
#t
guile> (gc-stats)
guile> (gc)
guile> (gc-stats)
((gc-time-taken . 2674) (gc-max-mark-stack-size . 256) (gc-n-collections . 48) (cells-allocated-1 . 2525107) (heap-size-1 . 3153920) (usable-heap-size-1 . 3147760) (gc-min-gc-yield-1 . 1049253) (cells-allocated-2 . 0) (heap-size-2 . 0) (usable-heap-size-2 . 0) (gc-min-gc-yield-2 . 0) (bytes-malloced . 164197) (gc-malloc-threshold . 181788) (cell-heap-segments (134586368 . 134520832) (1079349248 . 1075154944) (1083551744 . 1079357440) (1087754240 . 1083559936) (1091956736 . 1087762432) (1096159232 . 1091964928) (1100361728 . 1096167424)))
guile> (* 8 3153920)
25231360

I have cheated a bit by modifying the gc parameters. Without the
modification, it uses a bit less memory, but takes about twice as long
in the gc... even so, if the first one starts swapping, the doubled gc
time would look really great in comparison ;).

-- 
Greg