Re: Question about madvise(DONTNEED) in glibc malloc

[KOSAKI Motohiro <kosaki dot motohiro at gmail dot com>]

(4/15/13 5:45 AM), Rich Felker wrote:
> On Sun, Apr 14, 2013 at 10:01:52PM -0700, KOSAKI Motohiro wrote:
>> (4/14/13 9:28 PM), Rich Felker wrote:
>>> On Sun, Apr 14, 2013 at 09:37:16AM -0700, KOSAKI Motohiro wrote:
>>>> Hi all,
>>>>
>>>> Now, we linux MM folks discuss are discussing about new memory
>>>> discarding feature. (https://lkml.org/lkml/2013/3/12/105). The
>>>> motivation is similar wtih MADV_FREE, but more efficient.
>>>> (http://lwn.net/Articles/230799)
>>>
>>> If you're following this, do you know if vrange(VRANGE_NOVOLATILE) has
>>> failure cases? 
>>
>> If any pages have already been discarded, vrange(VRANGE_NOVOLATILE) return 1.
>> Subsequent memory access may cause minor page fault and makes page zero fill.
> 
> But vrange never fails as long as the mapping is valid?

Yes for current draft patch. but I don't expect final version also never fail.
It has several unimplement and every memory management syscall can fail. madvise() 
can fail too.

But, vrange() never makes implicit uncommit charge even though any irregular is happen.
Does this make sense? If you are interesting more corner case, you neeed to wait a while
because current patch is rough design and just for discussion.



> 
>>> This makes a big difference to how usable it is by
>>> malloc implementations. madvise re-mmapping new zero pages over the
>>> range are nice in that the range is not using physical memory, but
>>> still remains committed and immediately usable without any risk of
>>> failure to get it back. 
>>
>> Does you "commit" mean to talk about virtual address? If yes, vrange() never
>> call neither mmap/munmap implicitly.
>> If you are talk about physical pages, it may be discarded and disappeared. vrange()
>> user must not pass undroppable data.
> 
> I mean in the sense of commit charge. I would assume vrange does not
> adjust the commit charge, but this would be important to know.

Your assumption is correct.


>>> If vrange can fail to "get back" the range,
>>> this makes it a lot harder to use robustly.
>>
>> Any linux syscall _can_ return ENOMEM if system has really no memory.
>> but we've never seen practically because kernel handle memory starvation enough
>> clever.
> 
> No, there are plenty of syscalls that never fail. 

Ah, um getpid() and similar one never fail. yes. I only talked about memory
management syscall.


> And having this
> property is important. For example, if calling vrange would have to
> split up a range and thus add more ranges, the setting volatile mode
> could reasonably fail, 
> but setting non-volatile should just make the
> entire contained range non-volatile so that no splitting is required
> if it runs out of memory trying to do the split.

It depend. E.g. as you know, munmap() can increase number of maps and
fail by /proc/sys/vm/max_map_count limitation. In general, every range 
framework makes similar implicit merge (and subsequent splitting). To
avoid this, you need to gurantee neighber address region never be vrange()ed
area.