Hello everyone, here's a problem:
I have a file that is mmap'ped (read-only) into the process address space. However the file is big (say 1GB) and a really heavy random access is planned.
I need to map the file because several unrelated processes (i.e. different images, different users) may want to use the file simultaneously. Therefore, just allocating some memory with malloc (or mmap with MAP_ANON, etc.) and then read'ing it would be a painful waste of resources.
Computer has sufficient amount of memory so that aforementioned malloc would work well without any swapping.
The problem is that mapping a file that is not fully in disk cache usually means that each of the random accesses will cause a page fault, then disk read (that includes disk head seek), 100% disk busy, slow program and slow system.
I need to take additional steps around the time of the mmap call to make sure that my file is actually brought up into physical memory and (almost) no page faults will occur, that is, the result of (mmap + something) must be as convinient for random access as (malloc + read).
There's one solution that works pooly in certain conditions comparing to (malloc + read):
After mmap, scan ("precharge") the mapped memory area, that is, read 1-2 bytes from each page, altogether with using madvise(MADV_WILLNEED) upon the mapped area.
However, that does not work well when there's even a moderate disk activity in the system. Precharge step may even take minutes while read()'ing 1GB file still takes 4-10 seconds (250MB/s of sequential read from relatively fast SCSI array). That makes seemingly better and os-friendly solution of mmap lose to inefficient malloc+read.
How can I overcome that problem? I'm seeking advice either in a form of some userland code or even patching FreeBSD kernel.
Other advices from this thread that didn't help:
> try using mlock(2)
It is dangerous and actually works for super-user only.
I have a file that is mmap'ped (read-only) into the process address space. However the file is big (say 1GB) and a really heavy random access is planned.
I need to map the file because several unrelated processes (i.e. different images, different users) may want to use the file simultaneously. Therefore, just allocating some memory with malloc (or mmap with MAP_ANON, etc.) and then read'ing it would be a painful waste of resources.
Computer has sufficient amount of memory so that aforementioned malloc would work well without any swapping.
The problem is that mapping a file that is not fully in disk cache usually means that each of the random accesses will cause a page fault, then disk read (that includes disk head seek), 100% disk busy, slow program and slow system.
I need to take additional steps around the time of the mmap call to make sure that my file is actually brought up into physical memory and (almost) no page faults will occur, that is, the result of (mmap + something) must be as convinient for random access as (malloc + read).
There's one solution that works pooly in certain conditions comparing to (malloc + read):
After mmap, scan ("precharge") the mapped memory area, that is, read 1-2 bytes from each page, altogether with using madvise(MADV_WILLNEED) upon the mapped area.
However, that does not work well when there's even a moderate disk activity in the system. Precharge step may even take minutes while read()'ing 1GB file still takes 4-10 seconds (250MB/s of sequential read from relatively fast SCSI array). That makes seemingly better and os-friendly solution of mmap lose to inefficient malloc+read.
How can I overcome that problem? I'm seeking advice either in a form of some userland code or even patching FreeBSD kernel.
Other advices from this thread that didn't help:
> try using mlock(2)
It is dangerous and actually works for super-user only.