Solved How to get virtual memory size of current process in C/C++?

Hi,

I am porting my C++ application from Linux/macOS/Windows to FreeBSD and there is one detail I cannot solve.

How does a C/C++ application read its current virtual memory size (for the current process)?

The application is self-monitoring and periodically checks its usage of the system. Detecting that the virtual memory size always increases without reaching a stable point is a symptom of memory leak. Detecting it before it is too late (ie. crash) allows various forms of actions or recovery.

On Linux, I parse /proc/self/stat. On macOS, I call task_info(). On Windows, I call GetProcessMemoryInfo(). I am looking for something equivalent on FreeBSD but I couldn't find anything useful so far.

Thanks for any info.
 
libprocstat(3)
procstat_getvmmap
if you have proc mounted (not default) /proc/pid/map
I cannot assume that the user has mounted /proc. This is an open source application and users have various environments. I just installed my first FreeBSD VM for this port and /proc is not mounted. So, I assume that we should not expect it in the general case.
 
execl(const char *path,arg1,arg2,arg3, NULL);
path="/bin/ps"
arg1="-o"
arg2="vsz"
arg3="process_id"
Plus creating a pipe, reading it, parsing the output and synchronizing on process termination and pipe cleanup.

Thanks for the suggestion but, honestly, there must be something more straightforward for such a basic system information.
 
libprocstat(3)
procstat_getvmmap
if you have proc mounted (not default) /proc/pid/map
I may have misunderstood your response. Does procstat_getvmmap() access /proc or do you mention two independent methods, one using procstat_getvmmap (and not requiring /proc to be mounted) and the other one using /proc?
 
Plus creating a pipe, reading it, parsing the output and synchronizing on process termination and pipe cleanup.

Thanks for the suggestion but, honestly, there must be something more straightforward for such a basic system information.

Plus creating a pipe, reading it, parsing the output and synchronizing on process termination and pipe cleanup
My bad , small improvement,

Code:
char *comm="ps -o vsz= 1";
FILE *fp;
char bp[100];
fp=popen(comm,"r");
fgets(bp,99,fp);
printf("Virtual memory Size is : %s",bp);
pclose(fp);
 
Not necessary.
This works fine:
Code:
#include <stdio.h>
#include<signal.h>
#include<unistd.h>
void sig_handler(int signum){printf("\nInterrupt\n");}
int main(void){
    signal(SIGINT,sig_handler);
    char *comm="top -n 5 -d 1000 -b";
    FILE *fp;
    char bp[100];
    fp=popen(comm,"r");
    while(fgets(bp,99,fp)!=NULL)
        printf("%s",bp);
    pclose(fp);
    return 0;
}
 
popen(3) is never an improvement, quite to the contrary. Read the BUGS section. Be aware it additionally forks a shell, etc...

And apart from that, never ever fork an external process when there's some simple API to achieve the same.
 
procstat_getvmmap() works without procfs
the output of procstat -vm <pid> is similar with cat /proc/<pid>/map
I wonder where struct procstat , struct kfinfo, is defined a what it is or how to call it ?
Code:
procstat_getvmmap(struct procstat *procstat, struct kinfo_proc *kp,unsigned int *count);
[It's above my head]
 
I wonder where struct procstat , struct kfinfo, is defined a what it is or how to call it ?
Code:
procstat_getvmmap(struct procstat *procstat, struct kinfo_proc *kp,unsigned int *count);
[It's above my head]
That's what you call an opaque pointer. It isn't defined at all (except in the implementation code itself). As a consumer, you must not care what exactly it points to. You obtain it with procstat_open_sysctl(), and when you're done with it, you call procstat_close() on it to clean up.

edit, in case you ask "why": An opaque pointer enables strong information hiding in C. All you tell the consuming code is "a struct by that name exists", which enables this code to work with pointers to that struct, but never the struct itself. Opaque pointers are commonly used in OOP-style APIs written in C, the pointer acts as the "object reference".
 
procstat_getvmmap() works without procfs
the output of procstat -vm <pid> is similar with cat /proc/<pid>/map
Thanks covacat for pointing me to procstat_getvmmap().

I have written some sample code below to get an idea of what is available from the struct kinfo_vmentry. To characterize the impact of memory allocation on VM size, the command line parameter is the size of an area to malloc(). The area is also written to make that the pages are committed (don't know if this is the right work for FreeBSD).

In each struct kinfo_vmentry, there is a start and end virtual addresses which look properly page-aligned. To get the total VM size of the process, I sum the end-start of each struct kinfo_vmentry.

The result is a bit weird and is probably incorrect. And definitely inconsistent. I would like your advice on it. See code and its output below.
  • The total VM size is 550 MB, a bit high for such a simple application.
  • There are 36 segments, one of them accounting for 536 MB (at address 00007FFFDFFFF000)
  • For each run, the segment addresses and sizes are strictly identical, even though on one run I malloc 100 MB and nothing on another run. Nowhere we can see the malloc'ated 100 MB.
I can understand that some very large segments are mapped, so I do not worry too much about the high value. However, it is absolutely inconsistent that the exact same VM size is seen when we allocate 100 MB. I even allocated 1 GB and the total VM size is still 550 MB.

So, I do not know exactly what the struct kinfo_vmentry contains but it cannot be the state of the calling process.

Or is there an error in initializing with procstat_getprocs(pstat, KERN_PROC_PID, getpid(), &kproc_count); ?

As a side question, the addresses are the same at each run, isn't there any form of ASLR in FreeBSD?

Thanks for your help.

C:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <locale.h>
#include <unistd.h>
#include <kvm.h>
#include <sys/param.h>
#include <sys/queue.h>
#include <sys/socket.h>
#include <sys/sysctl.h>
#include <sys/user.h>
#include <libprocstat.h>

int main(int argc, char* argv[])
{
    setlocale(LC_NUMERIC, "en_US.UTF-8");

    int size = 0;
    if (argc > 1 && (size = atoi(argv[1])) > 0) {
        printf("Allocating %'d bytes\n", size);
        memset(malloc(size), 0x15, size);
    }

    struct procstat* pstat = procstat_open_sysctl();
    if (pstat == NULL) {
        perror("procstat_open_sysctl");
        return EXIT_FAILURE;
    }

    unsigned int kproc_count = 0;
    struct kinfo_proc* kproc = procstat_getprocs(pstat, KERN_PROC_PID, getpid(), &kproc_count);
    if (kproc == NULL) {
        perror("procstat_getprocs");
        return EXIT_FAILURE;
    }
    printf("Got %d struct kinfo_proc\n", kproc_count);
    if (kproc_count == 0) {
        printf("No process found\n");
        return EXIT_FAILURE;
    }

    unsigned int vmmap_count = 0;
    struct kinfo_vmentry* kvm = procstat_getvmmap(pstat, kproc, &vmmap_count);
    if (kvm == NULL) {
        perror("procstat_getvmmap");
        return EXIT_FAILURE;
    }
    printf("Got %d struct kinfo_vmentry\n", vmmap_count);

    uint64_t vsize = 0;
    for (unsigned int i = 0; i < vmmap_count; ++i) {
        printf("%2d:  %016lX-%016lX  %12'lu bytes\n", i, kvm[i].kve_start, kvm[i].kve_end, kvm[i].kve_end - kvm[i].kve_start);
        vsize += kvm[i].kve_end - kvm[i].kve_start;
    }
    printf("VM size: %'lu bytes\n", vsize);

    procstat_freevmmap(pstat, kvm);
    procstat_freeprocs(pstat, kproc);
    procstat_close(pstat);
    return EXIT_SUCCESS;
}

And the output is, regardless of the size of the malloc'ated area:
Code:
$ ./vmmap 100000000
Allocating 100,000,000 bytes
Got 1 struct kinfo_proc
Got 36 struct kinfo_vmentry
 0:  0000000000200000-0000000000201000         4,096 bytes
 1:  0000000000201000-0000000000202000         4,096 bytes
 2:  0000000000202000-0000000000204000         8,192 bytes
 3:  0000000000204000-0000000000205000         4,096 bytes
 4:  0000000800204000-000000080020A000        24,576 bytes
 5:  000000080020A000-0000000800221000        94,208 bytes
 6:  0000000800221000-0000000800222000         4,096 bytes
 7:  0000000800222000-0000000800245000       143,360 bytes
 8:  0000000800246000-000000080024A000        16,384 bytes
 9:  000000080024A000-0000000800252000        32,768 bytes
10:  0000000800252000-0000000800253000         4,096 bytes
11:  0000000800253000-0000000800254000         4,096 bytes
12:  0000000800254000-00000008002D8000       540,672 bytes
13:  00000008002D8000-0000000800424000     1,359,872 bytes
14:  0000000800424000-000000080042C000        32,768 bytes
15:  000000080042C000-000000080042D000         4,096 bytes
16:  000000080042D000-0000000800434000        28,672 bytes
17:  0000000800434000-000000080065E000     2,269,184 bytes
18:  000000080065E000-0000000800665000        28,672 bytes
19:  0000000800665000-0000000800677000        73,728 bytes
20:  0000000800677000-0000000800678000         4,096 bytes
21:  0000000800678000-0000000800679000         4,096 bytes
22:  0000000800679000-0000000800680000        28,672 bytes
23:  0000000800680000-000000080068B000        45,056 bytes
24:  000000080068B000-000000080068C000         4,096 bytes
25:  000000080068C000-000000080068E000         8,192 bytes
26:  000000080068E000-0000000800696000        32,768 bytes
27:  0000000800696000-00000008006A1000        45,056 bytes
28:  00000008006A1000-00000008006A2000         4,096 bytes
29:  00000008006A2000-00000008006A3000         4,096 bytes
30:  00000008006A3000-00000008006A4000         4,096 bytes
31:  00000008006A4000-00000008006A6000         8,192 bytes
32:  0000000800800000-0000000801000000     8,388,608 bytes
33:  00007FFFDFFFF000-00007FFFFFFDF000   536,739,840 bytes
34:  00007FFFFFFDF000-00007FFFFFFFF000       131,072 bytes
35:  00007FFFFFFFF000-0000800000000000         4,096 bytes
VM size: 550,137,856 bytes
 
look at /usr/src/usr.sbin/procstat/procstat_vm.c
and output of procstat -v $$
look at kve_type flags it might be a guard zone / not actually mapped (see mmap MAP_GUARD)
 
OK, I found the issue. The compiler aggressively optimized away the following line:
Code:
memset(malloc(size), 0x15, size);

I realized that when "allocating" 100 GB. The program should have crashed. I do not have 100 GB of mem/swap in the VM. Instead, it immediately succeeded.

The following code no longer optimizes the calls to malloc() and memset() because the address is printed:
Code:
void* mem = malloc(size);
printf("mem:  %016lX\n", (unsigned long)(mem));
memset(mem, 0x15, size);

Now, I can see the difference in virtual size when malloc'ating different sizes.

Second point, it seems that the "actual" vsize of the process is the sum of all kinfo_vmentry sizes, except the big ones with 536 MB at addresses 00007FFFDFFFF000 and higher. If you substract this, you get the vsize as displayed by ps.

What are these segments? Kernel space mapping?

Third point, looking at the kinfo_proc, there is a field defined as follow in sys/user.h:
Code:
vm_size_t ki_size;              /* virtual size */
And, as expected, it contains the vsize, as displayed by ps, as evaluated by summing the kinfo_vmentry with "low" addresses.

As a conclusion, the answer to my original question - getting the total VM size if the process - is in procstat_getprocs(). No need to call procstat_getvmmap(), unless you need an analysis of each segment.

Thanks again for your help.
 
For the record this looks like the right function,
Code:
struct kinfo_proc {
    int    ki_structsize;        /* size of this structure */
    int    ki_layout;        /* reserved: layout identifier */
    struct    pargs *ki_args;        /* address of command arguments */
    struct    proc *ki_paddr;        /* address of proc */
    struct    user *ki_addr;        /* kernel virtual addr of u-area */
    struct    vnode *ki_tracep;    /* pointer to trace file */
    struct    vnode *ki_textvp;    /* pointer to executable file */
    struct    filedesc *ki_fd;    /* pointer to open file info */
    struct    vmspace *ki_vmspace;    /* pointer to kernel vmspace struct */
    const void *ki_wchan;        /* sleep address */
    pid_t    ki_pid;            /* Process identifier */
    pid_t    ki_ppid;        /* parent process id */
    pid_t    ki_pgid;        /* process group id */
    pid_t    ki_tpgid;        /* tty process group id */
    pid_t    ki_sid;            /* Process session ID */
    pid_t    ki_tsid;        /* Terminal session ID */
    short    ki_jobc;        /* job control counter */
    short    ki_spare_short1;    /* unused (just here for alignment) */
    uint32_t ki_tdev_freebsd11;    /* controlling tty dev */
    sigset_t ki_siglist;        /* Signals arrived but not delivered */
    sigset_t ki_sigmask;        /* Current signal mask */
    sigset_t ki_sigignore;        /* Signals being ignored */
    sigset_t ki_sigcatch;        /* Signals being caught by user */
    uid_t    ki_uid;            /* effective user id */
    uid_t    ki_ruid;        /* Real user id */
    uid_t    ki_svuid;        /* Saved effective user id */
    gid_t    ki_rgid;        /* Real group id */
    gid_t    ki_svgid;        /* Saved effective group id */
    short    ki_ngroups;        /* number of groups */
    short    ki_spare_short2;    /* unused (just here for alignment) */
    gid_t    ki_groups[KI_NGROUPS];    /* groups */
    vm_size_t ki_size;        /* virtual size */
    segsz_t ki_rssize;        /* current resident set size in pages */
    segsz_t ki_swrss;        /* resident set size before last swap */
    segsz_t ki_tsize;        /* text size (pages) XXX */
    segsz_t ki_dsize;        /* data size (pages) XXX */
    segsz_t ki_ssize;        /* stack size (pages) */
    u_short    ki_xstat;        /* Exit status for wait & stop signal */
    u_short    ki_acflag;        /* Accounting flags */
    fixpt_t    ki_pctcpu;         /* %cpu for process during ki_swtime */
    u_int    ki_estcpu;         /* Time averaged value of ki_cpticks */
    u_int    ki_slptime;         /* Time since last blocked */
    u_int    ki_swtime;         /* Time swapped in or out */
    u_int    ki_cow;            /* number of copy-on-write faults */
    u_int64_t ki_runtime;        /* Real time in microsec */
    struct    timeval ki_start;    /* starting time */
    struct    timeval ki_childtime;    /* time used by process children */
    long    ki_flag;        /* P_* flags */
    long    ki_kiflag;        /* KI_* flags (below) */
    int    ki_traceflag;        /* Kernel trace points */
    char    ki_stat;        /* S* process status */
    signed char ki_nice;        /* Process "nice" value */
    char    ki_lock;        /* Process lock (prevent swap) count */
    char    ki_rqindex;        /* Run queue index */
    u_char    ki_oncpu_old;        /* Which cpu we are on (legacy) */
    u_char    ki_lastcpu_old;        /* Last cpu we were on (legacy) */
    char    ki_tdname[TDNAMLEN+1];    /* thread name */
    char    ki_wmesg[WMESGLEN+1];    /* wchan message */
    char    ki_login[LOGNAMELEN+1];    /* setlogin name */
    char    ki_lockname[LOCKNAMELEN+1]; /* lock name */
    char    ki_comm[COMMLEN+1];    /* command name */
    char    ki_emul[KI_EMULNAMELEN+1];  /* emulation name */
    char    ki_loginclass[LOGINCLASSLEN+1]; /* login class */
    char    ki_moretdname[MAXCOMLEN-TDNAMLEN+1];    /* more thread name */
    /*
     * When adding new variables, take space for char-strings from the
     * front of ki_sparestrings, and ints from the end of ki_spareints.
     * That way the spare room from both arrays will remain contiguous.
     */
    char    ki_sparestrings[46];    /* spare string space */
    int    ki_spareints[KI_NSPARE_INT];    /* spare room for growth */
    uint64_t ki_tdev;        /* controlling tty dev */
    int    ki_oncpu;        /* Which cpu we are on */
    int    ki_lastcpu;        /* Last cpu we were on */
    int    ki_tracer;        /* Pid of tracing process */
    int    ki_flag2;        /* P2_* flags */
    int    ki_fibnum;        /* Default FIB number */
    u_int    ki_cr_flags;        /* Credential flags */
    int    ki_jid;            /* Process jail ID */
    int    ki_numthreads;        /* XXXKSE number of threads in total */
    lwpid_t    ki_tid;            /* XXXKSE thread id */
    struct    priority ki_pri;    /* process priority */
    struct    rusage ki_rusage;    /* process rusage statistics */
    /* XXX - most fields in ki_rusage_ch are not (yet) filled in */
    struct    rusage ki_rusage_ch;    /* rusage of children processes */
    struct    pcb *ki_pcb;        /* kernel virtual addr of pcb */
    void    *ki_kstack;        /* kernel virtual addr of stack */
    void    *ki_udata;        /* User convenience pointer */
    struct    thread *ki_tdaddr;    /* address of thread */
    /*
     * When adding new variables, take space for pointers from the
     * front of ki_spareptrs, and longs from the end of ki_sparelongs.
     * That way the spare room from both arrays will remain contiguous.
     */
    struct    pwddesc *ki_pd;    /* pointer to process paths info */
    void    *ki_spareptrs[KI_NSPARE_PTR];    /* spare room for growth */
    long    ki_sparelongs[KI_NSPARE_LONG];    /* spare room for growth */
    long    ki_sflag;        /* PS_* flags */
    long    ki_tdflags;        /* XXXKSE kthread flag */
};
Containing,
Code:
vm_size_t ki_size;        /* virtual size */

E.g.,
Code:
#include <sys/types.h>
#include <sys/user.h>
#include <libutil.h>
#include <stdio.h>
int main(void){
    int procesid=1;
    struct kinfo_proc * k=kinfo_getproc(procesid);
    vm_size_t s=k->ki_size;
    printf("%lu",s/1024);
    return 0;
}
How did you find this function ?
 
That's reserved space (not fully allocated) for stack on 64b system by default. Will match the ulimit -s size.
Thanks. Is there a way to know how much was allocated so far in that space? Reserving virtual addresses is one thing, mapping virtual memory is another one. Is this accounted in ki_size? When we are interested in the VM resource of a process, we need to include stack space.

I do not know how the FreeBSD kernel works. From my experiences with other systems, I am used to large reserved stack addresses with a current allocation boundary. When the stack grows beyond the lowest allocated address of the stack, the invalid memory access exception which is generated enlarges the stack allocation, moving the new boundary below. But at any time, it is possible to identify where the current boundary is and consequently how much VM it uses.

This is just out of curiosity. My initial question was motivated by an early identification of potential memory leaks. And memory leaks don't come from the stack.
 
This is just out of curiosity. My initial question was motivated by an early identification of potential memory leaks. And memory leaks don't come from the stack.

If you want to detect leaks, then I recommend
  1. heaptrack (quite fast but won't handle custom allocators, has its own nice GUI).
  2. Valgrind memcheck (much slower, also detects other issues, can handle custom allocators but requires code instrumentation)
  3. Valgrind massif (not really for leak detection but when used with --pages-as-heap=yes it will really handle any kind of memory allocation)
I don't have much experience with heaptrack on FreeBSD.

Other than instrumentation for custom allocators none of the above require any modification.

I also saw that some work has been done on leak sanitizer, but I'm not sure what the state of play is. That would require rebuilding your executable.
 
I am used to large reserved stack addresses with a current allocation boundary. When the stack grows beyond the lowest allocated address of the stack, the invalid memory access exception which is generated enlarges the stack allocation, moving the new boundary below.
Map boundaries are boundaries for valid page access. Actions are taken depending on what that page is pointing to. Stepping outside the region means segmentation fault. You can move boundaries (sbrk), you can create new regions (mmap).

There is lot of good stuff written to FreeBSD virtual memory management. Summary of FreeBSD virtual memory. struct wise I'd start from the struct thread, go through vmspace and vm_map structure.
Recently I showed how to use dtrace to check these structures.

FreeBSD has one handy tool - procstat(1) - which can be handy. As an example csh:
Code:
# procstat vm $$
  PID              START                END PRT  RES PRES REF SHD FLAG  TP PATH
 1292          0x1021000          0x1038000 r--   17  102   5   3 CN--- vn /bin/csh
 1292          0x1038000          0x1089000 r-x   81  102   5   3 CN--- vn /bin/csh
 1292          0x1089000          0x108a000 r--    1    0   1   0 CN--- vn /bin/csh
 1292          0x108a000          0x108b000 rw-    1    0   1   0 CN--- vn /bin/csh
 1292          0x108b000          0x108f000 rw-    4    0   1   0 CN--- vn /bin/csh
..
..
 1292        0x801600000        0x801e00000 rw-  212  212   1   0 C---- df
 1292     0x7fffdffff000     0x7ffffffdf000 ---    0    0   0   0 ----- gd    <-- stack reserved
 1292     0x7ffffffdf000     0x7ffffffff000 rw-    9    9   1   0 C--D- df
 1292     0x7ffffffff000     0x800000000000 r-x    1    1  27   0 ----- ph
FreeBSD is "chopping off" from the reserved region and allocating it in chunks. Diffeerent approach to let's say Linux. Flag "D" is a very good indication we are talking about the stack as that's how stack behaves on x86 (side note though: MAP_STACK ; nothing prevents you do do a custom mmap).
That would be my approach to tell how much stack space was actively requested.
 
Back
Top