RAM from 4GB to 6GB on i386 with ZFS Help!

[freebuser]

Hi,

I am planning to upgrade my RAM from 4GB to 6GB.

The machine is ML350 G3 / 2.8GHz (I don't think it is x64 capable).

After creating 2 mirrored pools (2x146 + 2x78) I compiled a custom kernel with

KVA_PAGES=640

but no PAE.

1. If I upgrade my RAM then do I need to recompile the kernel with PAE?
2. I read about PAE still not stable enough is it still the case?
3. What other factors I should look or options I have?

Thank you all.

 

[Terri_Kennedy]

1. If I upgrade my RAM then do I need to recompile the kernel with PAE?

If you want to use more memory than the system maps into the first 4GB (usually 256MB to 512MB less than 4GB, due to the peripheral I/O address space at the top) you will either need to use PAE or the amd64 architecture (if supported by your CPU).

2. I read about PAE still not stable enough is it still the case?

I don't think PAE gets a lot of developer attention these days, since newer systems have a much better solution (amd64). There are a lot of subtle things that can cause problems in a PAE environment and there may be more of them now that fewer people use PAE. Additionally, there's a bunch of things that are known to not work with PAE. Take a look at /sys/i386/conf/PAE - it is a list of the things that don't work right in a PAE environment, and there are some show-stoppers in there - if your disk controller or network card isn't supported in PAE, you won't be able to get a usable system booted.

3. What other factors I should look or options I have?

The obvious question is: Do you actually need more memory in this system? If it is running with 4GB now and not swapping heavily, more memory may not be necessary. One common reason for wanting more memory is to use ZFS, but unless you're willing to invest heavily in tuning (and possibly re-tuning after each new FreeBSD version) you'd be much better off with an amd64-capable system. Whether that's practical will depend a lot on how much of your existing hardware you can re-use and your budget.

 

[freebuser]

After reading some more information of 3GB limits by OSs I am more confused than before.

My server ML350G3 is able to support upto 8GB RAM. Atleast I am clear on that, i.e. hardware side of it.

But I am not sure whether the 3GB limit is imposed only for the kernel or for the entire OS - I am considering OS as including all the softwares (eg. Java).

The reason I like to upgrade my RAM is the Java VM I am currently running (namely OpenKM) on my server.

By increasing the RAM to say 6GB without making any modifications to the kernel, will other programs like Java be able access the RAM above the 4GB mark?

Cheers,

 

[Terri_Kennedy]

After reading some more information of 3GB limits by OSs I am more confused than before.

You're welcome. :e

My server ML350G3 is able to support upto 8GB RAM. Atleast I am clear on that, i.e. hardware side of it.

Many systems (mostly "server class", but some "workstation" systems as well) of that era supported more than 4GB. It isn't obvious how many of those systems were actually configured with more than 4GB.

But I am not sure whether the 3GB limit is imposed only for the kernel or for the entire OS - I am considering OS as including all the softwares (eg. Java).

A 32-bit non-PAE operating system can only address the first 4GB of address space. As I mentioned in my earlier reply, part of that space is reserved for I/O peripherals and thus the usable address space will be less. Exactly how much less depends on how clever the hardware is as well as how tolerant the operating system is of small, non-contiguous memory chunks.

I have one i386-architecture system which does a pretty good job:

CPU: Intel(R) Pentium(R) 4 CPU 3.40GHz (3400.14-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0xf34  Stepping = 4
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x441d<SSE3,RSVD2,MON,DS_CPL,CNXT-ID,xTPR>
  Logical CPUs per core: 2
real memory  = 4160487424 (3967 MB)
avail memory = 4073869312 (3885 MB)

and another that does an outstanding job:

CPU: Intel(R) Xeon(TM) CPU 3.06GHz (3065.81-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0xf29  Family = f  Model = 2  Stepping = 9
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,C
MOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x4400<CNXT-ID,xTPR>
real memory  = 4294967296 (4096 MB)
avail memory = 4073127936 (3884 MB)

Note that this second system reports "384K of memory above 4GB ignored" because its BIOS relocates the memory displaced by the I/O region to above 4GB.

Here is a Microsoft TechNet blog entry which describes some of the issues with PAE in the Windows environment. In particular: "What they found was that many of the systems would crash, hang, or become unbootable because some device drivers, commonly those for video and audio devices that are found typically on clients but not servers, were not programmed to expect physical addresses larger than 4GB. As a result, the drivers truncated such addresses, resulting in memory corruptions and corruption side effects. Server systems commonly have more generic devices and with simpler and more stable drivers, and therefore hadn't generally surfaced these problems. The problematic client driver ecosystem led to the decision for client SKUs to ignore physical memory that resides above 4GB, even though they can theoretically address it."

The reason I like to upgrade my RAM is the Java VM I am currently running (namely OpenKM) on my server.

I ask again - does your system indicate it has low free memory? You can use # pstat -s to check swap usage (note that a small amount of usage - on the order of 64 blocks or so - is normal even on a system with lots of free memory). You can also use # vmstat to see how much memory is available (in the "memory / fre" column). You can also try # vmstat -w 5 to continuously report the number every 5 seconds. If you look at the "page" columns, you can see if your system is paging. Again, some small number of page faults is to be expected as images are activated by paging them in.

By increasing the RAM to say 6GB without making any modifications to the kernel, will other programs like Java be able access the RAM above the 4GB mark?

The answer, like so many other things, is "it depends". The short answer is that no single process will be able to allocate more than (approximately) 4GB. But multiple processes can each have an allocation of up to 4GB, assuming sufficient free memory is available. From the pae(4) manpage: "With the PAE option, memory above 4 gigabytes is simply added to the general page pool. The system makes no distinction between memory above or below 4 gigabytes, and no specific facility is provided for a process or the kernel to access more memory than they would otherwise be able to access, through a sliding window or otherwise."

This is another advantage of running in amd64 mode - a 64-bit process (the normal result of compiling and linking under amd64) can allocate as much memory as it wants, even beyond 4GB. A 32-bit process (either a binary installed from a vendor, or a program specially compiled and linked on amd64) will still be limited to roughly 4GB of memory.

 

[freebuser]

Thanks very much for your reply Terry,

From all of your commands above, there is no paging activity and all of the swap space is available.

procs      memory      page                    disks     faults         cpu
 r b w     avm    fre   flt  re  pi  po    fr  sr da0 da1   in   sy   cs us sy id
 0 0 0   1215M  3022M    45   2   1   0   112   0   0   0   32 1078 1003  1  0 99

With 3GB free memory I am still not able to start java.

I am trying to recompile the kernel with KVA_PAGES=512 (Earlier it was 640) to see whether that helps.

Cheers,

 

[Terri_Kennedy]

With 3GB free memory I am still not able to start java.

Ok, now we're getting somewhere. What error message do you get?
What Java version are you running? To display it, use: [cmd=""]java -version[/cmd]
What is the full command line you're trying to use to start your Java application?

Note that the command line may be buried inside a startup script (a quick perusal of the OpenKM wiki says that it is run.sh). You can use something like [cmd=""]ktrace -i -d run.sh[/cmd] and then [cmd=""]kdump > outfile.txt[/cmd] to capture everything that's going on. Note that this output will be HUGE - a trace file from a simple -version command is 4MB on my system and the resulting kdump output file is 8.5MB. That's just the way it is with complex programs like Java. You can either look through the output yourself or put it on a server where other people can look at it for you. If you compress the kdump output, it will get a lot smaller (in the case of my -version test, it reduced to 1.5MB). Of course, if the program you are tracing is running with elevated privileges or it does its own password management, passwords may be included in the trace, so either use disposable passwords (and change them afterwards) or only share the output file(s) with trusted individuals.

 

[freebuser]

Hi Terry,

It is very interesting to see that when I boot with the GENERIC kernel (KVA_PAGES=256) OpenKM starts without any issues, when I recompile the kernel with KVA_PAGES=512 no issues in starting OpenKM but with KVA_PAGES=640 not enough memory.

It looks like regardless of the vm.kmem.size_max the kernel reserves the memory on KVA_PAGES setting.

cheers,

 

[Terri_Kennedy]

It is very interesting to see that when I boot with the GENERIC kernel (KVA_PAGES=256) OpenKM starts without any issues, when I recompile the kernel with KVA_PAGES=512 no issues in starting OpenKM but with KVA_PAGES=640 not enough memory.

Take a look at /sys/i386/conf/NOTES, which says

#
# Change the size of the kernel virtual address space.  Due to
# constraints in loader(8) on i386, this must be a multiple of 4.
# 256 = 1 GB of kernel address space.  Increasing this also causes
# a reduction of the address space in user processes.  512 splits
# the 4GB cpu address space in half (2GB user, 2GB kernel).  For PAE
# kernels, the value will need to be double non-PAE.  A value of 1024
# for PAE kernels is necessary to split the address space in half.
# This will likely need to be increased to handle memory sizes >4GB.
# PAE kernels default to a value of 512.
#
options         KVA_PAGES=260