Hey,
I'm experimenting with ZFS in combination with tmpfs(5). I don't use critical data. If it's gone I don't care
I have about 70GB memory. I'll create a tmpfs(5) (60GB) mount point and create a ZFS pool attached to tmpfs(5) (60gb) storage. After that I'll have 10 GB for ARC (5gb) and system process (5gb).
When I'll create a NFS share (sync=disabled) and attach to ESX, I can start a VM. With great performance, but when I create a second VM (deploy template), all of the performance is gone.
First VM performance:
0.7 ms latency read/write
Second VM Performance:
10ms latency read/write.
I'll use the settings:
dedup on
compresson off
/etc/sysctl.conf
I have also tried without any ARC because tmpfs(5) its fast enough. But can someone help me why all the next VM are slow?
I'm using FreeBSD 10.2
Update: When I don't use tmpfs(5) as storage disk and use ARC of 50GB with 50GB storage, everything is very fast. I'll install 1 VM and get performance latency of 0.7 read/write
At this point I shutdown the first VM install the second VM (100% identical copy ore linked clone same problem) Run the same test 5ms latency read/write... Ok strange? Shutdown the Second VM and start the first VM and now i have the same 5ms latency of read/write.
It seems when the data has being deduped everything is slow.
Update2: First VM(2012R2) 0.7 read/write. Second VM(2008R2) also 0.7 read/write. Even when I run both VM.
Its look like dedup is the problem.
Update3: Config:
I'm experimenting with ZFS in combination with tmpfs(5). I don't use critical data. If it's gone I don't care
I have about 70GB memory. I'll create a tmpfs(5) (60GB) mount point and create a ZFS pool attached to tmpfs(5) (60gb) storage. After that I'll have 10 GB for ARC (5gb) and system process (5gb).
When I'll create a NFS share (sync=disabled) and attach to ESX, I can start a VM. With great performance, but when I create a second VM (deploy template), all of the performance is gone.
First VM performance:
0.7 ms latency read/write
Second VM Performance:
10ms latency read/write.
I'll use the settings:
dedup on
compresson off
/etc/sysctl.conf
Code:
vfs.zfs.prefetch_disable="1"
vfs.zfs.txg.timeout="5"I have also tried without any ARC because tmpfs(5) its fast enough
. But can someone help me why all the next VM are slow?I'm using FreeBSD 10.2
Update: When I don't use tmpfs(5) as storage disk and use ARC of 50GB with 50GB storage, everything is very fast. I'll install 1 VM and get performance latency of 0.7 read/write
At this point I shutdown the first VM install the second VM (100% identical copy ore linked clone same problem) Run the same test 5ms latency read/write... Ok strange? Shutdown the Second VM and start the first VM and now i have the same 5ms latency of read/write.
It seems when the data has being deduped everything is slow.
Update2: First VM(2012R2) 0.7 read/write. Second VM(2008R2) also 0.7 read/write. Even when I run both VM.
Its look like dedup is the problem.
Update3: Config:
Code:
------------------------------------------------------------------------
ZFS Subsystem Report Mon Jan 11 12:18:17 2016
------------------------------------------------------------------------
System Information:
Kernel Version: 1002000 (osreldate)
Hardware Platform: amd64
Processor Architecture: amd64
ZFS Storage pool Version: 5000
ZFS Filesystem Version: 5
FreeBSD 10.2-RELEASE #0: Fri Jan 8 09:45:21 CET 2016 root
12:18PM up 3:41, 1 user, load averages: 3.19, 2.78, 2.18
------------------------------------------------------------------------
System Memory:
0.01% 5.16 MiB Active, 0.07% 48.71 MiB Inact
31.22% 21.29 GiB Wired, 0.00% 0 Cache
68.70% 46.84 GiB Free, 0.00% 4.00 KiB Gap
Real Installed: 80.00 GiB
Real Available: 87.48% 69.98 GiB
Real Managed: 97.42% 68.18 GiB
Logical Total: 80.00 GiB
Logical Used: 41.40% 33.12 GiB
Logical Free: 58.60% 46.88 GiB
Kernel Memory: 561.82 MiB
Data: 96.68% 543.18 MiB
Text: 3.32% 18.64 MiB
Kernel Memory Map: 68.18 GiB
Size: 3.00% 2.05 GiB
Free: 97.00% 66.13 GiB
------------------------------------------------------------------------
ARC Summary: (HEALTHY)
Memory Throttle Count: 0
ARC Misc:
Deleted: 6
Recycle Misses: 0
Mutex Misses: 0
Evict Skips: 1
ARC Size: 32.93% 16.47 GiB
Target Size: (Adaptive) 100.00% 50.00 GiB
Min Size (Hard Limit): 80.00% 40.00 GiB
Max Size (High Water): 1:1 50.00 GiB
ARC Size Breakdown:
Recently Used Cache Size: 50.00% 25.00 GiB
Frequently Used Cache Size: 50.00% 25.00 GiB
ARC Hash Breakdown:
Elements Max: 3.60m
Elements Current: 99.67% 3.59m
Collisions: 1.84m
Chain Max: 5
Chains: 353.61k
------------------------------------------------------------------------
ARC Efficiency: 105.05m
Cache Hit Ratio: 99.87% 104.91m
Cache Miss Ratio: 0.13% 134.27k
Actual Hit Ratio: 99.87% 104.91m
Data Demand Efficiency: 99.81% 3.55m
CACHE HITS BY CACHE LIST:
Most Recently Used: 6.50% 6.81m
Most Frequently Used: 93.50% 98.10m
Most Recently Used Ghost: 0.00% 0
Most Frequently Used Ghost: 0.00% 0
CACHE HITS BY DATA TYPE:
Demand Data: 3.37% 3.54m
Prefetch Data: 0.00% 0
Demand Metadata: 96.63% 101.37m
Prefetch Metadata: 0.00% 0
CACHE MISSES BY DATA TYPE:
Demand Data: 5.11% 6.86k
Prefetch Data: 0.00% 0
Demand Metadata: 94.70% 127.16k
Prefetch Metadata: 0.19% 257
------------------------------------------------------------------------
L2ARC is disabled
------------------------------------------------------------------------
------------------------------------------------------------------------
VDEV cache is disabled
------------------------------------------------------------------------
ZFS Tunables (sysctl):
kern.maxusers 4814
vm.kmem_size 73202900992
vm.kmem_size_scale 1
vm.kmem_size_min 0
vm.kmem_size_max 1319413950874
vfs.zfs.trim.max_interval 1
vfs.zfs.trim.timeout 30
vfs.zfs.trim.txg_delay 32
vfs.zfs.trim.enabled 1
vfs.zfs.vol.unmap_enabled 1
vfs.zfs.vol.mode 1
vfs.zfs.version.zpl 5
vfs.zfs.version.spa 5000
vfs.zfs.version.acl 1
vfs.zfs.version.ioctl 4
vfs.zfs.debug 0
vfs.zfs.super_owner 0
vfs.zfs.sync_pass_rewrite 2
vfs.zfs.sync_pass_dont_compress 5
vfs.zfs.sync_pass_deferred_free 2
vfs.zfs.zio.exclude_metadata 0
vfs.zfs.zio.use_uma 1
vfs.zfs.cache_flush_disable 1
vfs.zfs.zil_replay_disable 0
vfs.zfs.min_auto_ashift 9
vfs.zfs.max_auto_ashift 13
vfs.zfs.vdev.trim_max_pending 10000
vfs.zfs.vdev.bio_delete_disable 0
vfs.zfs.vdev.bio_flush_disable 0
vfs.zfs.vdev.write_gap_limit 4096
vfs.zfs.vdev.read_gap_limit 32768
vfs.zfs.vdev.aggregation_limit 131072
vfs.zfs.vdev.trim_max_active 64
vfs.zfs.vdev.trim_min_active 1
vfs.zfs.vdev.scrub_max_active 2
vfs.zfs.vdev.scrub_min_active 1
vfs.zfs.vdev.async_write_max_active 10
vfs.zfs.vdev.async_write_min_active 1
vfs.zfs.vdev.async_read_max_active 3
vfs.zfs.vdev.async_read_min_active 1
vfs.zfs.vdev.sync_write_max_active 10
vfs.zfs.vdev.sync_write_min_active 10
vfs.zfs.vdev.sync_read_max_active 10
vfs.zfs.vdev.sync_read_min_active 10
vfs.zfs.vdev.max_active 1000
vfs.zfs.vdev.async_write_active_max_dirty_percent60
vfs.zfs.vdev.async_write_active_min_dirty_percent30
vfs.zfs.vdev.mirror.non_rotating_seek_inc1
vfs.zfs.vdev.mirror.non_rotating_inc 0
vfs.zfs.vdev.mirror.rotating_seek_offset1048576
vfs.zfs.vdev.mirror.rotating_seek_inc 5
vfs.zfs.vdev.mirror.rotating_inc 0
vfs.zfs.vdev.trim_on_init 1
vfs.zfs.vdev.cache.bshift 16
vfs.zfs.vdev.cache.size 0
vfs.zfs.vdev.cache.max -1837105152
vfs.zfs.vdev.metaslabs_per_vdev 200
vfs.zfs.txg.timeout 2
vfs.zfs.space_map_blksz 4096
vfs.zfs.spa_slop_shift 5
vfs.zfs.spa_asize_inflation 24
vfs.zfs.deadman_enabled 0
vfs.zfs.deadman_checktime_ms 5000
vfs.zfs.deadman_synctime_ms 1000000
vfs.zfs.recover 0
vfs.zfs.spa_load_verify_data 1
vfs.zfs.spa_load_verify_metadata 1
vfs.zfs.spa_load_verify_maxinflight 10000
vfs.zfs.check_hostid 1
vfs.zfs.mg_fragmentation_threshold 85
vfs.zfs.mg_noalloc_threshold 0
vfs.zfs.condense_pct 200
vfs.zfs.metaslab.bias_enabled 1
vfs.zfs.metaslab.lba_weighting_enabled 1
vfs.zfs.metaslab.fragmentation_factor_enabled1
vfs.zfs.metaslab.preload_enabled 1
vfs.zfs.metaslab.preload_limit 3
vfs.zfs.metaslab.unload_delay 8
vfs.zfs.metaslab.load_pct 50
vfs.zfs.metaslab.min_alloc_size 33554432
vfs.zfs.metaslab.df_free_pct 4
vfs.zfs.metaslab.df_alloc_threshold 131072
vfs.zfs.metaslab.debug_unload 0
vfs.zfs.metaslab.debug_load 0
vfs.zfs.metaslab.fragmentation_threshold70
vfs.zfs.metaslab.gang_bang 16777217
vfs.zfs.free_max_blocks -1
vfs.zfs.no_scrub_prefetch 0
vfs.zfs.no_scrub_io 0
vfs.zfs.resilver_min_time_ms 3000
vfs.zfs.free_min_time_ms 1000
vfs.zfs.scan_min_time_ms 1000
vfs.zfs.scan_idle 50
vfs.zfs.scrub_delay 4
vfs.zfs.resilver_delay 2
vfs.zfs.top_maxinflight 32
vfs.zfs.zfetch.array_rd_sz 1048576
vfs.zfs.zfetch.block_cap 256
vfs.zfs.zfetch.min_sec_reap 2
vfs.zfs.zfetch.max_streams 8
vfs.zfs.prefetch_disable 1
vfs.zfs.delay_scale 500000
vfs.zfs.delay_min_dirty_percent 60
vfs.zfs.dirty_data_sync 67108864
vfs.zfs.dirty_data_max_percent 10
vfs.zfs.dirty_data_max_max 4294967296
vfs.zfs.dirty_data_max 4294967296
vfs.zfs.max_recordsize 1048576
vfs.zfs.mdcomp_disable 0
vfs.zfs.nopwrite_enabled 1
vfs.zfs.dedup.prefetch 1
vfs.zfs.l2c_only_size 0
vfs.zfs.mfu_ghost_data_lsize 0
vfs.zfs.mfu_ghost_metadata_lsize 0
vfs.zfs.mfu_ghost_size 0
vfs.zfs.mfu_data_lsize 1571403264
vfs.zfs.mfu_metadata_lsize 119364608
vfs.zfs.mfu_size 1691062784
vfs.zfs.mru_ghost_data_lsize 0
vfs.zfs.mru_ghost_metadata_lsize 0
vfs.zfs.mru_ghost_size 0
vfs.zfs.mru_data_lsize 12089954304
vfs.zfs.mru_metadata_lsize 1367170560
vfs.zfs.mru_size 14068405760
vfs.zfs.anon_data_lsize 0
vfs.zfs.anon_metadata_lsize 0
vfs.zfs.anon_size 74373120
vfs.zfs.l2arc_norw 1
vfs.zfs.l2arc_feed_again 1
vfs.zfs.l2arc_noprefetch 1
vfs.zfs.l2arc_feed_min_ms 200
vfs.zfs.l2arc_feed_secs 1
vfs.zfs.l2arc_headroom 2
vfs.zfs.l2arc_write_boost 8388608
vfs.zfs.l2arc_write_max 8388608
vfs.zfs.arc_meta_limit 13421772800
vfs.zfs.arc_free_target 123937
vfs.zfs.arc_shrink_shift 5
vfs.zfs.arc_average_blocksize 8192
vfs.zfs.arc_min 42949672960
vfs.zfs.arc_max 53687091200
------------------------------------------------------------------------
Last edited by a moderator:
