Server freezes when using Git to update ports tree

herrbischoff · Apr 6, 2023

cracauer@, here's a set of back-to-back attempts to update the same set of Git changes on the same machine both on ZFS and UFS.

The update on UFS went though:

Code:

procs     memory       page                      disks     faults       cpu
r b w     avm     fre  flt  re  pi  po    fr   sr da0 da1   in   sy   cs us sy id
 1  0  0 1036268  434904   574  10   0   0   602   80   0   0   48  1168 163342  2 12 86
 0  0  0 1046868  434648   439   0   0   0   193   20   0   0    2   660  2328  1  3 96
 5  0  0 1036252  434648    73   0   0   0     0   20   1   0    5   583   550  1  2 97
 0  0  0 1046868  434648   429   0   0   0   396   20 174   0  115   870  5674  0 19 81
 3  0  0 1340344  371896 17886   0   0   0   654   34  34   2   43  2842  4126  1 82 17
 2  0  0 2433496  367864  5879   0  34   0  1517   50  12  51  106  4668  3849  4 20 76
 0  0  0 2431548  355704  6283   0 607   0  1423   50   0 620  629  1617  4977  2  6 91
 2  0  0 2438204  332152  3726   0 770   0     0   50   0 763  772   417  4753  4  2 93
 2  0  0 2453188  312572  3687   0 571   0   277   50 102 563  630   686  4960  2 37 61
 0  0  0 2458308  296188  2942   0 508   0     0   50   0 504  510   414  6287  2 89  9
 0  0  0 2453564  271676  4350   0 575   0   120   59   0 564  578   471  3639  7  2 91
16  0  0 1332080  293468 10311   0 201   0 16120   55   6 245  258 78123  4245  5 55 40
15  0  0 1332080  292828    20   0   0   0     0   50   0 173  170 38096  2717  1 99  0
13  0  0 1342968  290300   498   0   0   0   356   55 150 452  524 21226  5498  1 99  0
12  0  0 1342968  290140    16   0   0   0     0   50   0  47   50 40199  1397  1 99  0
 0  0  0 2475744  263644  6940   0   0   0   120   53   0  56   64 22888  2585  3 97  0
 0  0  0 2475744  260060  4873   0   0   0  3673   59   0 263  241   986  4467  6 92  2
 0  0  0 1046868  364604  3057   0  36   0 28000   26  20  51   76  3077  3591  0 88 12
 0  0  0 1046868  364604    10   0   0   0     0   20 133   0   82   457  5021  0 96  3
 0  0  0 1035980  364636    89   0   0   0   120   20   0   0    7   548  4091  1 97  2
procs     memory       page                      disks     faults       cpu
r b w     avm     fre  flt  re  pi  po    fr   sr da0 da1   in   sy   cs us sy id
 0  0  0 1035980  364636     9   0   0   0     0   22   0   2    3   412   376  0  2 98

The update on ZFS froze the server:

Code:

procs     memory       page                      disks     faults       cpu
r b w     avm     fre  flt  re  pi  po    fr   sr da0 da1   in   sy   cs us sy id
 0  0  0 1047156  364764   575  10   0   0   603   80   0   0   48  1172 163259  2 12 86
 2  0  0 1036268  364764   275   0   0   0   120   20   3   0    6   622  3950  1 94  5
 1  0  0 1047264  364764    23   0   0   0     2   20   5   0   10   683  3379  0 99  1
 0  0  0 1112008  346204  3785   0  60   0   627   24 197   0  202  1542 1984693  1 80 19
 1  0  0 1332220  283228 14186   0  89   0   426   37 155   0  166  2443 173965  1 97  2
 1  0  0 1343108  276988   471   0  92   0   276   50 289   0  237   774 3107812  0 68 31
 0  0  0 2448244  266428   213   0 161   0     0   50 150   0  154   441 1779210  1 83 16
 1  0  0 2437356  254716   340   0 187   0   120   50 148   0  150   427 1936256  0 80 20
 0  0  0 2437356  239164   257   0 243   0     0   50 189   1  194   400  4509  0 99  1
 0  0  0 2439212  226876   216   0 194   0     0   50 225   0  209   462 5593953  0 60 39
 1  0  0 2450100  206204   689   0 313   0   276   50 358   0  327   671 5436596  1 59 39
 3  0  0 2450100  188124   310   0 278   0     0   55 247   0  253   552 2373683  0 52 48
 0  0  0 2454940  178780  1024   0 121   0   590   50 118   0  148  1531 2252365  1 34 65
 0  0  0 2457500  162780  1058   0 248   0     0   50 219  18  280  1842 2398488  1 78 21
 1  0  0 2433892  152044  1093   0 193   0   927   50 319  15  328  2301  6295  2 98  0
 3  0  0 2456556  139628  2192   0 125   0   288   50 182   0  177  3393 2687888  5 64 31
 1  0  0 2475500  117804  4301   0  93   0     0   52  91   0  102  1806 1868206 17 50 33
 0  0  0 2464612  108364   678   0 151   0   120   60 132   0  142  1853 2491127  4 65 31
 0  0  0 2464612   97868   444   0 152   0     0   66 161   0  182  2240 2419854  3 71 27
 0  0  0 2434224  117212  3563   0  68   0  8838   54 178   0  134  1778 3380066  4 60 36
procs     memory       page                      disks     faults       cpu
r b w     avm     fre  flt  re  pi  po    fr   sr da0 da1   in   sy   cs us sy id
 0  0  0 2445112  111868  2260   0  66   0   277   50 248   0  208   701 3170020  1 67 33
 0  0  0 2445112  101372  1885   0 156   0     0   50 125   0  129   434 3236114  1 64 35
 0  0  0 2437296   93596  1148   0 114   0   120   50 156   0  160   448 3056616  2 65 33
 0  0  0 2437296   80924   992   0 178   0     3   50 138   0  140   419 1623457  0 81 18
 0  0  0 2443208   71260  1064   0 140   0   342   51 369   0  327  2237 2006807  2 79 20
 0  0  0 2451768   59292  1438   0 145   0   524   59 275   0  257  2983 2941928  2 69 29
 0  0  0 2455864   46332   585   0 214   0     0   60 170   0  173   585 1740658  2 80 18
 2  0  0 2444976   55560   665   0 211   0  5787   66 157   0  165   480 1805441  1 80 19
 0  0  0 2444976   48988   266   0 105   0     0   60 163   0  164   390 2483707  1 75 24
 1  0  0 2444976   43408   285   0  93   0     0   60 235   0  201   397 2360408  0 77 23
 1  0  0 2460984   41120   813  97 145   0  2407  339 277   0  237   730 3102019  1 74 26
 3  0  0 2460984   42588   231 259  58   0  1545 2109 192   0  195   400 2581987  0 73 27
 0  0  0 2450096   42880   349  56  58   0  1480 1369 120   0  123   446 3866170  1 66 34
 0  0  0 2456240   43844  1559  37 178   0  4132 4255 134   0  137   431 3221963  1 68 32
 0  0  0 2456240   43204   780   0 231   0  3629 3697 311   0  260   383 2127489  0 79 20
 1  0  0 2467128   40776  1166   0 183   0  3039 3852 205   0  207   664 5695513  4 60 36
 1  0  0 2467128   40096   659   0 126   0  2599 2712 156   0  167   444 2729558  2 72 26
 1  0  0 2471600   38024  2676  48 207   0  4732 5196 183   0  184   435 1074676  5 85 11
 7  0  0 1340516   96460 31869 1989   1   0 66071 69297 427   0  373  4305 2643474  9 72 20
 3  0  0 1340516   47812    34 4384   0 129 32009 57218 1127   0  711  5868 6148600  0 57 43
procs     memory       page                      disks     faults       cpu
r b w     avm     fre  flt  re  pi  po    fr   sr da0 da1   in   sy   cs us sy id
 2  0  0 1351404   41564   546 240  16   0 31267 35575 502   0  427  3435 5071656  1 53 46
 0  0  0 1364436   52164   321 2511  21   0 38604 41066 599   0  467  4496 5720614  1 55 45
 0  0  0 1364436   48484    49 131   4 245 37162 39773 987   0  613  6189 5777297  1 57 42
 3  0  9 1353548   39548   391 877   2 1078 38190 285752 1833   0  816  6214 6281601  2 56 43

The update on the server with compress=off on zroot/usr/ports also went through.

Code:

procs     memory       page                      disks     faults       cpu
r b w     avm     fre  flt  re  pi  po    fr   sr da0 cd0   in   sy   cs us sy id
 1  0 19 4622944   69724   354   9   3   1   447  382 19230   0   27   765  3494  2  0 98
 1  0 19 4623084   43504  4080 1108 2318 192  7079 10292 2644   0 2374   551 99813  1  9 90
 0  0 18 4615268   42912  3716 2205 2261 1500 12236 23901 3782   0 2563  1175 114518  3 14 83
 0  0 17 4618852   44236  2999 1490 2168 2186  8081 21094 4602   0 2637   369 100968  1 16 83
 1  0 15 4754348   46268  8624 8280 2190 2044 17195 38221 4246   0 2599  7984 91644  4 19 77
 0  0 15 4759468   42228  3486 2085 2199 1618 12208 25881 3612   0 2317   647 70696  2 18 80
 1  0 15 4754724   43280  4530 4560 2133 1282 18246 35010 3303   0 2204   439 79924  6 16 78
 6  0 14 3634088   38580 11080 2077 647 399 26471 19359 13817   0 6170 93498 269024  6 67 27
 0  0 14 3634088   40676    10  12   0 335 11207 24251 926   0  376  4454 1109254  0 54 46
 0  0 17 3645336   36844  1633  61  26 1807 18666 48189 3482   0 1370 13929 2350245  1 57 42
 0  0 17 3657080   33676   547  57  76 3067 32053 83420 3624   0 1296  4017 3808337  2 55 43
 0  0 16 3646192   24716   236  20  33 2487 26320 76033 3258   0 1127  5608 4449700  2 57 41
 9  0 24 3646192   13052    13 335   1 4153 35838 149536 4885   0 1649  6376 3577191  2 59 39
 6  0 22 3654772   40220   155   0   0 5398 34103 607580 9567   0 3580 28399 2175933  1 70 29
 0  0 24 4799912   47444  7447 116 163 970 16863 224267 7306   0 4322 48928 312896  4 87  9
 1  0 24 4799928   61604  8837 983 2415 1466  8605 22945 3577   0 2425  1192 108099  2 20 78
 2  0 24 3360992  153412  3595 142 312 517 31352 14341 1060   0  580  2903 19118  7  9 84
 1  0 24 3360992  153412    11   0   0   0     0  230   1   0    2   333   253  0  0 100
 0  0 23 3350104  153492   155   0  13   0   120  253  27   0   27   467   570  1  0 98

herrbischoff · Apr 6, 2023

Tangentially related to the issue: SirDice suggested a different approach to building a handful of ports but did not respond yet whether Poudriere now works as documented, downloading official packages whenever possible instead of building them all. I set this up myself to test. I'm pleased to report that it appears to work. Apparently, there's an elegant way to use multiple package sources, obviating even the need to cache all required dependencies.

For now, my Poudriere solely builds Nginx with options. I added an additional pkg source in /usr/local/etc/pkg/repos like this:

Code:

Poudriere: {
    url: "https://pkg.example.com/packages/131x64-quarterly",
    mirror_type: "http",
    signature_type: "pubkey",
    pubkey: "/usr/local/etc/ssl/certs/poudriere.crt",
    enabled: yes,
    priority: 100
}

Since I'm using quarterly ports with both official packages and Poudriere, this approach should work, prioritising the Poudriere source. If I have overlooked something, I'd appreciate a pointer from whoever has more experience with this.

SirDice · Apr 6, 2023

herrbischoff said:
@SirDice suggested a different approach to building a handful of ports but did not respond yet whether Poudriere now works as documented

Yes, wanted to verify first, I build everything with poudriere and don't actually use that feature. Hadn't been able to do so, too busy at $DAYJOB

cracauer@ · Apr 6, 2023

Interesting results.

Did you think about turning on all the extra sanity checks in the kernel config, especially DEADLKRES? Maybe that gets you a useful error message instead of just a hang?

PMc · Apr 7, 2023

There are some tunables in git. This is what I am using:

Code:

export GIT_CONFIG_COUNT=3
# Git etwas braver machen, damit der Arc handhabbar bleibt.
export GIT_CONFIG_KEY_0=core.packedGitWindowSize
export GIT_CONFIG_VALUE_0=128m
export GIT_CONFIG_KEY_1=core.packedGitLimit
export GIT_CONFIG_VALUE_1=1g
export GIT_CONFIG_KEY_2=diff.renameLimit
export GIT_CONFIG_VALUE_2=16384

zirias@ · Apr 7, 2023

Any reason to use these cumbersome env variables instead of just the system-wide config? (/usr/local/etc/gitconfig)

herrbischoff · Apr 7, 2023

zirias@ said:
Any reason to use these cumbersome env variables instead of just the system-wide config? (/usr/local/etc/gitconfig)

That's what I was wondering and set the values up like this, just to test:

Code:

git config --system core.packedGitWindowSize 128m
git config --system core.packedGitLimit 1g
git config --system diff.renameLimit 16384

along with

Code:

git config --system core.preloadIndex false

I found suggested elsewhere. Will report on my findings using them.

cracauer@, I've also compiled a kernel with the following debug settings, to see if that yields any further information:

Code:

options     KDB            # Enable kernel debugger support.
options     DDB            # Support DDB.
options     GDB            # Support remote GDB.
options     DEADLKRES        # Enable the deadlock resolver
options     INVARIANTS        # Enable calls of extra sanity checking
options     INVARIANT_SUPPORT    # Extra sanity checks of internal structures, required by INVARIANTS
options     WITNESS            # Enable checks to detect deadlocks and cycles
options     WITNESS_SKIPSPIN    # Don't run witness on spinlocks for speed
options     DEBUG_VFS_LOCKS        # enable VFS lock debugging
options     ACPI_DEBUG        # include ACPI-debug code

As I don't have any kernel debugging experience to speak of, I'd appreciate if you could illustrate the steps that provide the most useful information for you to evaluate.

cracauer@ · Apr 7, 2023

herrbischoff said:
cracauer@, I've also compiled a kernel with the following debug settings, to see if that yields any further information:

Code:

options KDB # Enable kernel debugger support. options DDB # Support DDB. options GDB # Support remote GDB. options DEADLKRES # Enable the deadlock resolver options INVARIANTS # Enable calls of extra sanity checking options INVARIANT_SUPPORT # Extra sanity checks of internal structures, required by INVARIANTS options WITNESS # Enable checks to detect deadlocks and cycles options WITNESS_SKIPSPIN # Don't run witness on spinlocks for speed options DEBUG_VFS_LOCKS # enable VFS lock debugging options ACPI_DEBUG # include ACPI-debug code

As I don't have any kernel debugging experience to speak of, I'd appreciate if you could illustrate the steps that provide the most useful information for you to evaluate.

No action to take. The hope is that actual error messages will appear on the console before or at the hang.

There could be a serious issue hiding there that needs fixing.

PMc · Apr 7, 2023

zirias@ said:
Any reason to use these cumbersome env variables instead of just the system-wide config? (/usr/local/etc/gitconfig)

You are right, there are four or five possibilities to set such config variables.

on the commandline (local to the process)
in a repository (local to that repo)
in a user's homedir (only for that user, but for all repos)
in a system-wide config (for all users and all repos on that machine

Here I am setting up temporary bhyves for the buildworld. And an easy way to export these variables to a bhyve is to copy them from the env when writing the /etc/rc.local into the bhyve. (*)

My scripts for the builds have a config file where the sizing for the bhyves is described. And I wanted these variables at the same place. I would have needed to create some shell variable to carry them - but then git had that already.

One problem I encountered with automation is, one tends to forget all the places where something was adjusted to make things work. A year later it gets copied to a different machine - and starts to behave differently. And then the big searching starts: this did work at one time, but why? ...

(*)My bhyves need git, because the dependency tree for portbuild is computed there under the correct OS version, and then stored into the ports repo and pushed back, so that changed dependencies will trigger the necessary rebuilds. Without that one cannot change a default version in /etc/make.conf and then build only the concerned ports.
(Commit the dependency tree into the ports repo, get a new commit id, write that commit id into the pkg annotations. At the next time fetch the annotations with pkg rquery, and diff that commit id against the current options+dependencies.)

herrbischoff · Apr 9, 2023

cracauer@ said:
No action to take. The hope is that actual error messages will appear on the console before or at the hang.

There could be a serious issue hiding there that needs fixing.

Today, I triggered the same behaviour with the debug kernel in place. Unfortunately, still no console output. The machine just stops responding dead in its tracks.
Please advise if you'd like me to try anything else to hopefully produce useful data. Otherwise, I will just permanently forgo ports via Git on smaller instances.

cracauer@ · Apr 9, 2023

herrbischoff said:
Today, I triggered the same behaviour with the debug kernel in place. Unfortunately, still no console output. The machine just stops responding dead in its tracks.
Please advise if you'd like me to try anything else to hopefully produce useful data. Otherwise, I will just permanently forgo ports via Git on smaller instances.

I'm wondering whether I can reproduce it here. Might be a real bug hiding (well, not so well) here.

herrbischoff · Apr 18, 2023

cracauer@ said:
I'm wondering whether I can reproduce it here. Might be a real bug hiding (well, not so well) here.

I have set up a VirtualBox instance with largely the same setup but couldn't get it to crash yet.

ketas · Feb 15, 2024

that thing did bite my ass right now

it took a long time to diagnose it

basically, git told kernel to allocate all my ram. so kernel did it really well. i think it mmaps files, but i'm not that strong in that field. what i saw is, in few seconds, wired mem grew to >3.6g. with only 4g ram in machine. then, kernel killed all of userland off because it needed more. in the end kernel was only thing that ran on that machine

to recover from that runaway, mu solution was:
watchdogd_enable="YES"
watchdogd_flags="-e date -s 15 -t 120 -T 30 -w --pretimeout 90 --pretimeout-action log,printf,panic"

usually when "normal" memory or cpu runaway happens, you have time and massive lag to still kill or ^c things, this isn't the case

i wonder if this is bug or feature

or how to limit it besides git options

richardtoohey2 · Feb 15, 2024

Did you have any swap?

Anything in /var/log/messages?

ketas · Feb 16, 2024

swap is not helping here at all, i have 16g of swap (8g on each mirror device), which is even too much, i get maxswzone complaints

and since kernel kills things quickly, syslog won't be able to even save it. however i get:

Unread portion of the kernel message buffer:
<3>pid 2617 (named), jid 0, uid 53, was killed: failed to reclaim memory
<3>pid 22253 (csh), jid 0, uid 0, was killed: failed to reclaim memory

when i did "git -C /usr/ports pull" and opened another window with top, this was last thing that came out over ssh:

last pid: 16826; load averages: 0.24, 0.22, 0.17 up 0+14:15:07 09:01:09
621 threads: 7 running, 576 sleeping, 38 waiting
CPU 0: 0.0% user, 0.0% nice, 15.5% system, 1.8% interrupt, 82.7% idle
CPU 1: 1.2% user, 0.0% nice, 19.0% system, 0.0% interrupt, 79.8% idle
Mem: 124M Active, 388K Inact, 182M Laundry, 3540M Wired, 10M Free
ARC: 410M Total, 137M MFU, 100M MRU, 2218K Anon, 2482K Header, 168M Other
35M Compressed, 204M Uncompressed, 5.77:1 Ratio
Swap: 16G Total, 370M Used, 16G Free, 2% Inuse, 220K In, 68M Out

ketas · Feb 16, 2024

by that time, i tried setting arc max to 512m. whatever happened, happened outside of that

richardtoohey2 · Feb 16, 2024

ketas said:
swap is not helping here at all

I was curious; any time I've had a process chew up all of RAM I can see it munching through RAM, then hitting swap until that runs out and then processes start to get killed (and that's been logged in /var/log/messages). But all on UFS so no ARC/ZFS in the mix.

This issue seems different.

ketas · Feb 16, 2024

yeah it's different. kernel memory can't be swapped out. i was surprised that it didn't even try to swap anything out. once git started whatever it did, there was slight lag, i tried to switch the tmux window, it didn't work anymore, in last top screen i saw wired through the roof. what, how, i don't know. either something allocated or leaked it all. i don't have ufs to try. it also didn't happen in src. nor ports, before, much. i bet some extra large change came from ports, and unrestrained git took it all down. unsure why git doesn't limit itself, maybe it could. i heard git taking a lot of memory but i thought ok it will swap then. i sometimes build things in there which take 2 days and allocate 10g swap which makes build really slow, but machine stays responsive. what git does is different and i don't even know what really happens there

grahamperrin · Feb 17, 2024

ketas said:
that thing

Which version of FreeBSD, exactly?

Packages of ports from quarterly, or latest?

freebsd--version -kru ; uname -aKU

pkg -vv | grep -e url -e enabled -e priority

cracauer@ · Feb 17, 2024

ketas said:
yeah it's different. kernel memory can't be swapped out. i was surprised that it didn't even try to swap anything out. once git started whatever it did, there was slight lag, i tried to switch the tmux window, it didn't work anymore, in last top screen i saw wired through the roof. what, how, i don't know. either something allocated or leaked it all. i don't have ufs to try. it also didn't happen in src. nor ports, before, much. i bet some extra large change came from ports, and unrestrained git took it all down. unsure why git doesn't limit itself, maybe it could. i heard git taking a lot of memory but i thought ok it will swap then. i sometimes build things in there which take 2 days and allocate 10g swap which makes build really slow, but machine stays responsive. what git does is different and i don't even know what really happens there

So what kind of git tree is it? FreeBSD src? ports? Anything public? If not how large is it?

Can you run `vmstat 30` during your git operation and post the output?

ketas · Feb 17, 2024

13.2-RELEASE-p10

zfs

i pull src for release, src for current and latest ports. that's where i also install everything. directly from ports
only ports does it, and it is indeed huge repo. funnily git is not that much slower if i limit it

do i need to run vmstat exactly like that with or without limits? it takes only 10s or so to bring everything down if git runs and enters that part in it's operation.

vmstat with ports main pull is here:

01:24,ketas@green:~> vmstat 30
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr ad0 ad1 in sy cs us sy id
1 1 5 7.0G 277M 2.4K 13 3 0 2.7K 452 0 0 1480 6.5K 7.9K 34 4 62
0 1 5 7.3G 117M 2.9K 0 26 0 1.8K 178 0 0 638 2.3K 2.9K 3 3 95
0 1 5 7.3G 94M 20 0 0 0 21 180 0 0 549 1.4K 2.7K 1 1 98
0 1 5 7.3G 124M 38 0 0 0 559 180 0 0 535 1.5K 2.6K 1 1 98
0 1 5 7.3G 106M 19 0 0 0 392 180 0 0 562 1.8K 2.7K 0 2 98
0 1 5 7.3G 108M 206 0 0 0 623 179 0 0 564 1.6K 2.8K 1 1 98
0 1 5 7.3G 102M 150 0 0 0 418 180 0 0 569 1.5K 3.3K 0 2 98
0 1 5 7.3G 99M 432 0 0 0 454 180 0 0 605 2.0K 4.2K 1 1 98
0 1 5 7.3G 99M 22 0 0 0 20 180 0 0 545 1.5K 3.7K 1 1 98
0 1 5 7.3G 99M 40 0 0 0 38 180 0 0 581 1.6K 4.1K 1 2 98
0 1 5 7.3G 99M 19 0 0 0 20 180 0 0 582 1.6K 4.4K 1 1 98
0 1 5 7.3G 88M 1.4K 0 0 0 1.2K 180 0 0 680 2.6K 4.9K 2 2 95
0 1 5 7.0G 202M 454 0 2 0 1.5K 160 0 0 514 1.1K 3.0K 1 2 97
^C
01:34,ketas@green:~>

ketas · Feb 17, 2024

Code:

01:24,ketas@green:~> vmstat 30
 procs    memory    page                      disks     faults       cpu
 r  b  w  avm  fre  flt  re  pi  po   fr   sr ad0 ad1   in   sy   cs us sy id
 1  1  5 7.0G 277M 2.4K  13   3   0 2.7K  452   0   0 1480 6.5K 7.9K 34  4 62
 0  1  5 7.3G 117M 2.9K   0  26   0 1.8K  178   0   0  638 2.3K 2.9K  3  3 95
 0  1  5 7.3G  94M   20   0   0   0   21  180   0   0  549 1.4K 2.7K  1  1 98
 0  1  5 7.3G 124M   38   0   0   0  559  180   0   0  535 1.5K 2.6K  1  1 98
 0  1  5 7.3G 106M   19   0   0   0  392  180   0   0  562 1.8K 2.7K  0  2 98
 0  1  5 7.3G 108M  206   0   0   0  623  179   0   0  564 1.6K 2.8K  1  1 98
 0  1  5 7.3G 102M  150   0   0   0  418  180   0   0  569 1.5K 3.3K  0  2 98
 0  1  5 7.3G  99M  432   0   0   0  454  180   0   0  605 2.0K 4.2K  1  1 98
 0  1  5 7.3G  99M   22   0   0   0   20  180   0   0  545 1.5K 3.7K  1  1 98
 0  1  5 7.3G  99M   40   0   0   0   38  180   0   0  581 1.6K 4.1K  1  2 98
 0  1  5 7.3G  99M   19   0   0   0   20  180   0   0  582 1.6K 4.4K  1  1 98
 0  1  5 7.3G  88M 1.4K   0   0   0 1.2K  180   0   0  680 2.6K 4.9K  2  2 95
 0  1  5 7.0G 202M  454   0   2   0 1.5K  160   0   0  514 1.1K 3.0K  1  2 97
^C
01:34,ketas@green:~>

cracauer@ · Feb 17, 2024

ketas said:

Code:

01:24,ketas@green:~> vmstat 30
 procs    memory    page                      disks     faults       cpu
 r  b  w  avm  fre  flt  re  pi  po   fr   sr ad0 ad1   in   sy   cs us sy id
 1  1  5 7.0G 277M 2.4K  13   3   0 2.7K  452   0   0 1480 6.5K 7.9K 34  4 62
 0  1  5 7.3G 117M 2.9K   0  26   0 1.8K  178   0   0  638 2.3K 2.9K  3  3 95
 0  1  5 7.3G  94M   20   0   0   0   21  180   0   0  549 1.4K 2.7K  1  1 98
 0  1  5 7.3G 124M   38   0   0   0  559  180   0   0  535 1.5K 2.6K  1  1 98
 0  1  5 7.3G 106M   19   0   0   0  392  180   0   0  562 1.8K 2.7K  0  2 98
 0  1  5 7.3G 108M  206   0   0   0  623  179   0   0  564 1.6K 2.8K  1  1 98
 0  1  5 7.3G 102M  150   0   0   0  418  180   0   0  569 1.5K 3.3K  0  2 98
 0  1  5 7.3G  99M  432   0   0   0  454  180   0   0  605 2.0K 4.2K  1  1 98
 0  1  5 7.3G  99M   22   0   0   0   20  180   0   0  545 1.5K 3.7K  1  1 98
 0  1  5 7.3G  99M   40   0   0   0   38  180   0   0  581 1.6K 4.1K  1  2 98
 0  1  5 7.3G  99M   19   0   0   0   20  180   0   0  582 1.6K 4.4K  1  1 98
 0  1  5 7.3G  88M 1.4K   0   0   0 1.2K  180   0   0  680 2.6K 4.9K  2  2 95
 0  1  5 7.0G 202M  454   0   2   0 1.5K  160   0   0  514 1.1K 3.0K  1  2 97
^C
01:34,ketas@green:~>

No pageout at all. Are you sure your swapspace is working?

Do you have dedup on in ZFS?

ketas · Feb 18, 2024

no dedup. would be insane with 4g ram anyway? and benefits of that was also questionable i read

i could take limits off and let machine fail again. let vmstat run until sshd is killed off. it goes really fast. but from top run, i saw that swap was not used. all the memory was wired. so there was nothing to swap anyway. there was some minimal swapping only. as others said, they are more familiar with things swapping "normally". i mean things you can replicate with sh -c 'while true; do a="$a$a$a.aaa"; done'

here, i don't know what happens. is this a bug? just allocating memory is not a bug. even if it's kernel one. even allocating and running out is not a bug. but here i don't know. limits? automatic limits not working for just 4g of ram? people with 64g or ram could fit entire ports tree there without any issues

but still, is this something that needs fix? i don't even know what it is as i'm not familiar with what zfs actually does

grahamperrin · Feb 18, 2024

ketas said:
… i pull src for release, src for current and latest ports. …

Is FreeBSD-CURRENT somehow involved?

Server freezes when using Git to update ports tree

Administrator