Solved Portupgrade Building Rust stopped with signal: 11 (SIGSEGV) (core dumped)

T-Aoki · Saturday at 11:12 PM

_martin said:
But it will be the same amount of physical memory and it will still be the same pfn, i.e. same physical address.

This is one of the reasons why I said "depending on implementation".

If any additional pages (including dummy) are mapped into physical memory pages, means, mapped physical pages increase, possibility to hit broken memory cell increases.

If randomizations are always assured to be done "per page" basis and the dummy page to differenciate start address for randomization are not actually mapped into physical pages, you're correct.

Note that I've not looked into with the ASLR implementations on FreeBSD, so all these are theoretical.

Rob215x · Saturday at 11:28 PM

T-Aoki said:
Another place to compare between affected and sane computers would be /usr/local/lib/compat/pkg. If there are more ones in affected computer, moving the additional ones into different place that libraries are NOT looked for and try again.

Okay this is interesting... here are the differences. Do you see anything problematic or weird?

Production machine (Rust doesn't build)

 

/usr/local/lib/compat/pkg # ls -l

total 14552

lrwxr-xr-x  1 root  wheel       21 Nov  9  2023 libanonymous.so.3 -> libanonymous.so.3.0.0

-rwxr-xr-x  1 root  wheel    17520 Nov  9  2023 libanonymous.so.3.0.0

lrwxr-xr-x  1 root  wheel       19 Nov  9  2023 libcrammd5.so.3 -> libcrammd5.so.3.0.0

-rwxr-xr-x  1 root  wheel    20168 Nov  9  2023 libcrammd5.so.3.0.0

lrwxr-xr-x  1 root  wheel       18 Apr 18  2024 libdb-5.3.so.0 -> db5/libdb-5.3.so.0

-rwxr-xr-x  1 root  wheel  1710000 Apr 18  2024 libdb-5.3.so.0.0.0

lrwxr-xr-x  1 root  wheel       14 Apr 18  2024 libdb-5.so.0 -> libdb-5.3.so.0

lrwxr-xr-x  1 root  wheel       22 Apr 18  2024 libdb_cxx-5.3.so.0 -> db5/libdb_cxx-5.3.so.0

-rwxr-xr-x  1 root  wheel  1862232 Apr 18  2024 libdb_cxx-5.3.so.0.0.0

lrwxr-xr-x  1 root  wheel       18 Apr 18  2024 libdb_cxx-5.so.0 -> libdb_cxx-5.3.so.0

lrwxr-xr-x  1 root  wheel       22 Apr 18  2024 libdb_stl-5.3.so.0 -> db5/libdb_stl-5.3.so.0

-rwxr-xr-x  1 root  wheel  1935840 Apr 18  2024 libdb_stl-5.3.so.0.0.0

lrwxr-xr-x  1 root  wheel       18 Apr 18  2024 libdb_stl-5.so.0 -> libdb_stl-5.3.so.0

lrwxr-xr-x  1 root  wheel       21 Nov  9  2023 libdigestmd5.so.3 -> libdigestmd5.so.3.0.0

-rwxr-xr-x  1 root  wheel    57552 Nov  9  2023 libdigestmd5.so.3.0.0

lrwxr-xr-x  1 root  wheel       16 Nov  8  2023 libgit2.so.1.6 -> libgit2.so.1.6.4

-rwxr-xr-x  1 root  wheel  1227264 Nov  8  2023 libgit2.so.1.6.4

lrwxr-xr-x  1 root  wheel       23 Nov  8  2023 libhttp_parser.so.2.9 -> libhttp_parser.so.2.9.4

-rwxr-xr-x  1 root  wheel    33416 Nov  8  2023 libhttp_parser.so.2.9.4

lrwxr-xr-x  1 root  wheel       17 Nov  9  2023 liblogin.so.3 -> liblogin.so.3.0.0

-rwxr-xr-x  1 root  wheel    18144 Nov  9  2023 liblogin.so.3.0.0

lrwxr-xr-x  1 root  wheel       20 Jun 20  2023 libmysqlclient_r.so.20 -> libmysqlclient.so.20

lrwxr-xr-x  1 root  wheel       16 Nov  9  2023 libntlm.so.3 -> libntlm.so.3.0.0

-rwxr-xr-x  1 root  wheel    37480 Nov  9  2023 libntlm.so.3.0.0

lrwxr-xr-x  1 root  wheel       15 Nov  9  2023 libotp.so.3 -> libotp.so.3.0.0

-rwxr-xr-x  1 root  wheel    27216 Nov  9  2023 libotp.so.3.0.0

lrwxr-xr-x  1 root  wheel       17 Nov  7  2023 libperl.so.5.34 -> libperl.so.5.34.1

-r--r--r--  1 root  wheel  3727648 Nov  7  2023 libperl.so.5.34.1

lrwxr-xr-x  1 root  wheel       17 Nov  9  2023 libperl.so.5.36 -> libperl.so.5.36.1

-r--r--r--  1 root  wheel  3804648 Nov  9  2023 libperl.so.5.36.1

lrwxr-xr-x  1 root  wheel       17 Nov  9  2023 libplain.so.3 -> libplain.so.3.0.0

-rwxr-xr-x  1 root  wheel    18264 Nov  9  2023 libplain.so.3.0.0

lrwxr-xr-x  1 root  wheel       18 Nov  9  2023 libsasldb.so.3 -> libsasldb.so.3.0.0

-rwxr-xr-x  1 root  wheel    24064 Nov  9  2023 libsasldb.so.3.0.0

lrwxr-xr-x  1 root  wheel       17 Nov  9  2023 libscram.so.3 -> libscram.so.3.0.0

-rwxr-xr-x  1 root  wheel    41136 Nov  9  2023 libscram.so.3.0.0

Backup machine, where Rust DID build successfully:

 

/usr/local/lib/compat/pkg # ls -l

total 19828

lrwxr-xr-x  1 root  wheel       21 Apr  1  2023 libanonymous.so.3 -> libanonymous.so.3.0.0

-rwxr-xr-x  1 root  wheel    17520 Apr  1  2023 libanonymous.so.3.0.0

lrwxr-xr-x  1 root  wheel       19 Apr  1  2023 libcrammd5.so.3 -> libcrammd5.so.3.0.0

-rwxr-xr-x  1 root  wheel    20168 Apr  1  2023 libcrammd5.so.3.0.0

lrwxr-xr-x  1 root  wheel       18 Feb 28  2024 libdb-5.3.so.0 -> db5/libdb-5.3.so.0

-rwxr-xr-x  1 root  wheel  1710000 Feb 28  2024 libdb-5.3.so.0.0.0

lrwxr-xr-x  1 root  wheel       14 Feb 28  2024 libdb-5.so.0 -> libdb-5.3.so.0

lrwxr-xr-x  1 root  wheel       22 Feb 28  2024 libdb_cxx-5.3.so.0 -> db5/libdb_cxx-5.3.so.0

-rwxr-xr-x  1 root  wheel  1862232 Feb 28  2024 libdb_cxx-5.3.so.0.0.0

lrwxr-xr-x  1 root  wheel       18 Feb 28  2024 libdb_cxx-5.so.0 -> libdb_cxx-5.3.so.0

lrwxr-xr-x  1 root  wheel       22 Feb 28  2024 libdb_stl-5.3.so.0 -> db5/libdb_stl-5.3.so.0

-rwxr-xr-x  1 root  wheel  1935840 Feb 28  2024 libdb_stl-5.3.so.0.0.0

lrwxr-xr-x  1 root  wheel       18 Feb 28  2024 libdb_stl-5.so.0 -> libdb_stl-5.3.so.0

lrwxr-xr-x  1 root  wheel       21 Apr  1  2023 libdigestmd5.so.3 -> libdigestmd5.so.3.0.0

-rwxr-xr-x  1 root  wheel    57552 Apr  1  2023 libdigestmd5.so.3.0.0

lrwxr-xr-x  1 root  wheel       17 Apr  1  2023 liblogin.so.3 -> liblogin.so.3.0.0

-rwxr-xr-x  1 root  wheel    18144 Apr  1  2023 liblogin.so.3.0.0

lrwxr-xr-x  1 root  wheel       16 Apr  1  2023 libntlm.so.3 -> libntlm.so.3.0.0

-rwxr-xr-x  1 root  wheel    37480 Apr  1  2023 libntlm.so.3.0.0

lrwxr-xr-x  1 root  wheel       15 Apr  1  2023 libotp.so.3 -> libotp.so.3.0.0

-rwxr-xr-x  1 root  wheel    27216 Apr  1  2023 libotp.so.3.0.0

lrwxr-xr-x  1 root  wheel       17 Dec  4  2020 libperl.so.5.28 -> libperl.so.5.28.3

-r--r--r--  1 root  wheel  3253512 Dec  4  2020 libperl.so.5.28.3

lrwxr-xr-x  1 root  wheel       17 Aug  5  2021 libperl.so.5.32 -> libperl.so.5.32.1

-r--r--r--  1 root  wheel  3773384 Aug  5  2021 libperl.so.5.32.1

lrwxr-xr-x  1 root  wheel       17 Nov  4  2023 libperl.so.5.36 -> libperl.so.5.36.1

-r--r--r--  1 root  wheel  3804648 Nov  4  2023 libperl.so.5.36.1

lrwxr-xr-x  1 root  wheel       17 Apr  1  2023 libplain.so.3 -> libplain.so.3.0.0

-rwxr-xr-x  1 root  wheel    18264 Apr  1  2023 libplain.so.3.0.0

-rw-r--r--  1 root  wheel  3284480 Aug  5  2021 libpython3.8.so.1.0

lrwxr-xr-x  1 root  wheel       18 Apr  1  2023 libsasldb.so.3 -> libsasldb.so.3.0.0

-rwxr-xr-x  1 root  wheel    24064 Apr  1  2023 libsasldb.so.3.0.0

lrwxr-xr-x  1 root  wheel       17 Apr  1  2023 libscram.so.3 -> libscram.so.3.0.0

-rwxr-xr-x  1 root  wheel    41136 Apr  1  2023 libscram.so.3.0.0

Rob215x · Saturday at 11:36 PM

I have a silly question... if one of my websites is busy, could that impact the rust build? I was thinking about it and the backup server has no traffic but the production server gets some traffic. I never see much load in htop, even when one of my websites is "busy" so I figured the websites didn't really affect builds.

In the past, I've always had everything running while I did Portupgrade

cracauer@ · Saturday at 11:47 PM

Rob215x said:
I have a silly question... if one of my websites is busy, could that impact the rust build?

No.

_martin · Saturday at 11:47 PM

Just for clarification: I didn't believe from the beginning this is a HW/memory issue.

T-Aoki said:
If any additional pages (including dummy) are mapped into physical memory pages, means, mapped physical pages increase, possibility to hit broken memory cell increases.

I think we are in agreement that more use of memory increases a chance to hit the issue (if there's a faulty location in physical memory somewhere). I agree with what you said in that thread, just not about ASLR.

T-Aoki said:
If randomizations are always assured to be done "per page"

It's always page aligned. You are always addressing memory in pages (size can differ). vaddr (virtual address) provides a mechanism how to divide virtual space and map it deterministicaly (page tables). OS assignes physical memory (divided into page frames, hence pfn) to that entry. It's just that ASLR, i.e. virtual address itself, doesn't play a role. It doesn't matter if your process starts at 0xc00lc0de000 or 0xbadc0de000, it will be mapped to a certain pfn (I'm not saying to the same one because that would steal this thread completely away

. When exec is done and kernel assignes pfn to the process it doesn't matter if it maps to vaddr1 or vaddr2.

Rob215x If the server backup machine is the same version of FreeBSD you can create a package from the current installation and moved it to prod server:

Code:

[root@dev ~]# pkg info | grep rust
rust-1.91.1                    Language with a focus on memory safety and concurrency
[root@dev ~]#

[root@dev ~]# pkg create rust-1.91.1
Creating package for rust-1.91.1
[root@dev ~]#

Copy pkg file to prod and do:

Code:

prod(~)# pkg add  rust-1.91.1.pkg
Installing rust-1.91.1...
Extracting rust-1.91.1: 100%
prod(~)#

You may need to resolve pkg dependecies. In my test it was only gdb which was missing.

Rob215x · Saturday at 11:59 PM

_martin does it matter if the 2 machines are different hardware?

Backup machine:

 

Copyright (c) 1992-2021 The FreeBSD Project.

Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994

    The Regents of the University of California. All rights reserved.

FreeBSD is a registered trademark of The FreeBSD Foundation.

FreeBSD 13.5-RELEASE releng/13.5-n259162-882b9f3f2218 GENERIC amd64

FreeBSD clang version 19.1.7 (https://github.com/llvm/llvm-project.git llvmorg-19.1.7-0-gcd708029e0b2)

VT(vga): resolution 640x480

CPU: Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz (2926.10-MHz K8-class CPU)

  Origin="GenuineIntel"  Id=0x106a5  Family=0x6  Model=0x1a  Stepping=5

  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>

  Features2=0x9ce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT>

  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>

  AMD Features2=0x1<LAHF>

  Structured Extended Features3=0x9c000000<IBPB,STIBP,L1DFL,SSBD>

  VT-x: (disabled in BIOS) PAT,HLT,MTF,PAUSE,EPT,VPID

  TSC: P-state invariant, performance statistics

real memory  = 68719476736 (65536 MB)

avail memory = 66798718976 (63704 MB)

Production machine:

 

Copyright (c) 1992-2021 The FreeBSD Project.

Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994

    The Regents of the University of California. All rights reserved.

FreeBSD is a registered trademark of The FreeBSD Foundation.

FreeBSD 13.5-RELEASE-p6 GENERIC amd64

FreeBSD clang version 19.1.7 (https://github.com/llvm/llvm-project.git llvmorg-19.1.7-0-gcd708029e0b2)

VT(vga): resolution 640x480

CPU: Intel(R) Xeon(R) E-2236 CPU @ 3.40GHz (3400.00-MHz K8-class CPU)

  Origin="GenuineIntel"  Id=0x906ea  Family=0x6  Model=0x9e  Stepping=10

  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>

  Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>

  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>

  AMD Features2=0x121<LAHF,ABM,Prefetch>

  Structured Extended Features=0x29c6fbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>

  Structured Extended Features3=0x9c002600<MCUOPT,MD_CLEAR,TSXFA,IBPB,STIBP,L1DFL,SSBD>

  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>

  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID

  TSC: P-state invariant, performance statistics

real memory  = 34359738368 (32768 MB)

avail memory = 33235881984 (31696 MB)

_martin · Sunday at 12:15 AM

No, it's ok. They are the same arch (amd64) with alike versions. You can always try and if it doesn't work you just remove the package.

cracauer@ · Sunday at 12:18 AM

Why do you think your services would be down when you try the maybe broken /usr/local on the backup machine? They wouldn't.

T-Aoki · Sunday at 12:32 AM

Rob215x said:
Okay this is interesting... here are the differences. Do you see anything problematic or weird?

Production machine (Rust doesn't build)
/usr/local/lib/compat/pkg # ls -l total 14552 lrwxr-xr-x 1 root wheel 21 Nov 9 2023 libanonymous.so.3 -> libanonymous.so.3.0.0 -rwxr-xr-x 1 root wheel 17520 Nov 9 2023 libanonymous.so.3.0.0 lrwxr-xr-x 1 root wheel 19 Nov 9 2023 libcrammd5.so.3 -> libcrammd5.so.3.0.0 -rwxr-xr-x 1 root wheel 20168 Nov 9 2023 libcrammd5.so.3.0.0 lrwxr-xr-x 1 root wheel 18 Apr 18 2024 libdb-5.3.so.0 -> db5/libdb-5.3.so.0 -rwxr-xr-x 1 root wheel 1710000 Apr 18 2024 libdb-5.3.so.0.0.0 lrwxr-xr-x 1 root wheel 14 Apr 18 2024 libdb-5.so.0 -> libdb-5.3.so.0 lrwxr-xr-x 1 root wheel 22 Apr 18 2024 libdb_cxx-5.3.so.0 -> db5/libdb_cxx-5.3.so.0 -rwxr-xr-x 1 root wheel 1862232 Apr 18 2024 libdb_cxx-5.3.so.0.0.0 lrwxr-xr-x 1 root wheel 18 Apr 18 2024 libdb_cxx-5.so.0 -> libdb_cxx-5.3.so.0 lrwxr-xr-x 1 root wheel 22 Apr 18 2024 libdb_stl-5.3.so.0 -> db5/libdb_stl-5.3.so.0 -rwxr-xr-x 1 root wheel 1935840 Apr 18 2024 libdb_stl-5.3.so.0.0.0 lrwxr-xr-x 1 root wheel 18 Apr 18 2024 libdb_stl-5.so.0 -> libdb_stl-5.3.so.0 lrwxr-xr-x 1 root wheel 21 Nov 9 2023 libdigestmd5.so.3 -> libdigestmd5.so.3.0.0 -rwxr-xr-x 1 root wheel 57552 Nov 9 2023 libdigestmd5.so.3.0.0 lrwxr-xr-x 1 root wheel 16 Nov 8 2023 libgit2.so.1.6 -> libgit2.so.1.6.4 -rwxr-xr-x 1 root wheel 1227264 Nov 8 2023 libgit2.so.1.6.4 lrwxr-xr-x 1 root wheel 23 Nov 8 2023 libhttp_parser.so.2.9 -> libhttp_parser.so.2.9.4 -rwxr-xr-x 1 root wheel 33416 Nov 8 2023 libhttp_parser.so.2.9.4 lrwxr-xr-x 1 root wheel 17 Nov 9 2023 liblogin.so.3 -> liblogin.so.3.0.0 -rwxr-xr-x 1 root wheel 18144 Nov 9 2023 liblogin.so.3.0.0 lrwxr-xr-x 1 root wheel 20 Jun 20 2023 libmysqlclient_r.so.20 -> libmysqlclient.so.20 lrwxr-xr-x 1 root wheel 16 Nov 9 2023 libntlm.so.3 -> libntlm.so.3.0.0 -rwxr-xr-x 1 root wheel 37480 Nov 9 2023 libntlm.so.3.0.0 lrwxr-xr-x 1 root wheel 15 Nov 9 2023 libotp.so.3 -> libotp.so.3.0.0 -rwxr-xr-x 1 root wheel 27216 Nov 9 2023 libotp.so.3.0.0 lrwxr-xr-x 1 root wheel 17 Nov 7 2023 libperl.so.5.34 -> libperl.so.5.34.1 -r--r--r-- 1 root wheel 3727648 Nov 7 2023 libperl.so.5.34.1 lrwxr-xr-x 1 root wheel 17 Nov 9 2023 libperl.so.5.36 -> libperl.so.5.36.1 -r--r--r-- 1 root wheel 3804648 Nov 9 2023 libperl.so.5.36.1 lrwxr-xr-x 1 root wheel 17 Nov 9 2023 libplain.so.3 -> libplain.so.3.0.0 -rwxr-xr-x 1 root wheel 18264 Nov 9 2023 libplain.so.3.0.0 lrwxr-xr-x 1 root wheel 18 Nov 9 2023 libsasldb.so.3 -> libsasldb.so.3.0.0 -rwxr-xr-x 1 root wheel 24064 Nov 9 2023 libsasldb.so.3.0.0 lrwxr-xr-x 1 root wheel 17 Nov 9 2023 libscram.so.3 -> libscram.so.3.0.0 -rwxr-xr-x 1 root wheel 41136 Nov 9 2023 libscram.so.3.0.0

Backup machine, where Rust DID build successfully:
/usr/local/lib/compat/pkg # ls -l total 19828 lrwxr-xr-x 1 root wheel 21 Apr 1 2023 libanonymous.so.3 -> libanonymous.so.3.0.0 -rwxr-xr-x 1 root wheel 17520 Apr 1 2023 libanonymous.so.3.0.0 lrwxr-xr-x 1 root wheel 19 Apr 1 2023 libcrammd5.so.3 -> libcrammd5.so.3.0.0 -rwxr-xr-x 1 root wheel 20168 Apr 1 2023 libcrammd5.so.3.0.0 lrwxr-xr-x 1 root wheel 18 Feb 28 2024 libdb-5.3.so.0 -> db5/libdb-5.3.so.0 -rwxr-xr-x 1 root wheel 1710000 Feb 28 2024 libdb-5.3.so.0.0.0 lrwxr-xr-x 1 root wheel 14 Feb 28 2024 libdb-5.so.0 -> libdb-5.3.so.0 lrwxr-xr-x 1 root wheel 22 Feb 28 2024 libdb_cxx-5.3.so.0 -> db5/libdb_cxx-5.3.so.0 -rwxr-xr-x 1 root wheel 1862232 Feb 28 2024 libdb_cxx-5.3.so.0.0.0 lrwxr-xr-x 1 root wheel 18 Feb 28 2024 libdb_cxx-5.so.0 -> libdb_cxx-5.3.so.0 lrwxr-xr-x 1 root wheel 22 Feb 28 2024 libdb_stl-5.3.so.0 -> db5/libdb_stl-5.3.so.0 -rwxr-xr-x 1 root wheel 1935840 Feb 28 2024 libdb_stl-5.3.so.0.0.0 lrwxr-xr-x 1 root wheel 18 Feb 28 2024 libdb_stl-5.so.0 -> libdb_stl-5.3.so.0 lrwxr-xr-x 1 root wheel 21 Apr 1 2023 libdigestmd5.so.3 -> libdigestmd5.so.3.0.0 -rwxr-xr-x 1 root wheel 57552 Apr 1 2023 libdigestmd5.so.3.0.0 lrwxr-xr-x 1 root wheel 17 Apr 1 2023 liblogin.so.3 -> liblogin.so.3.0.0 -rwxr-xr-x 1 root wheel 18144 Apr 1 2023 liblogin.so.3.0.0 lrwxr-xr-x 1 root wheel 16 Apr 1 2023 libntlm.so.3 -> libntlm.so.3.0.0 -rwxr-xr-x 1 root wheel 37480 Apr 1 2023 libntlm.so.3.0.0 lrwxr-xr-x 1 root wheel 15 Apr 1 2023 libotp.so.3 -> libotp.so.3.0.0 -rwxr-xr-x 1 root wheel 27216 Apr 1 2023 libotp.so.3.0.0 lrwxr-xr-x 1 root wheel 17 Dec 4 2020 libperl.so.5.28 -> libperl.so.5.28.3 -r--r--r-- 1 root wheel 3253512 Dec 4 2020 libperl.so.5.28.3 lrwxr-xr-x 1 root wheel 17 Aug 5 2021 libperl.so.5.32 -> libperl.so.5.32.1 -r--r--r-- 1 root wheel 3773384 Aug 5 2021 libperl.so.5.32.1 lrwxr-xr-x 1 root wheel 17 Nov 4 2023 libperl.so.5.36 -> libperl.so.5.36.1 -r--r--r-- 1 root wheel 3804648 Nov 4 2023 libperl.so.5.36.1 lrwxr-xr-x 1 root wheel 17 Apr 1 2023 libplain.so.3 -> libplain.so.3.0.0 -rwxr-xr-x 1 root wheel 18264 Apr 1 2023 libplain.so.3.0.0 -rw-r--r-- 1 root wheel 3284480 Aug 5 2021 libpython3.8.so.1.0 lrwxr-xr-x 1 root wheel 18 Apr 1 2023 libsasldb.so.3 -> libsasldb.so.3.0.0 -rwxr-xr-x 1 root wheel 24064 Apr 1 2023 libsasldb.so.3.0.0 lrwxr-xr-x 1 root wheel 17 Apr 1 2023 libscram.so.3 -> libscram.so.3.0.0 -rwxr-xr-x 1 root wheel 41136 Apr 1 2023 libscram.so.3.0.0

I've saved both list as:

the former (affected): /tmp/list_affected.txt
the latter (backup, not affected): /tmp/list_normal.txt

and performed below.

Code:

% cut -w -f9,10,11 /tmp/list_normal.txt | sort > /tmp/list_normal2.txt
% cut -w -f9,10,11 /tmp/list_affected.txt | sort > /tmp/list_affected2.txt
% diff -u -p /tmp/list_normal2.txt /tmp/list_affected2.txt | lv

--- /tmp/list_normal2.txt       2025-11-23 08:59:03.455167000 +0900
+++ /tmp/list_affected2.txt     2025-11-23 08:59:19.496596000 +0900
@@ -15,21 +15,23 @@ libdigestmd5.so.3.0.0
 libdb-5.so.0   ->      libdb-5.3.so.0
 libdigestmd5.so.3      ->      libdigestmd5.so.3.0.0
 libdigestmd5.so.3.0.0
+libgit2.so.1.6 ->      libgit2.so.1.6.4
+libgit2.so.1.6.4
+libhttp_parser.so.2.9  ->      libhttp_parser.so.2.9.4
+libhttp_parser.so.2.9.4
 liblogin.so.3  ->      liblogin.so.3.0.0
 liblogin.so.3.0.0
+libmysqlclient_r.so.20 ->      libmysqlclient.so.20
 libntlm.so.3   ->      libntlm.so.3.0.0
 libntlm.so.3.0.0
 libotp.so.3    ->      libotp.so.3.0.0
 libotp.so.3.0.0
-libperl.so.5.28        ->      libperl.so.5.28.3
-libperl.so.5.28.3
-libperl.so.5.32        ->      libperl.so.5.32.1
-libperl.so.5.32.1
+libperl.so.5.34        ->      libperl.so.5.34.1
+libperl.so.5.34.1
 libperl.so.5.36        ->      libperl.so.5.36.1
 libperl.so.5.36.1
 libplain.so.3  ->      libplain.so.3.0.0
 libplain.so.3.0.0
-libpython3.8.so.1.0
 libsasldb.so.3 ->      libsasldb.so.3.0.0
 libsasldb.so.3.0.0
 libscram.so.3  ->      libscram.so.3.0.0

The lines headed by

- means existing on normal case only,
+ means existing on affected case only, and
(space) means that matches.

and among things headed by +, devel/libgit2, which is required by devel/cargo-c that is related with Rust, requires it.

So, does things differ if you rename libgit2.so.1.6 and libgit2.so.1.6.4 in /usr/local/lib/compat/pkg on affected computer to something like libgit2.so.1.6.orig and libgit2.so.1.6.4.orig?

Anyway, not all of libraries there wouldn't actually needed, as the directory is (if I recall correctly) first introduced by portupgrade to preserve old libraries on upgrading not to break anything until all upgrades finishes sanely.
This worked quite fine as there were no other subdirectories in /usr/local/lib at the era portupgrade first appeared and /usr/local/lib/compat/pkg was looked for only when there's no wanted libraries directly under /lib, /usr/lib or /usr/local/lib.

But things are changed significantly and many ports create their own subdirectories under /usr/local/lib now.

Because of this, if libraries in /usr/local/lib/compat/pkg is/are loaded BEFORE actually needed libraries, it hurts.

Not always 100% assured to be safe, but usually deleting everything under /usr/local/lib/compat/pkg is safe, if you're NOT pending to upgrade some port and NOT keeping ports which are dissappeared from ports tree, or nothing failed for upgrading exist.

Rob215x · Sunday at 12:37 AM

At what point does py311-maturin-1.10.1 come into play?

Backup server (Rust compiled with no issues)

 

# pkg info | grep rust

py311-maturin-1.10.1           Build and publish crates with pyo3, rust-cpython and cffi bindings

py311-setuptools-rust-1.12.0   Setuptools Rust extension plugin

rust-1.91.1                    Language with a focus on memory safety and concurrency

Production server (Rust 1.91.1 make FAILED)

 

# pkg info | grep rust

py311-setuptools-rust-1.12.0   Setuptools Rust extension plugin

rust-1.89.0_1                  Language with a focus on memory safety and concurrency

Also, I just created the rust-1.91.1.pkg on my backup machine. It took 10 minutes to create it.

But I wonder if I should install it on the production machine before figuring out where I'm actually having the issue?

T-Aoki · Sunday at 1:07 AM

Rob215x said:
At what point does py311-maturin-1.10.1 come into play?

I don't think it affects Rust builds, but seemingly one of the consumers of Rust.
devel/py-maturin appears in "This port is required by: for Build" in entry for lang/rust at FreshPorts, on the other hand, lang/rust appears as "Build dependencies:" in entry for devel/py-maturin at FreshPorts.

Rob215x · Sunday at 2:03 AM

_martin

I copied the rust-1.91.1.pkg to my production machine and...

 

# pkg add rust-1.91.1.pkg

Installing rust-1.91.1...

the most recent version of rust-1.89.0_1 is already installed

does that make sense? Sorry I haven't done upgrades this way before. Do I need to pkg remove rust-1.89.0_1 first?

T-Aoki · Sunday at 7:32 AM

Rob215x said:
Do I need to pkg remove rust-1.89.0_1 first?

Yes.
What portupgrade (and anything alike appeared after it) does on upgrading is:

build the new version
if successful, backup currently installed old version (by default, libraries are copied into /usr/local/lib/compat/pkg/
once the backup succeeds, deinstall currently installed version
install the built new version
if the installation somehow failed, delete what's written into running system and restore previous version from backup

You should have seen the transcript above when you ran portupgrade.
So doing pkg install not after pkg remove (should be the alias of pkg delete) means skipping step 3 above.

Note that, as far as I know, portmaster doesn't copy old libraries into /usr/local/lib/compat/pkg/ by default (possible with option, though).

Rob215x · Sunday at 8:18 AM

T-Aoki I certainly have seen portupgrade take those steps many times. I understand that Portupgrade is basically building each port based on its local config (how I customize my PHP and Apache), while pkg is installing a port that was built somewhere else.

What I'm not familiar with is what steps pkg does (or doesn't do) automatically.

I appreciate you and everyone in this thread. I'm learning new things!

Also, I thought it was weird for it to say "the most recent version of rust-1.89.0_1 is already installed" because if I do pkg version it says my rust needs to be updated to 1.91.1

_martin · Sunday at 6:33 PM

My method of doing so (manually installing) was to provide you a way to get to the rust version you may need on prod server. I'd personally avoid compiling anything on prod server, that's what you have dev/test systems for. With that you can bring tested packages to prod systems (if you can't use prebuilt packages from FreeBSD).

You could also create a backup of the currently installed package on prod by doing pkg create rust-1.89.0_1 and have that ready in case you need to rollback. Manual uninstall/install is needed.

Rob215x · Monday at 7:08 AM

Hey, I just wanted to report that _martin gave me the idea of making the Rust 1.91 pkg on my backup server, then copying it to my production server and it worked!!

I just ran Portupgrade and it took a while to rebuild cargo-c. Then it built several other things, including py311-maturin-1.10.1 like on my backup server.

Portupgrade completed with no errors!

T-Aoki · Monday at 8:57 AM

Maybe you would already know, but portupgrade has option -p to create pkg along with upgrading, saved into /usr/ports/packages/All/ by default.
Possibly pkg create after upgrading is faster, though.

Rob215x · Monday at 9:02 PM

T-Aoki said:
Maybe you would already know, but portupgrade has option -p to create pkg along with upgrading, saved into /usr/ports/packages/All/ by default.
Possibly pkg create after upgrading is faster, though.

Thanks I did not know that. I typically do portupgrade -urRac