Hi!
I am trying to run software that utilises RocksDB as a data storage backend, the data structures are quite large and on Linux this software opens almost 130k files.
On FreeBSD initialisation fails because RocksDB breaks down with "Too many open files" (EMFILE) error precisely at 32768'th file.
After spending considerable time on troubleshooting I am sure that this is not user or kernel limits, those are way up, instead I believe that the way how RocksDB opens files causes FreeBSD to hit the fd > SHRT_MAX check and return the EMFILE error: https://github.com/freebsd/freebsd-src/blob/main/lib/libc/stdio/fdopen.c#L64-L67
Here is the part in RocksDB code that opens files: https://github.com/facebook/rocksdb/blob/master/env/fs_posix.cc#L164-L191
I am almost sure that this is not FreeBSD problem but issue with RocksDB but unfortunately I am not very familiar with FreeBSD C Development and do not know how to do this properly, I would be very thankful for advise on how to open files properly in order to avoid this limit, so I can patch RocksDB.
Thank you!
I am trying to run software that utilises RocksDB as a data storage backend, the data structures are quite large and on Linux this software opens almost 130k files.
On FreeBSD initialisation fails because RocksDB breaks down with "Too many open files" (EMFILE) error precisely at 32768'th file.
After spending considerable time on troubleshooting I am sure that this is not user or kernel limits, those are way up, instead I believe that the way how RocksDB opens files causes FreeBSD to hit the fd > SHRT_MAX check and return the EMFILE error: https://github.com/freebsd/freebsd-src/blob/main/lib/libc/stdio/fdopen.c#L64-L67
Here is the part in RocksDB code that opens files: https://github.com/facebook/rocksdb/blob/master/env/fs_posix.cc#L164-L191
I am almost sure that this is not FreeBSD problem but issue with RocksDB but unfortunately I am not very familiar with FreeBSD C Development and do not know how to do this properly, I would be very thankful for advise on how to open files properly in order to avoid this limit, so I can patch RocksDB.
Thank you!