fgrep excessive time increase after upgrade from 12.4 to 14.3 - regex library issue?

R

rowan194

This is a weird one. Since upgrading FreeBSD (jumping a couple of major versions), a script that used to run in less than a minute now takes literal hours. I've managed to distill the delay down to this use of fgrep:

cat filename.txt | fgrep --text -if filter.txt

- filename.txt contains 64 million lines, 2.4 GB
- filter.txt contains 615 lines, 6 KB

fgrep CPU usage is at 100% consistently.

I do have a backup of the 12 system, and I did notice one possible explanation:

fgrep 12.4: libgnuregex.so.5
fgrep 14.3: libregex.so.1

I cannot find any information about this change, or anyone else mentioning a regression. Hope someone has some insight. Thanks.
 
  • Like
Reactions: mro
Yes, this is normal. Unfortunately. I think it might be included in the foundation's SIMD efforts for string functions in FreeBSD.

There is a port for the GNU grep versions.
 
covacat said:
-i sucks in the bsd lib
because instead of a normal no case comparison it expands the search text from fubarbaz to [Ff][uU][bB][aA][rR][aA][zZ]
Click to expand...
Ouch, that might explain it.

I use grep -i in so many places, routinely, I'm now wondering where else may have been impacted...

In this instance, I've installed gnugrep from packages, and the filtering now takes 25 seconds. A massive difference.

grep: 41 lines/sec
ggrep: 4,000 lines/sec
 
here is the patch to make libregex less sucky
here is the speed diff
i cat-ed the files in /usr/share/dict 30 times in a big file
then grep-ed with both libs
Code: 
[titus@utmbox ~/builds/tr]$ time grep -i bollock /tmp/huge.txt |md5
561e3653847646190f3e581c6e5c0fb4

real    0m1.028s
user    0m0.992s
sys    0m0.032s
[titus@utmbox ~/builds/tr]$ unset LD_LIBRARY_PATH
[titus@utmbox ~/builds/tr]$ time grep -i bollock /tmp/huge.txt |md5
561e3653847646190f3e581c6e5c0fb4

real    0m23.005s
user    0m22.895s
sys    0m0.114s
[titus@utmbox ~/builds/tr]$ wc -l /tmp/biga.txt
 16495170 /tmp/biga.txt

i tried netbsd's libregex testsuite and it passes all
 

Attachments

it has some bugs relative to wide chars É é Ç but should be fixable

the problem with the current code and RE_ICASE is that it emits sets of [nN] for each char and won't be able to create a *must_have* contiguous pattern and use Boyer-Moore on it
 
i should remove wide chars from the optimization and then the boyer-moore pattern will be ascii only
this is easy and for strict latin grep-ers is best
running boyer-moore which searches backwards on an utf-8 string (variable width characters) is a bitch
also im not even sure that wide chars upper and lowercase have the same number of bytes
 
to build and test do this
apply the patch
cd /usr/src/lib/libnetbsd
make
cd /usr/src/lib/libregex
make
export LD_LIBRARY_PATH=/usr/obj/home/titus/builds/usr/src/arm64.aarch64/lib/libregex/ (replace with the actual build dir)
now grep and other tools will use your new lib
unset LD_LIBRARY_PATH will get you to the original state (tools will use the stock libregex)
 
Code: 
# time ggrep "^\*" oix-full-snapshot-latest.dat > b
        6.49 real         3.18 user         3.17 sys

# time grep "^\*" oix-full-snapshot-latest.dat > b
      683.32 real        38.91 user       639.51 sys

Another real-world case. System grep takes more than 100 times longer than GNU grep
 
its not because matching speed (mostly) its because our grep flushes stdout after every line printed regardless of output being a terminal or not
on the same file grep -c is only about 3x slower than gnu grep
but that file has 18 million lines matched so the 18 million flushes come in the way
 
Code: 
[titus@z ~]$ time /usr/obj/usr/src/amd64.amd64/usr.bin/grep/grep '^*' oix-full-snapshot-latest.dat > zzz

real    0m8.277s
user    0m6.826s
sys    0m1.397s
[titus@z ~]$ time grep '^*' oix-full-snapshot-latest.dat > zzz

real    1m14.028s
user    0m14.133s
sys    0m59.773s
[titus@z ~]$ time grep '^*' oix-full-snapshot-latest.dat > yyy

[titus@z ~]$ md5 zzz yyy
MD5 (zzz) = 154d6a383993547fee2e9f5ccafb9cf0
MD5 (yyy) = 154d6a383993547fee2e9f5ccafb9cf0
and the patch
Diff: 
--- /tmp/g/grep.c    2026-11-10 13:20:42.626287000 +0200
+++ ./grep.c    2026-11-10 13:15:36.579330000 +0200
@@ -104,6 +104,7 @@
 bool     wflag;        /* -w: pattern must start and end on word boundaries */
 bool     xflag;        /* -x: pattern must match entire line */
 bool     lbflag;    /* --line-buffered */
+bool     flflag;     /* use flush */
 bool     nullflag;    /* --null */
 char    *label;        /* --label */
 const char *color;    /* --color */
@@ -381,6 +382,9 @@
         aargc = argc;
     }

+    if ( isatty(STDOUT_FILENO))
+        flflag = true;
+
     while (((c = getopt_long(aargc, aargv, optstr, long_options, NULL)) !=
         -1)) {
         switch (c) {
@@ -597,6 +601,7 @@
             break;
         case LINEBUF_OPT:
             lbflag = true;
+            flflag = true;
             break;
         case NULL_OPT:
             nullflag = true;
diff -ubsw /tmp/g/grep.h ./grep.h
--- /tmp/g/grep.h    2026-11-10 13:20:42.759727000 +0200
+++ ./grep.h    2026-11-10 13:12:39.342249000 +0200
@@ -118,7 +118,7 @@
 extern bool     Eflag, Fflag, Gflag, Hflag, Lflag,
          bflag, cflag, hflag, iflag, lflag, mflag, nflag, oflag,
          qflag, sflag, vflag, wflag, xflag;
-extern bool     dexclude, dinclude, fexclude, finclude, lbflag, nullflag;
+extern bool     dexclude, dinclude, fexclude, finclude, lbflag, nullflag, flflag;
 extern long long Aflag, Bflag;
 extern long long mcount;
 extern long long mlimit;
Only in .: tests
diff -ubsw /tmp/g/util.c ./util.c
--- /tmp/g/util.c    2026-11-10 13:20:43.057778000 +0200
+++ ./util.c    2026-11-10 13:16:07.709757000 +0200
@@ -728,7 +728,7 @@
     printline_metadata(line, sep);
     fwrite(line->dat, line->len, 1, stdout);
     putchar(fileeol);
-
+    if (flflag)
         fflush(stdout);
 }

@@ -836,6 +836,7 @@
                 *last_out = pc->ln.len;
             }
             putchar('\n');
+            if (flflag)
                 fflush(stdout);
         } else if (!oflag) {
             /*
@@ -847,6 +848,7 @@
              */
             terminated = false;
         } else {
+            if (flflag)
                 fflush(stdout);
         }
     } else
 
You must log in or register to reply here.
Back
Top