Issue with ULE scheduler

CyberCr33p

CyberCr33p

CPU: AMD Ryzen 9 5950X 16 Cores (32 Threads)

I use this command to compress a lot of images:

find . -iname "*.jpg" -exec ~/graphicsmagick/bin/gm mogrify -strip -quality 75% {} > /dev/null 2>&1 \;


The issue is that GraphicsMagick (the same with ImageMagick) after some time hangs, the process usage is 100% and truss shows:

Code: 
sched_yield() = 0 (0x0)
sched_yield() = 0 (0x0)
sched_yield() = 0 (0x0)


Does anyone else has such issues with ULE scheduler ?


I found this https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=208421 which may be related.
 
What version of FreeBSD, what is the filesystem, is this in a jail?
Reason for asking is that bug is against FreeBSD 10.2 last update was back in 2018.
Have you tried other ways of accomplishing the task? Maybe do the find command to get a list of files, then a script that reads the file and does the gm command on each line?
Basically to rule out an interaction between the find and the gm command.
 
The same issue exist if I use a shell script with a for loop to do the compressions:

Code: 
#!/usr/local/bin/bash

for i in `cat list.txt`; do
  ~/graphicsmagick/bin/gm mogrify -strip -quality 75% "${i}"
done
 
  • Like
Reactions: mer
I don't have any answers, just trying to get whatever data we can.
If you do a single file by hand, does the same issue happen?
Did you build the gm executable against 13.1, install a binary from somewhere?
 
covacat said:
does it stall at the same image or random ?
Click to expand...
Random image

mer said:
I don't have any answers, just trying to get whatever data we can.
If you do a single file by hand, does the same issue happen?
Did you build the gm executable against 13.1, install a binary from somewhere?
Click to expand...
If I do a single file it doesn't hang but I am not sure for this as it hang in random file in the loop.

I build gm against 13.1.
 
  • Like
Reactions: mer
What version did you build? I'm asking because it's available in the ports/packages so a pkg install GraphicsMagick will install 1.3.37 it seems. If you don't have admin access to install packages, building under your user is fine, just want to eliminate variables.
EDIT 1.3.37 is the latest in quarterly packages, latest packages may have 1.3.38
 
mer said:
What version did you build? I'm asking because it's available in the ports/packages so a pkg install GraphicsMagick will install 1.3.37 it seems. If you don't have admin access to install packages, building under your user is fine, just want to eliminate variables.
Click to expand...
I use the latest version. The same issue happens with ImageMagick too.

The issue exist with "AMD Ryzen 9 5950X 16 Cores" and NVME disks but with "Intel i7-6700 Quad-Core Skylake" and mechanical disks doesn't exist.
 
If it's starting new process/thread before previous were stopped it can consume not only CPU but also RAM/swap - check if it's not the cause.
Edit: after closer inspection it looks like it's running sequentially, only one process/thread is created.
 
As a test in the scripted version, what happens if you add a "sleep 1" after the gm command so that the loop sleeps a little bit before the next iteration?
Yes I know it adds time if there are a lot of files, but it's a bit of a workaround and could help determine if it's a "process exit and cleanup" that's too slow and causing issues.
 
Running find . -iname "*.jpg" -exec jpegoptim -s -q {} \; the issue doesn't exist. So far I was able to reproduce it only with GraphicsMagick / ImageMagick.
 
I don't use graphicsmagick frequently so I didn't notice the same issue again. Maybe it's already fixed in 13.2.

With that program you have issue?
 
I'm trying to render out a video, which was created in Blender before.
The script I use is https://pypi.org/project/bpsrender/

It allows you to define the number of cores you want to use for rendering. Up to 3 cores the script runs smoothly, I get something like 35 frames per second rendered.
If I set it to use 4 or more cores it looks like it's pausing every some seconds for a 1 - 2 seconds before it continues to run.
 
You must log in or register to reply here.
Back
Top