Issues with python web applications in jails

S

sand_man

Hi all.
I currently have four jails. One is hosting DNS which is working, one is hosting a simple apache/php server which is also working.
I also have a jail hosting plexmediaserver and another jail hosting two python web applications (sickbeard and sabnzbdplus). Since upgrading to 10.2-RELEASE, the applications in these two jails no longer work. If I run top or ps I can see the process is there but a netstat -a does not show that there is a service listening on the ports I expect to see. Trying to browse using lynx both locally from the jail and from the host, just gets a connection refused.

Now, since upgrading to 10.2-RELEASE, I upgraded all ports using portmaster(8) as usual. However in both the problematic jails, when it tries to compile python2.7 it hangs during the Makefile check at:
Code: 
checking whether pthreads are available without options...
I can ctrl-c out but otherwise it just sits there indefinitely.
I have no idea if this is the cause of my problems but it's the best place to start. Does anyone have any idea or suggestion on where to go from here?
 
I am having exactly the same issues. Additionally I have a jail with a MySQL server and another jail with a Java app (Serviio mediaserver) which are also not working anymore.
OpenJDK 8 and 7 doesn't matter.
Also Samba 4.x (which relies on Python) doesn't work anymore.

I have tried to rollback to 10.1 - no change
I tried to reinstall all packages - no change
I created a new jail to test - same issues

sand_man: do you have java installed?
What does java -version return?

For me it returns nothing at all (hangs until ctrl+c is pressed)
 
If I compile lang/python27 with disabled "thread support" then it continues three lines further but it now hangs after that line (pthread)

Code: 
######
checking whether pthreads are available without options... no
checking whether cc accepts -Kpthread... no
checking whether cc accepts -Kthread... no
checking whether cc accepts -pthread...
 
Dr.Moe: No sorry I don't have Java installed but I did also try disabling thread support like you did with the same result. I thought it might have been an issue with clang so I tried to compile with lang/gcc48 but that made no difference.

yggdrasil: I guess I might have to submit a bug report. Thanks.
 
No I haven't reported it yet, sorry. I may get around to doing it later today if I have time. It's been a low priority for me.
You seem to be having the same problem so feel free to raise it yourself if you want. Let me know the bug report # if you do.
 
Actually I don't. Yet. I planned to upgrade to 10.2, but then saw this, so the update is on hold for me at the moment, until this is solved.
 
Is everyone running the most recent 10.2-RELEASE patch level? The closest errata I see is some very specific things regarding threading that don't seem to directly related to the issue we are seeing and I am having no issues building on both a patched 10.2-RELEASE-p5 and unpatched 10.2-RELEASE.
https://www.FreeBSD.org/security/advisories/FreeBSD-EN-15:17.libc.asc

In my Poudriere jail on 10.2-RELEASE-p5 my lang/python27 log reflects success:
Code: 
checking whether gcc supports ParseTuple __format__... no
checking whether pthreads are available without options... yes
checking whether c++ also accepts flags for thread support... no

In a normal 10.2-RELEASE jail I have for testing, cd /usr/ports/lang/python27; portmaster works just fine.

Dr.Moe said:
I am having exactly the same issues. Additionally I have a jail with a MySQL server and another jail with a Java app (Serviio mediaserver) which are also not working anymore. OpenJDK 8 and 7 doesn't matter. Also Samba 4.x (which relies on Python) doesn't work anymore.
Click to expand...

Logically, there is either more than one problem going on or all of these are symptoms of the same underlying issue. I don't know what that is just yet.

sand_man said:
...
Now, since upgrading to 10.2-RELEASE, I upgraded all ports using portmaster(8) as usual.
...
Click to expand...
How did you upgrade?

Can you both check in a completely clean jail or VM and see if the problem can be replicated?
 
I have the same problem, but with anything that seems to require threads inside jails (also for nodejs etc.)

It seems that processes get stuck in the umtxn state:

Code: 
# ps -aux | grep python
>> root  3846  0.4  0.3  114408  22492  1  S+J  4:56PM  0:00.44 /usr/local/bin/python2.7 /usr/local/bin/ipython << hangs in umtxn state
root  3853  0.0  0.0  18844  2172  2  S+  4:56PM  0:00.00 grep python
root@nergal:~ # procstat -kk 3846
  PID  TID COMM  TDNAME  KSTACK
3846 101222 python2.7  -  mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d umtxq_sleep+0x125 do_lock_umutex+0x1f74 __umtx_op_wait_umutex+0x78 amd64_syscall+0x357 Xfast_syscall+0xfb

I can kill the processes, but only from outside the current terminal (e.g. in another screen session).

Also in a fresh jail I see the same behaviour.

I'm now doing a clean rebuild of all my packages with poudriere with an updated build jail. See if that helps
 
Did a total rebuild of all my ports with an updated poudriere jail and a pkg upgrade -f inside the jail. The problem still persists. Note that everything runs fine outside of the jail.

Oh, I am running the latest 10.2-RELEASE-p5 (upgraded through freebsd-update. I run my jails with iocage.

Also of interest:

Code: 
$ python 
Python 2.7.10 (default, Oct 13 2015, 15:12:50)
[GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>> import threading
>>> thread = threading.Thread()
>>> thread.setDaemon(True) 
>>> thread.start()
[Hit Ctrl-T]
load: 0.41  cmd: python2.7 44300 [umtxn] 35.86r 0.02u 0.03s 0% 7668k
[Hit Ctrl-\]
^\[1]  44300 quit (core dumped)  python

For good measure, here is a backtrace of that core dump:

Code: 
#0  0x00000008008338ca in __error () from /lib/libthr.so.3
[New Thread 801c06400 (LWP 100370/python2.7)]
(gdb) bt
#0  0x00000008008338ca in __error () from /lib/libthr.so.3
#1  0x000000080082e40d in pthread_mutex_destroy () from /lib/libthr.so.3
#2  0x000000080082dfca in pthread_mutex_lock () from /lib/libthr.so.3
#3  0x00000008014ea679 in sbrk () from /lib/libc.so.7
#4  0x00000008014d5283 in syscall () from /lib/libc.so.7
#5  0x00000008014f4446 in calloc () from /lib/libc.so.7
#6  0x000000080082e4de in pthread_mutex_destroy () from /lib/libthr.so.3
#7  0x000000080082dfca in pthread_mutex_lock () from /lib/libthr.so.3
#8  0x00000008014ea679 in sbrk () from /lib/libc.so.7
#9  0x00000008014d5283 in syscall () from /lib/libc.so.7
#10 0x00000008014f4446 in calloc () from /lib/libc.so.7
#11 0x000000080082fbda in pthread_kill () from /lib/libthr.so.3
#12 0x0000000800827e3e in pthread_create () from /lib/libthr.so.3
#13 0x0000000800b5d408 in PyThread_start_new_thread () from /usr/local/lib/libpython2.7.so.1
#14 0x0000000800b61674 in initthread () from /usr/local/lib/libpython2.7.so.1
#15 0x0000000800b29bfe in PyEval_EvalFrameEx () from /usr/local/lib/libpython2.7.so.1
#16 0x0000000800b2c3d2 in _PyEval_SliceIndex () from /usr/local/lib/libpython2.7.so.1
#17 0x0000000800b27f2b in PyEval_EvalFrameEx () from /usr/local/lib/libpython2.7.so.1
#18 0x0000000800b24bc4 in PyEval_EvalCodeEx () from /usr/local/lib/libpython2.7.so.1
#19 0x0000000800b24536 in PyEval_EvalCode () from /usr/local/lib/libpython2.7.so.1
#20 0x0000000800b4c87d in PyRun_InteractiveOneFlags () from /usr/local/lib/libpython2.7.so.1
#21 0x0000000800b4c35e in PyRun_InteractiveLoopFlags () from /usr/local/lib/libpython2.7.so.1
#22 0x0000000800b4c1f2 in PyRun_AnyFileExFlags () from /usr/local/lib/libpython2.7.so.1
#23 0x0000000800b5f8d6 in Py_Main () from /usr/local/lib/libpython2.7.so.1
#24 0x000000000040089f in main ()
 
EDIT: Reported as https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203765

This code exhibits the problem. It is just a very simple threading example:

Code: 
#include <unistd.h>
#include <sys/types.h>
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include <string.h>

void  message(void *ptr);

typedef struct stdata {
  int  id;
  char  msg[100];
}  stdata;

int
main()
{
  pthread_t t1, t2;
  stdata d1, d2;

  d1.id = 1;
  strcpy(d1.msg, "Thread 1");

  d2.id = 2;
  strcpy(d2.msg, "Thread 2");

  pthread_create(&t1, NULL, (void *)&message, (void *)&d1);
  pthread_create(&t2, NULL, (void *)&message, (void *)&d2);

  pthread_join(t1, NULL);
  pthread_join(t2, NULL);

  exit(0);
}

void
message(void *ptr)
{
  stdata *data;

  data = (stdata *) ptr;

  printf("Thread %d message %s \n", data->id, data->msg);

  pthread_exit(0);
}

Demonstration:

Code: 
..[coolvibe@web] - [~/test/pthread] - [Wed Oct 14, 10:19]
..[$] <()> cc -o pthread pthread.c -pthread
..[coolvibe@web] - [~/test/pthread] - [Wed Oct 14, 10:20]
..[$] <()> ./pthread &
[1] 45949
..[coolvibe@web] - [~/test/pthread] - [Wed Oct 14, 10:20]
..[$] <()> ps -aux | grep pthread
coolvibe 45949  0.0  0.0  14564  1708  5  SNJ  10:20AM 0:00.00 ./pthread
..[coolvibe@web] - [~/test/pthread] - [Wed Oct 14, 10:21]
..[$] <()> procstat -kk 45949
  PID  TID COMM  TDNAME  KSTACK  
45949 101151 pthread  -  mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d umtxq_sleep+0x125 do_lock_umutex+0x1f74 __umtx_op_wait_umutex+0x78 amd64_syscall+0x357 Xfast_syscall+0xfb
..[coolvibe@web] - [~/test/pthread] - [Wed Oct 14, 10:21]
..[$] <()> fg
[1]  + 45949 running  ./pthread
load: 0.27  cmd: pthread 45949 [umtxn] 35.86r 0.00u 0.00s 0% 1692k
^\[1]  + 45949 quit (core dumped)  ./pthread
..[coolvibe@web] - [~/test/pthread] - [Wed Oct 14, 10:21]
..[$] <()>
 
I solved it.

For everyone that has this problem, check if libthr is the same as you have on your main host (where stuff does seem to work). So make sure all your libraries are of the right release (hint, check modification date/checksums).

I solved my problems by unpacking the base tarball again inside the jail root (be sure to exclude /etc and /var) and doing an # iocage update to get it up to speed.
 
You must log in or register to reply here.
Back
Top