Jitsi-meet / jicofo java truststore issue?

hendrack · Apr 20, 2023

I've been trying to configure a jitsi-meet server for a time now, following various guides, but I get this error in jicofo.log:

Code:

Jicofo 2023-04-20 19:17:28.889 INFO: [1] JicofoServices.<init>#134: Starting HTTP server with config: host=null, port=8888, tlsPort=8843, isTls=false, keyStorePath=null, sendServerVersion=true, [TLSv1.2, TLSv1.3]=[TLSv1.2, TLSv1.3], tlsCipherSuites=[TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256].
Jicofo 2023-04-20 19:17:29.004 SEVERE: [16] [xmpp_connection=client] XmppProvider.doConnect#179: Failed to connect/login: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
org.jivesoftware.smack.SmackException$SmackWrappedException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Followed by a loooong list of java exceptions.

Prosody only shows this when I try to connect to the jitsi-server:

Code:

info    Client disconnected: sslv3 alert certificate unknown
info    Client connected
info    Client disconnected: tlsv1 alert internal error

nginx/jitsi-videobridge log shows no error.

So I assume this java/ssl issue is the showstopper. The certificates are from lets encrypt and I generated the prosody certs and the jicofo truststore.jks, according to the honeyguide blog. This blog also mentions that the RC script does not read the configuration files, is this still an issue?

Any ideas? Thank you!

Edit: Maybe a problem with the prosody certificates, oh well: https://github.com/jitsi/jitsi-videobridge/issues/1402

jphovila · Aug 27, 2023

I'm fighting with the same exact problem.

It's not the first time I'm trying to figure out, how (exactly) to make a Java app trust self-signed certificates.

Googling pours out tons of different proposed solutions, none of which seem to work.

Based on what is said in https://www.bobeager.uk/pdf/jitsi.pdf I have tried, among others, using keytool as in this example:

$ keytool -noprompt \ –keystore /usr/local/etc/jitsi/jicofo/truststore.jks \ -importcert -alias prosody \ -file /var/db/prosody/auth.FQDN.crt

Didn't help.

In the same PDF document linked above, it also says:

Now trust the two certificates:
$ trust anchor /var/db/prosody/FQDN.crt
$ trust anchor /var/db/prosody/auth.FQDN.crt

The problem with that is, that there's no trust command in FreeBSD -- maybe it has existed in the past, I don't know, but now days it doesn't.

Issues like this are, IMHO, one of the most annoying ones.

It's clearly something very very basic, but you just can't figure it out. It's so great to spend hours and hours on something like this.

jphovila · Aug 27, 2023

Making some progress, finally.

While many would (for a good reason) say that this is not a real solution, you can disable SSL/TLS certificate checks for Jitsi video bridge to Prosody by adding the below line (in bold) to the section apis => xmpp-client => configs => xmpp-server-1 in jitsi-videobridge.conf:

apis {
xmpp-client {
configs {
xmpp-server-1 {
disable_certificate_verification = true
}
}
}
}

In my case, the above "fix", however, resulted in a new problem:

java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

According to https://medium.com/@gustavocalcater...hors-parameter-must-be-non-empty-7dd9cb300f43

The error trustAnchors parameter must be non-empty means the Java cannot find or read the client Truststore

Tile file is readable by the Jicofo process though, so something else must be wrong.

The trust store file, by the way, should be /usr/local/etc/jitsi/jicofo/truststore.jks -- not ~~/usr/local/etc/ssl/java.pem~~ (as claimed by the Honeyguide instructions).

To find out what's going on, I added this to the jicofo_start() function in /usr/local/etc/rc.d/jicofo:

-Djavax.net.debug=all \

Now, /var/log/jicofo.log tells me that

System property jdk.tls.client.cipherSuites is set to 'null'

~~Apparently, that results in all (actually) available cipher suites being rejected. Trying to sort that out next.~~
EDIT: The cipher suite related log entries apparently are normal and do not signal a problem. There are other issues remaining though. I'm working on those now and if/when I find a setup that works, I'll try to remember to post it here.

hendrack · Aug 28, 2023

jphovila said:
Making some progress, finally.

While many would (for a good reason) say that this is not a real solution, you can disable SSL/TLS certificate checks for Jitsi video bridge to Prosody by adding the below line (in bold) to the section apis => xmpp-client => configs => xmpp-server-1 in jitsi-videobridge.conf:

In my case, the above "fix", however, resulted in a new problem:

According to https://medium.com/@gustavocalcater...hors-parameter-must-be-non-empty-7dd9cb300f43

Tile file is readable by the Jicofo process though, so something else must be wrong.

The trust store file, by the way, should be /usr/local/etc/jitsi/jicofo/truststore.jks -- not ~~/usr/local/etc/ssl/java.pem~~ (as claimed by the Honeyguide instructions).

To find out what's going on, I added this to the jicofo_start() function in /usr/local/etc/rc.d/jicofo:

Now, /var/log/jicofo.log tells me that

~~Apparently, that results in all (actually) available cipher suites being rejected. Trying to sort that out next.~~
EDIT: The cipher suite related log entries apparently are normal and do not signal a problem. There are other issues remaining though. I'm working on those now and if/when I find a setup that works, I'll try to remember to post it here.

I have a working install on Devuan, when I have time I'll compare the config files with the FreeBSD guides. I have a gut feeling its just outdated guides because after 2-3 years the syntax changed?

jphovila · Aug 29, 2023

Yes, the guides being outdated is certainly at least part of the problem.

Also, Jitsi Meet is one of those otherwise wonderful open source products out there, that are a pain in the * to actually fully understand. I've been in [open source, Linux and BSD based] hosting for two decades and consider myself very good at figuring stuff out. Still, there are some of these products that really make me pull my hair off. Many times, it has something to do with [the lack of] documentation -- or, not exactly the lack of it, but the lack of descriptions that would allow admins new to that particular system to quickly grasp precise enough understanding of how (exactly) the different components "talk" with each other, what is their exact role in the big picture, etc.

I understand it's painful to write instructions about how to (exactly) install and configure anything that's even a bit more complicated than a basic daemon. Not to mention keeping the instructions up to date. Still, it just hurts the brain to see "guides" that assume only one possible approach to the process -- which now days often means not just the LAMP stack, but even worse: Docker. Not that this would be a problem with the two "Jitsi Meet for FreeBSD" guides I found; they are of course specifically for FreeBSD. They too, though, ignore tons of details that would most definitely be worth at least mentioning.

Yes yes, I should do it myself then.

hendrack · Aug 29, 2023

jphovila said:
Yes, the guides being outdated is certainly at least part of the problem.

Also, Jitsi Meet is one of those otherwise wonderful open source products out there, that are a pain in the * to actually fully understand. I've been in [open source, Linux and BSD based] hosting for two decades and consider myself very good at figuring stuff out. Still, there are some of these products that really make me pull my hair off. Many times, it has something to do with [the lack of] documentation -- or, not exactly the lack of it, but the lack of descriptions that would allow admins new to that particular system to quickly grasp precise enough understanding of how (exactly) the different components "talk" with each other, what is their exact role in the big picture, etc.

I understand it's painful to write instructions about how to (exactly) install and configure anything that's even a bit more complicated than a basic daemon. Not to mention keeping the instructions up to date. Still, it just hurts the brain to see "guides" that assume only one possible approach to the process -- which now days often means not just the LAMP stack, but even worse: Docker. Not that this would be a problem with the two "Jitsi Meet for FreeBSD" guides I found; they are of course specifically for FreeBSD. They too, though, ignore tons of details that would most definitely be worth at least mentioning.

Yes yes, I should do it myself then.

The .deb files from jitsi have a shell installer where you input the domain name, certificate provider etc. and some scripts do their magic. This might provide some insights too.

That was a sweet siren call to just slap Devuan on an unused PC and be done with it in a matter of minutes.

jphovila · Sep 1, 2023

Hi!

hendrack said:
That was a sweet siren call to just slap Devuan on an unused PC and be done with it in a matter of minutes.

Yup -- handy, great for "normal end-users". But not really useful for anyone (like me) who is attempting to build a publicly available service on a custom, self-built (FreeBSD based) platform.

I was able to extract some useful hints from https://github.com/jitsi/docker-jitsi-meet/blob/master/jicofo/rootfs/defaults/jicofo.conf. Most importantly, looking at it confirmed me that yes: both of the "Jitsi on FreeBSD" guides available online have, among others, serious mistakes regarding changes to some of the configuration files. For example, in one of the instructions it says:

Make [note: do NOT make -- notice the context above!] these changes to jicofo.conf:

JVB_XMPP_HOST=localhost
JVB_XMPP_DOMAIN=jitsi.honeyguide.net
JVB_XMPP_PORT=5347
JVB_XMPP_SECRET=KEYUSEDINCONFIG
JVB_XMPP_USER_DOMAIN=auth.jitsi.honeyguide.net
JVB_XMPP_USER_NAME=focus
JVB_XMPP_USER_SECRET=KEYUSEDINCONFIG

MAX_MEMORY=3072m

No, no, no. You can't just put lines like that into jicofo.conf. It seems like someone has just taken variables from a Docker configuration file template (

) with the assumption, that the lines can be entered into jicofo.conf as they are. That would not work at all, instead you'll have to figure out the correct form of changes based on the "tips" given by the guides. For example, the above JVB_XMPP_whatever would translate into something like this:

xmpp {
service {
hostname=localhost
port=5347
domain=auth.jitsi.honeyguide.net
username=focus
password=KEYUSEDINCONFIG
}
}

Note: this is not a functional configuration example; it's just an example of how you need to "filter" the [misleading] installation guides.

There's an identical mistake regarding /usr/local/etc/jitsi/videobridge/jitsi-videobridge.conf in the honeyguide version. It says:

As soon as reading from the config file is fixed, our config file will look like this:

JVB_XMPP_HOST=localhost
JVB_XMPP_DOMAIN=jitsi.honeyguide.net
JVB_XMPP_PORT=5347
JVB_XMPP_SECRET=KEYUSEDINCONFIG

VIDEOBRIDGE_MAX_MEMORY=3072m
# VIDEOBRIDGE_DEBUG_OPTIONS="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,addres

No, the configuration file will not look like that. Or if it does, your daemon most certainly won't start.

There's another potentially problematic detail in the honeyguide version as well. In their version, the Jicofo rc script gets edited (which probably isn't necessary any more, it seems to me that jicofo.conf is now being read properly) which causes the [FreeBSD] default location of Jicofo trust store to be changed from /usr/local/etc/jitsi/jicofo/truststore.jks to /usr/local/etc/ssl/java.pem.

I feel I'm getting closer to making it all work now. Just have to figure out all the (gaping) holes the (outdated and misleading) guides don't cover. I'll keep this thread updated and try to come up with a summary / solution that would be easy enough to understand without walking through the whole path of discovery...

hendrack · Sep 2, 2023

jphovila said:
I feel I'm getting closer to making it all work now. Just have to figure out all the (gaping) holes the (outdated and misleading) guides don't cover. I'll keep this thread updated and try to come up with a summary / solution that would be easy enough to understand without walking through the whole path of discovery...

Here are my config files of a working server on Devuan, I replaced my domain with example.com.

/etc/jitsi/jicofo/config

Code:

# adds java system props that are passed to jicofo (default are for home and logging config file)
JAVA_SYS_PROPS="-Dconfig.file=/etc/jitsi/jicofo/jicofo.conf -Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/etc/jitsi -Dnet.java.sip.communicator.SC_HOME_DIR_NAME=jicofo -Dnet.java.sip.communicator.SC_LOG_DIR_LOCATION=/var/log/jitsi -Djava.util.logging.config.file=/etc/jitsi/jicofo/logging.properties"

/etc/jitsi/jicofo/jicofo.conf

Code:

# Jicofo HOCON configuration. See reference.conf in /usr/share/jicofo/jicofo.jar for
#available options, syntax, and default values.
jicofo {
  xmpp: {
    client: {
      client-proxy: "focus.example.com"
      xmpp-domain: "example.com"
      domain: "auth.example.com"
      username: "focus"
      password: "password1"
    }
    trusted-domains: [ "recorder.example.com" ]
  }
  bridge: {
    brewery-jid: "JvbBrewery@internal.auth.example.com"
  }
}

/etc/jitsi/videobridge/config

Code:

JAVA_SYS_PROPS="-Dconfig.file=/etc/jitsi/videobridge/jvb.conf -Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/etc/jitsi -Dnet.java.sip.communicator.SC_HOME_DIR_NAME=videobridge -Dnet.java.sip.communicator.SC_LOG_DIR_LOCATION=/var/log/jitsi -Djava.util.logging.config.file=/etc/jitsi/videobridge/logging.properties"

/etc/jitsi/videobridge/jvb.conf

Code:

videobridge {
    http-servers {
        public {
            port = 9090
        }
    }
    websockets {
        enabled = true
        domain = "example.com:443"
        tls = true
    }
}

/etc/jitsi/videobridge/sip-communicator.properties

Code:

org.ice4j.ice.harvest.DISABLE_AWS_HARVESTER=true
org.ice4j.ice.harvest.STUN_MAPPING_HARVESTER_ADDRESSES=meet-jit-si-turnrelay.jitsi.net:443
org.jitsi.videobridge.ENABLE_STATISTICS=true
org.jitsi.videobridge.STATISTICS_TRANSPORT=muc
org.jitsi.videobridge.xmpp.user.shard.HOSTNAME=localhost
org.jitsi.videobridge.xmpp.user.shard.DOMAIN=auth.example.com
org.jitsi.videobridge.xmpp.user.shard.USERNAME=jvb
org.jitsi.videobridge.xmpp.user.shard.PASSWORD=password2
org.jitsi.videobridge.xmpp.user.shard.MUC_JIDS=JvbBrewery@internal.example.com
org.jitsi.videobridge.xmpp.user.shard.MUC_NICKNAME=f4f1a323-dc07-44fd-8e9a-e5630e780a83

/etc/jitsi/meet/example.com-config.js (relevant bit at the top)

Code:

/* eslint-disable comma-dangle, no-unused-vars, no-var, prefer-template, vars-on-top */

/*
 * NOTE: If you add a new option please remember to document it here:
 * https://jitsi.github.io/handbook/docs/dev-guide/dev-guide-configuration
 */

var subdir = '<!--# echo var="subdir" default="" -->';
var subdomain = '<!--# echo var="subdomain" default="" -->';

if (subdomain) {
    subdomain = subdomain.substr(0, subdomain.length - 1).split('.')
        .join('_')
        .toLowerCase() + '.';
}

// In case of no ssi provided by the webserver, use empty strings
if (subdir.startsWith('<!--')) {
    subdir = '';
}
if (subdomain.startsWith('<!--')) {
    subdomain = '';
}

var enableJaaS = false;

var config = {
    // Connection
    //

    hosts: {
        // XMPP domain.
        domain: 'example.com',

        // When using authentication, domain for guest users.
        // anonymousdomain: 'guest.example.com',

        // Domain for authenticated users. Defaults to <domain>.
        // authdomain: 'example.com',

        // Focus component domain. Defaults to focus.<domain>.
        // focus: 'focus.example.com',

        // XMPP MUC domain. FIXME: use XEP-0030 to discover it.
        muc: 'conference.' + subdomain + 'example.com',
    },

    // BOSH URL. FIXME: use XEP-0156 to discover it.
    bosh: 'https://example.com/' + subdir + 'http-bind',

jphovila · Sep 5, 2023

Just a brief update.

I have not yet figured out a solution to the Truststore issue, how ever many of the other mysteries are finally becoming more clear.

Since this journey has taken me to numerous different forums and open-ended threads, I'm trying to pull it all together and share my notes when I finally have a fully functional setup.

A quick and dirty (yet functional) temporary fix for certificate trust issues between [ Jicofo | Jitsi-Videobridge ] and Prosody is to add disable_certificate_verification = true into the appropriate XMPP client configuration sections of jicofo and/or jitsi-videobridge.

firemount · Sep 23, 2023

I've been fiddeling around trying to install jitsi on "FreeBSD 13.2-RELEASE-p2" and ran into the same problem with truststore and jicofo.conf being, well not according to what jicofo expects nowadays (maybe not even before). The excellent howtos seem to refer to some kind of hack to start jicofo by setting environment variables for /usr/local/etc/rc.d/jicofo, i.e the rc-script.

I have been looking around in old code and found that apparently jicofo java program seemed to ignore jicofo.conf in the ports version earlier and thus the hack being using jicofo.conf to set a number of variables for the rc-script

That does not seem to be the case anylonger.

Checking the java code out at github specifically these 2 links:
https://github.com/jitsi/jicofo/blob/master/jicofo/src/test/kotlin/org/jitsi/jicofo/xmpp/XmppConfigTest.kt and
https://github.com/jitsi/jicofo/blob/master/jicofo-selector/src/main/resources/reference.conf ,
reveals that there are two config formats "Legacy config" and "New Config" the latter being the one preferred and these values seems to take precedence over Legacy values.

The new format used is HOCON (Human-Optimized Config Object Notation) and by trying out a vanilla install on Ubuntu 22.04 it gave me this config which I placed in /usr/local/etc/jitsi/jicofo

cat /usr/local/etc/jitsi/jicofo/jicofo.conf
(I did s/mydomain/example.com/g)

Code:

# Jicofo HOCON configuration. See reference.conf in /usr/share/jicofo/jicofo.jar for
#available options, syntax, and default values.
jicofo {
  xmpp: {
    client: {
      client-proxy: "focus.meet.example.com"
      xmpp-domain: "meet.example.com"
      domain: "auth.meet.example.com"
      username: "focus"
      password: "supersecret"
    }
    trusted-domains: [ "recorder.meet.example.com" ]
  }
  bridge: {
    brewery-jid: "JvbBrewery@internal.auth.example.com"
  }
}

To check out prosody i did

  portsnap fetch

portsnap extract

cd /usr/ports/net-im/jitsi-prosody-plugin

make

find . -name prosody.\*lua

cmp ./work/jitsi-meet-stable-jitsi-meet_8319/prosody.cfg.lua ./work/stage/usr/local/share/examples/jitsi/prosody.cfg.lua

And checked out ./work/jitsi-meet-stable-jitsi-meet_8319/prosody.cfg.lua which says:
(I did s/mydomain/example.com/g)

Code:

-- Place this file in /usr/local/etc/prosody/prosody.cfg.lua
--
-- Then run
--
--    prosodyctl cert generate jitsi.example.com
--    prosodyctl cert generate auth.jitsi.example.com
--    keytool -noprompt -keystore /usr/local/etc/jitsi/jicofo/truststore.jks -importcert \
--      -alias prosody -file /var/db/prosody/auth.jitsi.example.com.crt
--
--    prosodyctl register jvb auth.jitsi.example.com "supersecret"
--    prosodyctl register focus auth.jitsi.example.com "supersecret"
--    prosodyctl mod_roster_command subscribe focus.jitsi.example.com focus@auth.jitsi.example.com "supersecret"

which was kind of handy

, so what I did to get rid of the trust Anchors parameter must be non-empty, PKIX path building failed was:

Code:

sed -e 's/jitsi.example.com/meet.example.com/g' -e 's/prosody.err/prosody\/prosody.err/' -e 's/prosody.log/prosody\/prosody.log/' ./work/jitsi-meet-stable-jitsi-meet_8319/prosody.cfg.lua > /usr/local/etc/prosody/prosody.cfg.lua

keytool -noprompt -keystore /usr/local/etc/jitsi/jicofo/truststore.jks -importcert -alias prosody -file /var/db/prosody/meet.example.com.cer -storepass supersecret
prosodyctl register jvb auth.meet.example.com "supersecret"
prosodyctl register focus auth.meet.example.com "supersecret"
prosodyctl mod_roster_command subscribe focus.meet.example.com focus@auth.meet.example.com "supersecret"

Right now I have prosody and jicofo running where jicofo authenticates to prosody using Lets-Encrypt-certs. Lots of other errors since I haven't bothered to install anything else yet.
Will keep digging. Hope this helps....

---------------------------------------------------------------------
/var/log/jicofo.log (I did s/mydomain/example.com/g)

Code:

Jicofo 2023-09-23 20:27:13.329 INFO: [1] Main.main#47: Starting Jicofo.
Jicofo 2023-09-23 20:27:13.522 INFO: [1] JitsiConfig.<clinit>#47: Initialized newConfig: merge of /usr/local/etc/jitsi/jicofo/jicofo.conf: 1,system properties,reference.conf @ jar:file:/usr/local/share/java/classes/jicofo.jar!/reference.conf: 1
Jicofo 2023-09-23 20:27:13.526 INFO: [1] ReadOnlyConfigurationService.reloadConfiguration#51: loading config file at path /usr/local/etc/jitsi/jicofo/sip-communicator.properties
Jicofo 2023-09-23 20:27:13.528 INFO: [1] ReadOnlyConfigurationService.reloadConfiguration#56: Error loading config file: java.io.FileNotFoundException: /usr/local/etc/jitsi/jicofo/sip-communicator.properties (No such file or directory)
Jicofo 2023-09-23 20:27:13.529 INFO: [1] JitsiConfig.<clinit>#68: Initialized legacyConfig: sip communicator props (no description provided)
Jicofo 2023-09-23 20:27:13.536 INFO: [1] JitsiConfig$Companion.reloadNewConfig#94: Reloading the Typesafe config source (previously reloaded 0 times).
Jicofo 2023-09-23 20:27:14.146 INFO: [1] JicofoServices.createAuthenticationAuthority#195: Authentication service disabled.
Jicofo 2023-09-23 20:27:14.312 INFO: [1] XmppServices.<init>#44: No dedicated Service XMPP connection configured, re-using the client XMPP connection.
Jicofo 2023-09-23 20:27:14.324 INFO: [1] XmppServices.<init>#71: No Jigasi detector configured.
Jicofo 2023-09-23 20:27:14.352 INFO: [1] BridgeSelector.<init>#57: Using org.jitsi.jicofo.bridge.SingleBridgeSelectionStrategy
Jicofo 2023-09-23 20:27:14.358 INFO: [16] [xmpp_connection=client] XmppProvider.doConnect#166: Connected, JID=null
Jicofo 2023-09-23 20:27:14.366 INFO: [1] [type=bridge brewery=jvbbrewery] BaseBrewery.<init>#101: Initialized with JID=jvbbrewery@internal.auth.meet.example.com
Jicofo 2023-09-23 20:27:14.369 INFO: [1] JicofoServices.<init>#109: No Jibri detector configured.
Jicofo 2023-09-23 20:27:14.369 INFO: [1] JicofoServices.<init>#115: No SIP Jibri detector configured.
Jicofo 2023-09-23 20:27:14.422 INFO: [1] JicofoServices.<init>#134: Starting HTTP server with config: host=null, port=8888, tlsPort=8843, isTls=false, keyStorePath=null, sendServerVersion=true, [TLSv1.2, TLSv1.3]=[TLSv1.2, TLSv1.3], tlsCipherSuites=[TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256].
Jicofo 2023-09-23 20:27:14.479 INFO: [16] [xmpp_connection=client] XmppProvider.fireRegistrationStateChanged#216: Set replyTimeout=PT15S
Jicofo 2023-09-23 20:27:14.479 INFO: [16] [xmpp_connection=client] XmppProvider$connectionListener$1.authenticated#97: Registered.
Jicofo 2023-09-23 20:27:14.480 INFO: [16] [xmpp_connection=client] XmppProvider$connectionListener$1.authenticated#99: Will discover components for meet.example.com
Jicofo 2023-09-23 20:27:14.496 SEVERE: [23] org.jivesoftware.smack.roster.Roster$5.processException: Exception reloading roster
org.jivesoftware.smack.XMPPException$XMPPErrorException: XMPPError: service-unavailable - cancel
    at org.jivesoftware.smack.XMPPException$XMPPErrorException.ifHasErrorThenThrow(XMPPException.java:171)
    at org.jivesoftware.smack.XMPPException$XMPPErrorException.ifHasErrorThenThrow(XMPPException.java:165)
    at org.jivesoftware.smack.AbstractXMPPConnection$6.processStanza(AbstractXMPPConnection.java:1968)
    at org.jivesoftware.smack.AbstractXMPPConnection$4.run(AbstractXMPPConnection.java:1601)
    at org.jivesoftware.smack.AbstractXMPPConnection$10.run(AbstractXMPPConnection.java:2146)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Jicofo 2023-09-23 20:27:14.505 WARNING: [24] org.jivesoftware.smack.roster.Roster$PresencePacketListener.processStanza: Roster not loaded while processing Presence Stanza [from=focus@auth.meet.example.com/focus,id=ZKJS8-1,type=available,prio=0,]
Jicofo 2023-09-23 20:27:14.522 WARNING: [21] [xmpp_connection=client] XmppProvider.setComponents#76: Discovered components: [Component(type=speakerstats, address=speakerstats.meet.example.com), Component(type=lobbyrooms, address=lobby.meet.example.com), Component(type=av_moderation, address=avmoderation.meet.example.com), Component(type=conference_duration, address=conferenceduration.meet.example.com), Component(type=breakout_rooms, address=breakout.meet.example.com)]
Jicofo 2023-09-23 20:27:14.524 INFO: [25] AvModerationHandler.componentsChanged#106: Using av_moderation component at avmoderation.meet.example.com.
Jicofo 2023-09-23 20:27:14.530 INFO: [26] ConferenceIqHandler.componentsChanged#189: Using breakout room component at breakout.meet.example.com.
Jicofo 2023-09-23 20:27:14.532 WARNING: [24] org.jivesoftware.smack.roster.Roster$PresencePacketListener.processStanza: Roster not loaded while processing Presence Stanza [to=focus@auth.meet.example.com/focus,from=jvbbrewery@internal.auth.meet.example.com/focus,id=ZKJS8-2,type=available,prio=0,]
Jicofo 2023-09-23 20:27:14.595 INFO: [1] org.eclipse.jetty.server.Server.doStart: jetty-11.0.10; built: 2022-06-16T20:42:17.891Z; git: d21dded5817960ec3c753a7ba02ef86f7c9ed89e; jvm 11.0.20+8-1
Jicofo 2023-09-23 20:27:14.610 INFO: [22] [type=bridge brewery=jvbbrewery] BaseBrewery.start#177: Joined the room.
Jicofo 2023-09-23 20:27:15.403 WARNING: [1] org.glassfish.jersey.server.wadl.WadlFeature.configure: JAXBContext implementation could not be found. WADL feature is disabled.
Jicofo 2023-09-23 20:27:15.499 WARNING: [1] org.glassfish.jersey.internal.inject.Providers.checkProviderRuntime: A provider org.jitsi.rest.Version registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.jitsi.rest.Version will be ignored.
Jicofo 2023-09-23 20:27:15.500 WARNING: [1] org.glassfish.jersey.internal.inject.Providers.checkProviderRuntime: A provider org.jitsi.rest.prometheus.Prometheus registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.jitsi.rest.prometheus.Prometheus will be ignored.
Jicofo 2023-09-23 20:27:15.500 WARNING: [1] org.glassfish.jersey.internal.inject.Providers.checkProviderRuntime: A provider org.jitsi.jicofo.rest.ConferenceRequest registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.jitsi.jicofo.rest.ConferenceRequest will be ignored.
Jicofo 2023-09-23 20:27:15.603 INFO: [1] org.eclipse.jetty.server.handler.ContextHandler.doStart: Started o.e.j.s.ServletContextHandler@56c8e6f0{/,null,AVAILABLE}
Jicofo 2023-09-23 20:27:15.613 INFO: [1] org.eclipse.jetty.server.AbstractConnector.doStart: Started ServerConnector@6c806c8b{HTTP/1.1, (http/1.1)}{0.0.0.0:8888}
Jicofo 2023-09-23 20:27:15.624 INFO: [1] org.eclipse.jetty.server.Server.doStart: Started Server@37b57b54{STARTING}[11.0.10,sto=0] @2626ms

and in /var/log/prosody/prosody.log (I did s/mydomain/example.com/g)

Code:

Sep 23 20:26:49 meet newsyslog[76588]: logfile first created
Sep 23 20:27:02 startup info    Hello and welcome to Prosody version 0.12.3
Sep 23 20:27:02 startup info    Prosody is using the epoll backend for connection handling
Sep 23 20:27:02 portmanager     info    Activated service 's2s' on [::]:5269, [*]:5269
Sep 23 20:27:02 portmanager     info    Activated service 's2s_direct_tls' on no ports
Sep 23 20:27:02 conferenceduration.meet.example.com:conference_duration_component  info    Starting conference duration timer for conference.meet.example.com
Sep 23 20:27:02 conferenceduration.meet.example.com:conference_duration_component  info    No muc component found, will listen for it: conference.meet.example.com
Sep 23 20:27:02 speakerstats.meet.example.com:speakerstats_component       info   Starting speakerstats for conference.meet.example.com
Sep 23 20:27:02 speakerstats.meet.example.com:speakerstats_component       info    Conference component loaded conference.meet.example.com
Sep 23 20:27:02 speakerstats.meet.example.com:speakerstats_component       info    Hook to muc events on table: 0x83e245500
Sep 23 20:27:02 speakerstats.meet.example.com:speakerstats_component       info    Main muc service table: 0x83e246040
Sep 23 20:27:02 conference.meet.example.com:muc_domain_mapper      info   Loading mod_muc_domain_mapper for host focus.meet.example.com!
Sep 23 20:27:02 conference.meet.example.com:muc_domain_mapper      info   Loading mod_muc_domain_mapper for host conference.meet.example.com!
Sep 23 20:27:02 conference.meet.example.com:muc_domain_mapper      info   Loading mod_muc_domain_mapper for host conferenceduration.meet.example.com!
Sep 23 20:27:02 conference.meet.example.com:muc_domain_mapper      info    Loading mod_muc_domain_mapper for host speakerstats.meet.example.com!
Sep 23 20:27:02 conference.meet.example.com:muc_domain_mapper      info   Loading mod_muc_domain_mapper for host internal.auth.meet.example.com!
Sep 23 20:27:02 conferenceduration.meet.example.com:conference_duration_component  info    Hook to muc events on conference.meet.example.com
Sep 23 20:27:02 conference.meet.example.com:muc_domain_mapper      info   Loading mod_muc_domain_mapper for host lobby.meet.example.com!
Sep 23 20:27:02 portmanager     info    Activated service 'http' on [::1]:5280, [127.0.0.1]:5280
Sep 23 20:27:02 portmanager     info    Activated service 'https' on no ports
Sep 23 20:27:02 meet.example.com:http      info    Serving 'bosh' at http://meet.example.com:5280/http-bind
Sep 23 20:27:02 portmanager     info    Activated service 'c2s' on [::]:5222, [*]:5222
Sep 23 20:27:02 portmanager     info    Activated service 'c2s_direct_tls' on no ports
Sep 23 20:27:02 portmanager     info    Activated service 'legacy_ssl' on no ports
Sep 23 20:27:02 meet.example.com:external_services warn    No services configured or all had errors
Sep 23 20:27:02 meet.example.com:muc_lobby_rooms   info    Lobby component loaded lobby.meet.example.com
Sep 23 20:27:02 meet.example.com:muc_breakout_rooms        info    Hook to muc events on conference.meet.example.com
Sep 23 20:27:02 conference.meet.example.com:muc_domain_mapper      info   Loading mod_muc_domain_mapper for host meet.example.com!
Sep 23 20:27:02 conference.meet.example.com:muc_domain_mapper      info   Loading mod_muc_domain_mapper for host avmoderation.meet.example.com!
Sep 23 20:27:02 avmoderation.meet.example.com:av_moderation_component     info    Starting av_moderation for conference.meet.example.com
Sep 23 20:27:02 avmoderation.meet.example.com:av_moderation_component     info    Hook to muc events on conference.meet.example.com
Sep 23 20:27:02 conference.meet.example.com:muc_domain_mapper      info   Loading mod_muc_domain_mapper for host auth.meet.example.com!
Sep 23 20:27:02 meet.example.com:muc_breakout_rooms        info    Breakout rooms component created breakout.meet.example.com
Sep 23 20:27:02 speakerstats.meet.example.com:speakerstats_component       info    Breakout component loaded breakout.meet.example.com
Sep 23 20:27:02 conference.meet.example.com:muc_domain_mapper      info   Loading mod_muc_domain_mapper for host breakout.meet.example.com!
Sep 23 20:27:02 meet.example.com:muc_breakout_rooms        info    Hook to muc events on breakout.meet.example.com
Sep 23 20:27:02 speakerstats.meet.example.com:speakerstats_component       info    Hook to muc events on table: 0x83e3a1ac0
Sep 23 20:27:02 breakout.meet.example.com:muc_domain_mapper        info   Loading mod_muc_domain_mapper for host focus.meet.example.com!
Sep 23 20:27:02 breakout.meet.example.com:muc_domain_mapper        info   Loading mod_muc_domain_mapper for host conferenceduration.meet.example.com!
Sep 23 20:27:02 breakout.meet.example.com:muc_domain_mapper        info    Loading mod_muc_domain_mapper for host speakerstats.meet.example.com!
Sep 23 20:27:02 breakout.meet.example.com:muc_domain_mapper        info   Loading mod_muc_domain_mapper for host internal.auth.meet.example.com!
Sep 23 20:27:02 breakout.meet.example.com:muc_domain_mapper        info   Loading mod_muc_domain_mapper for host conference.meet.example.com!
Sep 23 20:27:02 breakout.meet.example.com:muc_domain_mapper        info   Loading mod_muc_domain_mapper for host lobby.meet.example.com!
Sep 23 20:27:02 breakout.meet.example.com:muc_domain_mapper        info   Loading mod_muc_domain_mapper for host meet.example.com!
Sep 23 20:27:02 breakout.meet.example.com:muc_domain_mapper        info   Loading mod_muc_domain_mapper for host breakout.meet.example.com!
Sep 23 20:27:02 breakout.meet.example.com:muc_domain_mapper        info   Loading mod_muc_domain_mapper for host auth.meet.example.com!
Sep 23 20:27:02 breakout.meet.example.com:muc_domain_mapper        info   Loading mod_muc_domain_mapper for host avmoderation.meet.example.com!
Sep 23 20:27:02 mod_posix       info    Prosody is about to detach from the console, disabling further console output
Sep 23 20:27:02 mod_posix       info    Successfully daemonized to PID 718
Sep 23 20:27:14 c2s823e08a80    info    Client connected
Sep 23 20:27:14 c2s823e08a80    info    Authenticated as focus@auth.meet.example.com
Sep 23 20:27:14 auth.meet.example.com:limits_exception     info    Setting stanza size limits for focus@auth.meet.example.com to 10485760

rde42 · Dec 26, 2023

jphovila said:
Making some progress, finally.

While many would (for a good reason) say that this is not a real solution, you can disable SSL/TLS certificate checks for Jitsi video bridge to Prosody by adding the below line (in bold) to the section apis => xmpp-client => configs => xmpp-server-1 in jitsi-videobridge.conf:

In my case, the above "fix", however, resulted in a new problem:

According to https://medium.com/@gustavocalcater...hors-parameter-must-be-non-empty-7dd9cb300f43

Tile file is readable by the Jicofo process though, so something else must be wrong.

The trust store file, by the way, should be /usr/local/etc/jitsi/jicofo/truststore.jks -- not ~~/usr/local/etc/ssl/java.pem~~ (as claimed by the Honeyguide instructions).

To find out what's going on, I added this to the jicofo_start() function in /usr/local/etc/rc.d/jicofo:

Now, /var/log/jicofo.log tells me that

~~Apparently, that results in all (actually) available cipher suites being rejected. Trying to sort that out next.~~
EDIT: The cipher suite related log entries apparently are normal and do not signal a problem. There are other issues remaining though. I'm working on those now and if/when I find a setup that works, I'll try to remember to post it here.

When it says the trustAnchors parameter should be non-empty, it is misleading us. It's talking about the file, not the parameter.
Worse than that, the file isn't empty, it just has useless contents because it's in the wrong format! If you look, it's a PKS12 file, not a JKS file.
When creating the trust store, add -storetype jks to the call of keytool.