XMPP: basics, security, constrained networks

sidetone

Daemon

Reaction score: 592
Messages: 1,392

  • Core features of XMPP are defined by RFC's, which can be found at IETF.org
  • Extended features of XMPP are defined by XEP's, which can be found at XMPP.org/extensions
  • XMPP is an open standard, rather than open software (it includes plenty of open software as well)
  • The XMPP standard is approved by the IETF (Internet Engineering Task Force)
  • Aside from chatting, XMPP can be used for automation (via instructions), feeds and micro-blogging
  • 50 million XMPP users in 2009
  • Zoom, Kontalk and Jitsi messaging services are based on XMPP; combined, these consist of over 200 million users
  • There are over 1 billion users on XMPP based proprietary networks
  • Google and DuckDuckGo used to offer XMPP based services
To find XMPP/Jabber related ports: psearch -c net-im -so xmpp jabber. The s option allows a deep search into the long description; the o option is for or. As of now, there are a minimal amount of XMPP/Jabber related ports is in the net category.

XMPP user accounts
A Jabber Identifier (JID) consists of the username, and domain name of an XMPP user account. This can also include the extended resource part, which is an id for each device or connection.

XMPP server directory and test results by security grade:
Servers not listed yet or that haven't been checked recently can be tested on this website. Server to server (s2s), and client to server (c2s) connections can be checked. This security grade is by technical measures, so also, go by server's reputation.

Stream & Connection
An XMPP communication session is a stream over a TCP connection that lasts the length of a conversation. There are 2 streams established, 1 for each direction. The server and client each create their own XML XMPP document that is added to as a conversation progresses. A server doesn't poll the client for new messages or updates. Rather than constantly waiting for messages, the server sends or holds a message for sending as soon as it's received.

The route an XMPP communication takes is a direct route from client to server, to server, to client: this is called direct federation. (email uses indirect federation, which many indirect routes are taken between servers.)

In the traditional mode of XMPP communication, messages can get lost and aren't resent.

The resource part of a full JID can be used for further routing communication. Typically, only one device receives a message, which can be set by priority. It can be configured for many devices to receive a message.

The clients on their end close their XML/XMPP documents when a disruption in the stream is sensed.


XMPP components
When connection is started, XML begins as normal, with: <?xml version="1.0"?>

Under this is the <stream:stream> element, which doesn't close until the end of the connection. Then, after the opening <stream:stream> element, there's the <stream:features> self-closing element, which lists the features available for use in XMPP.

Three primary XML elements exchanged between server and client are called stanzas: MESSAGE, IQ and PRESENCE.

message stanza
Attribute type for message: error, normal, chat, groupchat, headline. Other attributes for message: to, from, id. Directs where message is sent in groupchat.

iq stanza
Is used for queries. In groupchat, can be used to change user permissions, or kick someone out of room. Can be used to reserve chatroom name.

presence stanza
Indicates users' availability. Extensions can indicate when user is typing, and another presence stanza is sent when there's a pause in typing. Join notification is indicated by this stanza. Tells when client leaves, unless the server to server connection goes down unexpectedly. Can indicate client's role (temporary status) or affiliation (persists across logins) in chatroom: visitor, participant, member, moderator, admin, owner.

File transfer
Bits of Binary (XEP-0231)/In-Band Bytestreams (IBB: XEP-0047) can be used to transfer small files over XMPP. For larger files, it is inefficient, and can cause congestion. It encodes the file into base 64, then breaks it down, to the a maximum size allowed for each message to be transferred. It is sent by message, with an indication of sequence. The IQ stanza then returns a receipt, that the messages were delivered.

Out-of-Band (OOB), Socks5 Bytestream (XEP-0065) is used to transfer messages bypassing the server, or directly from client to client. XMPP negotiates the transfer over the SOCKS5 protocol. The bytestream proxy can be on a separate machine than the XMPP server.

Jingle (XEP-0234) can be used for file transfer, as a more modern alternative to XEP-0065.

XEP-0087, XEP-0095 and XEP-0096 for transferring files are obsolete or retracted.

Server configuration (Prosody) and telephony messaging (Asterisk) are topics outside the scope of this thread.

XMPP extensions
For more features, see post #11 and other posts below. For a previous discussion about extensions, see: Thread xmpp-extensions.77183.


References (by order of use)
  • XMPP: Definitive Guide - Peter Saint-Andre, Kevin Smith, Remko Troncon. O'Reilly (2009)
  • xmpp.org
  • ietf.org
  • other XMPP related threads on this forum
  • Practical XMPP - Lloyd Watkins, David Koelle. Packt (2016)
 
Last edited:
OP
sidetone

sidetone

Daemon

Reaction score: 592
Messages: 1,392

The direct federation model has advantages over client to client negotiations, and the indirect federation model, for both security and for improved network reliability.


Security
The XMPP server coordinates encryption, spoof prevention and other security features. Security features can't be ignored from the client end. XMPP servers generally have encryption features available.

Because servers connect directly to each other, spoofing is prevented from originating between these two servers. Address spoofing originating from the client is also prevented, because the servers add this relevant information to the from attribute.

The ability for encryption comes with the core of XMPP (RFC 5246 and RFC 4422).

There is a certificate authority (CA) that is available for free, for XMPP servers: xmpp.org/ca. This is important to avoid the use of self-signed certificates, which can be a vulnerability for network spoofing. According to https://blog.prosody.im/mandatory-encryption-on-xmpp-starts-today/, there is intent within the XMPP community to replace CA's (Certificate Authorities) with models in the spirit of XMPP. DNSSEC, http://web.monkeysphere.info/, and fingerprints were mentioned.

Use TLS encryption plus SASL authentication.

TLS (there's no TLS subset of XMPP) is used to secure connections. The connection starts off unencrypted until TLS negotiates and encryption is set.

Don't use PLAIN text for sending authentication/password data, unless the connection is already encrypted with TLS. See about other SASL authentication methods. Password is sent over base64. Once authenticated, a new XML session is started, with <stream:features> set with the enabled security features. Anonymous is an option for allowing clients to use an XMPP server without a password, for instance, when a service will be used very few times.

As for connecting, avoid server dialback, as this can be spoofed. DNS validation needs DNSSEC for better security.

While messages to the servers are usually encrypted, the servers can still have access to this as unencrypted data for viewing and logging. As of 2009, there wasn't end to end encryption between clients.

The more recent Omemo (XEP-0384) offers end to end encryption of text. It is in experimental state. Jingle Encrypted Transports (JET, XEP-0391) and the additional Jingle Encrypted Transports - OMEMO (XEP-0396) offer file encryption.

Small filesize images can probably be transferred securely by in-band bytestream methods through Omemo encrypted chat.

If a XMPP connection manager (CM) is used (this relates to BOSH), that is another travel point that must be secured, unless it is on the same computer as the XMPP server.

Watch out for Unicode text that substitutes letters of recognized JID's, as an attempt to deceive XMPP users.

See XEP-0205 for best practices to discourage DDOS attacks. Also of concern is the reputation of the XMPP servers.

Serverless connections negotiated through XEP-0174 aren't secure. This document refers to using TLS and SASL through RFC 7622 to secure connections. XEP-0174 makes use of Multicast DNS (mDNS) & DNS Service Discovery (DNS-SD). For clients, this extension uses the .local tld (top level domain).


Firewall
For basic connection port numbers for XMPP, and firewall examples in PF, see: Thread firewall-settings-for-xmpp.61219.


Constrained/Intermittent Networks
Because XMPP uses a direct federation model, chose a suitable server relatively close to your region. Server to server connections, are often much more reliable. A shorter distance from a client on a constrained network to a server, would be much better, than a client having to send messages a longer distance to another client on its own.

The ability for data compression comes with the core of XMPP (rfc 3749). Streaming compression comes with the extensions XEP-0138 for zlib, or XEP-0229 for lzw. These go under the <stream:features> element. With these compressions enabled, XML data surrounding messages and presence is expected to lower bandwidth, for each transport of a stanza at a time.

It is better that the server handles most of the features, so that the client won't have to do as much heavy lifting.

Because XMPP uses streams that require a constant connection, it isn't suitable, as is, for mobile phones, intermittent Internet and other constrained networks. An intermittent Internet service causes the network to have to be reestablished, using up plenty of bandwidth. BOSH (Bidirectional-streams Over Synchronous HTTP) allows a connection manager (CM) to sit between the XMPP server and client, to take care of the XML end, and send (primarily stanza) data to the device, without it requiring a constant connection. BOSH is handled by XEP-0124, XEP-0156 and XEP-0206.

In many ways, WebSockets (RFC 7395) is more advanced than BOSH, but it is very inefficient (and not meant) for clients with intermittent connection or for clients on constrained networks.

XEP-0352: Client State Indication is another extension that saves bandwidth for mobile phones. It reduces the need for establishing XMPP connections and resending data required for interrupted XMPP streams. This extension is available on some mobile clients like Conversations. Client State Indication is in the proposed state.

Presence settings can typically use around 90% of bandwidth data. For constrained networks, use minimal presence settings, or turn this off depending on needs.

For transferring files, In-Band Bytestreams (XEP-0047), a kind of Bits of Binary, can be a source of congestion on a constrained network.
 
Last edited:

mjollnir

Daemon

Reaction score: 835
Messages: 1,267

  • FTR, most will already know this: XMPP is also known as Jabber. These are two names for the same protocol.
  • The (non-) availability/support of extensions of a particular client and/or server can be a source of failure.
  • You need to have a reliable password manager, since many XMPP/Jabber servers allow anonymous registration; i.e. when you forget your password and/or account name, you're lost... Depending on the configuration of the server, your account will be blocked after repeatedly failed connection attempts (for an hour or so). I.e. you can not connect during that period, even if you remembered the correct password.
  • Naturally, some extensions (e.g. STUN) or environments (firewall) require additional configuration that can be non-trivial & hairy for newbies.
  • Setting up a server can be non-trivial.
  • Many server software allow to set up bridges/gateways to other instant messaging systems.
  • XMPP/Jabber can be considered a well-established standard, it is used e.g. in military and governmental environments, and some proprietary IM systems use XMPP under the hood; but there seems to be a shift towards the newer Matrix protocol (due to the mentionend issues?).
 

vigole

Aspiring Daemon

Reaction score: 709
Messages: 660

there seems to be a shift towards the newer Matrix protocol
For whatever reason, everybody want to avoid XMPP. They present different rationale, but I think it's all about XML. I don't know why and certainly it's not my position, but most people don't like XML.
 

ekvz

Well-Known Member

Reaction score: 271
Messages: 431

  • FTR, most will already know this: XMPP is also known as Jabber. These are two names for the same protocol.

Yes, Jabber is pretty much the oldschool name for XMPP.

there seems to be a shift towards the newer Matrix protocol (due to the mentionend issues?).

While i am not very deep into this i think Matrix might profit a lot from the being new and the general hype around it. In any case i don't get the impression that it's technically less complicated than XMPP (rather the opposite). Sure it might be better at hiding this from users and given it's new it does not have to rely on a ton of extensions but i really don't feel comfortable with it's current state.

There seem to even be closed source components to it which wouldn't be much of a problem as the standard is actually open but from what i read those closed parts don't actually fully follow the published standard thereby complicating the development of open alternatives and resulting in what pretty much seems to be a semi centralized system right now. It's overall kinda messy and in my opinion privacy isn't taken very serious by the Matrix developers at all.

There is a bit of discussion here https://github.com/privacytools/services/issues/17 and there is more if one pokes around.
 

mjollnir

Daemon

Reaction score: 835
Messages: 1,267

IMHO it's not about XML, but the interoperability between different clients. E.g. I have 3 different authentication methods to verify an OTR encrypted channel, and the other side does not neccessarily support them. It's non-trivial, for newbies it's too complicated.
 

ekvz

Well-Known Member

Reaction score: 271
Messages: 431

IMHO it's not about XML, but the interoperability between different clients. E.g. I have 3 different authentication methods to verify an OTR encrypted channel, and the other side does not neccessarily support them. It's non-trivial, for newbies it's too complicated.

I agree. XML is not that bad but XMPP (the protocol) is simply braindamagingly complicated and convoluted with all it's extensions leading in part to said incompatibilities. As far as the choice of XML is concerned it's more about cleartext being a bad fit to transfer binary data than XML itself. XMPP is sadly pretty much stuck in a couple of ways and i think that's sad as i am watching recent developments like Matrix with a lot of scepticism.
 

vigole

Aspiring Daemon

Reaction score: 709
Messages: 660

Signaling is a problem: session, metadata, NAT, etc. WebRTC doesn't implement Signaling either. That's why there's ICE, STUN, etc.
OT: Signalling or Signaling? which one is correct!
 

mjollnir

Daemon

Reaction score: 835
Messages: 1,267

Signaling is a problem: session, metadata, NAT, etc. WebRTC doesn't implement Signaling either. That's why there's ICE, STUN, etc.
OT: Signalling or Signaling? which one is correct!
Now this should really go into a group chat... ;) I'd say signaling is correct.
 

vigole

Aspiring Daemon

Reaction score: 709
Messages: 660

I've recently read a lot of posts, on the topic of Jabber/XMPP, for the most part posted by [USER=45325]sidetone[/USER] and [USER=37644]mjollnir[/USER]
I've never had a Jabber ID. Now I have one! and I'm going to use it. I think it's a interesting protocol. I hope it gains some popularity.
Thank you: [USER=45325]sidetone[/USER] and [USER=37644]mjollnir[/USER]
 
OP
sidetone

sidetone

Daemon

Reaction score: 592
Messages: 1,392

Additional uses of XMPP
These generally require extensions.

Remote commands
Can be used to forward and manage XMPP messages, and to control client devices. An additional use is for automated devices (automation data). IQ stanza is to query capabilities. Data can also be sent to SOAP and RPC protocols. See: XEP-0050, XEP-0133, XEP-0146, and XEP-0244.

For IOT, see: XEP-0323, XEP-0324, XEP-0325, XEP-0326 and XEP-0347.

Pubsub
Publish/subscribe can be used for micro-blogging, and subscribing to other updates. Each subscription is a node on a Jabber ID (JID).

Data broadcasting
This is for use in Multi-user chat (MUC). MUC isn't only for chatrooms. MUC simply multiplies a message for many receivers. Data broadcasting (XEP-0045) is for relaying information to multiple devices.

XEP-0166 and XEP-0167, can be used for signaling for group sessions.

Mediated Information eXchange (MIX), XEP-0369, is intended to be a replacement for MUC, for its perceived shortcomings. This extension is in the experimental state.

Jingle
For Jingle, use ICE signaling, which is designed for optimal use with NAT. XMPP negotiates a client to client Jingle session. Previously mentioned is using Jingle for transferring files, such as images. Additional uses for Jingle is a topic outside the scope of this thread.

Other
Some of these features can be used with above capabilities.
  • vcard extension - For showing JID profiles. Can be queried from server, unless the information is wanted from a specific device on the base JID.
  • chat state notification - Turning this off, saves bandwidth.
  • privacy lists and blocking - Shows as if user hasn't logged on. Messages can get dropped by server.
  • receipts - Can be sent to sender of message, to know if important message was received.
  • message archiving
  • advanced message processing
  • discovery protocol - Find services on servers. disco#items discovers entities, and disco#info discovers features on these entities.
  • data forms
  • captcha forms
  • geolocation (XEP-0080)
  • atom feeds (RFC 4287) - Since Atom is XML, it can be used directly by XMPP.
  • jukebox/radio (XEP-0118)
  • application and gaming data
  • XEP-0333: Chat Markers - sends an indication to the sender when a message has been read.
 

ekvz

Well-Known Member

Reaction score: 271
Messages: 431

I think it's a interesting protocol.

In any case it's pretty much the most trustworthy and functional protocol there is right now in my opinion. Sadly that only means so much. There is quite some ugly stuff about it also (often more or less by design so it's unlikely to be fixed).

I hope it gains some popularity.

I hate to be pessimistic but i seriously doubt it. It's simply to much of a mess technically to really move forward.
 
OP
sidetone

sidetone

Daemon

Reaction score: 592
Messages: 1,392

BOSH
BOSH (Bidirectional-streams Over Synchronous HTTP, XEP-0124) uses tcp port 5280. https may be a fall-back port for BOSH.

XMPP clients with BOSH support
  • JWChat - jwchat.org uses JWChat webclient
  • Tigase messenger
  • Pidgin can be configured to use BOSH.
  • SparkWeb (uses Flash)
Webclients either generally or mostly use BOSH or WebSocket.

Other mentions of BOSH support
  • According to wiki.xmpp.org, BOSH support was being added to Gajim in 2008. However, it seems that it's not currently in Gajim. Gajim currently uses WebSocket.
  • Kiawa is a webchat application that likely uses BOSH. This can also be used at jwchat.org
  • Gloox is a C++ XMPP library that has BOSH support.
  • XIFF is a BOSH library for connection managers
  • net-im/py-punjab is an XMPP connection manager that uses BOSH. This would still require TLS encryption that is not based on a self-signed certificate, and would still require SASL authentication.

Further reading on XMPP over BOSH

Possible Alternatives or Complements to BOSH
As mentioned in an above post, WebSocket is more advanced than BOSH, except it's highly inefficient for intermittent and constrained networks.

Jabber HTTP Polling (XEP-0025) has been obsoleted by BOSH.

A possible addition for web management is Stream Management (XEP-0198), which synchronizes messages when XMPP reconnects.

XEP-0273: Stanza Interception and Filtering Technology (SIFT) blocks downloading unwanted stanzas (for instance, excessive presence updates) from the client end, to minimize bandwidth on constrained networks and mobile phones.

XEP-0352: Client State Indication (CSI) reduces bandwidth to mobile phones, including by reducing the need for reconnecting on intermittent networks. This has similarities to SIFT, but this is done on the server end. It can hold messages until user is active on client. CSI is possibly an alternative to BOSH. I haven't researched how well this would work in place of BOSH, or if this can be used in conjunction with BOSH.
 
Last edited:

mjollnir

Daemon

Reaction score: 835
Messages: 1,267

[USER=45325]sidetone[/USER]: your last two posts perfectly show the shortcomings of Jabber/XMPP: it's kind of nerdy in that it offers solutions for many issues via it's extensions, OTOH that makes it hairy to configure for the average, innocent user. AFAICT, TOX & maybe Matrix seem to be much easier to set up on the client-side.
 

vigole

Aspiring Daemon

Reaction score: 709
Messages: 660

it's kind of nerdy in that it offers solutions for many issues via it's extensions,
I had to spend a lot of time to choose and setup a perfect XMPP client. Also, one of the FreeBSD users gave me help with correct usage of XMPP client.
But If you only want to chat, i.e. not paying attention to details e.g. privacy and security, then XMPP web interface (register/usage) is very straightforward.
 

getopt

Aspiring Daemon

Reaction score: 590
Messages: 850

it's kind of nerdy in that it offers solutions for many issues via it's extensions,
Yes, XMPP has a learning curve, it includes learning about it's many XEPs. Or at least the ones you have in your client-app.

Lamenting about "nerdyness" is lamenting about a certain unwillingness to learn. XMPP may not be for Greti & Pleti, as FreeBSD is not for everyone. Both requires to RTFM. If that's too much effort there are apps for everything you can go shopping for.

for the average, innocent user.
I do not want to contribute to further off-topic in the Howtos & FAQs Forum. But coming here along with the attribute "innocent" is a funny entertainment.
 
OP
sidetone

sidetone

Daemon

Reaction score: 592
Messages: 1,392

it's kind of nerdy in that it offers solutions for many issues via it's extensions, OTOH that makes it hairy to configure for the average, innocent user.

Let the servers, connection managers and clients make a standardized offering or set (grouped/recommended extensions) on XMPP. For mobile phones, there's a need for BOSH and/or Client State Indication to become common. The XMPP platform is there as an open standard, that is ready for what is needed by any particular group, commonality or other entity.

Some features are only needed on one end, such as BOSH or other implements for constrained and other intermittent networks, until the server is reached.

End to end encryption is often needed on both ends. Encryption is in base, and ready to use, but it allows that traffic to be unencrypted at the server for viewing or logging. Omemo is becoming a new standard, replacing OTR but a bit difficult to use.

Let the server, connection manager and client be reputable, and let it inform the user how to ensure the needed features are enabled.

A lot of extensions are for nifty uses, and not necessary for typical person to person chatting.
 

ekvz

Well-Known Member

Reaction score: 271
Messages: 431

[USER=45325]sidetone[/USER]: your last two posts perfectly show the shortcomings of Jabber/XMPP: it's kind of nerdy in that it offers solutions for many issues via it's extensions, OTOH that makes it hairy to configure for the average, innocent user. AFAICT, TOX & maybe Matrix seem to be much easier to set up on the client-side.

Exactly and it's almost worse when having to deal with the actual implementation.

Yes, XMPP has a learning curve, it includes learning about it's many XEPs. Or at least the ones you have in your client-app.

While i agree that it's obviously possible to deal with it i think nerdyness is actually quite fitting. XMPP as a whole is quite nerdy. In my opinion the way it's protocol is designed also shows clear signs of nerdyness. I think it's obvious how having a protocol consist of abstract building blocks with potential reusability rather than simple feature definitions would be appealing to this group. Actually i can very much understand myself how this might look very elegant, clean and superior in theory. Sadly the practical side of things is a bit different.

This compresses data efficiently, because it is made for XML. Additional XML schemas (including XMPP extension data) can be on server to further improve efficiently.

While that's nice i can't help but notice how this wouldn't have been a problem in the first place if the designers just had accepted that binary protocols obviously have clear advantages over XML. My impression is that XML mostly won due to nerdyness while real world problems (transfering binary data...) were simply ignored.
 
OP
sidetone

sidetone

Daemon

Reaction score: 592
Messages: 1,392

Obviously, it's an XML format, and they should have known to use compression based on that in the beginning. Maybe hindsight is 20/20, or they used what was already available and didn't bother to make a more fitting adaption for compressing XML type data. That's a extension that should go into base, to a large extent.

I read somewhere, that there were disadvantages to using a format that wasn't XML. That XML is structured for keeping data organized, when transferring data/communicating. If bandwidth is the issue, XML can already be compressed, and an ideal means for it should be made standard.
 

ekvz

Well-Known Member

Reaction score: 271
Messages: 431

Obviously, it's an XML format, and they should have known to use compression based on that in the beginning. Maybe hindsight is 20/20, or they used what was already available and didn't bother to make a more fitting adaption for compressing XML type data. That's a extension that should go into base, to a large extent.

Size isn't really the major problem. As you stated that's even somewhat fixable. It's more about how XML is quite bad when dealing with binary data. I think that's at least part of the reason why for example XMPP's concepts of transferring images rely on external solutions. External solutions either being direct connections between clients (which are problematic for multiple reasons) or an external HTTP server which also isn't exactly nice (even if it's by far the better option out of the two). The fact that going for a text based protocol will force you to rely on an encoding like base64 if you ever have to deal with binary data is something that's actually visible from a mile away and it's hard to imagine that this being a requirement would simply be overlooked. XML itself wasting bandwidth is not that bad in most cases. I think in a general sense it becomes mostly noticeable where low latency would be desirable but TCP isn't great in that respect anyways so it might be somewhat excusable.

I read somewhere, that there were disadvantages to using a format that wasn't XML. That XML is structured for keeping data organized, when transferring data/communicating. If bandwidth is the issue, XML can already be compressed, and an ideal means for it should be made standard.

Sure, binary protocols aren't without drawbacks either but when you put things like having to deal with byte order in a couple of places in contrast to having to base64 encode every bit of binary data it think it's quite obvious which is the lesser evil. Structure is not much of an argument in my opinion as you could easily define a binary layout that's pretty much equivalent to XML if you really wanted to.
 
OP
sidetone

sidetone

Daemon

Reaction score: 592
Messages: 1,392

Further Reading

More XEP's
  • https://xmpp.org/extensions/
  • XEP-0127: Common Alerting Protocol (CAP) over XMPP - CAP is used by NOAA and EAS for notifications; robotic voice for tv, radio, telephone; text alerts for tv
  • XEP-0286: Mobile Considerations on LTE Networks
  • XEP-0289: Federated MUC for Constrained Environments
  • XEP-0361: Zero Handshake Server to Server Protocol - s2s connection for constrained networks; servers must be configured to work together beforehand
  • XEP-0365: Server to Server communication over STANAG 5066 ARQ - this is for military use, but is an example of how XMPP can be used over HF radio; this extension relies on XEP-0361
  • XEP-0423: XMPP Compliance Suites 2020 - yearly update on status changes on XMPP extensions and their categories
  • XEP-0443: XMPP Compliance Suites 2021

Additional Links
 
Last edited:
Top