Category Archives: XMPP

Things XMPP can do that IRC can't

This keeps popping up. I’m sure there’s a better article than this one. There’s a few things that I happen to value, even if I do use and like IRC as well.

So: what are some of the things XMPP can do that IRC can’t?

Widespread across all clients/servers:

  1. Per-person presence.
  2. Multi-line messages.
  3. Rich-text messages. 1 XEP-0071: XHTML-IM
  4. Avatars. XEP-0084: User Avatar, XEP-0153: vCard-based Avatars and more.
  5. Offline messages.
  6. Multiple concurrent logins with same credentials. Gets superpowers with XEP-0280: Message Carbons2 and XEP-0313: Message Archive Management.
  7. Cross-“domain”, nearly-trustless s2s federation.

Easy to set up server-side; sufficiently widely supported on mobile and desktop:

  1. Rich status updates. Particularly with extensions based on XEP-0163: Personal Eventing Protocol such as XEP-0118: User Tune or XEP-0080: User Location.
  2. Personal chat history.3 XEP-0313: Message Archive Management
  3. Chatroom history. 4 XEP-0313: Message Archive Management
  4. Chatroom backlog. Built into the multi-user chat specification.
  5. Per-chatroom nickname.
  6. Integrated HTTP file upload file sharing. Peer to peer with XEP-0234: Jingle File Transfer similar to IRC’s DCC, or, for a much better, offline-compatible and multi-client-friendly experience, XEP-0363: HTTP File Upload.
  7. Session resumption. XEP-0198: Stream Management.
  8. Standardized HTTP-based transports to support web-based clients. draft-ietf-xmpp-websocket-00, XEP-0124: BOSH, XEP-0206: XMPP over BOSH.

Rarely well supported on mobile, desktop or web:

  1. Microblogging. XEP-0277
  2. Integrated VVoIP. XEP-0166: Jingle-based XEP-0167: Jingle RTP Sessions.

A story

Now for a short use case description for something that XMPP allows me, and IRC doesn’t allow trivially…

As a side effect of some design choices in how per-protocol “gateways” operate, it’s trivial for a per-protocol “gateway” to also act as an equivalent of an IRC bouncer. For instance, I am logged into XMPP from multiple devices, and I have configured the gateway 5 to persist connection into some IRC channels. This means that I am effectively logged into IRC from multiple locations, and even if I lose the TCP connection from one of the clients (such as my phone), I won’t be kicked out. Plus Message Archive Management will let me obtain the backlog for the chatroom. Session resumption also means that, as far as the IRC component is aware, I will not have lost the connection anyway in most of the cases.

I am simply “available” in chatrooms, and particularly in IRC. And if I am not, I can catch up without referring to an external chatroom log service. Nor do I have to have a persistently-running IRC client in a tmux session, or an IRC bouncer; the IRC component connected to my self-hosted server is acting as a bouncer. If I were using a bouncer instead of a persistent client (enabling multiple connections and enabling logging), I would still easily lose some of the IRC history, as it would be trivial for me to see only the immediate backlog of ~20 messages.

Closing remarks

Are there other protocols offering openness, self-hosting, federation, presence, extensibility (which allows for a bunch of features quoted above)? Probably. None of them attracted me6. Despite low use of XMPP, other such protocols are used even less.

Can IRC be extended? Sure. How long before clients such as Irsii, Hexchat and mIRC are updated? If they are not, how long before they are pried from people’s hands? How long would it take for various IRC daemons to be updated? How long before daemon deployments are updated to support multiline messages?

Finally, if I am mistaken, and IRC has a capability I don’t know about or an upcoming protocol revision is supposed to provide a capability, leave a comment.

Updates

2018-02-27 16:30

The following were brought up on Twitter:


  1. Yes, technically, IRC has some support for message formatting in form of color, weight and slant control characters. 
  2. This enables realtime replication of messages aimed at one client onto other clients. 
  3. Stored server-side and retrievable incrementally. 
  4. Useful for catching up on offline discussions. 
  5. Despite occasional quirks, I highly recommend Biboumi as a featureful, yet sane and no-frills, IRC gateway component. 
  6. I had a very lengthy text here. tl;dr: XML’s namespacing eases extensibility and makes me tolerate XML’s verbosity. And HTTP-based protocols are good and acceptable, but also a sign of a modern-day “I have a hammer, every problem looks like a nail” attitude. 
  7. Apparently, there’s a way to use OTR with IRC using Pidgin.- 
  8. E2E interferes with my interest in keeping various devices in sync using MAM, without having to keep their encryption keys in sync. Instead I hope the system hosting my XMPP server is secure enough, and rarely use E2E. I perhaps shouldn’t, but I do. 
  9. Mobile push is important on platforms that don’t allow for long-running applications maintaining background connections out of energy concerns, and therefore require UI notifications to go through their systems. Notifications, out of necessity, have to go through client developer’s infrastructure, so they partially defeat the point of self-hosting. 

Why I choose not to use WhatsApp, Viber et al

There are many messenger apps these days that have very similar features, and are widely used. I’d usually describe them as “modern” messengers. I choose not to use them. I sometimes get into discussions about why. I’ll update this post if I get new perspectives or if I find better ways to clarify my opinions.

Here are some anti-features from my perspective, widely (but not universally shared), that make me strongly prefer not using these messengers:

  • Uploading all contacts. Many modern messengers use your phone’s addressbook as the primary source for the contact list. This is, in principle, laudable. One source of contacts is a good idea1.

    However, to enable the distinction between contacts that do not use this messenger and those that do, the clients have opted to query their servers for this piece of information. To do so, they upload all your contacts, and see whether the phone number is connected to the service or not.

    Despite not having a signal of whether a contact is ‘weak’ or ‘strong’ (occasional and mainly formal interactions vs daily friendly interactions), messengers can use this to form social graphs. I don’t have a reason to believe they are doing this or exploiting this information, however, I’d prefer a smaller number of companies to have access to my contact list. I’m sure my contacts would prefer that as well.

    For this reason, I’ve chosen that the primary company that’ll have access to this is the one that already syncs my contacts and sees all my email: Google. This means I get restricted to Hangouts (which is mid-way between ‘classical’ and ‘modern’ messengers) or Allo (which is slick, but underused, and has other flaws from this list).

    • Workaround: Messengers, please let me choose not to upload all contacts. Please don’t tell me I cannot block iOS and Android from you grabbing all my contacts. Please let me share only some contacts with you, or manually enter the phone number I want to reach out to.
  • Phone numbers only, please. I use many devices. Counting off the top of my head, I use 82 ‘smart’ devices regularly and 43 sporadically. This is not counting all the operating systems I have on them: both my desktop and my old laptop have 3. How about browsers? Anything with OS X has at least Safari and Chrome.

    I change environments multiple times a day. I’ve changed countries. I could change my phone number. It’s not unreasonable that I expect my conversation to continue from one environment to the other. If I’m on my desktop, I strongly prefer not to have to take out my phone just to see that Jack has said “hi” without any other followup. And doing this while I’m dealing with a page or writing or updating a very convoluted test is very distracting. It could be an important message — should I really have to decide between 15s to get the phone and see the “hi”, making me distracted for the next 5-10min, or leaving a possibly important message unseen?

    Tying a messenger to one phone number and thus one device is ridiculous.

    • Non-workaround: Browser-based solutions. I could receive and send messages from my desktop — hurray? While I do want a web-based client to be available when I’m on a Chromebook, due to e2e they’re usually convoluted and require messages to go through your phone, only to go back through the provider’s servers (presumably re-encrypted) and to be presented in a web UI. I object to this convoluted solution on moral grounds. 🙂

      I also don’t expect I get to integrate with my desktop environment that well.

    • Non-workaround: Wearable devices. While I can see the messages quickly, I have to actually own one. My Moto 360 broke down long time ago, and I’m still waiting for a decent, affordable Android Wear 2.0 device to become available in Ireland. (If I need to get a new one, why not get a proper upgrade?)

    • Counter-example: iMessage. Yep, in place of a workaround, I’m giving a specific “modern messenger” solution. I can not only tie multiple phone numbers, but also multiple email addresses, all on multiple devices, to the same account. Messages.app (formerly iChat), an OS X feature which integrates with iMessage, is a desktop, non-browser solution that neatly integrates with the OS as well.

      I would use this messenger much more if iMessage was available on non-Apple devices and on the web.

  • Ubiquitous e2e. In principle, I like encryption. It does come with huge costs. Most messengers that implement it (well) become terrible with syncing message archives, and become terrible storing them for prolonged periods of time.

    They also have to decide where to store the keys. To keep the whole contraption secure, they often choose a storage mechanism that makes it hard to exfiltrate the keys. This is a good thing — except it prevents sync from working, and it makes it hard to introduce new devices (or browsers!) into the mix. And as I said, I use many, many devices.

    Situations where I actually, genuinely care about e2e enough to break message sync, message archiving, and make provisioning new devices for the same account difficult or impossible — those situations are very rare. I can think of maybe 5-10 cases over the past 3 years, and I can’t even recall the specifics. Cases where I wanted to find details of an old conversation, or where I wanted to continue an old discussion, those are far more frequent.

    • Counter-example: iMessage as a service is doing somewhat well here again. I am just guessing, but it seems like, once provisioned, a message will be encrypted for a particular device’s key in addition to all other devices. If a device is under-used, the key gets phased out. Messages get synced while a device is provisioned.

      Where it’s not doing so well is in-browser support. Apple recently introduced Business Chat and iCloud syncing for messages. It seems to let third-party providers create integrations with iMessage, including web based. It’s for businesses only, from what I can tell!

    • Counter-example: What about, say, WhatsApp’s web UI? Link to your phone, and have all messages go through it; a secure solution, but which I object to morally. I was going to say “I have no idea how message sync interacts with e2e with WhatsApp”, but for me it would be a non-problem with WhatsApp, as either I’d use web UI (which would presumably fetch messages from the phone), or I would not have message sync (as only one phone has a particular phone number). Possibly the key and messages get backed up to Google Drive on Android, but that solves the problem of “I’m changing the phone”, not “I’m using multiple phone numbers and non-phone devices concurrently”.

    • Workaround: What I’d really like to see happen is optional e2e. At the very least, let users agree not to e2e, and reap the benefits of message sync, nice and slick web UI, easier provisioning of new devices. When I use XMPP, I don’t bother at all turning on OTR, OMEMO, or OpenGPG (mechanisms supported in Conversations, top of the line messenger for Android) — but I strongly care about support for Message Carbons (“deliver messages to all online clients”) and Message Archive Management (“archive messages on the server and let clients request the archive”). I own my domain, so I get the benefit of not being tied to a single provider. Friends who use my secondary domain are also welcome to request archive export should they choose to spin up their own server — I’ll gladly spend the time providing them this data. (I’ll also delete the data from their archives on my server, as well, but otherwise I’d expect that I can keep my own records of these chats.)

I could also simply not worry about these problems.

For example, my personal social graph is not going to be important or even a useful source of information to sell me things4. That said, I don’t know what all the contacts in my addressbook are up to. Do I want everyone to tie me to them? It probably does not matter, but I choose to draw the line there.

I could also choose to use the messengers only on one device, and ignore notifications that come while I am focused. I could choose to accept e2e and all the downsides it brings to the sync table. I could choose to use iMessage with my Apple-toting buddies (hint: there aren’t many!). I could choose to install Facebook Messenger, tolerate battery drain, and tolerate having an additional company have access to my communications.

All that said… I don’t get that many benefits from any of these messengers. I can easily reach people I care with XMPP, Hangouts or even SMS. If SMS fails, I can, occasionally, even reactivate the Facebook account and reach out to people using Facebook Messenger on the desktop. I don’t have a good reason to compromise, or to figure out a workaround such as setting up an XMPP transport for WhatsApp. People who happen to be using WhatsApp — I can reach them through SMS as well, and often through Hangouts as well.


  1. Android allowed the apps to do it the other way around, too; applications should integrate with the Contacts app. In practice, social network apps, even if they integrated with Android’s contacts, chose to remove the integration many years ago. This is disappointing. 
  2. Phones: Nexus 6P (personal), iPhone 7 (work). Tablets: iPad Air (personal). Computers: desktop, Macbook Pro 2016, Digital Ocean VPS (personal), workstation, HP Chromebook (work). Other: Nvidia Shield Android TV, Samsung 6400 TV, QNAP TS-509 NAS w/ debian (personal). 
  3. Phones: Nexus 5, Jolla (personal). Tablets: original iPad w/ iOS 5.1.1 (personal), Nexus 7 (work). Computers: Macbook unibody late 2009, Chromebook (personal). 
  4. I mean, I rarely buy exactly the same product just because a friend has it. 
But my dog does.

Migrating Prosody from text store and sqlite3 to PostgreSQL

Assumptions

  • You started off from the basic storage config:
    • Regular data is in filesystem.
    • mam (xep0313) message archive is in SQLite3.
  • You want to transition to using just PostgreSQL.
  • PostgreSQL version is 9.4.

Installation

# # largest version of pgsql you have
~# apt install postgresql-9.4-client

# # get lua-dbi module for pgsql
~# apt install lua-dbi-postgresql

Creating PostgreSQL user

# # PostgreSQL trusts users connecting over unix domain socket to be the
# # same as their local account.
# # Therefore, become postgres -- the admin account.
user:~$ sudo su postgres

# # run user creation
postgres:~$ createuser --interactive
# # name: prosody
# # no other administrative options

# # run postgresql client
postgres:~$ psql
-- create database
CREATE DATABASE prosody;

-- give the prosody user all rights on it
GRANT ALL ON DATABASE prosody TO prosody;

-- in case of connecting over network and using md5 trust,
-- set prosody account password:
ALTER ROLE prosody WITH ENCRYPTED PASSWORD 'here_some_password';

Over network?

Assuming you want to connect over the network, edit /etc/postgresql/9.4/main/pg_hba.conf. Append:

# type, database, user, address, auth method
host prosody prosody 172.16.0.0/16 md5
  • Try to minimize your permitted netmask.
  • Can you configure a more secure auth method than md5? Do so.

Migrate data from filesystem

# # as prosody local user

prosody:$ cd prosody-hg/tools/migrator

# # overwrite the config.
prosody:$ cat > migrator.cfg.lua << _EOF
local data_path = "../../data";

input {
        type = "prosody_files";
        path = data_path;
}
output {
        type = "prosody_sql";
        driver = "PostgreSQL";
        database = "prosody";
        username = "prosody";
        password = "here_some_password";
        host = "database.host.here"; -- this assumes network connection; migration with local user credentials was not attempted.
}
_EOF

# # run the migrator in ~/prosody-hg/tools/migrator
prosody:$ lua prosody-migrator.lua input output

Migrate mam archive from SQLite3

# # as prosody user

prosody:~$ cd prosody-hg/data

# # Having first verified there is nothing in Prosody table...
prosody:$ sqlite3 prosody.sqlite 'SELECT COUNT(*) FROM prosody;'
# # ...drop prosody table. All its data (roster etc) was until now stored on the filesystem.
prosody:$ sqlite3 prosody.sqlite 'DROP TABLE prosody;'

# # dump and massage the sqlite3 output, piping it into psql.
# # psql authenticates as the 'prosody' user and does not require the password.
prosody:$ sqlite3 prosody.sqlite .dump | \
    grep -v 'BEGIN TRANSACTION;' | \
    sed 's/PRAGMA foreign_keys=OFF;/BEGIN TRANSACTION; SET CONSTRAINTS ALL DEFERRED;/' | \
    grep -v sqlite_sequence | \
    awk '/CREATE TABLE/{gsub("`","\"");} 1' | \
    awk '/CREATE UNIQUE INDEX/{gsub("`","\"");} 1' | \
    sed 's/INTEGER PRIMARY KEY AUTOINCREMENT/SERIAL PRIMARY KEY/' | \
    psql

# # manual step :(
# # fix the autoincrement.
prosody:$ sqlite3 prosody.sqlite 'SELECT COUNT(*) FROM prosodyarchive;'
# # use this number + 1 in:
prosody:$ echo 'ALTER SEQUENCE prosodyarchive_sort_id_seq RESTART WITH 123456;' | psql

Update Prosody config

In prosody.cfg.lua:

storage = {
        archive2 = "sql";
}
sql = { driver = "PostgreSQL", database = "prosody", username = "prosody", password = "here_some_password", host = "database.host.here" }

Tip for XMPP users: adjust your priority!

To declare which of my connections to my XMPP server has the ‘most important’ and ‘most chattable’ status, I use XMPP’s mechanism.

Basics of XMPP connections

XMPP connection is actually two streams of XML, one going from the client to the server, and the other from the server to the client. (If your client exposes a feature typically called “XML Console”, use it to see the traffic.) Each XMPP connection has a “resource” string attached to it (and generally requested by the client — though the server can opt to override it). Typically, clients will set it to a random string, to the client name, or to the hostname. Together with your account name (“Jabber ID”), resource forms a globally-unique way to reach you in the XMPP network; for example: username@example.net/GajimAtWork.

Basics of XMPP presences

Each XMPP connection’s status text and status type are declared using a tag. Just an empty tag means “My status type is ‘online’, with no status text and with priority set to zero”. To specify each of these, according to RFC3921’s section 2.2.2, you add extra tags inside the 1:


  away
  This is my status
  20

The will be broadcast to users subscribed to them, whom you authorized to receive them.

How priority is used

The of a is used by clients for a variety of things, including prioritizing which status to display to users. And, even more importantly, servers use to determine where to deliver s that are directed towards a bare JID, and not towards a full JID. (A bare JID does not include the resource string, and thus describes an account; a full JID includes a resource string, and thus described a connection.)

Servers will deliver s aimed at a bare JID to all connections that have the top priority. For example, if you have the following connections:

Resource Status Priority
GajimAtWork online 15
PidginAtWork online 15
OldPhone online 14
NewPhone online 14
GajimAtHome away 2
Tablet away 1
Webmail online 1

you will receive the message to GajimAtWork and PidginAtWork. (Specifics of this may be overridden by the server, especially if some XEP2 such as ‘carbon copies’ is in use.) See more information in RFC3921, section 11.1 which discusses how the server should handle incoming stanzas (incl. those directed at bare JID).

And let’s say you absolutely don’t want to be disturbed to a certain device, unless this device is directly contacted (by specifying full JID). In that case, specifying a negative priority (say, -1) is handy, which tells the server not to deliver the message at all, even if it’s the top priority. You can still initiate outgoing chats; receiving a message commonly makes the client switch to sending to a bare JID from which it was received.

How priority is useful

Many of the better clients let you associate a priority with a status type; that is, if you set your status type () to dnd3, you can declare that your should also change to 20; and if you set your status type to online (that is, is missing), your should change to 40.

Combined with the fact that, all things equal, I would prefer to answer to incoming chats on my desktop, I began using the following setup in clients that support the aforementioned functionality:

show Usual name in UI Device type Priority
chat Free for chat desktop 50
none Available desktop 30
dnd Busy desktop 25
away Away desktop 10
xa Extended Away desktop 2

For mobile, just decrease by one:

show Usual name in UI Device type Priority
chat Free for chat mobile 49
none Available mobile 29
dnd Busy mobile 24
away Away mobile 9
xa Extended Away mobile 1

What happens if your client does not support ?

Useful clients which don’t support setting — for example, Conversations for Android (source) — will have priority set to zero. Such a client, however, will be useful mainly because it supports and uses replacement XEPs such as XEP-0280: Message Carbons, which will ensure the message is still delivered to that device. There will be dark sorcery involving XEP-0333: Chat Markers which will help to reduce the number of devices that are making noises, similar to experience in Hangouts.

Some say statuses and priority are not useful

Some say statuses and priority are not useful. I respect this opinion, but my personal experience with Hangouts where exactly this is the norm tells me otherwise. I’d rather automate declaring my status than have it disappear from my contacts’ feeds. “Locked my workstation? It’s 12:00-14:00? Probably at lunch.” and similar personalized heuristics. “I’ve been toying with my phone for more than 5 min? I am probably free for chat — but do tell the contacts that I am on my phone.”

That said, I do like and appreciate much of the modern experiences certain statusless client(s) have. There is something to be said for simply receiving messages where they should arrive and notify based on actual activity. I like the simplicity of it.

Then again, if I am at work, I probably don’t want to chat with you; how will non-personalized client know that I’m busy out of the box? Or even more importantly, how will it relay that to my contacts? Can I more simply teach my phone to shut up based on my personal daily routine?

shrug I think I can. Your mileage may vary.


  1. s are used for more than this even in base RFCs. They can be directed towards one
    specific JID, instead of server just broadcasting them. This can in turn be used as a mechanism
    to declare that you would like to subscribe to a particular user’s presences. But, this is a
    discussion on tag, so refer to the RFC for more information. 

  2. XEP: XMPP Extension Protocol. 
  3. dnd maps to ‘busy` in UI. 

Stopping Gajim from autostarting OTR end-to-end encryption

Is Gajim auto-negotiating OTR? Does that annoy you by preventing server-side message archival and message carbons being delivered to other clients you use?

  • Go to Preferences (hit ctrl+p).
  • Go to Advanced tab
  • Under Advanced Configuration Editor click Open.
  • Use search to find ‘autonegotiate_esessions’ and ‘enable_esessions’ options
  • For each account, click on value ‘Activated’ to toggle it into ‘Deactivated’.

I definitely want the option to initiate crypto via GPG and OTR. I don’t want it started automatically and preventing other useful archival that I do on my self-hosted hardware.