On Mastodon and the “fediverse”

I do like the freedom from lock-in. That’s why I run my own federated XMPP server (though a domain whitelist applies due to spam – contact me if you want to interop).

I do want to regain control over my social postings. I don’t use Facebook; I’m mainly on Twitter. I don’t mind Twitter as much, but it would be nice to host my own posts.

That’s why I will not be signing up for a Mastodon instance. Allegedly migration to another instance is easy. But permalinks to posts would still be stored on a domain owned by someone else. Thank you – but then I might as well stay on Twitter.

What about running my own? I have a test deployment I can spin up, but I don’t want to pay for the resources that would be required to make it a permanent thing. Mastodon’s minimum requirements are huge.

Why not GNU Social? I am trying to reduce my consumption of PHP.

Why not Pleroma? Maybe I’ll do that. I need to check it out, however, I’m not well versed in Erlang and fediverse stuff seems like something I may want to customize.

Customize in what way? Bridging to XMPPs microblogging seems like an interesting possibility. And sharing more than just “notes” (Twitter’s tweets, Mastodon’s toots) seems like a good way of weaning myself off of WordPress. ActivityStreams vocabulary (which makes an appearance in OStatus and is basis for ActivityPub) has more than just Notes.

Enabling zoom ‘key’ and spell key on Microsoft Natural® Ergonomic Keyboard 4000

The procedure is not best described elsewhere on the web. This article is a mess, too, but it works for me.

Keys need to be remapped to something under keycode 256 in order to work under X11.

  • Try using evtest and pressing keys to see what the keys map to right now.
    • evtest can will also tell you what are all the events supported by the device.
    • evtest will show two devices; you are interested in the second one (which exposes all the extended keys, such as new, reply, open, send, etc.
  • Use xev to see whether the keys are recognized, and as what are they recognized, in X11.

Now for the juicy part:

# put this into: /etc/udev/hwdb.d/61-keyboard-custom.hwdb

# then to update:
#  sudo udevadm hwdb --update && sudo udevadm control --reload-rules && sudo udevadm control --reload
# and:
#  sudo udevadm trigger
# or:
#  for i in /sys/class/input/* ; do if [[ -e "$i"/id/vendor ]] && [[ -e "$i"/id/product ]] && [[ "$(cat "$i"/id/vendor)" == 045e ]] && [[ "$(cat "$i"/id/product)" == 00db ]] ;  then echo $i ; echo change | sudo tee $i/uevent ; fi ; done

# Natural Keyboard 4000
# formerly:
#keyboard:usb:v045Ep00DB*
# now:
evdev:input:b0003v045Ep00DB*
 KEYBOARD_KEY_0c01ab=finance             # KEY_SPELLCHECK    to KEY_FINANCE
 KEYBOARD_KEY_c022d=up
 KEYBOARD_KEY_c022e=down

We’re naming it 61-keyboard-custom.hwdb in order to have it come after /lib/udev/hwdb.d/60-keyboard.hwdb.

Instead of finance, up and down keys, try taking something from this list: quirk-keymap-list.txt — however, I am not certain how to determine which ones are under 256 except by looking at evtest‘s output.

You can map to keycode 255 and use xmodmap -e "keycode 255 = XF86ZoomIn" to map to a ‘proper’ zoom in key.

On a related note: If you want to remap scancodes to keycodes, you can do it on the fly using setkeycodes(8)

Some sources:

Things XMPP can do that IRC can’t

This keeps popping up. I’m sure there’s a better article than this one. There’s a few things that I happen to value, even if I do use and like IRC as well.

So: what are some of the things XMPP can do that IRC can’t?

Widespread across all clients/servers:

  1. Per-person presence.
  2. Multi-line messages.
  3. Rich-text messages. 1 XEP-0071: XHTML-IM
  4. Avatars. XEP-0084: User Avatar, XEP-0153: vCard-based Avatars and more.
  5. Offline messages.
  6. Multiple concurrent logins with same credentials. Gets superpowers with XEP-0280: Message Carbons2 and XEP-0313: Message Archive Management.
  7. Cross-“domain”, nearly-trustless s2s federation.

Easy to set up server-side; sufficiently widely supported on mobile and desktop:

  1. Rich status updates. Particularly with extensions based on XEP-0163: Personal Eventing Protocol such as XEP-0118: User Tune or XEP-0080: User Location.
  2. Personal chat history.3 XEP-0313: Message Archive Management
  3. Chatroom history. 4 XEP-0313: Message Archive Management
  4. Chatroom backlog. Built into the multi-user chat specification.
  5. Per-chatroom nickname.
  6. Integrated HTTP file upload file sharing. Peer to peer with XEP-0234: Jingle File Transfer similar to IRC’s DCC, or, for a much better, offline-compatible and multi-client-friendly experience, XEP-0363: HTTP File Upload.
  7. Session resumption. XEP-0198: Stream Management.
  8. Standardized HTTP-based transports to support web-based clients. draft-ietf-xmpp-websocket-00, XEP-0124: BOSH, XEP-0206: XMPP over BOSH.

Rarely well supported on mobile, desktop or web:

  1. Microblogging. XEP-0277
  2. Integrated VVoIP. XEP-0166: Jingle-based XEP-0167: Jingle RTP Sessions.

A story

Now for a short use case description for something that XMPP allows me, and IRC doesn’t allow trivially…

As a side effect of some design choices in how per-protocol “gateways” operate, it’s trivial for a per-protocol “gateway” to also act as an equivalent of an IRC bouncer. For instance, I am logged into XMPP from multiple devices, and I have configured the gateway 5 to persist connection into some IRC channels. This means that I am effectively logged into IRC from multiple locations, and even if I lose the TCP connection from one of the clients (such as my phone), I won’t be kicked out. Plus Message Archive Management will let me obtain the backlog for the chatroom. Session resumption also means that, as far as the IRC component is aware, I will not have lost the connection anyway in most of the cases.

I am simply “available” in chatrooms, and particularly in IRC. And if I am not, I can catch up without referring to an external chatroom log service. Nor do I have to have a persistently-running IRC client in a tmux session, or an IRC bouncer; the IRC component connected to my self-hosted server is acting as a bouncer. If I were using a bouncer instead of a persistent client (enabling multiple connections and enabling logging), I would still easily lose some of the IRC history, as it would be trivial for me to see only the immediate backlog of ~20 messages.

Closing remarks

Are there other protocols offering openness, self-hosting, federation, presence, extensibility (which allows for a bunch of features quoted above)? Probably. None of them attracted me6. Despite low use of XMPP, other such protocols are used even less.

Can IRC be extended? Sure. How long before clients such as Irsii, Hexchat and mIRC are updated? If they are not, how long before they are pried from people’s hands? How long would it take for various IRC daemons to be updated? How long before daemon deployments are updated to support multiline messages?

Finally, if I am mistaken, and IRC has a capability I don’t know about or an upcoming protocol revision is supposed to provide a capability, leave a comment.

Updates

2018-02-27 16:30

The following were brought up on Twitter:


  1. Yes, technically, IRC has some support for message formatting in form of color, weight and slant control characters. 
  2. This enables realtime replication of messages aimed at one client onto other clients. 
  3. Stored server-side and retrievable incrementally. 
  4. Useful for catching up on offline discussions. 
  5. Despite occasional quirks, I highly recommend Biboumi as a featureful, yet sane and no-frills, IRC gateway component. 
  6. I had a very lengthy text here. tl;dr: XML’s namespacing eases extensibility and makes me tolerate XML’s verbosity. And HTTP-based protocols are good and acceptable, but also a sign of a modern-day “I have a hammer, every problem looks like a nail” attitude. 
  7. Apparently, there’s a way to use OTR with IRC using Pidgin.- 
  8. E2E interferes with my interest in keeping various devices in sync using MAM, without having to keep their encryption keys in sync. Instead I hope the system hosting my XMPP server is secure enough, and rarely use E2E. I perhaps shouldn’t, but I do. 
  9. Mobile push is important on platforms that don’t allow for long-running applications maintaining background connections out of energy concerns, and therefore require UI notifications to go through their systems. Notifications, out of necessity, have to go through client developer’s infrastructure, so they partially defeat the point of self-hosting. 

AppArmor on Ubuntu and MySQL in custom directories

AppArmor profile on Ubuntu is (rightly) restrictive and prevents the daemon binary mysqld from writing to unexpected locations.

So here’s another one in my series of ‘stupid notes to self’ — things that may help a reader, things that will help me, but things that are not proper or full guides to solving a problem.

Starting an already initialized datadir:

normal-user$ /usr/sbin/mysqld --defaults-file=/tmp/barproject-mysql-my.cnf
2018-01-09T21:24:28.090896Z 0 [ERROR] InnoDB: The innodb_system data file 'ibdata1' must be writable
2018-01-09T21:24:28.090907Z 0 [ERROR] InnoDB: The innodb_system data file 'ibdata1' must be writable

Initializing a new one:

normal-user$ /usr/sbin/mysqld --defaults-file=/tmp/barproject-mysql-my.cnf --datadir="$(pwd)/mysql" --log-level-verbosity=VERBOSE --initialize-insecure
mysqld: Can't create directory '/home/foo/projects/bar/_dev/mysql/' (Errcode: 13 -     Permission denied)
2016-10-10T16:23:29.515470Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2016-10-10T16:23:29.519420Z 0 [ERROR] Aborting

(Note that /tmp/barproject-mysql-my.cnf has been created from a template prior to running either of these. It specifies many values including datadir.)

First, I worked under the assumption the daemon was running under the wrong user or that the directory has wrong permissions. However, changing settings to any reasonable value did not get rid of errors with either initialization step or run step.

AppArmor has profiles that may block accesses atypical for the program executed. One such profile is for /usr/sbin/mysqld and is located in /etc/apparmor.d/usr.sbin.mysqld.

I tried symlinking it to /etc/apparmor.d/disable directory and restarting apparmor with systemctl restart apparmor. This didn’t change anything. I also tried whitelisting the directory by adding a local configuration change to /etc/apparmor.d/local/usr.sbin.mysqld:

/home/foo/projects/bar/_dev/mysql/** rwk,

systemctl restart apparmor — i.e. restarting through systemd — did not help.

However telling apparmor to tear itself down using the service command, as well as telling it to reload its profile cache, did:

service apparmor stop
service apparmor teardown
service apparmor recache
service apparmor start

There’s probably a smarter way, but this is good enough for me.

Why I choose not to use WhatsApp, Viber et al

There are many messenger apps these days that have very similar features, and are widely used. I’d usually describe them as “modern” messengers. I choose not to use them. I sometimes get into discussions about why. I’ll update this post if I get new perspectives or if I find better ways to clarify my opinions.

Here are some anti-features from my perspective, widely (but not universally shared), that make me strongly prefer not using these messengers:

  • Uploading all contacts. Many modern messengers use your phone’s addressbook as the primary source for the contact list. This is, in principle, laudable. One source of contacts is a good idea1.

    However, to enable the distinction between contacts that do not use this messenger and those that do, the clients have opted to query their servers for this piece of information. To do so, they upload all your contacts, and see whether the phone number is connected to the service or not.

    Despite not having a signal of whether a contact is ‘weak’ or ‘strong’ (occasional and mainly formal interactions vs daily friendly interactions), messengers can use this to form social graphs. I don’t have a reason to believe they are doing this or exploiting this information, however, I’d prefer a smaller number of companies to have access to my contact list. I’m sure my contacts would prefer that as well.

    For this reason, I’ve chosen that the primary company that’ll have access to this is the one that already syncs my contacts and sees all my email: Google. This means I get restricted to Hangouts (which is mid-way between ‘classical’ and ‘modern’ messengers) or Allo (which is slick, but underused, and has other flaws from this list).

    • Workaround: Messengers, please let me choose not to upload all contacts. Please don’t tell me I cannot block iOS and Android from you grabbing all my contacts. Please let me share only some contacts with you, or manually enter the phone number I want to reach out to.
  • Phone numbers only, please. I use many devices. Counting off the top of my head, I use 82 ‘smart’ devices regularly and 43 sporadically. This is not counting all the operating systems I have on them: both my desktop and my old laptop have 3. How about browsers? Anything with OS X has at least Safari and Chrome.

    I change environments multiple times a day. I’ve changed countries. I could change my phone number. It’s not unreasonable that I expect my conversation to continue from one environment to the other. If I’m on my desktop, I strongly prefer not to have to take out my phone just to see that Jack has said “hi” without any other followup. And doing this while I’m dealing with a page or writing or updating a very convoluted test is very distracting. It could be an important message — should I really have to decide between 15s to get the phone and see the “hi”, making me distracted for the next 5-10min, or leaving a possibly important message unseen?

    Tying a messenger to one phone number and thus one device is ridiculous.

    • Non-workaround: Browser-based solutions. I could receive and send messages from my desktop — hurray? While I do want a web-based client to be available when I’m on a Chromebook, due to e2e they’re usually convoluted and require messages to go through your phone, only to go back through the provider’s servers (presumably re-encrypted) and to be presented in a web UI. I object to this convoluted solution on moral grounds. 🙂

      I also don’t expect I get to integrate with my desktop environment that well.

    • Non-workaround: Wearable devices. While I can see the messages quickly, I have to actually own one. My Moto 360 broke down long time ago, and I’m still waiting for a decent, affordable Android Wear 2.0 device to become available in Ireland. (If I need to get a new one, why not get a proper upgrade?)

    • Counter-example: iMessage. Yep, in place of a workaround, I’m giving a specific “modern messenger” solution. I can not only tie multiple phone numbers, but also multiple email addresses, all on multiple devices, to the same account. Messages.app (formerly iChat), an OS X feature which integrates with iMessage, is a desktop, non-browser solution that neatly integrates with the OS as well.

      I would use this messenger much more if iMessage was available on non-Apple devices and on the web.

  • Ubiquitous e2e. In principle, I like encryption. It does come with huge costs. Most messengers that implement it (well) become terrible with syncing message archives, and become terrible storing them for prolonged periods of time.

    They also have to decide where to store the keys. To keep the whole contraption secure, they often choose a storage mechanism that makes it hard to exfiltrate the keys. This is a good thing — except it prevents sync from working, and it makes it hard to introduce new devices (or browsers!) into the mix. And as I said, I use many, many devices.

    Situations where I actually, genuinely care about e2e enough to break message sync, message archiving, and make provisioning new devices for the same account difficult or impossible — those situations are very rare. I can think of maybe 5-10 cases over the past 3 years, and I can’t even recall the specifics. Cases where I wanted to find details of an old conversation, or where I wanted to continue an old discussion, those are far more frequent.

    • Counter-example: iMessage as a service is doing somewhat well here again. I am just guessing, but it seems like, once provisioned, a message will be encrypted for a particular device’s key in addition to all other devices. If a device is under-used, the key gets phased out. Messages get synced while a device is provisioned.

      Where it’s not doing so well is in-browser support. Apple recently introduced Business Chat and iCloud syncing for messages. It seems to let third-party providers create integrations with iMessage, including web based. It’s for businesses only, from what I can tell!

    • Counter-example: What about, say, WhatsApp’s web UI? Link to your phone, and have all messages go through it; a secure solution, but which I object to morally. I was going to say “I have no idea how message sync interacts with e2e with WhatsApp”, but for me it would be a non-problem with WhatsApp, as either I’d use web UI (which would presumably fetch messages from the phone), or I would not have message sync (as only one phone has a particular phone number). Possibly the key and messages get backed up to Google Drive on Android, but that solves the problem of “I’m changing the phone”, not “I’m using multiple phone numbers and non-phone devices concurrently”.

    • Workaround: What I’d really like to see happen is optional e2e. At the very least, let users agree not to e2e, and reap the benefits of message sync, nice and slick web UI, easier provisioning of new devices. When I use XMPP, I don’t bother at all turning on OTR, OMEMO, or OpenGPG (mechanisms supported in Conversations, top of the line messenger for Android) — but I strongly care about support for Message Carbons (“deliver messages to all online clients”) and Message Archive Management (“archive messages on the server and let clients request the archive”). I own my domain, so I get the benefit of not being tied to a single provider. Friends who use my secondary domain are also welcome to request archive export should they choose to spin up their own server — I’ll gladly spend the time providing them this data. (I’ll also delete the data from their archives on my server, as well, but otherwise I’d expect that I can keep my own records of these chats.)

I could also simply not worry about these problems.

For example, my personal social graph is not going to be important or even a useful source of information to sell me things4. That said, I don’t know what all the contacts in my addressbook are up to. Do I want everyone to tie me to them? It probably does not matter, but I choose to draw the line there.

I could also choose to use the messengers only on one device, and ignore notifications that come while I am focused. I could choose to accept e2e and all the downsides it brings to the sync table. I could choose to use iMessage with my Apple-toting buddies (hint: there aren’t many!). I could choose to install Facebook Messenger, tolerate battery drain, and tolerate having an additional company have access to my communications.

All that said… I don’t get that many benefits from any of these messengers. I can easily reach people I care with XMPP, Hangouts or even SMS. If SMS fails, I can, occasionally, even reactivate the Facebook account and reach out to people using Facebook Messenger on the desktop. I don’t have a good reason to compromise, or to figure out a workaround such as setting up an XMPP transport for WhatsApp. People who happen to be using WhatsApp — I can reach them through SMS as well, and often through Hangouts as well.


  1. Android allowed the apps to do it the other way around, too; applications should integrate with the Contacts app. In practice, social network apps, even if they integrated with Android’s contacts, chose to remove the integration many years ago. This is disappointing. 
  2. Phones: Nexus 6P (personal), iPhone 7 (work). Tablets: iPad Air (personal). Computers: desktop, Macbook Pro 2016, Digital Ocean VPS (personal), workstation, HP Chromebook (work). Other: Nvidia Shield Android TV, Samsung 6400 TV, QNAP TS-509 NAS w/ debian (personal). 
  3. Phones: Nexus 5, Jolla (personal). Tablets: original iPad w/ iOS 5.1.1 (personal), Nexus 7 (work). Computers: Macbook unibody late 2009, Chromebook (personal). 
  4. I mean, I rarely buy exactly the same product just because a friend has it. 

What are OSC terminal control sequences / escape codes?

Changelog for a recent release of SecureShell for Chrome
included a mention of “Support for iTerm2 OSC-9 growl notifications”. This is displayed in terminal when you’re SSHing into a machine, and this entry piqued my interest.

Turns out it’s from hterm’s changelog. And turns out hterm has a nice list of supported control sequences.

OSC are “operating system command” control sequences and there’s a few of them supported by hterm / SecureShell. I find a few super interesting (\033 being ESC in the doc linked above):

  • 0: set window title: echo -e "\033]0;This is the window title\a"
  • 9: show system notification (iTerm2 variant): echo -e "\033]9;This is the notification text\a"
  • 777: show system notification (rxvt-unicode / urxvt variant): echo -e "\033]777;notify;This is the notification title;This is the notification text\a"
  • 52: clipboard operation; hterm only supports c for copy: echo -e "\033]52;c;$(base64 <<< hello)\a"