Neil Alexander

After quite a significant gap between releases — version 0.3.11 was released at the beginning of March — we're happy to finally announce the release of Dendrite 0.4.0 today!

The full changelog for the release is available on GitHub, but we wanted to take the opportunity to talk a little about some of the changes that have gone into this release.

Recently our release cadence for Dendrite has slowed as we have spent more time within the team working on Pinecone and Low Bandwidth Matrix. These are major areas of research for us which we hope will unlock a number of new opportunities within the Matrix ecosystem, allowing us to build on Matrix anywhere and to reduce the protocol-level footprint. However, Dendrite has not been forgotten amidst the excitement and we will be spending more time working on Dendrite again in the coming months.

State storage

One of the major features in v0.4.0 is that we've introduced newly-refactored state storage in the roomserver database. The goal here is to make state storage significantly more efficient by ensuring that we deduplicate state blocks and snapshots wherever we can. By ensuring that all state blocks and snapshots are ordered strictly, and by enforcing uniqueness constraints on the hashes of the blocks/snapshots, we've been able to achieve this.

This was largely spurred on by watching dendrite.matrix.org consuming a rather alarming amount of disk space on a daily basis. In this particular instance, moving to the new state storage resulted in a 15x improvement on disk utilisation for state blocks and a further 2x improvement for state snapshot references immediately after the migration, and the growth rate of the database has slowed substantially since.

Ensuring that we don't waste disk space is one of the most important factors in ensuring that Dendrite operates well at any scale — future datacentre deployments supporting many users will find storage overheads decreased and small/embedded single-user deployments (such as P2P, on mobile devices or in the browser) will fit much more effectively onto resource-constrained targets.

After upgrading to v0.4.0, Dendrite will run an automatic migration to update your homeserver to the new state storage format. This might take a while on larger databases so please expect some downtime.

Optimisations

We've continued to squeeze further optimisations into the federation and state resolution code, aiming to reduce the amount of CPU burn and memory utilisation. Some of the feedback that we receive most often from those that have been experimenting with the Dendrite betas is around the sudden spikes in resource usage, especially when joined to large rooms.

The bulk of this resource usage comes either from attempting to reconcile missing events or running state resolution in rooms with lots of members, as potentially large state sets of events need to be brought into memory in order to do so. We've introduced some transaction-level caches for dealing with missing auth/prev events to reduce the memory pressure and we've also tweaked the caching around around /get_missing_events to ensure we don't duplicate any state events in memory.

Resource spikes aren't completely eliminated but this should smooth out CPU and memory utilisation significantly. In the case of dendrite.matrix.org, which is joined to some 6500 rooms at present, memory utilisation of the Dendrite process typically sits around 1.5GB at present.

State Resolution v2 has also seen further optimisations in the power-level checking, which should reduce CPU usage even more.

Bridges

Thanks to Half-Shot's perseverance and contributions, we've merged a couple PRs and worked on some further fixes for getting Application Services working correctly in Dendrite. Whilst not entirely feature-complete and with a number of features still to go, enough support is now present to support basic bridging functionality.

We've done quite a bit of preliminary testing with matrix-appservice-irc and have also heard a number of success stories from the community with mautrix-whatsapp and mautrix-telegram. Others may work too — let us know what you find!

Bug-hunting

A number of bugs in various places (including the roomserver, federation API and media API) which could cause Dendrite to crash have also been fixed. Some of these have been contributed by the community in pull requests, so we extend our thanks to anyone who has submitted a fix to the project.

A special mention also goes to Jakob Varmose Bentzen for reporting a security issue to us around the legacy /v1/register endpoint, where a flaw in the legacy shared secret registration allowed malicious users to create accounts. We've since removed this legacy endpoint and the vulnerability is now fixed.

What's next

There are still a number of missing user-facing features which we will be working on over the coming months, as well as some architectural issues that we will look to address.

A notable area of work involves attempting to remove the dependency on Kafka for polylith deployments. Kafka is very resource-heavy in operation and somewhat limits us to the types of interactions that we can perform between components. It's also very difficult to manage retention correctly, in the interests of not endlessly consuming disk space here either.

As usual, Dendrite is still considered beta so you may not want to rely on it for production systems, although it should be stable enough to experiment with. If you find any bugs or anything that doesn't look right, please let us know:

• #dendrite:matrix.org
• GitHub Issues

We're also open to contributions, so don't be afraid to open pull requests. Finally, thanks for your continued support!

— Team Dendrite

Since the end of 2019, we have spent quite a bit of time thinking about and exploring different technologies whilst building various demos for P2P Matrix. Our mission for P2P Matrix is to evolve Matrix into a hybrid between today's server-oriented network and a pure P2P network - empowering users to have total autonomy and privacy over their data if they want (by storing it in P2P Matrix, by embedding their server into their Matrix client), while also letting users store their data in serverside nodes if they so desire.

The goal is to protect metadata much better (as users no longer have to depend on a server run by someone else to communicate), as well as drive new features such as account portability, multi-homed accounts, low-bandwidth Matrix and smarter federation transports - and provide support for internet-less mesh communication via Matrix which can also interoperate with the wider network. You can read more about it in our Introducing P2P Matrix blog post from last summer, or watch our FOSDEM 2021 talk where we previewed Pinecone. It's important to note that this has been a small but important long-term project for Matrix, and has been progressing entirely outside our business-as-usual work of improving the core protocol and reference implementations.

As the project has progressed, we've built a variety of prototypes using existing libraries (go-libp2p, js-libp2p and Yggdrasil), demonstrating what an early P2P Matrix might feel like if it were running on a mobile device, in the web browser and so on using such an overlay network. Each of these demos has taught us something new, and so in October 2020 we decided to take this knowledge to build an experimental new overlay network of our own.

Pinecone is designed to provide end-to-end encrypted communications between devices, regardless of how they are connected to one another, in a lightweight and self-arranging fashion. The routing protocol is a hybrid, taking inspiration from Yggdrasil by building a global spanning tree, but rather than forwarding all traffic using the spanning tree topology, we use it as a bootstrap routing mechanism for a line/snake topology, ordered by their ed25519 public keys, which we have affectionately named SNEK (Sequentially Networked Edwards Key) routing.

Nodes seek out their closest keyspace neighbours on the network and paths are built between these pairs of nodes, similar to how a Chord DHT functions, populating the routing tables of intermediate nodes in the process. These paths are then used to forward traffic without having to perform up-front searches, allowing for very fast connection setups between overlay nodes. These paths are resilient to network topology changes and handle node mobility considerably better than any other name-independent routing scheme that we have seen — early results are very promising so far. We have also been experimenting with a combination of the μTP (Micro Transport Protocol) and TLS to provide stateful connection setup, congestion control and end-to-end encryption for all federation traffic carried over the Pinecone network.

If Pinecone works out, our intention is to collaborate with the libp2p and IPFS team to incorporate Pinecone routing into libp2p (if they'll have us!) while incorporating their gossipsub routing to improve Matrix federation... and get the best of both worlds :)

Today we're releasing the source code for our current early implementation of Pinecone — you can get it from GitHub right now! It's very experimental still and not very well optimised yet, but it is the foundation of our latest mobile P2P Matrix demos, which support P2P Matrix over both Bluetooth Low Energy mesh networks, multicast DNS discovery within a LAN, and/or by routing through static Pinecone peers on the Internet:

• Android: https://appdistribution.firebase.dev/i/394600067ea8ba37
• iOS: https://testflight.apple.com/join/Tgh2MEk6

Building a routing overlay is only the first step in the journey towards P2P Matrix. We will also be looking closely in the coming months at improving the Matrix federation protocol to work well in mixed-connectivity scenarios (rather than the full mesh approach used today) as well as decentralised identities, hybrid deployments with existing homeservers and getting Dendrite (the Matrix homeserver which is embedded into the current P2P demos) more stable and feature-complete.

The long-term plan could look something like this:

Most discussion around P2P Matrix takes place in #p2p:matrix.org, so if you are interested in what's going on, please join us there!

It's been a year since Dendrite development picked up again and it's certainly been a busy one at that! We started off 2020 by sprinting to complete the FOSDEM P2P demo and, since then, we have continued to develop Dendrite into a more featureful and stable homeserver.

In October, we moved Dendrite into beta, and have since released a number of releases. We've also seen quite a lot of interest from the community, so I'm here today to write about some of the fun things that have been going on in Dendrite-land.

Announcing dendrite.matrix.org

I'm happy to announce that we've finally deployed our own public Dendrite instance at dendrite.matrix.org! It's running the latest Dendrite code and is open for registration, so if you have been looking for an opportunity to play with Dendrite without hosting your own deployment, here's your chance!

There are still bugs and missing features, but overall the server is quite usable, so please feel free to register and try it and let us know how you get on.

This is the first deployment that we've built for any kind of scale, so we are cautious of the fact that there may be performance bottlenecks still. That said, over the last few weeks, a number of performance-improving changes have been merged, including:

• Around 20x performance improvement on the State Resolution v2 algorithm, which is used in room versions 2 and above
• Significantly reducing the amount of time spent recalculating event reference hashes and event IDs in the roomserver and sync API, reducing CPU usage
• Optimised memory usage and reduced database hits in the federation sender, which helps particularly in large rooms with lots of resident servers

We're optimistic that running this deployment will help us to identify scaling pain points and to make Dendrite leaner in the long run. Feel free to sign up and give it a spin! :-)

Features

Since the beginning of the year, we've added a number of new features, including but not limited to:

• Room versions support for all currently specced versions (v1-v6), including support for State Resolution v2
• SQLite storage support in addition to PostgreSQL, largely useful for the P2P demos
• Support for database schema upgrades, making updating Dendrite significantly easier
• Early end-to-end encryption support, including device list syncing and send-to-device messages, although with key backup and cross-signing still to come
• A number of federation features, including invites, retries and backing off unreachable homeservers
• User-interactive authentication (UIA), which is used in password changes and deleting devices from your device list
• Support for local pagination, backfilling over federation and fetching missing events
• Redaction of events both locally and over federation
• Entirely new microservices for managing server signing keys, E2E keys, user and device management
• Lots of great contributions from the community - including all of Read Receipts (thanks to S7evinK) and Read Markers (thanks to Lesterpig)!

...and of course entered Beta in October!

Spec compliance

Of course, Dendrite also needs to be able to fulfill the promise of being a usable next-generation Matrix homeserver at the same time as being a sci-fi development platform. We have spent much of the last year working specifically on this. Today, Dendrite's Sytest compliance sits at:

• 59% compliance for Client-Server APIs, up from 33% in May 2020
• 83% compliance for Server-Server APIs, up from 26% in May 2020

As you can see, these are significant leaps in the numbers of tests passing against Dendrite.

Experimental MSCs

We have been increasingly trying to use Dendrite for the development and testing of some new Matrix feature proposals. Recently we've seen early support added for Peeking (MSC2753) and there is work in progress on Peeking over Federation (MSC2444).

Peeking enables temporarily subscribing to a room for real-time events without joining the room. This will only be possible with rooms that are world-readable, but it reduces the overhead of looking into a room significantly as there is no need to update the room state for each peeking user/device.

In addition to that, we've also been working on Threading (MSC2836) support, which is the gateway to building some pretty new and interesting Matrix experiences. Twitter-like or Reddit-like social prototypes like this have traditionally been difficult to build on top of Matrix as the m.reference relation type from MSC1849 had never really been fleshed out.

Threading adds m.relationship fields for embedding these relationships, and also specifies an additional /event_relationships API endpoint for finding other events related to a given event in either direction. This makes it possible to build threads.

P2P Matrix

Dendrite has also been our primary development platform for P2P Matrix. This year we have released multiple P2P Matrix demos, including:

• p2p.riot.im, which uses libp2p rendezvous and embeds a full Dendrite homeserver into an in-browser Service Worker
• Element iOS P2P, available on TestFlight, which embeds a full Dendrite homeserver into the Element iOS app, initially using QUIC over Yggdrasil as a transport for federation traffic, but with more recent versions using QUIC over the experimental Pinecone protocol
• dendrite-demo-libp2p, a standalone binary which formed the basis of the FOSDEM 2020 P2P demo, using libp2p and local multicast discovery as a transport for federation traffic
• dendrite-demo-yggdrasil, another standalone binary like above, but using QUIC over Yggdrasil connectivity instead of libp2p as the transport for federation traffic

Each experiment teaches us more about potential issues that need to be resolved in order to bring P2P Matrix closer to being reality, and we are continuing to use Dendrite for this work. We'll be announcing more information in the New Year about our latest efforts and the Pinecone routing scheme that we are developing.

It's also worth highlighting that all of the other experimental work taking place right now, including Threading and Peeking, also work over P2P!

What's next

We'll be taking a short break for Christmas, but will then be continuing work on Dendrite in 2021, with the main aims being to add new features, improve spec compliance further, fix bugs and eventually exit beta. We'll also be continuing further experimental work in the P2P and Threading areas, as well as supporting the development of new MSCs such as Portable Identities (MSC2787).

We'd like to say thank you for the community support and interest, and also to send out a special thanks to our community contributors who have contributed a number of fixes and features in recent months! We always welcome code contributions via GitHub if you are an interested developer.

As always, stay tuned for more Dendrite updates either by joining us in #dendrite:matrix.org or by getting your regular dose of This Week in Matrix!

— Neil Alexander and Kegan