Merge branch 'master' into matthew/peeking-over-fed

.github/ISSUE_TEMPLATE/BUG_REPORT.md

@@ -0,0 +1,49 @@
+---
+name: Bug report
+about: Create a report to help us improve
+
+---
+
+<!-- All bug reports must provide the following background information -->
+
+### Background information
+
+- **Dendrite version or git SHA**: 
+- **Monolith or Polylith?**: 
+- **SQLite3 or Postgres?**: 
+- **Running in Docker?**: 
+- **`go version`**: 
+
+<!--
+
+This is a bug report template. By following the instructions below and
+filling out the sections with your information, you will help the us to get all
+the necessary data to fix your issue.
+
+You can also preview your report before submitting it. You may remove sections
+that aren't relevant to your particular case.
+
+Text between <!-- and --​> marks will be invisible in the report.
+
+-->
+
+### Description
+
+<!-- Describe here the problem that you are experiencing -->
+
+### Steps to reproduce
+
+- list the steps
+- that reproduce the bug
+- using hyphens as bullet points
+
+<!--
+Describe how what happens differs from what you expected.
+
+If you can identify any relevant log snippets from server logs, please include
+those (please be careful to remove any personal or private data). Please surround them with
+``` (three backticks, on a line on their own), so that they are formatted legibly.
+
+Alternatively, please send logs to @kegan:matrix.org or @neilalexander:matrix.org
+with a link to the respective Github issue, thanks!
+-->

.github/ISSUE_TEMPLATE/FEATURE_REQUEST.md

@@ -0,0 +1,14 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+
+---
+
+<!--
+Please do not open feature requests for missing parts of the Matrix specification.
+We are tracking those features under https://github.com/matrix-org/dendrite/issues?q=is%3Aissue+is%3Aopen+label%3Aare-we-synapse-yet
+-->
+
+**Description:**
+
+<!-- Describe here the feature you are requesting. -->

.github/PULL_REQUEST_TEMPLATE.md

@@ -2,5 +2,7 @@
 
 <!-- Please read CONTRIBUTING.md before submitting your pull request -->
 
-* [ ] I have added any new tests that need to pass to `testfile` as specified in [docs/sytest.md](https://github.com/matrix-org/dendrite/blob/master/docs/sytest.md)
+* [ ] I have added any new tests that need to pass to `sytest-whitelist` as specified in [docs/sytest.md](https://github.com/matrix-org/dendrite/blob/master/docs/sytest.md)
 * [ ] Pull request includes a [sign off](https://github.com/matrix-org/dendrite/blob/master/docs/CONTRIBUTING.md#sign-off)
+
+Signed-off-by: `Your Name <your@email.example.org>`

.github/workflows/codeql-analysis.yml

@@ -0,0 +1,34 @@
+name: "CodeQL"
+
+on:
+  push:
+    branches: [master]
+  pull_request:
+    branches: [master]
+
+jobs:
+  analyze:
+    name: Analyze
+    runs-on: ubuntu-latest
+
+    strategy:
+      fail-fast: false
+      matrix:
+        language: ['go']
+
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v2
+      with:
+        fetch-depth: 2
+
+    - run: git checkout HEAD^2
+      if: ${{ github.event_name == 'pull_request' }}
+
+    - name: Initialize CodeQL
+      uses: github/codeql-action/init@v1
+      with:
+        languages: ${{ matrix.language }}
+
+    - name: Perform CodeQL Analysis
+      uses: github/codeql-action/analyze@v1

CHANGES.md

@@ -0,0 +1,149 @@
+# Changelog
+
+## Dendrite 0.2.0 (2020-10-20)
+
+### Important
+
+* This release makes breaking changes for polylith deployments, since they now use the multi-personality binary rather than separate binary files
+  * Users of polylith deployments should revise their setups to use the new binary - see the Features section below
+* This release also makes breaking changes for Docker deployments, as are now publishing images to Docker Hub in separate repositories for monolith and polylith
+  * New repositories are as follows: [matrixdotorg/dendrite-monolith](https://hub.docker.com/repository/docker/matrixdotorg/dendrite-monolith) and [matrixdotorg/dendrite-polylith](https://hub.docker.com/repository/docker/matrixdotorg/dendrite-polylith)
+  * The new `latest` tag will be updated with the latest release, and new versioned tags, e.g. `v0.2.0`, will preserve specific release versions
+  * [Sample Compose configs](https://github.com/matrix-org/dendrite/tree/master/build/docker) have been updated - if you are running a Docker deployment, please review the changes
+  * Images for the client API proxy and federation API proxy are no longer provided as they are unsupported - please use [nginx](docs/nginx/) (or another reverse proxy) instead
+
+### Features
+
+* Dendrite polylith deployments now use a special multi-personality binary, rather than separate binaries
+  * This is cleaner, builds faster and simplifies deployment
+  * The first command line argument states the component to run, e.g. `./dendrite-polylith-multi roomserver`
+* Database migrations are now run at startup
+* Invalid UTF-8 in requests is now rejected (contributed by [Pestdoktor](https://github.com/Pestdoktor))
+* Fully read markers are now implemented in the client API (contributed by [Lesterpig](https://github.com/Lesterpig))
+* Missing auth events are now retrieved from other servers in the room, rather than just the event origin
+* `m.room.create` events are now validated properly when processing a `/send_join` response
+* The roomserver now implements `KindOld` for handling historic events without them becoming forward extremity candidates, i.e. for backfilled or missing events
+
+### Fixes
+
+* State resolution v2 performance has been improved dramatically when dealing with large state sets
+* The roomserver no longer processes outlier events if they are already known
+* A SQLite locking issue in the previous events updater has been fixed
+* The client API `/state` endpoint now correctly returns state after the leave event, if the user has left the room
+* The client API `/createRoom` endpoint now sends cumulative state to the roomserver for the initial room events
+* The federation API `/send` endpoint now correctly requests the entire room state from the roomserver when needed
+* Some internal HTTP API paths have been fixed in the user API (contributed by [S7evinK](https://github.com/S7evinK))
+* A race condition in the rate limiting code resulting in concurrent map writes has been fixed
+* Each component now correctly starts a consumer/producer connection in monolith mode (when using Kafka)
+* State resolution is no longer run for single trusted state snapshots that have been verified before
+* A crash when rolling back the transaction in the latest events updater has been fixed
+* Typing events are now ignored when the sender domain does not match the origin server
+* Duplicate redaction entries no longer result in database errors
+* Recursion has been removed from the code path for retrieving missing events
+* `QueryMissingAuthPrevEvents` now returns events that have no associated state as if they are missing
+* Signing key fetchers no longer ignore keys for the local domain, if retrieving a key that is not known in the local config
+* Federation timeouts have been adjusted so we don't give up on remote requests so quickly
+* `create-account` no longer relies on the device database (contributed by [ThatNerdyPikachu](https://github.com/ThatNerdyPikachu))
+
+### Known issues
+
+* Old events can incorrectly appear in `/sync` as if they are new when retrieving missing events from federated servers, causing them to appear at the bottom of the timeline in clients
+
+## Dendrite 0.1.0 (2020-10-08)
+
+First versioned release of Dendrite.
+
+## Client-Server API Features
+
+### Account registration and management
+
+* Registration: By password only.
+* Login: By password only. No fallback.
+* Logout: Yes.
+* Change password: Yes.
+* Link email/msisdn to account: No.
+* Deactivate account: Yes.
+* Check if username is available: Yes.
+* Account data: Yes.
+* OpenID: No.
+
+### Rooms
+
+* Room creation: Yes, including presets.
+* Joining rooms: Yes, including by alias or `?server_name=`.
+* Event sending: Yes, including transaction IDs.
+* Aliases: Yes.
+* Published room directory: Yes.
+* Kicking users: Yes.
+* Banning users: Yes.
+* Inviting users: Yes, but not third-party invites.
+* Forgetting rooms: No.
+* Room versions: All (v1 * v6)
+* Tagging: Yes.
+
+### User management
+
+* User directory: Basic support.
+* Ignoring users: No.
+* Groups/Communities: No.
+
+### Device management
+
+* Creating devices: Yes.
+* Deleting devices: Yes.
+* Send-to-device messaging: Yes.
+
+### Sync
+
+* Filters: Timeline limit only. Rest unimplemented.
+* Deprecated `/events` and `/initialSync`: No.
+
+### Room events
+
+* Typing: Yes.
+* Receipts: No.
+* Read Markers: No.
+* Presence: No.
+* Content repository (attachments): Yes.
+* History visibility: No, defaults to `joined`.
+* Push notifications: No.
+* Event context: No.
+* Reporting content: No.
+
+### End-to-End Encryption
+
+* Uploading device keys: Yes.
+* Downloading device keys: Yes.
+* Claiming one-time keys: Yes.
+* Querying key changes: Yes.
+* Cross-Signing: No.
+
+### Misc
+
+* Server-side search: No.
+* Guest access: Partial.
+* Room previews: No, partial support for Peeking via MSC2753.
+* Third-Party networks: No.
+* Server notices: No.
+* Policy lists: No.
+
+## Federation Features
+
+* Querying keys (incl. notary): Yes.
+* Server ACLs: Yes.
+* Sending transactions: Yes.
+* Joining rooms: Yes.
+* Inviting to rooms: Yes, but not third-party invites.
+* Leaving rooms: Yes.
+* Content repository: Yes.
+* Backfilling / get_missing_events: Yes.
+* Retrieving state of the room (`/state` and `/state_ids`): Yes.
+* Public rooms: Yes.
+* Querying profile data: Yes.
+* Device management: Yes.
+* Send-to-Device messaging: Yes.
+* Querying/Claiming E2E Keys: Yes.
+* Typing: Yes.
+* Presence: No.
+* Receipts: No.
+* OpenID: No.

README.md

@@ -1,13 +1,55 @@
-# Dendrite [![Build Status](https://badge.buildkite.com/4be40938ab19f2bbc4a6c6724517353ee3ec1422e279faf374.svg?branch=master)](https://buildkite.com/matrix-dot-org/dendrite) [![Dendrite Dev on Matrix](https://img.shields.io/matrix/dendrite-dev:matrix.org.svg?label=%23dendrite-dev%3Amatrix.org&logo=matrix&server_fqdn=matrix.org)](https://matrix.to/#/#dendrite-dev:matrix.org) [![Dendrite on Matrix](https://img.shields.io/matrix/dendrite:matrix.org.svg?label=%23dendrite%3Amatrix.org&logo=matrix&server_fqdn=matrix.org)](https://matrix.to/#/#dendrite:matrix.org)
+# Dendrite [![Build Status](https://badge.buildkite.com/4be40938ab19f2bbc4a6c6724517353ee3ec1422e279faf374.svg?branch=master)](https://buildkite.com/matrix-dot-org/dendrite) [![Dendrite](https://img.shields.io/matrix/dendrite:matrix.org.svg?label=%23dendrite%3Amatrix.org&logo=matrix&server_fqdn=matrix.org)](https://matrix.to/#/#dendrite:matrix.org) [![Dendrite Dev](https://img.shields.io/matrix/dendrite-dev:matrix.org.svg?label=%23dendrite-dev%3Amatrix.org&logo=matrix&server_fqdn=matrix.org)](https://matrix.to/#/#dendrite-dev:matrix.org)
 
-Dendrite is a second-generation Matrix homeserver written in Go. It is not recommended to use Dendrite as
-a production homeserver at this time as there is no stable release.
+Dendrite is a second-generation Matrix homeserver written in Go.
+It intends to provide an **efficient**, **reliable** and **scalable** alternative to Synapse:
+ - Efficient: A small memory footprint with better baseline performance than an out-of-the-box Synapse.
+ - Reliable: Implements the Matrix specification as written, using the
+   [same test suite](https://github.com/matrix-org/sytest) as Synapse as well as
+   a [brand new Go test suite](https://github.com/matrix-org/complement).
+ - Scalable: can run on multiple machines and eventually scale to massive homeserver deployments.
 
-Dendrite will start to receive versioned releases stable enough to run [once we enter beta](https://github.com/matrix-org/dendrite/milestone/8).
 
-# Quick start
+As of October 2020, Dendrite has now entered **beta** which means:
+- Dendrite is ready for early adopters. We recommend running in Monolith mode with a PostgreSQL database.
+- Dendrite has periodic semver releases. We intend to release new versions as we land significant features.
+- Dendrite supports database schema upgrades between releases. This means you should never lose your messages when upgrading Dendrite.
+- Breaking changes will not occur on minor releases. This means you can safely upgrade Dendrite without modifying your database or config file.
 
-Requires Go 1.13+ and SQLite3 (Postgres is also supported):
+This does not mean:
+ - Dendrite is bug-free. It has not yet been battle-tested in the real world and so will be error prone initially.
+ - All of the CS/Federation APIs are implemented. We are tracking progress via a script called 'Are We Synapse Yet?'. In particular,
+   read receipts, presence and push notifications are entirely missing from Dendrite. See [CHANGES.md](CHANGES.md) for updates.
+ - Dendrite is ready for massive homeserver deployments. You cannot shard each microservice, only run each one on a different machine.
+
+Currently, we expect Dendrite to function well for small (10s/100s of users) homeserver deployments as well as P2P Matrix nodes in-browser or on mobile devices.
+In the future, we will be able to scale up to gigantic servers (equivalent to matrix.org) via polylith mode. 
+
+Join us in:
+
+- **[#dendrite:matrix.org](https://matrix.to/#/#dendrite:matrix.org)** - General chat about the Dendrite project, for users and server admins alike
+- **[#dendrite-dev:matrix.org](https://matrix.to/#/#dendrite-dev:matrix.org)** - The place for developers, where all Dendrite development discussion happens
+- **[#dendrite-alerts:matrix.org](https://matrix.to/#/#dendrite-alerts:matrix.org)** - Release notifications and important info, highly recommended for all Dendrite server admins
+
+## Requirements
+
+To build Dendrite, you will need Go 1.13 or later. 
+
+For a usable federating Dendrite deployment, you will also need:
+- A domain name (or subdomain) 
+- A valid TLS certificate issued by a trusted authority for that domain
+- SRV records or a well-known file pointing to your deployment
+
+Also recommended are:
+- A PostgreSQL database engine, which will perform better than SQLite with many users and/or larger rooms
+- A reverse proxy server, such as nginx, configured [like this sample](https://github.com/matrix-org/dendrite/blob/master/docs/nginx/monolith-sample.conf)
+
+The [Federation Tester](https://federationtester.matrix.org) can be used to verify your deployment.
+
+## Get started
+
+If you wish to build a fully-federating Dendrite instance, see [INSTALL.md](docs/INSTALL.md). For running in Docker, see [build/docker](build/docker).
+
+The following instructions are enough to get Dendrite started as a non-federating test deployment using self-signed certificates and SQLite databases:
 
 ```bash
 $ git clone https://github.com/matrix-org/dendrite
@@ -27,14 +69,13 @@ $ go build ./cmd/dendrite-monolith-server
 $ ./dendrite-monolith-server --tls-cert server.crt --tls-key server.key --config dendrite.yaml
 ```
 
-Then point your favourite Matrix client at `http://localhost:8008`. For full installation information, see
-[INSTALL.md](docs/INSTALL.md). For running in Docker, see [build/docker](build/docker).
+Then point your favourite Matrix client at `http://localhost:8008`.
 
-# Progress
+## Progress
 
 We use a script called Are We Synapse Yet which checks Sytest compliance rates. Sytest is a black-box homeserver
 test rig with around 900 tests. The script works out how many of these tests are passing on Dendrite and it
-updates with CI. As of August 2020 we're at around 52% CS API coverage and 65% Federation coverage, though check
+updates with CI. As of October 2020 we're at around 56% CS API coverage and 77% Federation coverage, though check
 CI for the latest numbers. In practice, this means you can communicate locally and via federation with Synapse
 servers such as matrix.org reasonably well. There's a long list of features that are not implemented, notably:
  - Receipts
@@ -59,7 +100,7 @@ This means Dendrite supports amongst others:
  - E2E keys and device lists
 
 
-# Contributing
+## Contributing
 
 We would be grateful for any help on issues marked as
 [Are We Synapse Yet](https://github.com/matrix-org/dendrite/labels/are-we-synapse-yet). These issues
@@ -101,7 +142,7 @@ look for [Good First Issues](https://github.com/matrix-org/dendrite/labels/good%
 familiar with the project, look for [Help Wanted](https://github.com/matrix-org/dendrite/labels/help-wanted)
 issues.
 
-# Hardware requirements
+## Hardware requirements
 
 Dendrite in Monolith + SQLite works in a range of environments including iOS and in-browser via WASM.
 
@@ -112,12 +153,3 @@ encrypted rooms:
  - CPU: Brief spikes when processing events, typically idles at 1% CPU.
 
 This means Dendrite should comfortably work on things like Raspberry Pis.
-
-# Discussion
-
-For questions about Dendrite we have a dedicated room on Matrix
-[#dendrite:matrix.org](https://matrix.to/#/#dendrite:matrix.org). Development
-discussion should happen in
-[#dendrite-dev:matrix.org](https://matrix.to/#/#dendrite-dev:matrix.org).
-
-

appservice/appservice.go

@@ -30,6 +30,7 @@ import (
 	"github.com/matrix-org/dendrite/appservice/workers"
 	"github.com/matrix-org/dendrite/internal/config"
 	"github.com/matrix-org/dendrite/internal/setup"
+	"github.com/matrix-org/dendrite/internal/setup/kafka"
 	roomserverAPI "github.com/matrix-org/dendrite/roomserver/api"
 	userapi "github.com/matrix-org/dendrite/userapi/api"
 	"github.com/sirupsen/logrus"
@@ -47,6 +48,8 @@ func NewInternalAPI(
 	userAPI userapi.UserInternalAPI,
 	rsAPI roomserverAPI.RoomserverInternalAPI,
 ) appserviceAPI.AppServiceQueryAPI {
+	consumer, _ := kafka.SetupConsumerProducer(&base.Cfg.Global.Kafka)
+
 	// Create a connection to the appservice postgres DB
 	appserviceDB, err := storage.NewDatabase(&base.Cfg.AppServiceAPI.Database)
 	if err != nil {
@@ -86,7 +89,7 @@ func NewInternalAPI(
 	// We can't add ASes at runtime so this is safe to do.
 	if len(workerStates) > 0 {
 		consumer := consumers.NewOutputRoomEventConsumer(
-			base.Cfg, base.KafkaConsumer, appserviceDB,
+			base.Cfg, consumer, appserviceDB,
 			rsAPI, workerStates,
 		)
 		if err := consumer.Start(); err != nil {

build.sh

@@ -3,10 +3,19 @@
 # Put installed packages into ./bin
 export GOBIN=$PWD/`dirname $0`/bin
 
-export BRANCH=`(git symbolic-ref --short HEAD | cut -d'/' -f 3 )|| ""`
-export BUILD=`git rev-parse --short HEAD || ""`
+if [ -d ".git" ] 
+then
+    export BUILD=`git rev-parse --short HEAD || ""`
+    export BRANCH=`(git symbolic-ref --short HEAD | tr -d \/ ) || ""`
+    if [[ $BRANCH == "master" ]]
+    then
+        export BRANCH=""
+    fi
 
-export FLAGS="-X github.com/matrix-org/dendrite/internal.branch=$BRANCH -X github.com/matrix-org/dendrite/internal.build=$BUILD"
+    export FLAGS="-X github.com/matrix-org/dendrite/internal.branch=$BRANCH -X github.com/matrix-org/dendrite/internal.build=$BUILD"
+else
+    export FLAGS=""
+fi
 
 go install -trimpath -ldflags "$FLAGS" -v $PWD/`dirname $0`/cmd/...
 

build/docker/Dockerfile

@@ -1,4 +1,4 @@
-FROM docker.io/golang:1.13.7-alpine3.11 AS builder
+FROM docker.io/golang:1.15-alpine AS builder
 
 RUN apk --update --no-cache add bash build-base
 

build/docker/Dockerfile.component

@@ -1,13 +0,0 @@
-FROM matrixdotorg/dendrite:latest AS base
-
-FROM alpine:latest
-
-ARG component=monolith
-ENV entrypoint=${component}
-
-COPY --from=base /build/bin/${component} /usr/bin
-
-VOLUME /etc/dendrite
-WORKDIR /etc/dendrite
-
-ENTRYPOINT /usr/bin/${entrypoint} $@

build/docker/Dockerfile.monolith

@@ -0,0 +1,13 @@
+FROM matrixdotorg/dendrite:latest AS base
+
+FROM alpine:latest
+
+COPY --from=base /build/bin/dendrite-monolith-server /usr/bin
+COPY --from=base /build/bin/goose /usr/bin
+COPY --from=base /build/bin/create-account /usr/bin
+COPY --from=base /build/bin/generate-keys /usr/bin
+
+VOLUME /etc/dendrite
+WORKDIR /etc/dendrite
+
+ENTRYPOINT ["/usr/bin/dendrite-monolith-server"]

build/docker/Dockerfile.polylith

@@ -0,0 +1,13 @@
+FROM matrixdotorg/dendrite:latest AS base
+
+FROM alpine:latest
+
+COPY --from=base /build/bin/dendrite-polylith-multi /usr/bin
+COPY --from=base /build/bin/goose /usr/bin
+COPY --from=base /build/bin/create-account /usr/bin
+COPY --from=base /build/bin/generate-keys /usr/bin
+
+VOLUME /etc/dendrite
+WORKDIR /etc/dendrite
+
+ENTRYPOINT ["/usr/bin/dendrite-polylith-multi"]

build/docker/README.md

@@ -38,21 +38,38 @@ go run github.com/matrix-org/dendrite/cmd/generate-keys \
   --tls-key=server.key
 ```
 
-## Starting Dendrite
+## Starting Dendrite as a monolith deployment
 
-Once in place, start the dependencies:
+Create your config based on the `dendrite.yaml` configuration file in the `docker/config`
+folder in the [Dendrite repository](https://github.com/matrix-org/dendrite). Additionally,
+make the following changes to the configuration:
+
+- Enable Naffka: `use_naffka: true`
+
+Once in place, start the PostgreSQL dependency:
 
 ```
-docker-compose -f docker-compose.deps.yml up
+docker-compose -f docker-compose.deps.yml up postgres
 ```
 
-Wait a few seconds for Kafka and Postgres to finish starting up, and then start a monolith:
+Wait a few seconds for PostgreSQL to finish starting up, and then start a monolith:
 
 ```
 docker-compose -f docker-compose.monolith.yml up
 ```
 
-... or start the polylith components:
+## Starting Dendrite as a polylith deployment
+
+Create your config based on the `dendrite.yaml` configuration file in the `docker/config`
+folder in the [Dendrite repository](https://github.com/matrix-org/dendrite).
+
+Once in place, start all the dependencies:
+
+```
+docker-compose -f docker-compose.deps.yml up
+```
+
+Wait a few seconds for PostgreSQL and Kafka to finish starting up, and then start a polylith:
 
 ```
 docker-compose -f docker-compose.polylith.yml up

build/docker/config/dendrite-config.yaml

@@ -38,8 +38,13 @@ global:
   # The path to the signing private key file, used to sign requests and events.
   private_key: matrix_key.pem
 
-  # A unique identifier for this private key. Must start with the prefix "ed25519:".
-  key_id: ed25519:auto
+  # The paths and expiry timestamps (as a UNIX timestamp in millisecond precision)
+  # to old signing private keys that were formerly in use on this domain. These
+  # keys will not be used for federation request or event signing, but will be
+  # provided to any other homeserver that asks when trying to verify old events.
+  # old_private_keys:
+  # - private_key: old_matrix_key.pem
+  #   expired_at: 1601024554498
 
   # How long a remote server can cache our server signing key before requesting it
   # again. Increasing this number will reduce the number of requests made by other
@@ -71,7 +76,7 @@ global:
 
     # Naffka database options. Not required when using Kafka.
     naffka_database:
-      connection_string: file:naffka.db
+      connection_string: postgresql://dendrite:itsasecret@postgres/dendrite_naffka?sslmode=disable
       max_open_conns: 100
       max_idle_conns: 2
       conn_max_lifetime: -1
@@ -133,6 +138,14 @@ client_api:
     turn_username: ""
     turn_password: ""
 
+  # Settings for rate-limited endpoints. Rate limiting will kick in after the
+  # threshold number of "slots" have been taken by requests from a specific 
+  # host. Each "slot" will be released after the cooloff time in milliseconds.
+  rate_limiting:
+    enabled: true
+    threshold: 5
+    cooloff_ms: 500
+
 # Configuration for the EDU server.
 edu_server:
   internal_api:
@@ -240,12 +253,12 @@ room_server:
     conn_max_lifetime: -1
 
 # Configuration for the Server Key API (for server signing keys).
-server_key_api:
+signing_key_server:
   internal_api:
     listen: http://0.0.0.0:7780
-    connect: http://server_key_api:7780
+    connect: http://signing_key_server:7780
   database:
-    connection_string: postgresql://dendrite:itsasecret@postgres/dendrite_serverkey?sslmode=disable
+    connection_string: postgresql://dendrite:itsasecret@postgres/dendrite_signingkeyserver?sslmode=disable
     max_open_conns: 100
     max_idle_conns: 2
     conn_max_lifetime: -1
@@ -260,6 +273,11 @@ server_key_api:
       public_key: Noi6WqcDj0QmPxCNQqgezwTlBKrfqehY1u2FyWP9uYw
     - key_id: ed25519:a_RXGa
       public_key: l8Hft5qXKn1vfHrg3p4+W8gELQVo8N13JkluMfmn2sQ
+      
+  # This option will control whether Dendrite will prefer to look up keys directly
+  # or whether it should try perspective servers first, using direct fetches as a
+  # last resort.
+  prefer_direct_fetch: false
 
 # Configuration for the Sync API.
 sync_api:
@@ -291,6 +309,8 @@ user_api:
     conn_max_lifetime: -1
 
 # Configuration for Opentracing.
+# See https://github.com/matrix-org/dendrite/tree/master/docs/tracing for information on
+# how this works and how to set it up.
 tracing:
   enabled: false
   jaeger:

build/docker/docker-compose.deps.yml

@@ -2,10 +2,13 @@ version: "3.4"
 services:
   postgres:
     hostname: postgres
-    image: postgres:9.5
+    image: postgres:9.6
     restart: always
     volumes:
       - ./postgres/create_db.sh:/docker-entrypoint-initdb.d/20-create_db.sh
+    # To persist your PostgreSQL databases outside of the Docker image, to 
+    # prevent data loss, you will need to add something like this:
+    #  - ./path/to/persistent/storage:/var/lib/postgresql/data
     environment:
       POSTGRES_PASSWORD: itsasecret
       POSTGRES_USER: dendrite
@@ -26,6 +29,8 @@ services:
       KAFKA_ADVERTISED_HOST_NAME: "kafka"
       KAFKA_DELETE_TOPIC_ENABLE: "true"
       KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
+    ports:
+      - 9092:9092
     depends_on:
       - zookeeper
     networks:

build/docker/docker-compose.monolith.yml

@@ -2,9 +2,8 @@ version: "3.4"
 services:
   monolith:
     hostname: monolith
-    image: matrixdotorg/dendrite:monolith
+    image: matrixdotorg/dendrite-monolith:latest
     command: [
-      "--config=dendrite.yaml",
       "--tls-cert=server.crt",
       "--tls-key=server.key"
     ]

build/docker/docker-compose.polylith.yml

@@ -1,43 +1,18 @@
 version: "3.4"
 services:
-  client_api_proxy:
-    hostname: client_api_proxy
-    image: matrixdotorg/dendrite:clientproxy
-    command: [
-      "--bind-address=:8008",
-      "--client-api-server-url=http://client_api:8071",
-      "--sync-api-server-url=http://sync_api:8073",
-      "--media-api-server-url=http://media_api:8074"
-    ]
-    volumes:
-      - ./config:/etc/dendrite
-    networks:
-      - internal
-    depends_on:
-      - sync_api
-      - client_api
-      - media_api
-    ports:
-      - "8008:8008"
-
   client_api:
     hostname: client_api
-    image: matrixdotorg/dendrite:clientapi
-    command: [
-      "--config=dendrite.yaml"
-    ]
+    image: matrixdotorg/dendrite-polylith:latest
+    command: clientapi
     volumes:
       - ./config:/etc/dendrite
-      - room_server
     networks:
       - internal
 
   media_api:
     hostname: media_api
-    image: matrixdotorg/dendrite:mediaapi
-    command: [
-      "--config=dendrite.yaml"
-    ]
+    image: matrixdotorg/dendrite-polylith:latest
+    command: mediaapi
     volumes:
       - ./config:/etc/dendrite
     networks:
@@ -45,10 +20,8 @@ services:
 
   sync_api:
     hostname: sync_api
-    image: matrixdotorg/dendrite:syncapi
-    command: [
-      "--config=dendrite.yaml"
-    ]
+    image: matrixdotorg/dendrite-polylith:latest
+    command: syncapi
     volumes:
       - ./config:/etc/dendrite
     networks:
@@ -56,10 +29,8 @@ services:
 
   room_server:
     hostname: room_server
-    image: matrixdotorg/dendrite:roomserver
-    command: [
-      "--config=dendrite.yaml"
-    ]
+    image: matrixdotorg/dendrite-polylith:latest
+    command: roomserver
     volumes:
       - ./config:/etc/dendrite
     networks:
@@ -67,40 +38,17 @@ services:
 
   edu_server:
     hostname: edu_server
-    image: matrixdotorg/dendrite:eduserver
-    command: [
-      "--config=dendrite.yaml"
-    ]
+    image: matrixdotorg/dendrite-polylith:latest
+    command: eduserver
     volumes:
       - ./config:/etc/dendrite
     networks:
       - internal
 
-  federation_api_proxy:
-    hostname: federation_api_proxy
-    image: matrixdotorg/dendrite:federationproxy
-    command: [
-      "--bind-address=:8448",
-      "--federation-api-url=http://federation_api:8072",
-      "--media-api-server-url=http://media_api:8074"
-    ]
-    volumes:
-      - ./config:/etc/dendrite
-    depends_on:
-      - federation_api
-      - federation_sender
-      - media_api
-    networks:
-      - internal
-    ports:
-      - "8448:8448"
-
   federation_api:
     hostname: federation_api
-    image: matrixdotorg/dendrite:federationapi
-    command: [
-      "--config=dendrite.yaml"
-    ]
+    image: matrixdotorg/dendrite-polylith:latest
+    command: federationapi
     volumes:
       - ./config:/etc/dendrite
     networks:
@@ -108,10 +56,8 @@ services:
 
   federation_sender:
     hostname: federation_sender
-    image: matrixdotorg/dendrite:federationsender
-    command: [
-      "--config=dendrite.yaml"
-    ]
+    image: matrixdotorg/dendrite-polylith:latest
+    command: federationsender
     volumes:
       - ./config:/etc/dendrite
     networks:
@@ -119,21 +65,17 @@ services:
       
   key_server:
     hostname: key_server
-    image: matrixdotorg/dendrite:keyserver
-    command: [
-      "--config=dendrite.yaml"
-    ]
+    image: matrixdotorg/dendrite-polylith:latest
+    command: keyserver
     volumes:
       - ./config:/etc/dendrite
     networks:
         - internal
 
-  server_key_api:
-    hostname: server_key_api
-    image: matrixdotorg/dendrite:serverkeyapi
-    command: [
-      "--config=dendrite.yaml"
-    ]
+  signing_key_server:
+    hostname: signing_key_server
+    image: matrixdotorg/dendrite-polylith:latest
+    command: signingkeyserver
     volumes:
       - ./config:/etc/dendrite
     networks:
@@ -141,10 +83,8 @@ services:
 
   user_api:
     hostname: user_api
-    image: matrixdotorg/dendrite:userapi
-    command: [
-        "--config=dendrite.yaml"
-    ]
+    image: matrixdotorg/dendrite-polylith:latest
+    command: userapi
     volumes:
         - ./config:/etc/dendrite
     networks:
@@ -152,10 +92,8 @@ services:
 
   appservice_api:
     hostname: appservice_api
-    image: matrixdotorg/dendrite:appservice
-    command: [
-      "--config=dendrite.yaml"
-    ]
+    image: matrixdotorg/dendrite-polylith:latest
+    command: appservice
     volumes:
       - ./config:/etc/dendrite
     networks:

build/docker/images-build.sh

@@ -2,20 +2,11 @@
 
 cd $(git rev-parse --show-toplevel)
 
-docker build -f build/docker/Dockerfile -t matrixdotorg/dendrite:latest .
+TAG=${1:-latest}
 
-docker build -t matrixdotorg/dendrite:monolith          --build-arg component=dendrite-monolith-server          -f build/docker/Dockerfile.component .
+echo "Building tag '${TAG}'"
 
-docker build -t matrixdotorg/dendrite:appservice        --build-arg component=dendrite-appservice-server        -f build/docker/Dockerfile.component .
-docker build -t matrixdotorg/dendrite:clientapi         --build-arg component=dendrite-client-api-server        -f build/docker/Dockerfile.component .
-docker build -t matrixdotorg/dendrite:clientproxy       --build-arg component=client-api-proxy                  -f build/docker/Dockerfile.component .
-docker build -t matrixdotorg/dendrite:eduserver         --build-arg component=dendrite-edu-server               -f build/docker/Dockerfile.component .
-docker build -t matrixdotorg/dendrite:federationapi     --build-arg component=dendrite-federation-api-server    -f build/docker/Dockerfile.component .
-docker build -t matrixdotorg/dendrite:federationsender  --build-arg component=dendrite-federation-sender-server -f build/docker/Dockerfile.component .
-docker build -t matrixdotorg/dendrite:federationproxy   --build-arg component=federation-api-proxy              -f build/docker/Dockerfile.component .
-docker build -t matrixdotorg/dendrite:keyserver         --build-arg component=dendrite-key-server               -f build/docker/Dockerfile.component .
-docker build -t matrixdotorg/dendrite:mediaapi          --build-arg component=dendrite-media-api-server         -f build/docker/Dockerfile.component .
-docker build -t matrixdotorg/dendrite:roomserver        --build-arg component=dendrite-room-server              -f build/docker/Dockerfile.component .
-docker build -t matrixdotorg/dendrite:syncapi           --build-arg component=dendrite-sync-api-server          -f build/docker/Dockerfile.component .
-docker build -t matrixdotorg/dendrite:serverkeyapi      --build-arg component=dendrite-server-key-api-server    -f build/docker/Dockerfile.component .
-docker build -t matrixdotorg/dendrite:userapi           --build-arg component=dendrite-user-api-server          -f build/docker/Dockerfile.component .
+docker build -f build/docker/Dockerfile -t matrixdotorg/dendrite:${TAG} .
+
+docker build -t matrixdotorg/dendrite-monolith:${TAG}  -f build/docker/Dockerfile.monolith .
+docker build -t matrixdotorg/dendrite-polylith:${TAG}  -f build/docker/Dockerfile.polylith .

build/docker/images-pull.sh

@@ -1,16 +1,8 @@
 #!/bin/bash
 
-docker pull matrixdotorg/dendrite:monolith
+TAG=${1:-latest}
 
-docker pull matrixdotorg/dendrite:appservice
-docker pull matrixdotorg/dendrite:clientapi
-docker pull matrixdotorg/dendrite:clientproxy
-docker pull matrixdotorg/dendrite:eduserver
-docker pull matrixdotorg/dendrite:federationapi
-docker pull matrixdotorg/dendrite:federationsender
-docker pull matrixdotorg/dendrite:federationproxy
-docker pull matrixdotorg/dendrite:keyserver
-docker pull matrixdotorg/dendrite:mediaapi
-docker pull matrixdotorg/dendrite:roomserver
-docker pull matrixdotorg/dendrite:syncapi
-docker pull matrixdotorg/dendrite:userapi
+echo "Pulling tag '${TAG}'"
+
+docker pull matrixdotorg/dendrite-monolith:${TAG}
+docker pull matrixdotorg/dendrite-polylith:${TAG}

build/docker/images-push.sh

@@ -1,17 +1,8 @@
 #!/bin/bash
 
-docker push matrixdotorg/dendrite:monolith
+TAG=${1:-latest}
 
-docker push matrixdotorg/dendrite:appservice
-docker push matrixdotorg/dendrite:clientapi
-docker push matrixdotorg/dendrite:clientproxy
-docker push matrixdotorg/dendrite:eduserver
-docker push matrixdotorg/dendrite:federationapi
-docker push matrixdotorg/dendrite:federationsender
-docker push matrixdotorg/dendrite:federationproxy
-docker push matrixdotorg/dendrite:keyserver
-docker push matrixdotorg/dendrite:mediaapi
-docker push matrixdotorg/dendrite:roomserver
-docker push matrixdotorg/dendrite:syncapi
-docker push matrixdotorg/dendrite:serverkeyapi
-docker push matrixdotorg/dendrite:userapi
+echo "Pushing tag '${TAG}'"
+
+docker push matrixdotorg/dendrite-monolith:${TAG}
+docker push matrixdotorg/dendrite-polylith:${TAG}

build/docker/postgres/create_db.sh

@@ -1,5 +1,5 @@
-#!/bin/bash
+#!/bin/sh
 
-for db in account device mediaapi syncapi roomserver serverkey keyserver federationsender appservice e2ekey naffka; do
+for db in account device mediaapi syncapi roomserver signingkeyserver keyserver federationsender appservice naffka; do
     createdb -U dendrite -O dendrite dendrite_$db
 done

build/gobind/monolith.go

@@ -88,19 +88,18 @@ func (m *DendriteMonolith) Start() {
 	cfg.Global.PrivateKey = ygg.SigningPrivateKey()
 	cfg.Global.KeyID = gomatrixserverlib.KeyID(signing.KeyID)
 	cfg.Global.Kafka.UseNaffka = true
-	cfg.Global.Kafka.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-naffka.db", m.StorageDirectory))
-	cfg.UserAPI.AccountDatabase.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-account.db", m.StorageDirectory))
-	cfg.UserAPI.DeviceDatabase.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-device.db", m.StorageDirectory))
-	cfg.MediaAPI.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-mediaapi.db", m.StorageDirectory))
-	cfg.SyncAPI.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-syncapi.db", m.StorageDirectory))
-	cfg.RoomServer.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-roomserver.db", m.StorageDirectory))
-	cfg.ServerKeyAPI.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-serverkey.db", m.StorageDirectory))
-	cfg.KeyServer.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-keyserver.db", m.StorageDirectory))
-	cfg.FederationSender.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-federationsender.db", m.StorageDirectory))
-	cfg.AppServiceAPI.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-appservice.db", m.StorageDirectory))
+	cfg.Global.Kafka.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-p2p-naffka.db", m.StorageDirectory))
+	cfg.UserAPI.AccountDatabase.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-p2p-account.db", m.StorageDirectory))
+	cfg.UserAPI.DeviceDatabase.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-p2p-device.db", m.StorageDirectory))
+	cfg.MediaAPI.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-p2p-mediaapi.db", m.StorageDirectory))
+	cfg.SyncAPI.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-p2p-syncapi.db", m.StorageDirectory))
+	cfg.RoomServer.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-p2p-roomserver.db", m.StorageDirectory))
+	cfg.SigningKeyServer.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-p2p-signingkeyserver.db", m.StorageDirectory))
+	cfg.KeyServer.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-p2p-keyserver.db", m.StorageDirectory))
+	cfg.FederationSender.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-p2p-federationsender.db", m.StorageDirectory))
+	cfg.AppServiceAPI.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-p2p-appservice.db", m.StorageDirectory))
 	cfg.MediaAPI.BasePath = config.Path(fmt.Sprintf("%s/tmp", m.StorageDirectory))
 	cfg.MediaAPI.AbsBasePath = config.Path(fmt.Sprintf("%s/tmp", m.StorageDirectory))
-	cfg.FederationSender.FederationMaxRetries = 8
 	if err = cfg.Derive(); err != nil {
 		panic(err)
 	}
@@ -113,7 +112,7 @@ func (m *DendriteMonolith) Start() {
 
 	serverKeyAPI := &signing.YggdrasilKeys{}
 	keyRing := serverKeyAPI.KeyRing()
-	keyAPI := keyserver.NewInternalAPI(&base.Cfg.KeyServer, federation, base.KafkaProducer)
+	keyAPI := keyserver.NewInternalAPI(&base.Cfg.KeyServer, federation)
 	userAPI := userapi.NewInternalAPI(accountDB, &cfg.UserAPI, cfg.Derived.ApplicationServices, keyAPI)
 	keyAPI.SetUserAPI(userAPI)
 
@@ -147,13 +146,11 @@ func (m *DendriteMonolith) Start() {
 	rsAPI.SetFederationSenderAPI(fsAPI)
 
 	monolith := setup.Monolith{
-		Config:        base.Cfg,
-		AccountDB:     accountDB,
-		Client:        ygg.CreateClient(base),
-		FedClient:     federation,
-		KeyRing:       keyRing,
-		KafkaConsumer: base.KafkaConsumer,
-		KafkaProducer: base.KafkaProducer,
+		Config:    base.Cfg,
+		AccountDB: accountDB,
+		Client:    ygg.CreateClient(base),
+		FedClient: federation,
+		KeyRing:   keyRing,
 
 		AppserviceAPI:       asAPI,
 		EDUInternalAPI:      eduInputAPI,

clientapi/auth/user_interactive.go

@@ -103,7 +103,8 @@ type userInteractiveFlow struct {
 // the user already has a valid access token, but we want to double-check
 // that it isn't stolen by re-authenticating them.
 type UserInteractive struct {
-	Flows []userInteractiveFlow
+	Completed []string
+	Flows     []userInteractiveFlow
 	// Map of login type to implementation
 	Types map[string]Type
 	// Map of session ID to completed login types, will need to be extended in future
@@ -117,6 +118,7 @@ func NewUserInteractive(getAccByPass GetAccountByPassword, cfg *config.ClientAPI
 	}
 	// TODO: Add SSO login
 	return &UserInteractive{
+		Completed: []string{},
 		Flows: []userInteractiveFlow{
 			{
 				Stages: []string{typePassword.Name()},
@@ -140,6 +142,7 @@ func (u *UserInteractive) IsSingleStageFlow(authType string) bool {
 
 func (u *UserInteractive) AddCompletedStage(sessionID, authType string) {
 	// TODO: Handle multi-stage flows
+	u.Completed = append(u.Completed, authType)
 	delete(u.Sessions, sessionID)
 }
 
@@ -148,11 +151,13 @@ func (u *UserInteractive) Challenge(sessionID string) *util.JSONResponse {
 	return &util.JSONResponse{
 		Code: 401,
 		JSON: struct {
-			Flows   []userInteractiveFlow `json:"flows"`
-			Session string                `json:"session"`
+			Completed []string              `json:"completed"`
+			Flows     []userInteractiveFlow `json:"flows"`
+			Session   string                `json:"session"`
 			// TODO: Return any additional `params`
 			Params map[string]interface{} `json:"params"`
 		}{
+			u.Completed,
 			u.Flows,
 			sessionID,
 			make(map[string]interface{}),

clientapi/clientapi.go

@@ -15,7 +15,6 @@
 package clientapi
 
 import (
-	"github.com/Shopify/sarama"
 	"github.com/gorilla/mux"
 	appserviceAPI "github.com/matrix-org/dendrite/appservice/api"
 	"github.com/matrix-org/dendrite/clientapi/api"
@@ -24,6 +23,7 @@ import (
 	eduServerAPI "github.com/matrix-org/dendrite/eduserver/api"
 	federationSenderAPI "github.com/matrix-org/dendrite/federationsender/api"
 	"github.com/matrix-org/dendrite/internal/config"
+	"github.com/matrix-org/dendrite/internal/setup/kafka"
 	"github.com/matrix-org/dendrite/internal/transactions"
 	keyserverAPI "github.com/matrix-org/dendrite/keyserver/api"
 	roomserverAPI "github.com/matrix-org/dendrite/roomserver/api"
@@ -36,7 +36,6 @@ import (
 func AddPublicRoutes(
 	router *mux.Router,
 	cfg *config.ClientAPI,
-	producer sarama.SyncProducer,
 	accountsDB accounts.Database,
 	federation *gomatrixserverlib.FederationClient,
 	rsAPI roomserverAPI.RoomserverInternalAPI,
@@ -48,6 +47,8 @@ func AddPublicRoutes(
 	keyAPI keyserverAPI.KeyInternalAPI,
 	extRoomsProvider api.ExtraPublicRoomsProvider,
 ) {
+	_, producer := kafka.SetupConsumerProducer(&cfg.Matrix.Kafka)
+
 	syncProducer := &producers.SyncAPIProducer{
 		Producer: producer,
 		Topic:    cfg.Matrix.Kafka.TopicFor(config.TopicOutputClientData),

clientapi/httputil/httputil.go

@@ -16,7 +16,9 @@ package httputil
 
 import (
 	"encoding/json"
+	"io/ioutil"
 	"net/http"
+	"unicode/utf8"
 
 	"github.com/matrix-org/dendrite/clientapi/jsonerror"
 	"github.com/matrix-org/util"
@@ -25,7 +27,23 @@ import (
 // UnmarshalJSONRequest into the given interface pointer. Returns an error JSON response if
 // there was a problem unmarshalling. Calling this function consumes the request body.
 func UnmarshalJSONRequest(req *http.Request, iface interface{}) *util.JSONResponse {
-	if err := json.NewDecoder(req.Body).Decode(iface); err != nil {
+	// encoding/json allows invalid utf-8, matrix does not
+	// https://matrix.org/docs/spec/client_server/r0.6.1#api-standards
+	body, err := ioutil.ReadAll(req.Body)
+	if err != nil {
+		util.GetLogger(req.Context()).WithError(err).Error("ioutil.ReadAll failed")
+		resp := jsonerror.InternalServerError()
+		return &resp
+	}
+
+	if !utf8.Valid(body) {
+		return &util.JSONResponse{
+			Code: http.StatusBadRequest,
+			JSON: jsonerror.NotJSON("Body contains invalid UTF-8"),
+		}
+	}
+
+	if err := json.Unmarshal(body, iface); err != nil {
 		// TODO: We may want to suppress the Error() return in production? It's useful when
 		// debugging because an error will be produced for both invalid/malformed JSON AND
 		// valid JSON with incorrect types for values.

clientapi/routing/account_data.go

@@ -20,8 +20,10 @@ import (
 	"io/ioutil"
 	"net/http"
 
+	"github.com/matrix-org/dendrite/clientapi/httputil"
 	"github.com/matrix-org/dendrite/clientapi/jsonerror"
 	"github.com/matrix-org/dendrite/clientapi/producers"
+	roomserverAPI "github.com/matrix-org/dendrite/roomserver/api"
 	"github.com/matrix-org/dendrite/userapi/api"
 
 	"github.com/matrix-org/util"
@@ -91,6 +93,13 @@ func SaveAccountData(
 		}
 	}
 
+	if dataType == "m.fully_read" {
+		return util.JSONResponse{
+			Code: http.StatusForbidden,
+			JSON: jsonerror.Forbidden("Unable to set read marker"),
+		}
+	}
+
 	body, err := ioutil.ReadAll(req.Body)
 	if err != nil {
 		util.GetLogger(req.Context()).WithError(err).Error("ioutil.ReadAll failed")
@@ -112,7 +121,7 @@ func SaveAccountData(
 	}
 	dataRes := api.InputAccountDataResponse{}
 	if err := userAPI.InputAccountData(req.Context(), &dataReq, &dataRes); err != nil {
-		util.GetLogger(req.Context()).WithError(err).Error("userAPI.QueryAccountData failed")
+		util.GetLogger(req.Context()).WithError(err).Error("userAPI.InputAccountData failed")
 		return util.ErrorResponse(err)
 	}
 
@@ -127,3 +136,67 @@ func SaveAccountData(
 		JSON: struct{}{},
 	}
 }
+
+type readMarkerJSON struct {
+	FullyRead string `json:"m.fully_read"`
+	Read      string `json:"m.read"`
+}
+
+type fullyReadEvent struct {
+	EventID string `json:"event_id"`
+}
+
+// SaveReadMarker implements POST /rooms/{roomId}/read_markers
+func SaveReadMarker(
+	req *http.Request, userAPI api.UserInternalAPI, rsAPI roomserverAPI.RoomserverInternalAPI,
+	syncProducer *producers.SyncAPIProducer, device *api.Device, roomID string,
+) util.JSONResponse {
+	// Verify that the user is a member of this room
+	resErr := checkMemberInRoom(req.Context(), rsAPI, device.UserID, roomID)
+	if resErr != nil {
+		return *resErr
+	}
+
+	var r readMarkerJSON
+	resErr = httputil.UnmarshalJSONRequest(req, &r)
+	if resErr != nil {
+		return *resErr
+	}
+
+	if r.FullyRead == "" {
+		return util.JSONResponse{
+			Code: http.StatusBadRequest,
+			JSON: jsonerror.BadJSON("Missing m.fully_read mandatory field"),
+		}
+	}
+
+	data, err := json.Marshal(fullyReadEvent{EventID: r.FullyRead})
+	if err != nil {
+		return jsonerror.InternalServerError()
+	}
+
+	dataReq := api.InputAccountDataRequest{
+		UserID:      device.UserID,
+		DataType:    "m.fully_read",
+		RoomID:      roomID,
+		AccountData: data,
+	}
+	dataRes := api.InputAccountDataResponse{}
+	if err := userAPI.InputAccountData(req.Context(), &dataReq, &dataRes); err != nil {
+		util.GetLogger(req.Context()).WithError(err).Error("userAPI.InputAccountData failed")
+		return util.ErrorResponse(err)
+	}
+
+	if err := syncProducer.SendData(device.UserID, roomID, "m.fully_read"); err != nil {
+		util.GetLogger(req.Context()).WithError(err).Error("syncProducer.SendData failed")
+		return jsonerror.InternalServerError()
+	}
+
+	// TODO handle the read receipt that may be included in the read marker
+	// See https://matrix.org/docs/spec/client_server/r0.6.0#post-matrix-client-r0-rooms-roomid-read-markers
+
+	return util.JSONResponse{
+		Code: http.StatusOK,
+		JSON: struct{}{},
+	}
+}

clientapi/routing/capabilities.go

@@ -23,8 +23,8 @@ import (
 	"github.com/matrix-org/util"
 )
 
-// SendMembership implements PUT /rooms/{roomID}/(join|kick|ban|unban|leave|invite)
-// by building a m.room.member event then sending it to the room server
+// GetCapabilities returns information about the server's supported feature set
+// and other relevant capabilities to an authenticated user.
 func GetCapabilities(
 	req *http.Request, rsAPI roomserverAPI.RoomserverInternalAPI,
 ) util.JSONResponse {
@@ -41,6 +41,9 @@ func GetCapabilities(
 
 	response := map[string]interface{}{
 		"capabilities": map[string]interface{}{
+			"m.change_password": map[string]bool{
+				"enabled": true,
+			},
 			"m.room_versions": roomVersionsQueryRes,
 		},
 	}

clientapi/routing/createroom.go

@@ -339,12 +339,22 @@ func createRoom(
 			util.GetLogger(req.Context()).WithError(err).Error("authEvents.AddEvent failed")
 			return jsonerror.InternalServerError()
 		}
-	}
 
-	// send events to the room server
-	if err = roomserverAPI.SendEvents(req.Context(), rsAPI, builtEvents, cfg.Matrix.ServerName, nil); err != nil {
-		util.GetLogger(req.Context()).WithError(err).Error("SendEvents failed")
-		return jsonerror.InternalServerError()
+		accumulated := gomatrixserverlib.UnwrapEventHeaders(builtEvents)
+		if err = roomserverAPI.SendEventWithState(
+			req.Context(),
+			rsAPI,
+			roomserverAPI.KindNew,
+			&gomatrixserverlib.RespState{
+				StateEvents: accumulated,
+				AuthEvents:  accumulated,
+			},
+			ev.Headered(roomVersion),
+			nil,
+		); err != nil {
+			util.GetLogger(req.Context()).WithError(err).Error("SendEventWithState failed")
+			return jsonerror.InternalServerError()
+		}
 	}
 
 	// TODO(#269): Reserve room alias while we create the room. This stops us

clientapi/routing/deactivate.go

@@ -0,0 +1,55 @@
+package routing
+
+import (
+	"io/ioutil"
+	"net/http"
+
+	"github.com/matrix-org/dendrite/clientapi/auth"
+	"github.com/matrix-org/dendrite/clientapi/jsonerror"
+	"github.com/matrix-org/dendrite/userapi/api"
+	"github.com/matrix-org/gomatrixserverlib"
+	"github.com/matrix-org/util"
+)
+
+// Deactivate handles POST requests to /account/deactivate
+func Deactivate(
+	req *http.Request,
+	userInteractiveAuth *auth.UserInteractive,
+	userAPI api.UserInternalAPI,
+	deviceAPI *api.Device,
+) util.JSONResponse {
+	ctx := req.Context()
+	defer req.Body.Close() // nolint:errcheck
+	bodyBytes, err := ioutil.ReadAll(req.Body)
+	if err != nil {
+		return util.JSONResponse{
+			Code: http.StatusBadRequest,
+			JSON: jsonerror.BadJSON("The request body could not be read: " + err.Error()),
+		}
+	}
+
+	login, errRes := userInteractiveAuth.Verify(ctx, bodyBytes, deviceAPI)
+	if errRes != nil {
+		return *errRes
+	}
+
+	localpart, _, err := gomatrixserverlib.SplitID('@', login.User)
+	if err != nil {
+		util.GetLogger(req.Context()).WithError(err).Error("gomatrixserverlib.SplitID failed")
+		return jsonerror.InternalServerError()
+	}
+
+	var res api.PerformAccountDeactivationResponse
+	err = userAPI.PerformAccountDeactivation(ctx, &api.PerformAccountDeactivationRequest{
+		Localpart: localpart,
+	}, &res)
+	if err != nil {
+		util.GetLogger(ctx).WithError(err).Error("userAPI.PerformAccountDeactivation failed")
+		return jsonerror.InternalServerError()
+	}
+
+	return util.JSONResponse{
+		Code: http.StatusOK,
+		JSON: struct{}{},
+	}
+}

clientapi/routing/device.go

@@ -15,11 +15,11 @@
 package routing
 
 import (
-	"encoding/json"
 	"io/ioutil"
 	"net/http"
 
 	"github.com/matrix-org/dendrite/clientapi/auth"
+	"github.com/matrix-org/dendrite/clientapi/httputil"
 	"github.com/matrix-org/dendrite/clientapi/jsonerror"
 	"github.com/matrix-org/dendrite/userapi/api"
 	userapi "github.com/matrix-org/dendrite/userapi/api"
@@ -121,9 +121,8 @@ func UpdateDeviceByID(
 
 	payload := deviceUpdateJSON{}
 
-	if err := json.NewDecoder(req.Body).Decode(&payload); err != nil {
-		util.GetLogger(req.Context()).WithError(err).Error("json.NewDecoder.Decode failed")
-		return jsonerror.InternalServerError()
+	if resErr := httputil.UnmarshalJSONRequest(req, &payload); resErr != nil {
+		return *resErr
 	}
 
 	var performRes api.PerformDeviceUpdateResponse
@@ -211,9 +210,8 @@ func DeleteDevices(
 	ctx := req.Context()
 	payload := devicesDeleteJSON{}
 
-	if err := json.NewDecoder(req.Body).Decode(&payload); err != nil {
-		util.GetLogger(ctx).WithError(err).Error("json.NewDecoder.Decode failed")
-		return jsonerror.InternalServerError()
+	if resErr := httputil.UnmarshalJSONRequest(req, &payload); resErr != nil {
+		return *resErr
 	}
 
 	defer req.Body.Close() // nolint: errcheck

clientapi/routing/joinroom.go

@@ -16,9 +16,11 @@ package routing
 
 import (
 	"net/http"
+	"time"
 
 	"github.com/matrix-org/dendrite/clientapi/auth/authtypes"
 	"github.com/matrix-org/dendrite/clientapi/httputil"
+	"github.com/matrix-org/dendrite/clientapi/jsonerror"
 	roomserverAPI "github.com/matrix-org/dendrite/roomserver/api"
 	"github.com/matrix-org/dendrite/userapi/api"
 	"github.com/matrix-org/dendrite/userapi/storage/accounts"
@@ -74,16 +76,32 @@ func JoinRoomByIDOrAlias(
 	}
 
 	// Ask the roomserver to perform the join.
-	rsAPI.PerformJoin(req.Context(), &joinReq, &joinRes)
-	if joinRes.Error != nil {
-		return joinRes.Error.JSONResponse()
-	}
+	done := make(chan util.JSONResponse, 1)
+	go func() {
+		defer close(done)
+		rsAPI.PerformJoin(req.Context(), &joinReq, &joinRes)
+		if joinRes.Error != nil {
+			done <- joinRes.Error.JSONResponse()
+		} else {
+			done <- util.JSONResponse{
+				Code: http.StatusOK,
+				// TODO: Put the response struct somewhere internal.
+				JSON: struct {
+					RoomID string `json:"room_id"`
+				}{joinRes.RoomID},
+			}
+		}
+	}()
 
-	return util.JSONResponse{
-		Code: http.StatusOK,
-		// TODO: Put the response struct somewhere internal.
-		JSON: struct {
-			RoomID string `json:"room_id"`
-		}{joinRes.RoomID},
+	// Wait either for the join to finish, or for us to hit a reasonable
+	// timeout, at which point we'll just return a 200 to placate clients.
+	select {
+	case <-time.After(time.Second * 20):
+		return util.JSONResponse{
+			Code: http.StatusAccepted,
+			JSON: jsonerror.Unknown("The room join will continue in the background."),
+		}
+	case result := <-done:
+		return result
 	}
 }

clientapi/routing/login.go

@@ -79,7 +79,7 @@ func Login(
 			return *authErr
 		}
 		// make a device/access token
-		return completeAuth(req.Context(), cfg.Matrix.ServerName, userAPI, login)
+		return completeAuth(req.Context(), cfg.Matrix.ServerName, userAPI, login, req.RemoteAddr, req.UserAgent())
 	}
 	return util.JSONResponse{
 		Code: http.StatusMethodNotAllowed,
@@ -89,6 +89,7 @@ func Login(
 
 func completeAuth(
 	ctx context.Context, serverName gomatrixserverlib.ServerName, userAPI userapi.UserInternalAPI, login *auth.Login,
+	ipAddr, userAgent string,
 ) util.JSONResponse {
 	token, err := auth.GenerateAccessToken()
 	if err != nil {
@@ -108,6 +109,8 @@ func completeAuth(
 		DeviceID:          login.DeviceID,
 		AccessToken:       token,
 		Localpart:         localpart,
+		IPAddr:            ipAddr,
+		UserAgent:         userAgent,
 	}, &performRes)
 	if err != nil {
 		return util.JSONResponse{

clientapi/routing/membership.go

@@ -76,6 +76,7 @@ func sendMembership(ctx context.Context, accountDB accounts.Database, device *us
 
 	if err = roomserverAPI.SendEvents(
 		ctx, rsAPI,
+		api.KindNew,
 		[]gomatrixserverlib.HeaderedEvent{event.Event.Headered(roomVer)},
 		cfg.Matrix.ServerName,
 		nil,

clientapi/routing/profile.go

@@ -170,7 +170,7 @@ func SetAvatarURL(
 		return jsonerror.InternalServerError()
 	}
 
-	if err := api.SendEvents(req.Context(), rsAPI, events, cfg.Matrix.ServerName, nil); err != nil {
+	if err := api.SendEvents(req.Context(), rsAPI, api.KindNew, events, cfg.Matrix.ServerName, nil); err != nil {
 		util.GetLogger(req.Context()).WithError(err).Error("SendEvents failed")
 		return jsonerror.InternalServerError()
 	}
@@ -288,7 +288,7 @@ func SetDisplayName(
 		return jsonerror.InternalServerError()
 	}
 
-	if err := api.SendEvents(req.Context(), rsAPI, events, cfg.Matrix.ServerName, nil); err != nil {
+	if err := api.SendEvents(req.Context(), rsAPI, api.KindNew, events, cfg.Matrix.ServerName, nil); err != nil {
 		util.GetLogger(req.Context()).WithError(err).Error("SendEvents failed")
 		return jsonerror.InternalServerError()
 	}

clientapi/routing/rate_limiting.go

@@ -13,6 +13,7 @@ import (
 type rateLimits struct {
 	limits           map[string]chan struct{}
 	limitsMutex      sync.RWMutex
+	cleanMutex       sync.RWMutex
 	enabled          bool
 	requestThreshold int64
 	cooloffDuration  time.Duration
@@ -38,6 +39,7 @@ func (l *rateLimits) clean() {
 		// empty. If they are then we will close and delete them,
 		// freeing up memory.
 		time.Sleep(time.Second * 30)
+		l.cleanMutex.Lock()
 		l.limitsMutex.Lock()
 		for k, c := range l.limits {
 			if len(c) == 0 {
@@ -46,6 +48,7 @@ func (l *rateLimits) clean() {
 			}
 		}
 		l.limitsMutex.Unlock()
+		l.cleanMutex.Unlock()
 	}
 }
 
@@ -55,12 +58,12 @@ func (l *rateLimits) rateLimit(req *http.Request) *util.JSONResponse {
 		return nil
 	}
 
-	// Lock the map long enough to check for rate limiting. We hold it
-	// for longer here than we really need to but it makes sure that we
-	// also don't conflict with the cleaner goroutine which might clean
-	// up a channel after we have retrieved it otherwise.
-	l.limitsMutex.RLock()
-	defer l.limitsMutex.RUnlock()
+	// Take a read lock out on the cleaner mutex. The cleaner expects to
+	// be able to take a write lock, which isn't possible while there are
+	// readers, so this has the effect of blocking the cleaner goroutine
+	// from doing its work until there are no requests in flight.
+	l.cleanMutex.RLock()
+	defer l.cleanMutex.RUnlock()
 
 	// First of all, work out if X-Forwarded-For was sent to us. If not
 	// then we'll just use the IP address of the caller.
@@ -69,12 +72,19 @@ func (l *rateLimits) rateLimit(req *http.Request) *util.JSONResponse {
 		caller = forwardedFor
 	}
 
-	// Look up the caller's channel, if they have one. If they don't then
-	// let's create one.
+	// Look up the caller's channel, if they have one.
+	l.limitsMutex.RLock()
 	rateLimit, ok := l.limits[caller]
+	l.limitsMutex.RUnlock()
+
+	// If the caller doesn't have a channel, create one and write it
+	// back to the map.
 	if !ok {
-		l.limits[caller] = make(chan struct{}, l.requestThreshold)
-		rateLimit = l.limits[caller]
+		rateLimit = make(chan struct{}, l.requestThreshold)
+
+		l.limitsMutex.Lock()
+		l.limits[caller] = rateLimit
+		l.limitsMutex.Unlock()
 	}
 
 	// Check if the user has got free resource slots for this request.

clientapi/routing/redaction.go

@@ -121,7 +121,7 @@ func SendRedaction(
 			JSON: jsonerror.NotFound("Room does not exist"),
 		}
 	}
-	if err = roomserverAPI.SendEvents(context.Background(), rsAPI, []gomatrixserverlib.HeaderedEvent{*e}, cfg.Matrix.ServerName, nil); err != nil {
+	if err = roomserverAPI.SendEvents(context.Background(), rsAPI, api.KindNew, []gomatrixserverlib.HeaderedEvent{*e}, cfg.Matrix.ServerName, nil); err != nil {
 		util.GetLogger(req.Context()).WithError(err).Errorf("failed to SendEvents")
 		return jsonerror.InternalServerError()
 	}

clientapi/routing/register.go

@@ -543,6 +543,8 @@ func handleGuestRegistration(
 		Localpart:         res.Account.Localpart,
 		DeviceDisplayName: r.InitialDisplayName,
 		AccessToken:       token,
+		IPAddr:            req.RemoteAddr,
+		UserAgent:         req.UserAgent(),
 	}, &devRes)
 	if err != nil {
 		return util.JSONResponse{
@@ -691,7 +693,7 @@ func handleApplicationServiceRegistration(
 	// Don't need to worry about appending to registration stages as
 	// application service registration is entirely separate.
 	return completeRegistration(
-		req.Context(), userAPI, r.Username, "", appserviceID,
+		req.Context(), userAPI, r.Username, "", appserviceID, req.RemoteAddr, req.UserAgent(),
 		r.InhibitLogin, r.InitialDisplayName, r.DeviceID,
 	)
 }
@@ -710,7 +712,7 @@ func checkAndCompleteFlow(
 	if checkFlowCompleted(flow, cfg.Derived.Registration.Flows) {
 		// This flow was completed, registration can continue
 		return completeRegistration(
-			req.Context(), userAPI, r.Username, r.Password, "",
+			req.Context(), userAPI, r.Username, r.Password, "", req.RemoteAddr, req.UserAgent(),
 			r.InhibitLogin, r.InitialDisplayName, r.DeviceID,
 		)
 	}
@@ -762,10 +764,10 @@ func LegacyRegister(
 			return util.MessageResponse(http.StatusForbidden, "HMAC incorrect")
 		}
 
-		return completeRegistration(req.Context(), userAPI, r.Username, r.Password, "", false, nil, nil)
+		return completeRegistration(req.Context(), userAPI, r.Username, r.Password, "", req.RemoteAddr, req.UserAgent(), false, nil, nil)
 	case authtypes.LoginTypeDummy:
 		// there is nothing to do
-		return completeRegistration(req.Context(), userAPI, r.Username, r.Password, "", false, nil, nil)
+		return completeRegistration(req.Context(), userAPI, r.Username, r.Password, "", req.RemoteAddr, req.UserAgent(), false, nil, nil)
 	default:
 		return util.JSONResponse{
 			Code: http.StatusNotImplemented,
@@ -812,7 +814,7 @@ func parseAndValidateLegacyLogin(req *http.Request, r *legacyRegisterRequest) *u
 func completeRegistration(
 	ctx context.Context,
 	userAPI userapi.UserInternalAPI,
-	username, password, appserviceID string,
+	username, password, appserviceID, ipAddr, userAgent string,
 	inhibitLogin eventutil.WeakBoolean,
 	displayName, deviceID *string,
 ) util.JSONResponse {
@@ -880,6 +882,8 @@ func completeRegistration(
 		AccessToken:       token,
 		DeviceDisplayName: displayName,
 		DeviceID:          deviceID,
+		IPAddr:            ipAddr,
+		UserAgent:         userAgent,
 	}, &devRes)
 	if err != nil {
 		return util.JSONResponse{

clientapi/routing/routing.go

@@ -23,6 +23,7 @@ import (
 	appserviceAPI "github.com/matrix-org/dendrite/appservice/api"
 	"github.com/matrix-org/dendrite/clientapi/api"
 	"github.com/matrix-org/dendrite/clientapi/auth"
+	clientutil "github.com/matrix-org/dendrite/clientapi/httputil"
 	"github.com/matrix-org/dendrite/clientapi/jsonerror"
 	"github.com/matrix-org/dendrite/clientapi/producers"
 	eduServerAPI "github.com/matrix-org/dendrite/eduserver/api"
@@ -435,6 +436,15 @@ func Setup(
 		}),
 	).Methods(http.MethodPost, http.MethodOptions)
 
+	r0mux.Handle("/account/deactivate",
+		httputil.MakeAuthAPI("deactivate", userAPI, func(req *http.Request, device *userapi.Device) util.JSONResponse {
+			if r := rateLimits.rateLimit(req); r != nil {
+				return *r
+			}
+			return Deactivate(req, userInteractiveAuth, userAPI, device)
+		}),
+	).Methods(http.MethodPost, http.MethodOptions)
+
 	// Stub endpoints required by Riot
 
 	r0mux.Handle("/login",
@@ -650,8 +660,9 @@ func Setup(
 				SearchString string `json:"search_term"`
 				Limit        int    `json:"limit"`
 			}{}
-			if err := json.NewDecoder(req.Body).Decode(&postContent); err != nil {
-				return util.ErrorResponse(err)
+
+			if resErr := clientutil.UnmarshalJSONRequest(req, &postContent); resErr != nil {
+				return *resErr
 			}
 			return *SearchUserDirectory(
 				req.Context(),
@@ -686,12 +697,15 @@ func Setup(
 	).Methods(http.MethodGet, http.MethodOptions)
 
 	r0mux.Handle("/rooms/{roomID}/read_markers",
-		httputil.MakeExternalAPI("rooms_read_markers", func(req *http.Request) util.JSONResponse {
+		httputil.MakeAuthAPI("rooms_read_markers", userAPI, func(req *http.Request, device *userapi.Device) util.JSONResponse {
 			if r := rateLimits.rateLimit(req); r != nil {
 				return *r
 			}
-			// TODO: return the read_markers.
-			return util.JSONResponse{Code: http.StatusOK, JSON: struct{}{}}
+			vars, err := httputil.URLDecodeMapValues(mux.Vars(req))
+			if err != nil {
+				return util.ErrorResponse(err)
+			}
+			return SaveReadMarker(req, userAPI, rsAPI, syncProducer, device, vars["roomID"])
 		}),
 	).Methods(http.MethodPost, http.MethodOptions)
 
@@ -793,7 +807,7 @@ func Setup(
 			}
 			return GetCapabilities(req, rsAPI)
 		}),
-	).Methods(http.MethodGet)
+	).Methods(http.MethodGet, http.MethodOptions)
 
 	// Supplying a device ID is deprecated.
 	r0mux.Handle("/keys/upload/{deviceID}",

clientapi/routing/sendevent.go

@@ -92,6 +92,7 @@ func SendEvent(
 	// event ID in case of duplicate transaction is discarded
 	if err := api.SendEvents(
 		req.Context(), rsAPI,
+		api.KindNew,
 		[]gomatrixserverlib.HeaderedEvent{
 			e.Headered(verRes.RoomVersion),
 		},

clientapi/routing/state.go

@@ -141,7 +141,7 @@ func OnIncomingStateRequest(ctx context.Context, device *userapi.Device, rsAPI a
 			util.GetLogger(ctx).WithError(err).Error("Failed to QueryMembershipForUser")
 			return jsonerror.InternalServerError()
 		}
-		for _, ev := range stateRes.StateEvents {
+		for _, ev := range stateAfterRes.StateEvents {
 			stateEvents = append(
 				stateEvents,
 				gomatrixserverlib.HeaderedToClientEvent(ev, gomatrixserverlib.FormatAll),

clientapi/threepid/invites.go

@@ -361,6 +361,7 @@ func emit3PIDInviteEvent(
 
 	return api.SendEvents(
 		ctx, rsAPI,
+		api.KindNew,
 		[]gomatrixserverlib.HeaderedEvent{
 			(*event).Headered(queryRes.RoomVersion),
 		},

cmd/create-account/main.go

@@ -22,7 +22,6 @@ import (
 
 	"github.com/matrix-org/dendrite/internal/config"
 	"github.com/matrix-org/dendrite/userapi/storage/accounts"
-	"github.com/matrix-org/dendrite/userapi/storage/devices"
 	"github.com/matrix-org/gomatrixserverlib"
 )
 
@@ -39,7 +38,6 @@ var (
 	username      = flag.String("username", "", "The user ID localpart to register e.g 'alice' in '@alice:localhost'.")
 	password      = flag.String("password", "", "Optional. The password to register with. If not specified, this account will be password-less.")
 	serverNameStr = flag.String("servername", "localhost", "The Matrix server domain which will form the domain part of the user ID.")
-	accessToken   = flag.String("token", "", "Optional. The desired access_token to have. If not specified, a random access_token will be made.")
 )
 
 func main() {
@@ -78,29 +76,5 @@ func main() {
 		os.Exit(1)
 	}
 
-	deviceDB, err := devices.NewDatabase(&config.DatabaseOptions{
-		ConnectionString: config.DataSource(*database),
-	}, serverName)
-	if err != nil {
-		fmt.Println(err.Error())
-		os.Exit(1)
-	}
-
-	if *accessToken == "" {
-		t := "token_" + *username
-		accessToken = &t
-	}
-
-	device, err := deviceDB.CreateDevice(
-		context.Background(), *username, nil, *accessToken, nil,
-	)
-	if err != nil {
-		fmt.Println(err.Error())
-		os.Exit(1)
-	}
-
-	fmt.Println("Created account:")
-	fmt.Printf("user_id      = %s\n", device.UserID)
-	fmt.Printf("device_id    = %s\n", device.ID)
-	fmt.Printf("access_token = %s\n", device.AccessToken)
+	fmt.Println("Created account")
 }

cmd/dendrite-demo-libp2p/main.go

@@ -36,7 +36,7 @@ import (
 	"github.com/matrix-org/dendrite/internal/setup"
 	"github.com/matrix-org/dendrite/keyserver"
 	"github.com/matrix-org/dendrite/roomserver"
-	"github.com/matrix-org/dendrite/serverkeyapi"
+	"github.com/matrix-org/dendrite/signingkeyserver"
 	"github.com/matrix-org/dendrite/userapi"
 	"github.com/matrix-org/gomatrixserverlib"
 
@@ -89,7 +89,7 @@ func createClient(
 		"matrix",
 		p2phttp.NewTransport(base.LibP2P, p2phttp.ProtocolOption("/matrix")),
 	)
-	return gomatrixserverlib.NewClientWithTransport(true, tr)
+	return gomatrixserverlib.NewClientWithTransport(tr)
 }
 
 func main() {
@@ -125,7 +125,7 @@ func main() {
 	cfg.MediaAPI.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s-mediaapi.db", *instanceName))
 	cfg.SyncAPI.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s-syncapi.db", *instanceName))
 	cfg.RoomServer.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s-roomserver.db", *instanceName))
-	cfg.ServerKeyAPI.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s-serverkey.db", *instanceName))
+	cfg.SigningKeyServer.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s-signingkeyserver.db", *instanceName))
 	cfg.FederationSender.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s-federationsender.db", *instanceName))
 	cfg.AppServiceAPI.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s-appservice.db", *instanceName))
 	cfg.Global.Kafka.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s-naffka.db", *instanceName))
@@ -139,12 +139,12 @@ func main() {
 
 	accountDB := base.Base.CreateAccountsDB()
 	federation := createFederationClient(base)
-	keyAPI := keyserver.NewInternalAPI(&base.Base.Cfg.KeyServer, federation, base.Base.KafkaProducer)
+	keyAPI := keyserver.NewInternalAPI(&base.Base.Cfg.KeyServer, federation)
 	userAPI := userapi.NewInternalAPI(accountDB, &cfg.UserAPI, nil, keyAPI)
 	keyAPI.SetUserAPI(userAPI)
 
-	serverKeyAPI := serverkeyapi.NewInternalAPI(
-		&base.Base.Cfg.ServerKeyAPI, federation, base.Base.Caches,
+	serverKeyAPI := signingkeyserver.NewInternalAPI(
+		&base.Base.Cfg.SigningKeyServer, federation, base.Base.Caches,
 	)
 	keyRing := serverKeyAPI.KeyRing()
 	createKeyDB(
@@ -169,13 +169,11 @@ func main() {
 	}
 
 	monolith := setup.Monolith{
-		Config:        base.Base.Cfg,
-		AccountDB:     accountDB,
-		Client:        createClient(base),
-		FedClient:     federation,
-		KeyRing:       keyRing,
-		KafkaConsumer: base.Base.KafkaConsumer,
-		KafkaProducer: base.Base.KafkaProducer,
+		Config:    base.Base.Cfg,
+		AccountDB: accountDB,
+		Client:    createClient(base),
+		FedClient: federation,
+		KeyRing:   keyRing,
 
 		AppserviceAPI:          asAPI,
 		EDUInternalAPI:         eduInputAPI,

cmd/dendrite-demo-yggdrasil/main.go

@@ -73,13 +73,12 @@ func main() {
 	cfg.Global.PrivateKey = ygg.SigningPrivateKey()
 	cfg.Global.KeyID = gomatrixserverlib.KeyID(signing.KeyID)
 	cfg.Global.Kafka.UseNaffka = true
-	cfg.FederationSender.FederationMaxRetries = 8
 	cfg.UserAPI.AccountDatabase.ConnectionString = config.DataSource(fmt.Sprintf("file:%s-account.db", *instanceName))
 	cfg.UserAPI.DeviceDatabase.ConnectionString = config.DataSource(fmt.Sprintf("file:%s-device.db", *instanceName))
 	cfg.MediaAPI.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s-mediaapi.db", *instanceName))
 	cfg.SyncAPI.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s-syncapi.db", *instanceName))
 	cfg.RoomServer.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s-roomserver.db", *instanceName))
-	cfg.ServerKeyAPI.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s-serverkey.db", *instanceName))
+	cfg.SigningKeyServer.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s-signingkeyserver.db", *instanceName))
 	cfg.KeyServer.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s-keyserver.db", *instanceName))
 	cfg.FederationSender.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s-federationsender.db", *instanceName))
 	cfg.AppServiceAPI.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s-appservice.db", *instanceName))
@@ -97,7 +96,7 @@ func main() {
 	serverKeyAPI := &signing.YggdrasilKeys{}
 	keyRing := serverKeyAPI.KeyRing()
 
-	keyAPI := keyserver.NewInternalAPI(&base.Cfg.KeyServer, federation, base.KafkaProducer)
+	keyAPI := keyserver.NewInternalAPI(&base.Cfg.KeyServer, federation)
 	userAPI := userapi.NewInternalAPI(accountDB, &cfg.UserAPI, nil, keyAPI)
 	keyAPI.SetUserAPI(userAPI)
 
@@ -130,13 +129,11 @@ func main() {
 	rsComponent.SetFederationSenderAPI(fsAPI)
 
 	monolith := setup.Monolith{
-		Config:        base.Cfg,
-		AccountDB:     accountDB,
-		Client:        ygg.CreateClient(base),
-		FedClient:     federation,
-		KeyRing:       keyRing,
-		KafkaConsumer: base.KafkaConsumer,
-		KafkaProducer: base.KafkaProducer,
+		Config:    base.Cfg,
+		AccountDB: accountDB,
+		Client:    ygg.CreateClient(base),
+		FedClient: federation,
+		KeyRing:   keyRing,
 
 		AppserviceAPI:       asAPI,
 		EDUInternalAPI:      eduInputAPI,

cmd/dendrite-demo-yggdrasil/yggconn/client.go

@@ -33,7 +33,7 @@ func (n *Node) CreateClient(
 			},
 		},
 	)
-	return gomatrixserverlib.NewClientWithTransport(true, tr)
+	return gomatrixserverlib.NewClientWithTransport(tr)
 }
 
 func (n *Node) CreateFederationClient(

cmd/dendrite-demo-yggdrasil/yggconn/node.go

@@ -205,13 +205,11 @@ func (n *Node) SessionCount() int {
 
 func (n *Node) KnownNodes() []gomatrixserverlib.ServerName {
 	nodemap := map[string]struct{}{
-		"b5ae50589e50991dd9dd7d59c5c5f7a4521e8da5b603b7f57076272abc58b374": {},
+		//"b5ae50589e50991dd9dd7d59c5c5f7a4521e8da5b603b7f57076272abc58b374": {},
+	}
+	for _, peer := range n.core.GetSwitchPeers() {
+		nodemap[hex.EncodeToString(peer.SigPublicKey[:])] = struct{}{}
 	}
-	/*
-		for _, peer := range n.core.GetSwitchPeers() {
-			nodemap[hex.EncodeToString(peer.PublicKey[:])] = struct{}{}
-		}
-	*/
 	n.sessions.Range(func(_, v interface{}) bool {
 		session, ok := v.(quic.Session)
 		if !ok {

cmd/dendrite-monolith-server/main.go

@@ -27,14 +27,15 @@ import (
 	"github.com/matrix-org/dendrite/keyserver"
 	"github.com/matrix-org/dendrite/roomserver"
 	"github.com/matrix-org/dendrite/roomserver/api"
-	"github.com/matrix-org/dendrite/serverkeyapi"
+	"github.com/matrix-org/dendrite/signingkeyserver"
 	"github.com/matrix-org/dendrite/userapi"
-	"github.com/matrix-org/gomatrixserverlib"
+	"github.com/sirupsen/logrus"
 )
 
 var (
 	httpBindAddr   = flag.String("http-bind-address", ":8008", "The HTTP listening port for the server")
 	httpsBindAddr  = flag.String("https-bind-address", ":8448", "The HTTPS listening port for the server")
+	apiBindAddr    = flag.String("api-bind-address", "localhost:18008", "The HTTP listening port for the internal HTTP APIs (if -api is enabled)")
 	certFile       = flag.String("tls-cert", "", "The PEM formatted X509 certificate to use for TLS")
 	keyFile        = flag.String("tls-key", "", "The PEM private key to use for TLS")
 	enableHTTPAPIs = flag.Bool("api", false, "Use HTTP APIs instead of short-circuiting (warning: exposes API endpoints!)")
@@ -45,22 +46,25 @@ func main() {
 	cfg := setup.ParseFlags(true)
 	httpAddr := config.HTTPAddress("http://" + *httpBindAddr)
 	httpsAddr := config.HTTPAddress("https://" + *httpsBindAddr)
+	httpAPIAddr := httpAddr
 
 	if *enableHTTPAPIs {
+		logrus.Warnf("DANGER! The -api option is enabled, exposing internal APIs on %q!", *apiBindAddr)
+		httpAPIAddr = config.HTTPAddress("http://" + *apiBindAddr)
 		// If the HTTP APIs are enabled then we need to update the Listen
 		// statements in the configuration so that we know where to find
 		// the API endpoints. They'll listen on the same port as the monolith
 		// itself.
-		cfg.AppServiceAPI.InternalAPI.Connect = httpAddr
-		cfg.ClientAPI.InternalAPI.Connect = httpAddr
-		cfg.EDUServer.InternalAPI.Connect = httpAddr
-		cfg.FederationAPI.InternalAPI.Connect = httpAddr
-		cfg.FederationSender.InternalAPI.Connect = httpAddr
-		cfg.KeyServer.InternalAPI.Connect = httpAddr
-		cfg.MediaAPI.InternalAPI.Connect = httpAddr
-		cfg.RoomServer.InternalAPI.Connect = httpAddr
-		cfg.ServerKeyAPI.InternalAPI.Connect = httpAddr
-		cfg.SyncAPI.InternalAPI.Connect = httpAddr
+		cfg.AppServiceAPI.InternalAPI.Connect = httpAPIAddr
+		cfg.ClientAPI.InternalAPI.Connect = httpAPIAddr
+		cfg.EDUServer.InternalAPI.Connect = httpAPIAddr
+		cfg.FederationAPI.InternalAPI.Connect = httpAPIAddr
+		cfg.FederationSender.InternalAPI.Connect = httpAPIAddr
+		cfg.KeyServer.InternalAPI.Connect = httpAPIAddr
+		cfg.MediaAPI.InternalAPI.Connect = httpAPIAddr
+		cfg.RoomServer.InternalAPI.Connect = httpAPIAddr
+		cfg.SigningKeyServer.InternalAPI.Connect = httpAPIAddr
+		cfg.SyncAPI.InternalAPI.Connect = httpAPIAddr
 	}
 
 	base := setup.NewBaseDendrite(cfg, "Monolith", *enableHTTPAPIs)
@@ -69,14 +73,14 @@ func main() {
 	accountDB := base.CreateAccountsDB()
 	federation := base.CreateFederationClient()
 
-	serverKeyAPI := serverkeyapi.NewInternalAPI(
-		&base.Cfg.ServerKeyAPI, federation, base.Caches,
+	skAPI := signingkeyserver.NewInternalAPI(
+		&base.Cfg.SigningKeyServer, federation, base.Caches,
 	)
 	if base.UseHTTPAPIs {
-		serverkeyapi.AddInternalRoutes(base.InternalAPIMux, serverKeyAPI, base.Caches)
-		serverKeyAPI = base.ServerKeyAPIClient()
+		signingkeyserver.AddInternalRoutes(base.InternalAPIMux, skAPI, base.Caches)
+		skAPI = base.SigningKeyServerHTTPClient()
 	}
-	keyRing := serverKeyAPI.KeyRing()
+	keyRing := skAPI.KeyRing()
 
 	rsImpl := roomserver.NewInternalAPI(
 		base, keyRing,
@@ -104,7 +108,7 @@ func main() {
 	// This is different to rsAPI which can be the http client which doesn't need this dependency
 	rsImpl.SetFederationSenderAPI(fsAPI)
 
-	keyAPI := keyserver.NewInternalAPI(&base.Cfg.KeyServer, fsAPI, base.KafkaProducer)
+	keyAPI := keyserver.NewInternalAPI(&base.Cfg.KeyServer, fsAPI)
 	userAPI := userapi.NewInternalAPI(accountDB, &cfg.UserAPI, cfg.Derived.ApplicationServices, keyAPI)
 	keyAPI.SetUserAPI(userAPI)
 
@@ -123,19 +127,17 @@ func main() {
 	}
 
 	monolith := setup.Monolith{
-		Config:        base.Cfg,
-		AccountDB:     accountDB,
-		Client:        gomatrixserverlib.NewClient(cfg.FederationSender.DisableTLSValidation),
-		FedClient:     federation,
-		KeyRing:       keyRing,
-		KafkaConsumer: base.KafkaConsumer,
-		KafkaProducer: base.KafkaProducer,
+		Config:    base.Cfg,
+		AccountDB: accountDB,
+		Client:    base.CreateClient(),
+		FedClient: federation,
+		KeyRing:   keyRing,
 
 		AppserviceAPI:       asAPI,
 		EDUInternalAPI:      eduInputAPI,
 		FederationSenderAPI: fsAPI,
 		RoomserverAPI:       rsAPI,
-		ServerKeyAPI:        serverKeyAPI,
+		ServerKeyAPI:        skAPI,
 		UserAPI:             userAPI,
 		KeyAPI:              keyAPI,
 	}
@@ -149,18 +151,18 @@ func main() {
 	// Expose the matrix APIs directly rather than putting them under a /api path.
 	go func() {
 		base.SetupAndServeHTTP(
-			config.HTTPAddress(httpAddr), // internal API
-			config.HTTPAddress(httpAddr), // external API
-			nil, nil,                     // TLS settings
+			httpAPIAddr, // internal API
+			httpAddr,    // external API
+			nil, nil,    // TLS settings
 		)
 	}()
 	// Handle HTTPS if certificate and key are provided
 	if *certFile != "" && *keyFile != "" {
 		go func() {
 			base.SetupAndServeHTTP(
-				config.HTTPAddress(httpsAddr), // internal API
-				config.HTTPAddress(httpsAddr), // external API
-				certFile, keyFile,             // TLS settings
+				setup.NoListener,  // internal API
+				httpsAddr,         // external API
+				certFile, keyFile, // TLS settings
 			)
 		}()
 	}

cmd/dendrite-polylith-multi/main.go

@@ -0,0 +1,78 @@
+// Copyright 2020 The Matrix.org Foundation C.I.C.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+	"flag"
+	"os"
+	"strings"
+
+	"github.com/matrix-org/dendrite/cmd/dendrite-polylith-multi/personalities"
+	"github.com/matrix-org/dendrite/internal/config"
+	"github.com/matrix-org/dendrite/internal/setup"
+	"github.com/sirupsen/logrus"
+)
+
+type entrypoint func(base *setup.BaseDendrite, cfg *config.Dendrite)
+
+func main() {
+	cfg := setup.ParseFlags(true)
+
+	component := ""
+	if flag.NFlag() > 0 {
+		component = flag.Arg(0) // ./dendrite-polylith-multi --config=... clientapi
+	} else if len(os.Args) > 1 {
+		component = os.Args[1] // ./dendrite-polylith-multi clientapi
+	}
+
+	components := map[string]entrypoint{
+		"appservice":       personalities.Appservice,
+		"clientapi":        personalities.ClientAPI,
+		"eduserver":        personalities.EDUServer,
+		"federationapi":    personalities.FederationAPI,
+		"federationsender": personalities.FederationSender,
+		"keyserver":        personalities.KeyServer,
+		"mediaapi":         personalities.MediaAPI,
+		"roomserver":       personalities.RoomServer,
+		"signingkeyserver": personalities.SigningKeyServer,
+		"syncapi":          personalities.SyncAPI,
+		"userapi":          personalities.UserAPI,
+	}
+
+	start, ok := components[component]
+	if !ok {
+		if component == "" {
+			logrus.Errorf("No component specified")
+			logrus.Info("The first argument on the command line must be the name of the component to run")
+		} else {
+			logrus.Errorf("Unknown component %q specified", component)
+		}
+
+		var list []string
+		for c := range components {
+			list = append(list, c)
+		}
+		logrus.Infof("Valid components: %s", strings.Join(list, ", "))
+
+		os.Exit(1)
+	}
+
+	logrus.Infof("Starting %q component", component)
+
+	base := setup.NewBaseDendrite(cfg, component, false) // TODO
+	defer base.Close()                                   // nolint: errcheck
+
+	start(base, cfg)
+}

cmd/dendrite-polylith-multi/personalities/appservice.go

@@ -1,4 +1,4 @@
-// Copyright 2018 Vector Creations Ltd
+// Copyright 2020 The Matrix.org Foundation C.I.C.
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,18 +12,15 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-package main
+package personalities
 
 import (
 	"github.com/matrix-org/dendrite/appservice"
+	"github.com/matrix-org/dendrite/internal/config"
 	"github.com/matrix-org/dendrite/internal/setup"
 )
 
-func main() {
-	cfg := setup.ParseFlags(false)
-	base := setup.NewBaseDendrite(cfg, "AppServiceAPI", true)
-
-	defer base.Close() // nolint: errcheck
+func Appservice(base *setup.BaseDendrite, cfg *config.Dendrite) {
 	userAPI := base.UserAPIClient()
 	rsAPI := base.RoomserverHTTPClient()
 
@@ -31,8 +28,8 @@ func main() {
 	appservice.AddInternalRoutes(base.InternalAPIMux, intAPI)
 
 	base.SetupAndServeHTTP(
-		base.Cfg.AppServiceAPI.InternalAPI.Listen,
-		setup.NoExternalListener,
+		base.Cfg.AppServiceAPI.InternalAPI.Listen, // internal listener
+		setup.NoListener, // external listener
 		nil, nil,
 	)
 }

cmd/dendrite-polylith-multi/personalities/clientapi.go

@@ -1,4 +1,4 @@
-// Copyright 2017 Vector Creations Ltd
+// Copyright 2020 The Matrix.org Foundation C.I.C.
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,20 +12,16 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-package main
+package personalities
 
 import (
 	"github.com/matrix-org/dendrite/clientapi"
+	"github.com/matrix-org/dendrite/internal/config"
 	"github.com/matrix-org/dendrite/internal/setup"
 	"github.com/matrix-org/dendrite/internal/transactions"
 )
 
-func main() {
-	cfg := setup.ParseFlags(false)
-
-	base := setup.NewBaseDendrite(cfg, "ClientAPI", true)
-	defer base.Close() // nolint: errcheck
-
+func ClientAPI(base *setup.BaseDendrite, cfg *config.Dendrite) {
 	accountDB := base.CreateAccountsDB()
 	federation := base.CreateFederationClient()
 
@@ -37,7 +33,7 @@ func main() {
 	keyAPI := base.KeyServerHTTPClient()
 
 	clientapi.AddPublicRoutes(
-		base.PublicClientAPIMux, &base.Cfg.ClientAPI, base.KafkaProducer, accountDB, federation,
+		base.PublicClientAPIMux, &base.Cfg.ClientAPI, accountDB, federation,
 		rsAPI, eduInputAPI, asQuery, transactions.New(), fsAPI, userAPI, keyAPI, nil,
 	)
 

cmd/dendrite-polylith-multi/personalities/eduserver.go

@@ -1,3 +1,5 @@
+// Copyright 2020 The Matrix.org Foundation C.I.C.
+//
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
 // You may obtain a copy of the License at
@@ -10,32 +12,22 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-package main
+package personalities
 
 import (
-	_ "net/http/pprof"
-
 	"github.com/matrix-org/dendrite/eduserver"
 	"github.com/matrix-org/dendrite/eduserver/cache"
+	"github.com/matrix-org/dendrite/internal/config"
 	"github.com/matrix-org/dendrite/internal/setup"
-	"github.com/sirupsen/logrus"
 )
 
-func main() {
-	cfg := setup.ParseFlags(false)
-	base := setup.NewBaseDendrite(cfg, "EDUServerAPI", true)
-	defer func() {
-		if err := base.Close(); err != nil {
-			logrus.WithError(err).Warn("BaseDendrite close failed")
-		}
-	}()
-
+func EDUServer(base *setup.BaseDendrite, cfg *config.Dendrite) {
 	intAPI := eduserver.NewInternalAPI(base, cache.New(), base.UserAPIClient())
 	eduserver.AddInternalRoutes(base.InternalAPIMux, intAPI)
 
 	base.SetupAndServeHTTP(
-		base.Cfg.EDUServer.InternalAPI.Listen,
-		setup.NoExternalListener,
+		base.Cfg.EDUServer.InternalAPI.Listen, // internal listener
+		setup.NoListener,                      // external listener
 		nil, nil,
 	)
 }

cmd/dendrite-polylith-multi/personalities/federationapi.go

@@ -1,4 +1,4 @@
-// Copyright 2017 Vector Creations Ltd
+// Copyright 2020 The Matrix.org Foundation C.I.C.
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,21 +12,18 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-package main
+package personalities
 
 import (
 	"github.com/matrix-org/dendrite/federationapi"
+	"github.com/matrix-org/dendrite/internal/config"
 	"github.com/matrix-org/dendrite/internal/setup"
 )
 
-func main() {
-	cfg := setup.ParseFlags(false)
-	base := setup.NewBaseDendrite(cfg, "FederationAPI", true)
-	defer base.Close() // nolint: errcheck
-
+func FederationAPI(base *setup.BaseDendrite, cfg *config.Dendrite) {
 	userAPI := base.UserAPIClient()
 	federation := base.CreateFederationClient()
-	serverKeyAPI := base.ServerKeyAPIClient()
+	serverKeyAPI := base.SigningKeyServerHTTPClient()
 	keyRing := serverKeyAPI.KeyRing()
 	fsAPI := base.FederationSenderHTTPClient()
 	rsAPI := base.RoomserverHTTPClient()

cmd/dendrite-polylith-multi/personalities/federationsender.go

@@ -1,4 +1,4 @@
-// Copyright 2017 Vector Creations Ltd
+// Copyright 2020 The Matrix.org Foundation C.I.C.
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,21 +12,18 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-package main
+package personalities
 
 import (
 	"github.com/matrix-org/dendrite/federationsender"
+	"github.com/matrix-org/dendrite/internal/config"
 	"github.com/matrix-org/dendrite/internal/setup"
 )
 
-func main() {
-	cfg := setup.ParseFlags(false)
-	base := setup.NewBaseDendrite(cfg, "FederationSender", true)
-	defer base.Close() // nolint: errcheck
-
+func FederationSender(base *setup.BaseDendrite, cfg *config.Dendrite) {
 	federation := base.CreateFederationClient()
 
-	serverKeyAPI := base.ServerKeyAPIClient()
+	serverKeyAPI := base.SigningKeyServerHTTPClient()
 	keyRing := serverKeyAPI.KeyRing()
 
 	rsAPI := base.RoomserverHTTPClient()
@@ -36,8 +33,8 @@ func main() {
 	federationsender.AddInternalRoutes(base.InternalAPIMux, fsAPI)
 
 	base.SetupAndServeHTTP(
-		base.Cfg.FederationSender.InternalAPI.Listen,
-		setup.NoExternalListener,
+		base.Cfg.FederationSender.InternalAPI.Listen, // internal listener
+		setup.NoListener, // external listener
 		nil, nil,
 	)
 }

cmd/dendrite-polylith-multi/personalities/keyserver.go

@@ -12,26 +12,23 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-package main
+package personalities
 
 import (
+	"github.com/matrix-org/dendrite/internal/config"
 	"github.com/matrix-org/dendrite/internal/setup"
 	"github.com/matrix-org/dendrite/keyserver"
 )
 
-func main() {
-	cfg := setup.ParseFlags(false)
-	base := setup.NewBaseDendrite(cfg, "KeyServer", true)
-	defer base.Close() // nolint: errcheck
-
-	intAPI := keyserver.NewInternalAPI(&base.Cfg.KeyServer, base.CreateFederationClient(), base.KafkaProducer)
+func KeyServer(base *setup.BaseDendrite, cfg *config.Dendrite) {
+	intAPI := keyserver.NewInternalAPI(&base.Cfg.KeyServer, base.CreateFederationClient())
 	intAPI.SetUserAPI(base.UserAPIClient())
 
 	keyserver.AddInternalRoutes(base.InternalAPIMux, intAPI)
 
 	base.SetupAndServeHTTP(
-		base.Cfg.KeyServer.InternalAPI.Listen,
-		setup.NoExternalListener,
+		base.Cfg.KeyServer.InternalAPI.Listen, // internal listener
+		setup.NoListener,                      // external listener
 		nil, nil,
 	)
 }

cmd/dendrite-polylith-multi/personalities/mediaapi.go

@@ -1,4 +1,4 @@
-// Copyright 2017 Vector Creations Ltd
+// Copyright 2020 The Matrix.org Foundation C.I.C.
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,21 +12,17 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-package main
+package personalities
 
 import (
+	"github.com/matrix-org/dendrite/internal/config"
 	"github.com/matrix-org/dendrite/internal/setup"
 	"github.com/matrix-org/dendrite/mediaapi"
-	"github.com/matrix-org/gomatrixserverlib"
 )
 
-func main() {
-	cfg := setup.ParseFlags(false)
-	base := setup.NewBaseDendrite(cfg, "MediaAPI", true)
-	defer base.Close() // nolint: errcheck
-
+func MediaAPI(base *setup.BaseDendrite, cfg *config.Dendrite) {
 	userAPI := base.UserAPIClient()
-	client := gomatrixserverlib.NewClient(cfg.FederationSender.DisableTLSValidation)
+	client := base.CreateClient()
 
 	mediaapi.AddPublicRoutes(base.PublicMediaAPIMux, &base.Cfg.MediaAPI, userAPI, client)
 

cmd/dendrite-polylith-multi/personalities/roomserver.go

@@ -1,4 +1,4 @@
-// Copyright 2017 Vector Creations Ltd
+// Copyright 2020 The Matrix.org Foundation C.I.C.
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,19 +12,16 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-package main
+package personalities
 
 import (
+	"github.com/matrix-org/dendrite/internal/config"
 	"github.com/matrix-org/dendrite/internal/setup"
 	"github.com/matrix-org/dendrite/roomserver"
 )
 
-func main() {
-	cfg := setup.ParseFlags(false)
-	base := setup.NewBaseDendrite(cfg, "RoomServerAPI", true)
-	defer base.Close() // nolint: errcheck
-
-	serverKeyAPI := base.ServerKeyAPIClient()
+func RoomServer(base *setup.BaseDendrite, cfg *config.Dendrite) {
+	serverKeyAPI := base.SigningKeyServerHTTPClient()
 	keyRing := serverKeyAPI.KeyRing()
 
 	fsAPI := base.FederationSenderHTTPClient()
@@ -33,8 +30,8 @@ func main() {
 	roomserver.AddInternalRoutes(base.InternalAPIMux, rsAPI)
 
 	base.SetupAndServeHTTP(
-		base.Cfg.RoomServer.InternalAPI.Listen,
-		setup.NoExternalListener,
+		base.Cfg.RoomServer.InternalAPI.Listen, // internal listener
+		setup.NoListener,                       // external listener
 		nil, nil,
 	)
 }

cmd/dendrite-polylith-multi/personalities/signingkeyserver.go

@@ -12,26 +12,23 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-package main
+package personalities
 
 import (
+	"github.com/matrix-org/dendrite/internal/config"
 	"github.com/matrix-org/dendrite/internal/setup"
-	"github.com/matrix-org/dendrite/serverkeyapi"
+	"github.com/matrix-org/dendrite/signingkeyserver"
 )
 
-func main() {
-	cfg := setup.ParseFlags(false)
-	base := setup.NewBaseDendrite(cfg, "ServerKeyAPI", true)
-	defer base.Close() // nolint: errcheck
-
+func SigningKeyServer(base *setup.BaseDendrite, cfg *config.Dendrite) {
 	federation := base.CreateFederationClient()
 
-	intAPI := serverkeyapi.NewInternalAPI(&base.Cfg.ServerKeyAPI, federation, base.Caches)
-	serverkeyapi.AddInternalRoutes(base.InternalAPIMux, intAPI, base.Caches)
+	intAPI := signingkeyserver.NewInternalAPI(&base.Cfg.SigningKeyServer, federation, base.Caches)
+	signingkeyserver.AddInternalRoutes(base.InternalAPIMux, intAPI, base.Caches)
 
 	base.SetupAndServeHTTP(
-		base.Cfg.ServerKeyAPI.InternalAPI.Listen,
-		setup.NoExternalListener,
+		base.Cfg.SigningKeyServer.InternalAPI.Listen,
+		setup.NoListener,
 		nil, nil,
 	)
 }

cmd/dendrite-polylith-multi/personalities/syncapi.go

@@ -1,4 +1,4 @@
-// Copyright 2017 Vector Creations Ltd
+// Copyright 2020 The Matrix.org Foundation C.I.C.
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,25 +12,22 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-package main
+package personalities
 
 import (
+	"github.com/matrix-org/dendrite/internal/config"
 	"github.com/matrix-org/dendrite/internal/setup"
 	"github.com/matrix-org/dendrite/syncapi"
 )
 
-func main() {
-	cfg := setup.ParseFlags(false)
-	base := setup.NewBaseDendrite(cfg, "SyncAPI", true)
-	defer base.Close() // nolint: errcheck
-
+func SyncAPI(base *setup.BaseDendrite, cfg *config.Dendrite) {
 	userAPI := base.UserAPIClient()
 	federation := base.CreateFederationClient()
 
 	rsAPI := base.RoomserverHTTPClient()
 
 	syncapi.AddPublicRoutes(
-		base.PublicClientAPIMux, base.KafkaConsumer, userAPI, rsAPI,
+		base.PublicClientAPIMux, userAPI, rsAPI,
 		base.KeyServerHTTPClient(),
 		federation, &cfg.SyncAPI,
 	)

cmd/dendrite-polylith-multi/personalities/userapi.go

@@ -1,4 +1,4 @@
-// Copyright 2017 Vector Creations Ltd
+// Copyright 2020 The Matrix.org Foundation C.I.C.
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,18 +12,15 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-package main
+package personalities
 
 import (
+	"github.com/matrix-org/dendrite/internal/config"
 	"github.com/matrix-org/dendrite/internal/setup"
 	"github.com/matrix-org/dendrite/userapi"
 )
 
-func main() {
-	cfg := setup.ParseFlags(false)
-	base := setup.NewBaseDendrite(cfg, "UserAPI", true)
-	defer base.Close() // nolint: errcheck
-
+func UserAPI(base *setup.BaseDendrite, cfg *config.Dendrite) {
 	accountDB := base.CreateAccountsDB()
 
 	userAPI := userapi.NewInternalAPI(accountDB, &cfg.UserAPI, cfg.Derived.ApplicationServices, base.KeyServerHTTPClient())
@@ -31,8 +28,8 @@ func main() {
 	userapi.AddInternalRoutes(base.InternalAPIMux, userAPI)
 
 	base.SetupAndServeHTTP(
-		base.Cfg.UserAPI.InternalAPI.Listen,
-		setup.NoExternalListener,
+		base.Cfg.UserAPI.InternalAPI.Listen, // internal listener
+		setup.NoListener,                    // external listener
 		nil, nil,
 	)
 }

cmd/dendritejs/main.go

@@ -141,14 +141,14 @@ func createFederationClient(cfg *config.Dendrite, node *go_http_js_libp2p.P2pLoc
 	fed := gomatrixserverlib.NewFederationClient(
 		cfg.Global.ServerName, cfg.Global.KeyID, cfg.Global.PrivateKey, true,
 	)
-	fed.Client = *gomatrixserverlib.NewClientWithTransport(true, tr)
+	fed.Client = *gomatrixserverlib.NewClientWithTransport(tr)
 
 	return fed
 }
 
 func createClient(node *go_http_js_libp2p.P2pLocalNode) *gomatrixserverlib.Client {
 	tr := go_http_js_libp2p.NewP2pTransport(node)
-	return gomatrixserverlib.NewClientWithTransport(true, tr)
+	return gomatrixserverlib.NewClientWithTransport(tr)
 }
 
 func createP2PNode(privKey ed25519.PrivateKey) (serverName string, node *go_http_js_libp2p.P2pLocalNode) {
@@ -168,7 +168,7 @@ func main() {
 	cfg.FederationSender.Database.ConnectionString = "file:/idb/dendritejs_fedsender.db"
 	cfg.MediaAPI.Database.ConnectionString = "file:/idb/dendritejs_mediaapi.db"
 	cfg.RoomServer.Database.ConnectionString = "file:/idb/dendritejs_roomserver.db"
-	cfg.ServerKeyAPI.Database.ConnectionString = "file:/idb/dendritejs_serverkey.db"
+	cfg.SigningKeyServer.Database.ConnectionString = "file:/idb/dendritejs_signingkeyserver.db"
 	cfg.SyncAPI.Database.ConnectionString = "file:/idb/dendritejs_syncapi.db"
 	cfg.KeyServer.Database.ConnectionString = "file:/idb/dendritejs_e2ekey.db"
 	cfg.Global.Kafka.UseNaffka = true
@@ -190,7 +190,7 @@ func main() {
 
 	accountDB := base.CreateAccountsDB()
 	federation := createFederationClient(cfg, node)
-	keyAPI := keyserver.NewInternalAPI(&base.Cfg.KeyServer, federation, base.KafkaProducer)
+	keyAPI := keyserver.NewInternalAPI(&base.Cfg.KeyServer, federation)
 	userAPI := userapi.NewInternalAPI(accountDB, &cfg.UserAPI, nil, keyAPI)
 	keyAPI.SetUserAPI(userAPI)
 
@@ -212,13 +212,11 @@ func main() {
 	p2pPublicRoomProvider := NewLibP2PPublicRoomsProvider(node, fedSenderAPI, federation)
 
 	monolith := setup.Monolith{
-		Config:        base.Cfg,
-		AccountDB:     accountDB,
-		Client:        createClient(node),
-		FedClient:     federation,
-		KeyRing:       &keyRing,
-		KafkaConsumer: base.KafkaConsumer,
-		KafkaProducer: base.KafkaProducer,
+		Config:    base.Cfg,
+		AccountDB: accountDB,
+		Client:    createClient(node),
+		FedClient: federation,
+		KeyRing:   &keyRing,
 
 		AppserviceAPI:       asQuery,
 		EDUInternalAPI:      eduInputAPI,

cmd/furl/main.go

@@ -0,0 +1,124 @@
+package main
+
+import (
+	"bufio"
+	"bytes"
+	"context"
+	"crypto/ed25519"
+	"encoding/json"
+	"encoding/pem"
+	"flag"
+	"fmt"
+	"io/ioutil"
+	"net/url"
+	"os"
+
+	"github.com/matrix-org/gomatrixserverlib"
+)
+
+var requestFrom = flag.String("from", "", "the server name that the request should originate from")
+var requestKey = flag.String("key", "matrix_key.pem", "the private key to use when signing the request")
+var requestPost = flag.Bool("post", false, "send a POST request instead of GET (pipe input into stdin or type followed by Ctrl-D)")
+
+// nolint:gocyclo
+func main() {
+	flag.Parse()
+
+	if requestFrom == nil || *requestFrom == "" {
+		fmt.Println("expecting: furl -from origin.com [-key matrix_key.pem] https://path/to/url")
+		fmt.Println("supported flags:")
+		flag.PrintDefaults()
+		os.Exit(1)
+	}
+
+	data, err := ioutil.ReadFile(*requestKey)
+	if err != nil {
+		panic(err)
+	}
+
+	var privateKey ed25519.PrivateKey
+	keyBlock, _ := pem.Decode(data)
+	if keyBlock == nil {
+		panic("keyBlock is nil")
+	}
+	if keyBlock.Type == "MATRIX PRIVATE KEY" {
+		_, privateKey, err = ed25519.GenerateKey(bytes.NewReader(keyBlock.Bytes))
+		if err != nil {
+			panic(err)
+		}
+	} else {
+		panic("unexpected key block")
+	}
+
+	client := gomatrixserverlib.NewFederationClient(
+		gomatrixserverlib.ServerName(*requestFrom),
+		gomatrixserverlib.KeyID(keyBlock.Headers["Key-ID"]),
+		privateKey,
+		false,
+	)
+
+	u, err := url.Parse(flag.Arg(0))
+	if err != nil {
+		panic(err)
+	}
+
+	var bodyObj interface{}
+	var bodyBytes []byte
+	method := "GET"
+	if *requestPost {
+		method = "POST"
+		fmt.Println("Waiting for JSON input. Press Enter followed by Ctrl-D when done...")
+
+		scan := bufio.NewScanner(os.Stdin)
+		for scan.Scan() {
+			bytes := scan.Bytes()
+			bodyBytes = append(bodyBytes, bytes...)
+		}
+		fmt.Println("Done!")
+		if err = json.Unmarshal(bodyBytes, &bodyObj); err != nil {
+			panic(err)
+		}
+	}
+
+	req := gomatrixserverlib.NewFederationRequest(
+		method,
+		gomatrixserverlib.ServerName(u.Host),
+		u.RequestURI(),
+	)
+
+	if *requestPost {
+		if err = req.SetContent(bodyObj); err != nil {
+			panic(err)
+		}
+	}
+
+	if err = req.Sign(
+		gomatrixserverlib.ServerName(*requestFrom),
+		gomatrixserverlib.KeyID(keyBlock.Headers["Key-ID"]),
+		privateKey,
+	); err != nil {
+		panic(err)
+	}
+
+	httpReq, err := req.HTTPRequest()
+	if err != nil {
+		panic(err)
+	}
+
+	var res interface{}
+	err = client.DoRequestAndParseResponse(
+		context.TODO(),
+		httpReq,
+		&res,
+	)
+	if err != nil {
+		panic(err)
+	}
+
+	j, err := json.MarshalIndent(res, "", "  ")
+	if err != nil {
+		panic(err)
+	}
+
+	fmt.Println(string(j))
+}

cmd/generate-config/main.go

@@ -27,7 +27,7 @@ func main() {
 			},
 		},
 	}
-	cfg.ServerKeyAPI.KeyPerspectives = config.KeyPerspectives{
+	cfg.SigningKeyServer.KeyPerspectives = config.KeyPerspectives{
 		{
 			ServerName: "matrix.org",
 			Keys: []config.KeyPerspectiveTrustKey{

cmd/goose/README.md

@@ -104,4 +104,6 @@ You __must__ import the package in `/cmd/goose/main.go` so `func init()` gets ca
 #### Database limitations
 
 - SQLite3 does NOT support `ALTER TABLE table_name DROP COLUMN` - you would have to rename the column or drop the table
-  entirely and recreate it.
+  entirely and recreate it. ([example](https://github.com/matrix-org/dendrite/blob/master/userapi/storage/accounts/sqlite3/deltas/20200929203058_is_active.sql))
+  
+  More information: [sqlite.org](https://www.sqlite.org/lang_altertable.html)

cmd/goose/main.go

@@ -8,19 +8,38 @@ import (
 	"log"
 	"os"
 
-	// Example complex Go migration import:
-	// _ "github.com/matrix-org/dendrite/serverkeyapi/storage/postgres/deltas"
+	pgaccounts "github.com/matrix-org/dendrite/userapi/storage/accounts/postgres/deltas"
+	slaccounts "github.com/matrix-org/dendrite/userapi/storage/accounts/sqlite3/deltas"
+	pgdevices "github.com/matrix-org/dendrite/userapi/storage/devices/postgres/deltas"
+	sldevices "github.com/matrix-org/dendrite/userapi/storage/devices/sqlite3/deltas"
 	"github.com/pressly/goose"
 
 	_ "github.com/lib/pq"
 	_ "github.com/mattn/go-sqlite3"
 )
 
-var (
-	flags = flag.NewFlagSet("goose", flag.ExitOnError)
-	dir   = flags.String("dir", ".", "directory with migration files")
+const (
+	AppService       = "appservice"
+	FederationSender = "federationsender"
+	KeyServer        = "keyserver"
+	MediaAPI         = "mediaapi"
+	RoomServer       = "roomserver"
+	SigningKeyServer = "signingkeyserver"
+	SyncAPI          = "syncapi"
+	UserAPIAccounts  = "userapi_accounts"
+	UserAPIDevices   = "userapi_devices"
 )
 
+var (
+	dir       = flags.String("dir", "", "directory with migration files")
+	flags     = flag.NewFlagSet("goose", flag.ExitOnError)
+	component = flags.String("component", "", "dendrite component name")
+	knownDBs  = []string{
+		AppService, FederationSender, KeyServer, MediaAPI, RoomServer, SigningKeyServer, SyncAPI, UserAPIAccounts, UserAPIDevices,
+	}
+)
+
+// nolint: gocyclo
 func main() {
 	err := flags.Parse(os.Args[1:])
 	if err != nil {
@@ -37,19 +56,20 @@ Drivers:
     sqlite3
 
 Examples:
-    goose -d roomserver/storage/sqlite3/deltas sqlite3 ./roomserver.db status
-    goose -d roomserver/storage/sqlite3/deltas sqlite3 ./roomserver.db up
+    goose -component roomserver sqlite3 ./roomserver.db status
+    goose -component roomserver sqlite3 ./roomserver.db up
 
-    goose -d roomserver/storage/postgres/deltas postgres "user=dendrite dbname=dendrite sslmode=disable" status
+    goose -component roomserver postgres "user=dendrite dbname=dendrite sslmode=disable" status
 
 Options:
-
-  -dir string
-    	directory with migration files (default ".")
+  -component string
+    	Dendrite component name e.g roomserver, signingkeyserver, clientapi, syncapi
   -table string
     	migrations table name (default "goose_db_version")
   -h	print help
   -v	enable verbose mode
+  -dir string
+        directory with migration files, only relevant when creating new migrations.
   -version
     	print version
 
@@ -74,6 +94,25 @@ Commands:
 		fmt.Println("engine must be one of 'sqlite3' or 'postgres'")
 		return
 	}
+
+	knownComponent := false
+	for _, c := range knownDBs {
+		if c == *component {
+			knownComponent = true
+			break
+		}
+	}
+	if !knownComponent {
+		fmt.Printf("component must be one of %v\n", knownDBs)
+		return
+	}
+
+	if engine == "sqlite3" {
+		loadSQLiteDeltas(*component)
+	} else {
+		loadPostgresDeltas(*component)
+	}
+
 	dbstring, command := args[1], args[2]
 
 	db, err := goose.OpenDBWithDriver(engine, dbstring)
@@ -92,7 +131,30 @@ Commands:
 		arguments = append(arguments, args[3:]...)
 	}
 
-	if err := goose.Run(command, db, *dir, arguments...); err != nil {
+	// goose demands a directory even though we don't use it for upgrades
+	d := *dir
+	if d == "" {
+		d = os.TempDir()
+	}
+	if err := goose.Run(command, db, d, arguments...); err != nil {
 		log.Fatalf("goose %v: %v", command, err)
 	}
 }
+
+func loadSQLiteDeltas(component string) {
+	switch component {
+	case UserAPIAccounts:
+		slaccounts.LoadFromGoose()
+	case UserAPIDevices:
+		sldevices.LoadFromGoose()
+	}
+}
+
+func loadPostgresDeltas(component string) {
+	switch component {
+	case UserAPIAccounts:
+		pgaccounts.LoadFromGoose()
+	case UserAPIDevices:
+		pgdevices.LoadFromGoose()
+	}
+}

dendrite-config.yaml

@@ -252,13 +252,13 @@ room_server:
     max_idle_conns: 2
     conn_max_lifetime: -1
 
-# Configuration for the Server Key API (for server signing keys).
-server_key_api:
+# Configuration for the Signing Key Server (for server signing keys).
+signing_key_server:
   internal_api:
     listen: http://localhost:7780
     connect: http://localhost:7780
   database:
-    connection_string: file:serverkeyapi.db
+    connection_string: file:signingkeyserver.db
     max_open_conns: 100
     max_idle_conns: 2
     conn_max_lifetime: -1
@@ -274,6 +274,11 @@ server_key_api:
     - key_id: ed25519:a_RXGa
       public_key: l8Hft5qXKn1vfHrg3p4+W8gELQVo8N13JkluMfmn2sQ
 
+  # This option will control whether Dendrite will prefer to look up keys directly
+  # or whether it should try perspective servers first, using direct fetches as a
+  # last resort.
+  prefer_direct_fetch: false
+
 # Configuration for the Sync API.
 sync_api:
   internal_api:

docs/CONTRIBUTING.md

@@ -72,7 +72,7 @@ issue](https://github.com/matrix-org/dendrite/labels/good%20first%20issue).
 These should be well-contained, small pieces of work that can be picked up to
 help you get familiar with the code base.
 
-Once you're comfortable with hacking on Dendrite there are issues lablled as
+Once you're comfortable with hacking on Dendrite there are issues labelled as
 [help wanted](https://github.com/matrix-org/dendrite/labels/help-wanted),
 these are often slightly larger or more complicated pieces of work but are
 hopefully nonetheless fairly well-contained.

docs/INSTALL.md

@@ -24,7 +24,7 @@ use in production environments just yet!
 Dendrite requires:
 
 * Go 1.13 or higher
-* Postgres 9.5 or higher (if using Postgres databases, not needed for SQLite)
+* Postgres 9.6 or higher (if using Postgres databases, not needed for SQLite)
 
 If you want to run a polylith deployment, you also need:
 
@@ -93,12 +93,12 @@ brew services start kafka
 ### SQLite database setup
 
 Dendrite can use the built-in SQLite database engine for small setups.
-The SQLite databases do not need to be preconfigured - Dendrite will
+The SQLite databases do not need to be pre-built - Dendrite will
 create them automatically at startup.
 
 ### Postgres database setup
 
-Assuming that Postgres 9.5 (or later) is installed:
+Assuming that Postgres 9.6 (or later) is installed:
 
 * Create role, choosing a new password when prompted:
 
@@ -109,7 +109,7 @@ Assuming that Postgres 9.5 (or later) is installed:
 * Create the component databases:
 
   ```bash
-  for i in account device mediaapi syncapi roomserver serverkey federationsender appservice e2ekey naffka; do
+  for i in mediaapi syncapi roomserver signingkeyserver federationsender appservice keyserver userapi_account userapi_device naffka; do
       sudo -u postgres createdb -O dendrite dendrite_$i
   done
   ```
@@ -120,7 +120,10 @@ Assuming that Postgres 9.5 (or later) is installed:
 
 Each Dendrite server requires unique server keys.
 
-Generate the self-signed SSL certificate for federation and the server signing key:
+In order for an instance to federate correctly, you should have a valid
+certificate issued by a trusted authority, and private key to match. If you
+don't and just want to test locally, generate the self-signed SSL certificate
+for federation and the server signing key:
 
 ```bash
 ./bin/generate-keys --private-key matrix_key.pem --tls-cert server.crt --tls-key server.key
@@ -132,8 +135,8 @@ Create config file, based on `dendrite-config.yaml`. Call it `dendrite.yaml`. Th
 
 * The `server_name` entry to reflect the hostname of your Dendrite server
 * The `database` lines with an updated connection string based on your
-  desired setup, e.g. replacing `component` with the name of the component:
-  * For Postgres: `postgres://dendrite:password@localhost/component`
+  desired setup, e.g. replacing `database` with the name of the database:
+  * For Postgres: `postgres://dendrite:password@localhost/database`
   * For SQLite on disk: `file:component.db` or `file:///path/to/component.db`
   * Postgres and SQLite can be mixed and matched.
 * The `use_naffka` option if using Naffka in a monolith deployment
@@ -144,6 +147,10 @@ then configuring `key_perspectives` (like `matrix.org` in the sample) can
 help to improve reliability considerably by allowing your homeserver to fetch
 public keys for dead homeservers from somewhere else.
 
+**WARNING:** Dendrite supports running all components from the same database in
+Postgres mode, but this is **NOT** a supported configuration with SQLite. When
+using SQLite, all components **MUST** use their own database file.
+
 ## Starting a monolith server
 
 It is possible to use Naffka as an in-process replacement to Kafka when using
@@ -164,30 +171,17 @@ as shown below, it will also listen for HTTPS connections on port 8448.
 
 The following contains scripts which will run all the required processes in order to point a Matrix client at Dendrite.
 
-### Client proxy
+### nginx (or other reverse proxy)
 
-This is what Matrix clients will talk to. If you use the script below, point
-your client at `http://localhost:8008`.
+This is what your clients and federated hosts will talk to. It must forward
+requests onto the correct API server based on URL:
 
-```bash
-./bin/client-api-proxy \
---bind-address ":8008" \
---client-api-server-url "http://localhost:7771" \
---sync-api-server-url "http://localhost:7773" \
---media-api-server-url "http://localhost:7774" \
-```
+* `/_matrix/client` to the client API server
+* `/_matrix/federation` to the federation API server
+* `/_matrix/key` to the federation API server
+* `/_matrix/media` to the media API server
 
-### Federation proxy
-
-This is what Matrix servers will talk to. This is only required if you want
-to support federation.
-
-```bash
-./bin/federation-api-proxy \
---bind-address ":8448" \
---federation-api-url "http://localhost:7772" \
---media-api-server-url "http://localhost:7774" \
-```
+See `docs/nginx/polylith-sample.conf` for a sample configuration.
 
 ### Client API server
 
@@ -195,7 +189,7 @@ This is what implements CS API endpoints. Clients talk to this via the proxy in
 order to send messages, create and join rooms, etc.
 
 ```bash
-./bin/dendrite-client-api-server --config=dendrite.yaml
+./bin/dendrite-polylith-multi --config=dendrite.yaml clientapi
 ```
 
 ### Sync server
@@ -204,7 +198,7 @@ This is what implements `/sync` requests. Clients talk to this via the proxy
 in order to receive messages.
 
 ```bash
-./bin/dendrite-sync-api-server --config dendrite.yaml
+./bin/dendrite-polylith-multi --config=dendrite.yaml syncapi
 ```
 
 ### Media server
@@ -213,7 +207,7 @@ This implements `/media` requests. Clients talk to this via the proxy in
 order to upload and retrieve media.
 
 ```bash
-./bin/dendrite-media-api-server --config dendrite.yaml
+./bin/dendrite-polylith-multi --config=dendrite.yaml mediaapi
 ```
 
 ### Federation API server
@@ -223,7 +217,7 @@ order to send transactions.  This is only required if you want to support
 federation.
 
 ```bash
-./bin/dendrite-federation-api-server --config dendrite.yaml
+./bin/dendrite-polylith-multi --config=dendrite.yaml federationapi
 ```
 
 ### Internal components
@@ -236,7 +230,7 @@ contacted by other components. This includes the following components.
 This is what implements the room DAG. Clients do not talk to this.
 
 ```bash
-./bin/dendrite-room-server --config=dendrite.yaml
+./bin/dendrite-polylith-multi --config=dendrite.yaml roomserver
 ```
 
 #### Federation sender
@@ -245,7 +239,7 @@ This sends events from our users to other servers.  This is only required if
 you want to support federation.
 
 ```bash
-./bin/dendrite-federation-sender-server --config dendrite.yaml
+./bin/dendrite-polylith-multi --config=dendrite.yaml federationsender
 ```
 
 #### Appservice server
@@ -256,7 +250,7 @@ running locally.  This is only required if you want to support running
 application services on your homeserver.
 
 ```bash
-./bin/dendrite-appservice-server --config dendrite.yaml
+./bin/dendrite-polylith-multi --config=dendrite.yaml appservice
 ```
 
 #### Key server
@@ -264,15 +258,15 @@ application services on your homeserver.
 This manages end-to-end encryption keys for users.
 
 ```bash
-./bin/dendrite-key-server --config dendrite.yaml
+./bin/dendrite-polylith-multi --config=dendrite.yaml keyserver
 ```
 
-#### Server Key server
+#### Signing key server
 
 This manages signing keys for servers.
 
 ```bash
-./bin/dendrite-server-key-api-server --config dendrite.yaml
+./bin/dendrite-polylith-multi --config=dendrite.yaml signingkeyserver
 ```
 
 #### EDU server
@@ -280,7 +274,7 @@ This manages signing keys for servers.
 This manages processing EDUs such as typing, send-to-device events and presence. Clients do not talk to
 
 ```bash
-./bin/dendrite-edu-server --config dendrite.yaml
+./bin/dendrite-polylith-multi --config=dendrite.yaml eduserver
 ```
 
 #### User server
@@ -289,6 +283,6 @@ This manages user accounts, device access tokens and user account data,
 amongst other things.
 
 ```bash
-./bin/dendrite-user-api-server --config dendrite.yaml
+./bin/dendrite-polylith-multi --config=dendrite.yaml userapi
 ```
 

docs/nginx/monolith-sample.conf

@@ -0,0 +1,24 @@
+server {
+    listen 443 ssl;
+    server_name my.hostname.com;
+
+    ssl_certificate /path/to/fullchain.pem;
+    ssl_certificate_key /path/to/privkey.pem;
+    ssl_dhparam /path/to/ssl-dhparams.pem;
+
+    proxy_set_header Host      $host;
+    proxy_set_header X-Real-IP $remote_addr;
+    proxy_read_timeout         600;
+
+    location /.well-known/matrix/server {
+        return 200 '{ "m.server": "my.hostname.com:443" }';
+    }
+
+    location /.well-known/matrix/client {
+        return 200 '{ "m.homeserver": { "base_url": "https://my.hostname.com" } }';
+    }
+
+    location /_matrix {
+        proxy_pass http://monolith:8008;
+    }
+}

docs/nginx/polylith-sample.conf

@@ -0,0 +1,47 @@
+server {
+    listen 443 ssl;
+    server_name my.hostname.com;
+
+    ssl_certificate /path/to/fullchain.pem;
+    ssl_certificate_key /path/to/privkey.pem;
+    ssl_dhparam /path/to/ssl-dhparams.pem;
+
+    proxy_set_header Host      $host;
+    proxy_set_header X-Real-IP $remote_addr;
+    proxy_read_timeout         600;
+
+    location /.well-known/matrix/server {
+        return 200 '{ "m.server": "my.hostname.com:443" }';
+    }
+
+    location /.well-known/matrix/client {
+        return 200 '{ "m.homeserver": { "base_url": "https://my.hostname.com" } }';
+    }
+
+    # route requests to:
+    # /_matrix/client/.*/sync
+    # /_matrix/client/.*/user/{userId}/filter
+    # /_matrix/client/.*/user/{userId}/filter/{filterID}
+    # /_matrix/client/.*/keys/changes
+    # /_matrix/client/.*/rooms/{roomId}/messages
+    # to sync_api
+    location ~ /_matrix/client/.*?/(sync|user/.*?/filter/?.*|keys/changes|rooms/.*?/messages)$  {
+        proxy_pass http://sync_api:8073;
+    }
+
+    location /_matrix/client {
+        proxy_pass http://client_api:8071;
+    }
+
+    location /_matrix/federation {
+        proxy_pass http://federation_api:8072;
+    }
+
+    location /_matrix/key {
+        proxy_pass http://federation_api:8072;
+    }
+
+    location /_matrix/media {
+        proxy_pass http://media_api:8074;
+    }
+}

docs/systemd/monolith-example.service

@@ -0,0 +1,17 @@
+[Unit]
+Description=Dendrite (Matrix Homeserver)
+After=syslog.target
+After=network.target
+After=postgresql.service
+
+[Service]
+RestartSec=2s
+Type=simple
+User=dendrite
+Group=dendrite
+WorkingDirectory=/opt/dendrite/
+ExecStart=/opt/dendrite/bin/dendrite-monolith-server
+Restart=always
+
+[Install]
+WantedBy=multi-user.target

eduserver/eduserver.go

@@ -24,6 +24,7 @@ import (
 	"github.com/matrix-org/dendrite/eduserver/inthttp"
 	"github.com/matrix-org/dendrite/internal/config"
 	"github.com/matrix-org/dendrite/internal/setup"
+	"github.com/matrix-org/dendrite/internal/setup/kafka"
 	userapi "github.com/matrix-org/dendrite/userapi/api"
 )
 
@@ -41,10 +42,13 @@ func NewInternalAPI(
 	userAPI userapi.UserInternalAPI,
 ) api.EDUServerInputAPI {
 	cfg := &base.Cfg.EDUServer
+
+	_, producer := kafka.SetupConsumerProducer(&cfg.Matrix.Kafka)
+
 	return &input.EDUServerInputAPI{
 		Cache:                        eduCache,
 		UserAPI:                      userAPI,
-		Producer:                     base.KafkaProducer,
+		Producer:                     producer,
 		OutputTypingEventTopic:       string(cfg.Matrix.Kafka.TopicFor(config.TopicOutputTypingEvent)),
 		OutputSendToDeviceEventTopic: string(cfg.Matrix.Kafka.TopicFor(config.TopicOutputSendToDeviceEvent)),
 		ServerName:                   cfg.Matrix.ServerName,

federationapi/routing/devices.go

@@ -40,6 +40,7 @@ func GetUserDevices(
 	response := gomatrixserverlib.RespUserDevices{
 		UserID:   userID,
 		StreamID: res.StreamID,
+		Devices:  []gomatrixserverlib.RespUserDevice{},
 	}
 
 	for _, dev := range res.Devices {

federationapi/routing/join.go

@@ -192,9 +192,7 @@ func SendJoin(
 	if event.StateKey() == nil || event.StateKeyEquals("") {
 		return util.JSONResponse{
 			Code: http.StatusBadRequest,
-			JSON: jsonerror.BadJSON(
-				fmt.Sprintf("No state key was provided in the join event."),
-			),
+			JSON: jsonerror.BadJSON("No state key was provided in the join event."),
 		}
 	}
 
@@ -292,6 +290,7 @@ func SendJoin(
 	if !alreadyJoined {
 		if err = api.SendEvents(
 			httpReq.Context(), rsAPI,
+			api.KindNew,
 			[]gomatrixserverlib.HeaderedEvent{
 				event.Headered(stateAndAuthChainResponse.RoomVersion),
 			},

federationapi/routing/keys.go

@@ -136,7 +136,6 @@ func localKeys(cfg *config.FederationAPI, validUntil time.Time) (*gomatrixserver
 	var keys gomatrixserverlib.ServerKeys
 
 	keys.ServerName = cfg.Matrix.ServerName
-	keys.TLSFingerprints = cfg.TLSFingerPrints
 	keys.ValidUntilTS = gomatrixserverlib.AsTimestamp(validUntil)
 
 	publicKey := cfg.Matrix.PrivateKey.Public().(ed25519.PublicKey)
@@ -151,7 +150,7 @@ func localKeys(cfg *config.FederationAPI, validUntil time.Time) (*gomatrixserver
 	for _, oldVerifyKey := range cfg.Matrix.OldVerifyKeys {
 		keys.OldVerifyKeys[oldVerifyKey.KeyID] = gomatrixserverlib.OldVerifyKey{
 			VerifyKey: gomatrixserverlib.VerifyKey{
-				Key: gomatrixserverlib.Base64Bytes(oldVerifyKey.PrivateKey),
+				Key: gomatrixserverlib.Base64Bytes(oldVerifyKey.PrivateKey.Public().(ed25519.PublicKey)),
 			},
 			ExpiredTS: oldVerifyKey.ExpiredAt,
 		}

federationapi/routing/leave.go

@@ -256,6 +256,7 @@ func SendLeave(
 	// the room, so set SendAsServer to cfg.Matrix.ServerName
 	if err = api.SendEvents(
 		httpReq.Context(), rsAPI,
+		api.KindNew,
 		[]gomatrixserverlib.HeaderedEvent{
 			event.Headered(verRes.RoomVersion),
 		},

federationapi/routing/missingevents.go

@@ -26,7 +26,7 @@ type getMissingEventRequest struct {
 	EarliestEvents []string `json:"earliest_events"`
 	LatestEvents   []string `json:"latest_events"`
 	Limit          int      `json:"limit"`
-	MinDepth       int64    `json:"min_depth"`
+	MinDepth       int64    `json:"min_depth"` // not used
 }
 
 // GetMissingEvents returns missing events between earliest_events & latest_events.
@@ -59,7 +59,7 @@ func GetMissingEvents(
 		return jsonerror.InternalServerError()
 	}
 
-	eventsResponse.Events = filterEvents(eventsResponse.Events, gme.MinDepth, roomID)
+	eventsResponse.Events = filterEvents(eventsResponse.Events, roomID)
 
 	resp := gomatrixserverlib.RespMissingEvents{
 		Events: gomatrixserverlib.UnwrapEventHeaders(eventsResponse.Events),
@@ -71,13 +71,13 @@ func GetMissingEvents(
 	}
 }
 
-// filterEvents returns only those events with matching roomID and having depth greater than minDepth
+// filterEvents returns only those events with matching roomID
 func filterEvents(
-	events []gomatrixserverlib.HeaderedEvent, minDepth int64, roomID string,
+	events []gomatrixserverlib.HeaderedEvent, roomID string,
 ) []gomatrixserverlib.HeaderedEvent {
 	ref := events[:0]
 	for _, ev := range events {
-		if ev.Depth() >= minDepth && ev.RoomID() == roomID {
+		if ev.RoomID() == roomID {
 			ref = append(ref, ev)
 		}
 	}

federationapi/routing/send.go

@@ -16,9 +16,11 @@ package routing
 
 import (
 	"context"
+	"database/sql"
 	"encoding/json"
 	"fmt"
 	"net/http"
+	"sync"
 	"time"
 
 	"github.com/matrix-org/dendrite/clientapi/jsonerror"
@@ -80,7 +82,7 @@ func Send(
 	t.TransactionID = txnID
 	t.Destination = cfg.Matrix.ServerName
 
-	util.GetLogger(httpReq.Context()).Infof("Received transaction %q containing %d PDUs, %d EDUs", txnID, len(t.PDUs), len(t.EDUs))
+	util.GetLogger(httpReq.Context()).Infof("Received transaction %q from %q containing %d PDUs, %d EDUs", txnID, request.Origin(), len(t.PDUs), len(t.EDUs))
 
 	resp, jsonErr := t.processTransaction(httpReq.Context())
 	if jsonErr != nil {
@@ -181,7 +183,7 @@ func (t *txnReq) processTransaction(ctx context.Context) (*gomatrixserverlib.Res
 
 	// Process the events.
 	for _, e := range pdus {
-		if err := t.processEvent(ctx, e.Unwrap(), true); err != nil {
+		if err := t.processEvent(ctx, e.Unwrap()); err != nil {
 			// If the error is due to the event itself being bad then we skip
 			// it and move onto the next event. We report an error so that the
 			// sender knows that we have skipped processing it.
@@ -205,10 +207,10 @@ func (t *txnReq) processTransaction(ctx context.Context) (*gomatrixserverlib.Res
 				return nil, &jsonErr
 			} else {
 				// Auth errors mean the event is 'rejected' which have to be silent to appease sytest
+				errMsg := ""
 				_, rejected := err.(*gomatrixserverlib.NotAllowed)
-				errMsg := err.Error()
-				if rejected {
-					errMsg = ""
+				if !rejected {
+					errMsg = err.Error()
 				}
 				util.GetLogger(ctx).WithError(err).WithField("event_id", e.EventID()).WithField("rejected", rejected).Warn(
 					"Failed to process incoming federation event, skipping",
@@ -233,17 +235,10 @@ func (t *txnReq) processTransaction(ctx context.Context) (*gomatrixserverlib.Res
 // we should stop processing the transaction, and returns false if it
 // is just some less serious error about a specific event.
 func isProcessingErrorFatal(err error) bool {
-	switch err.(type) {
-	case roomNotFoundError:
-	case *gomatrixserverlib.NotAllowed:
-	case missingPrevEventsError:
-	default:
-		switch err {
-		case context.Canceled:
-		case context.DeadlineExceeded:
-		default:
-			return true
-		}
+	switch err {
+	case sql.ErrConnDone:
+	case sql.ErrTxDone:
+		return true
 	}
 	return false
 }
@@ -251,9 +246,6 @@ func isProcessingErrorFatal(err error) bool {
 type roomNotFoundError struct {
 	roomID string
 }
-type unmarshalError struct {
-	err error
-}
 type verifySigError struct {
 	eventID string
 	err     error
@@ -264,7 +256,6 @@ type missingPrevEventsError struct {
 }
 
 func (e roomNotFoundError) Error() string { return fmt.Sprintf("room %q not found", e.roomID) }
-func (e unmarshalError) Error() string    { return fmt.Sprintf("unable to parse event: %s", e.err) }
 func (e verifySigError) Error() string {
 	return fmt.Sprintf("unable to verify signature of event %q: %s", e.eventID, e.err)
 }
@@ -298,6 +289,15 @@ func (t *txnReq) processEDUs(ctx context.Context) {
 				util.GetLogger(ctx).WithError(err).Error("Failed to unmarshal typing event")
 				continue
 			}
+			_, domain, err := gomatrixserverlib.SplitID('@', typingPayload.UserID)
+			if err != nil {
+				util.GetLogger(ctx).WithError(err).Error("Failed to split domain from typing event sender")
+				continue
+			}
+			if domain != t.Origin {
+				util.GetLogger(ctx).Warnf("Dropping typing event where sender domain (%q) doesn't match origin (%q)", domain, t.Origin)
+				continue
+			}
 			if err := eduserverAPI.SendTyping(ctx, t.eduAPI, typingPayload.UserID, typingPayload.RoomID, typingPayload.Typing, 30*1000); err != nil {
 				util.GetLogger(ctx).WithError(err).Error("Failed to send typing event to edu server")
 			}
@@ -343,19 +343,36 @@ func (t *txnReq) processDeviceListUpdate(ctx context.Context, e gomatrixserverli
 	}
 }
 
-func (t *txnReq) processEvent(ctx context.Context, e gomatrixserverlib.Event, isInboundTxn bool) error {
-	prevEventIDs := e.PrevEventIDs()
-
-	// Fetch the state needed to authenticate the event.
-	needed := gomatrixserverlib.StateNeededForAuth([]gomatrixserverlib.Event{e})
-	stateReq := api.QueryStateAfterEventsRequest{
-		RoomID:       e.RoomID(),
-		PrevEventIDs: prevEventIDs,
-		StateToFetch: needed.Tuples(),
+func (t *txnReq) getServers(ctx context.Context, roomID string) []gomatrixserverlib.ServerName {
+	servers := []gomatrixserverlib.ServerName{t.Origin}
+	serverReq := &api.QueryServerJoinedToRoomRequest{
+		RoomID: roomID,
 	}
-	var stateResp api.QueryStateAfterEventsResponse
-	if err := t.rsAPI.QueryStateAfterEvents(ctx, &stateReq, &stateResp); err != nil {
-		return err
+	serverRes := &api.QueryServerJoinedToRoomResponse{}
+	if err := t.rsAPI.QueryServerJoinedToRoom(ctx, serverReq, serverRes); err == nil {
+		servers = append(servers, serverRes.ServerNames...)
+		util.GetLogger(ctx).Infof("Found %d server(s) to query for missing events in %q", len(servers), roomID)
+	}
+	return servers
+}
+
+func (t *txnReq) processEvent(ctx context.Context, e gomatrixserverlib.Event) error {
+	logger := util.GetLogger(ctx).WithField("event_id", e.EventID()).WithField("room_id", e.RoomID())
+
+	// Work out if the roomserver knows everything it needs to know to auth
+	// the event. This includes the prev_events and auth_events.
+	// NOTE! This is going to include prev_events that have an empty state
+	// snapshot. This is because we will need to re-request the event, and
+	// it's /state_ids, in order for it to exist in the roomserver correctly
+	// before the roomserver tries to work out
+	stateReq := api.QueryMissingAuthPrevEventsRequest{
+		RoomID:       e.RoomID(),
+		AuthEventIDs: e.AuthEventIDs(),
+		PrevEventIDs: e.PrevEventIDs(),
+	}
+	var stateResp api.QueryMissingAuthPrevEventsResponse
+	if err := t.rsAPI.QueryMissingAuthPrevEvents(ctx, &stateReq, &stateResp); err != nil {
+		return fmt.Errorf("t.rsAPI.QueryMissingAuthPrevEvents: %w", err)
 	}
 
 	if !stateResp.RoomExists {
@@ -368,8 +385,16 @@ func (t *txnReq) processEvent(ctx context.Context, e gomatrixserverlib.Event, is
 		return roomNotFoundError{e.RoomID()}
 	}
 
-	if !stateResp.PrevEventsExist {
-		return t.processEventWithMissingState(ctx, e, stateResp.RoomVersion, isInboundTxn)
+	if len(stateResp.MissingAuthEventIDs) > 0 {
+		logger.Infof("Event refers to %d unknown auth_events", len(stateResp.MissingAuthEventIDs))
+		if err := t.retrieveMissingAuthEvents(ctx, e, &stateResp); err != nil {
+			return fmt.Errorf("t.retrieveMissingAuthEvents: %w", err)
+		}
+	}
+
+	if len(stateResp.MissingPrevEventIDs) > 0 {
+		logger.Infof("Event refers to %d unknown prev_events", len(stateResp.MissingPrevEventIDs))
+		return t.processEventWithMissingState(ctx, e, stateResp.RoomVersion)
 	}
 
 	// pass the event to the roomserver which will do auth checks
@@ -378,6 +403,7 @@ func (t *txnReq) processEvent(ctx context.Context, e gomatrixserverlib.Event, is
 	return api.SendEvents(
 		context.Background(),
 		t.rsAPI,
+		api.KindNew,
 		[]gomatrixserverlib.HeaderedEvent{
 			e.Headered(stateResp.RoomVersion),
 		},
@@ -386,6 +412,60 @@ func (t *txnReq) processEvent(ctx context.Context, e gomatrixserverlib.Event, is
 	)
 }
 
+func (t *txnReq) retrieveMissingAuthEvents(
+	ctx context.Context, e gomatrixserverlib.Event, stateResp *api.QueryMissingAuthPrevEventsResponse,
+) error {
+	logger := util.GetLogger(ctx).WithField("event_id", e.EventID()).WithField("room_id", e.RoomID())
+
+	missingAuthEvents := make(map[string]struct{})
+	for _, missingAuthEventID := range stateResp.MissingAuthEventIDs {
+		missingAuthEvents[missingAuthEventID] = struct{}{}
+	}
+
+	servers := t.getServers(ctx, e.RoomID())
+	if len(servers) > 5 {
+		servers = servers[:5]
+	}
+withNextEvent:
+	for missingAuthEventID := range missingAuthEvents {
+	withNextServer:
+		for _, server := range servers {
+			logger.Infof("Retrieving missing auth event %q from %q", missingAuthEventID, server)
+			tx, err := t.federation.GetEvent(ctx, server, missingAuthEventID)
+			if err != nil {
+				logger.WithError(err).Warnf("Failed to retrieve auth event %q", missingAuthEventID)
+				continue withNextServer
+			}
+			ev, err := gomatrixserverlib.NewEventFromUntrustedJSON(tx.PDUs[0], stateResp.RoomVersion)
+			if err != nil {
+				logger.WithError(err).Warnf("Failed to unmarshal auth event %q", missingAuthEventID)
+				continue withNextServer
+			}
+			if err = api.SendInputRoomEvents(
+				context.Background(),
+				t.rsAPI,
+				[]api.InputRoomEvent{
+					{
+						Kind:         api.KindOutlier,
+						Event:        ev.Headered(stateResp.RoomVersion),
+						AuthEventIDs: ev.AuthEventIDs(),
+						SendAsServer: api.DoNotSendToOtherServers,
+					},
+				},
+			); err != nil {
+				return fmt.Errorf("api.SendEvents: %w", err)
+			}
+			delete(missingAuthEvents, missingAuthEventID)
+			continue withNextEvent
+		}
+	}
+
+	if missing := len(missingAuthEvents); missing > 0 {
+		return fmt.Errorf("Event refers to %d auth_events which we failed to fetch", missing)
+	}
+	return nil
+}
+
 func checkAllowedByState(e gomatrixserverlib.Event, stateEvents []gomatrixserverlib.Event) error {
 	authUsingState := gomatrixserverlib.NewAuthEvents(nil)
 	for i := range stateEvents {
@@ -397,7 +477,8 @@ func checkAllowedByState(e gomatrixserverlib.Event, stateEvents []gomatrixserver
 	return gomatrixserverlib.Allowed(e, &authUsingState)
 }
 
-func (t *txnReq) processEventWithMissingState(ctx context.Context, e gomatrixserverlib.Event, roomVersion gomatrixserverlib.RoomVersion, isInboundTxn bool) error {
+// nolint:gocyclo
+func (t *txnReq) processEventWithMissingState(ctx context.Context, e gomatrixserverlib.Event, roomVersion gomatrixserverlib.RoomVersion) error {
 	// Do this with a fresh context, so that we keep working even if the
 	// original request times out. With any luck, by the time the remote
 	// side retries, we'll have fetched the missing state.
@@ -423,65 +504,148 @@ func (t *txnReq) processEventWithMissingState(ctx context.Context, e gomatrixser
 	// - fill in the gap completely then process event `e` returning no backwards extremity
 	// - fail to fill in the gap and tell us to terminate the transaction err=not nil
 	// - fail to fill in the gap and tell us to fetch state at the new backwards extremity, and to not terminate the transaction
-	backwardsExtremity, err := t.getMissingEvents(gmectx, e, roomVersion, isInboundTxn)
+	newEvents, err := t.getMissingEvents(gmectx, e, roomVersion)
 	if err != nil {
 		return err
 	}
-	if backwardsExtremity == nil {
-		// we filled in the gap!
+	if len(newEvents) == 0 {
 		return nil
 	}
 
-	// at this point we know we're going to have a gap: we need to work out the room state at the new backwards extremity.
-	// security: we have to do state resolution on the new backwards extremity (TODO: WHY)
-	// Therefore, we cannot just query /state_ids with this event to get the state before. Instead, we need to query
-	// the state AFTER all the prev_events for this event, then apply state resolution to that to get the state before the event.
-	var states []*gomatrixserverlib.RespState
-	needed := gomatrixserverlib.StateNeededForAuth([]gomatrixserverlib.Event{*backwardsExtremity}).Tuples()
-	for _, prevEventID := range backwardsExtremity.PrevEventIDs() {
-		var prevState *gomatrixserverlib.RespState
-		prevState, err = t.lookupStateAfterEvent(gmectx, roomVersion, backwardsExtremity.RoomID(), prevEventID, needed)
-		if err != nil {
-			util.GetLogger(ctx).WithError(err).Errorf("Failed to lookup state after prev_event: %s", prevEventID)
-			return err
-		}
-		states = append(states, prevState)
-	}
-	resolvedState, err := t.resolveStatesAndCheck(gmectx, roomVersion, states, backwardsExtremity)
-	if err != nil {
-		util.GetLogger(ctx).WithError(err).Errorf("Failed to resolve state conflicts for event %s", backwardsExtremity.EventID())
-		return err
+	backwardsExtremity := &newEvents[0]
+	newEvents = newEvents[1:]
+
+	type respState struct {
+		// A snapshot is considered trustworthy if it came from our own roomserver.
+		// That's because the state will have been through state resolution once
+		// already in QueryStateAfterEvent.
+		trustworthy bool
+		*gomatrixserverlib.RespState
 	}
 
-	// pass the event along with the state to the roomserver using a background context so we don't
-	// needlessly expire
-	headeredEvent := e.Headered(roomVersion)
-	return api.SendEventWithState(context.Background(), t.rsAPI, resolvedState, headeredEvent, t.haveEventIDs())
+	// at this point we know we're going to have a gap: we need to work out the room state at the new backwards extremity.
+	// Therefore, we cannot just query /state_ids with this event to get the state before. Instead, we need to query
+	// the state AFTER all the prev_events for this event, then apply state resolution to that to get the state before the event.
+	var states []*respState
+	for _, prevEventID := range backwardsExtremity.PrevEventIDs() {
+		// Look up what the state is after the backward extremity. This will either
+		// come from the roomserver, if we know all the required events, or it will
+		// come from a remote server via /state_ids if not.
+		prevState, trustworthy, lerr := t.lookupStateAfterEvent(gmectx, roomVersion, backwardsExtremity.RoomID(), prevEventID)
+		if lerr != nil {
+			util.GetLogger(ctx).WithError(lerr).Errorf("Failed to lookup state after prev_event: %s", prevEventID)
+			return lerr
+		}
+		// Append the state onto the collected state. We'll run this through the
+		// state resolution next.
+		states = append(states, &respState{trustworthy, prevState})
+	}
+
+	// Now that we have collected all of the state from the prev_events, we'll
+	// run the state through the appropriate state resolution algorithm for the
+	// room if needed. This does a couple of things:
+	// 1. Ensures that the state is deduplicated fully for each state-key tuple
+	// 2. Ensures that we pick the latest events from both sets, in the case that
+	//    one of the prev_events is quite a bit older than the others
+	resolvedState := &gomatrixserverlib.RespState{}
+	switch len(states) {
+	case 0:
+		extremityIsCreate := backwardsExtremity.Type() == gomatrixserverlib.MRoomCreate && backwardsExtremity.StateKeyEquals("")
+		if !extremityIsCreate {
+			// There are no previous states and this isn't the beginning of the
+			// room - this is an error condition!
+			util.GetLogger(ctx).Errorf("Failed to lookup any state after prev_events")
+			return fmt.Errorf("expected %d states but got %d", len(backwardsExtremity.PrevEventIDs()), len(states))
+		}
+	case 1:
+		// There's only one previous state - if it's trustworthy (came from a
+		// local state snapshot which will already have been through state res),
+		// use it as-is. There's no point in resolving it again.
+		if states[0].trustworthy {
+			resolvedState = states[0].RespState
+			break
+		}
+		// Otherwise, if it isn't trustworthy (came from federation), run it through
+		// state resolution anyway for safety, in case there are duplicates.
+		fallthrough
+	default:
+		respStates := make([]*gomatrixserverlib.RespState, len(states))
+		for i := range states {
+			respStates[i] = states[i].RespState
+		}
+		// There's more than one previous state - run them all through state res
+		resolvedState, err = t.resolveStatesAndCheck(gmectx, roomVersion, respStates, backwardsExtremity)
+		if err != nil {
+			util.GetLogger(ctx).WithError(err).Errorf("Failed to resolve state conflicts for event %s", backwardsExtremity.EventID())
+			return err
+		}
+	}
+
+	// First of all, send the backward extremity into the roomserver with the
+	// newly resolved state. This marks the "oldest" point in the backfill and
+	// sets the baseline state for any new events after this.
+	err = api.SendEventWithState(
+		context.Background(),
+		t.rsAPI,
+		api.KindOld,
+		resolvedState,
+		backwardsExtremity.Headered(roomVersion),
+		t.haveEventIDs(),
+	)
+	if err != nil {
+		return fmt.Errorf("api.SendEventWithState: %w", err)
+	}
+
+	// Then send all of the newer backfilled events, of which will all be newer
+	// than the backward extremity, into the roomserver without state. This way
+	// they will automatically fast-forward based on the room state at the
+	// extremity in the last step.
+	headeredNewEvents := make([]gomatrixserverlib.HeaderedEvent, len(newEvents))
+	for i, newEvent := range newEvents {
+		headeredNewEvents[i] = newEvent.Headered(roomVersion)
+	}
+	if err = api.SendEvents(
+		context.Background(),
+		t.rsAPI,
+		api.KindOld,
+		append(headeredNewEvents, e.Headered(roomVersion)),
+		api.DoNotSendToOtherServers,
+		nil,
+	); err != nil {
+		return fmt.Errorf("api.SendEvents: %w", err)
+	}
+
+	return nil
 }
 
 // lookupStateAfterEvent returns the room state after `eventID`, which is the state before eventID with the state of `eventID` (if it's a state event)
 // added into the mix.
-func (t *txnReq) lookupStateAfterEvent(ctx context.Context, roomVersion gomatrixserverlib.RoomVersion, roomID, eventID string, needed []gomatrixserverlib.StateKeyTuple) (*gomatrixserverlib.RespState, error) {
+func (t *txnReq) lookupStateAfterEvent(ctx context.Context, roomVersion gomatrixserverlib.RoomVersion, roomID, eventID string) (*gomatrixserverlib.RespState, bool, error) {
 	// try doing all this locally before we resort to querying federation
-	respState := t.lookupStateAfterEventLocally(ctx, roomID, eventID, needed)
+	respState := t.lookupStateAfterEventLocally(ctx, roomID, eventID)
 	if respState != nil {
-		return respState, nil
+		return respState, true, nil
 	}
 
 	respState, err := t.lookupStateBeforeEvent(ctx, roomVersion, roomID, eventID)
 	if err != nil {
-		return nil, err
+		return nil, false, fmt.Errorf("t.lookupStateBeforeEvent: %w", err)
+	}
+
+	servers := t.getServers(ctx, roomID)
+	if len(servers) > 5 {
+		servers = servers[:5]
 	}
 
 	// fetch the event we're missing and add it to the pile
-	h, err := t.lookupEvent(ctx, roomVersion, eventID, false)
+	h, err := t.lookupEvent(ctx, roomVersion, eventID, false, servers)
 	switch err.(type) {
 	case verifySigError:
-		return respState, nil
+		return respState, false, nil
 	case nil:
 		// do nothing
 	default:
-		return nil, err
+		return nil, false, fmt.Errorf("t.lookupEvent: %w", err)
 	}
 	t.haveEvents[h.EventID()] = h
 	if h.StateKey() != nil {
@@ -499,15 +663,14 @@ func (t *txnReq) lookupStateAfterEvent(ctx context.Context, roomVersion gomatrix
 		}
 	}
 
-	return respState, nil
+	return respState, false, nil
 }
 
-func (t *txnReq) lookupStateAfterEventLocally(ctx context.Context, roomID, eventID string, needed []gomatrixserverlib.StateKeyTuple) *gomatrixserverlib.RespState {
+func (t *txnReq) lookupStateAfterEventLocally(ctx context.Context, roomID, eventID string) *gomatrixserverlib.RespState {
 	var res api.QueryStateAfterEventsResponse
 	err := t.rsAPI.QueryStateAfterEvents(ctx, &api.QueryStateAfterEventsRequest{
 		RoomID:       roomID,
 		PrevEventIDs: []string{eventID},
-		StateToFetch: needed,
 	}, &res)
 	if err != nil || !res.PrevEventsExist {
 		util.GetLogger(ctx).WithError(err).Warnf("failed to query state after %s locally", eventID)
@@ -558,18 +721,12 @@ func (t *txnReq) lookupStateAfterEventLocally(ctx context.Context, roomID, event
 // lookuptStateBeforeEvent returns the room state before the event e, which is just /state_ids and/or /state depending on what
 // the server supports.
 func (t *txnReq) lookupStateBeforeEvent(ctx context.Context, roomVersion gomatrixserverlib.RoomVersion, roomID, eventID string) (
-	respState *gomatrixserverlib.RespState, err error) {
+	*gomatrixserverlib.RespState, error) {
 
 	util.GetLogger(ctx).Infof("lookupStateBeforeEvent %s", eventID)
 
 	// Attempt to fetch the missing state using /state_ids and /events
-	respState, err = t.lookupMissingStateViaStateIDs(ctx, roomID, eventID, roomVersion)
-	if err != nil {
-		// Fallback to /state
-		util.GetLogger(ctx).WithError(err).Warn("lookupStateBeforeEvent failed to /state_ids, falling back to /state")
-		respState, err = t.lookupMissingStateViaState(ctx, roomID, eventID, roomVersion)
-	}
-	return
+	return t.lookupMissingStateViaStateIDs(ctx, roomID, eventID, roomVersion)
 }
 
 func (t *txnReq) resolveStatesAndCheck(ctx context.Context, roomVersion gomatrixserverlib.RoomVersion, states []*gomatrixserverlib.RespState, backwardsExtremity *gomatrixserverlib.Event) (*gomatrixserverlib.RespState, error) {
@@ -588,7 +745,11 @@ retryAllowedState:
 	if err = checkAllowedByState(*backwardsExtremity, resolvedStateEvents); err != nil {
 		switch missing := err.(type) {
 		case gomatrixserverlib.MissingAuthEventError:
-			h, err2 := t.lookupEvent(ctx, roomVersion, missing.AuthEventID, true)
+			servers := t.getServers(ctx, backwardsExtremity.RoomID())
+			if len(servers) > 5 {
+				servers = servers[:5]
+			}
+			h, err2 := t.lookupEvent(ctx, roomVersion, missing.AuthEventID, true, servers)
 			switch err2.(type) {
 			case verifySigError:
 				return &gomatrixserverlib.RespState{
@@ -616,12 +777,9 @@ retryAllowedState:
 // getMissingEvents returns a nil backwardsExtremity if missing events were fetched and handled, else returns the new backwards extremity which we should
 // begin from. Returns an error only if we should terminate the transaction which initiated /get_missing_events
 // This function recursively calls txnReq.processEvent with the missing events, which will be processed before this function returns.
-// This means that we may recursively call this function, as we spider back up prev_events to the min depth.
-func (t *txnReq) getMissingEvents(ctx context.Context, e gomatrixserverlib.Event, roomVersion gomatrixserverlib.RoomVersion, isInboundTxn bool) (backwardsExtremity *gomatrixserverlib.Event, err error) {
-	if !isInboundTxn {
-		// we've recursed here, so just take a state snapshot please!
-		return &e, nil
-	}
+// This means that we may recursively call this function, as we spider back up prev_events.
+// nolint:gocyclo
+func (t *txnReq) getMissingEvents(ctx context.Context, e gomatrixserverlib.Event, roomVersion gomatrixserverlib.RoomVersion) (newEvents []gomatrixserverlib.Event, err error) {
 	logger := util.GetLogger(ctx).WithField("event_id", e.EventID()).WithField("room_id", e.RoomID())
 	needed := gomatrixserverlib.StateNeededForAuth([]gomatrixserverlib.Event{e})
 	// query latest events (our trusted forward extremities)
@@ -632,26 +790,50 @@ func (t *txnReq) getMissingEvents(ctx context.Context, e gomatrixserverlib.Event
 	var res api.QueryLatestEventsAndStateResponse
 	if err = t.rsAPI.QueryLatestEventsAndState(ctx, &req, &res); err != nil {
 		logger.WithError(err).Warn("Failed to query latest events")
-		return &e, nil
+		return nil, err
 	}
 	latestEvents := make([]string, len(res.LatestEvents))
 	for i := range res.LatestEvents {
 		latestEvents[i] = res.LatestEvents[i].EventID
 	}
-	// this server just sent us an event for which we do not know its prev_events - ask that server for those prev_events.
-	minDepth := int(res.Depth) - 20
-	if minDepth < 0 {
-		minDepth = 0
+
+	servers := []gomatrixserverlib.ServerName{t.Origin}
+	serverReq := &api.QueryServerJoinedToRoomRequest{
+		RoomID: e.RoomID(),
+	}
+	serverRes := &api.QueryServerJoinedToRoomResponse{}
+	if err = t.rsAPI.QueryServerJoinedToRoom(ctx, serverReq, serverRes); err == nil {
+		servers = append(servers, serverRes.ServerNames...)
+		logger.Infof("Found %d server(s) to query for missing events", len(servers))
+	}
+
+	var missingResp *gomatrixserverlib.RespMissingEvents
+	for _, server := range servers {
+		var m gomatrixserverlib.RespMissingEvents
+		if m, err = t.federation.LookupMissingEvents(ctx, server, e.RoomID(), gomatrixserverlib.MissingEvents{
+			Limit: 20,
+			// The latest event IDs that the sender already has. These are skipped when retrieving the previous events of latest_events.
+			EarliestEvents: latestEvents,
+			// The event IDs to retrieve the previous events for.
+			LatestEvents: []string{e.EventID()},
+		}, roomVersion); err == nil {
+			missingResp = &m
+			break
+		} else {
+			logger.WithError(err).Errorf("%s pushed us an event but %q did not respond to /get_missing_events", t.Origin, server)
+		}
+	}
+
+	if missingResp == nil {
+		logger.WithError(err).Errorf(
+			"%s pushed us an event but %d server(s) couldn't give us details about prev_events via /get_missing_events - dropping this event until it can",
+			t.Origin, len(servers),
+		)
+		return nil, missingPrevEventsError{
+			eventID: e.EventID(),
+			err:     err,
+		}
 	}
-	missingResp, err := t.federation.LookupMissingEvents(ctx, t.Origin, e.RoomID(), gomatrixserverlib.MissingEvents{
-		Limit: 20,
-		// synapse uses the min depth they've ever seen in that room
-		MinDepth: minDepth,
-		// The latest event IDs that the sender already has. These are skipped when retrieving the previous events of latest_events.
-		EarliestEvents: latestEvents,
-		// The event IDs to retrieve the previous events for.
-		LatestEvents: []string{e.EventID()},
-	}, roomVersion)
 
 	// security: how we handle failures depends on whether or not this event will become the new forward extremity for the room.
 	// There's 2 scenarios to consider:
@@ -664,20 +846,10 @@ func (t *txnReq) getMissingEvents(ctx context.Context, e gomatrixserverlib.Event
 	// https://github.com/matrix-org/synapse/pull/3456
 	// https://github.com/matrix-org/synapse/blob/229eb81498b0fe1da81e9b5b333a0285acde9446/synapse/handlers/federation.py#L335
 	// For now, we do not allow Case B, so reject the event.
-	if err != nil {
-		logger.WithError(err).Errorf(
-			"%s pushed us an event but couldn't give us details about prev_events via /get_missing_events - dropping this event until it can",
-			t.Origin,
-		)
-		return nil, missingPrevEventsError{
-			eventID: e.EventID(),
-			err:     err,
-		}
-	}
 	logger.Infof("get_missing_events returned %d events", len(missingResp.Events))
 
 	// topologically sort and sanity check that we are making forward progress
-	newEvents := gomatrixserverlib.ReverseTopologicalOrdering(missingResp.Events, gomatrixserverlib.TopologicalOrderByPrevEvents)
+	newEvents = gomatrixserverlib.ReverseTopologicalOrdering(missingResp.Events, gomatrixserverlib.TopologicalOrderByPrevEvents)
 	shouldHaveSomeEventIDs := e.PrevEventIDs()
 	hasPrevEvent := false
 Event:
@@ -700,16 +872,9 @@ Event:
 			err:     err,
 		}
 	}
-	// process the missing events then the event which started this whole thing
-	for _, ev := range append(newEvents, e) {
-		err := t.processEvent(ctx, ev, false)
-		if err != nil {
-			return nil, err
-		}
-	}
 
 	// we processed everything!
-	return nil, nil
+	return newEvents, nil
 }
 
 func (t *txnReq) lookupMissingStateViaState(ctx context.Context, roomID, eventID string, roomVersion gomatrixserverlib.RoomVersion) (
@@ -725,6 +890,7 @@ func (t *txnReq) lookupMissingStateViaState(ctx context.Context, roomID, eventID
 	return &state, nil
 }
 
+// nolint:gocyclo
 func (t *txnReq) lookupMissingStateViaStateIDs(ctx context.Context, roomID, eventID string, roomVersion gomatrixserverlib.RoomVersion) (
 	*gomatrixserverlib.RespState, error) {
 	util.GetLogger(ctx).Infof("lookupMissingStateViaStateIDs %s", eventID)
@@ -762,52 +928,120 @@ func (t *txnReq) lookupMissingStateViaStateIDs(ctx context.Context, roomID, even
 		}
 	}
 
+	concurrentRequests := 8
+	missingCount := len(missing)
+
+	// If over 50% of the auth/state events from /state_ids are missing
+	// then we'll just call /state instead, otherwise we'll just end up
+	// hammering the remote side with /event requests unnecessarily.
+	if missingCount > concurrentRequests && missingCount > len(wantIDs)/2 {
+		util.GetLogger(ctx).WithFields(logrus.Fields{
+			"missing":           missingCount,
+			"event_id":          eventID,
+			"room_id":           roomID,
+			"total_state":       len(stateIDs.StateEventIDs),
+			"total_auth_events": len(stateIDs.AuthEventIDs),
+		}).Info("Fetching all state at event")
+		return t.lookupMissingStateViaState(ctx, roomID, eventID, roomVersion)
+	}
+
 	util.GetLogger(ctx).WithFields(logrus.Fields{
-		"missing":           len(missing),
-		"event_id":          eventID,
-		"room_id":           roomID,
-		"total_state":       len(stateIDs.StateEventIDs),
-		"total_auth_events": len(stateIDs.AuthEventIDs),
+		"missing":             missingCount,
+		"event_id":            eventID,
+		"room_id":             roomID,
+		"total_state":         len(stateIDs.StateEventIDs),
+		"total_auth_events":   len(stateIDs.AuthEventIDs),
+		"concurrent_requests": concurrentRequests,
 	}).Info("Fetching missing state at event")
 
+	// Get a list of servers to fetch from.
+	servers := t.getServers(ctx, roomID)
+	if len(servers) > 5 {
+		servers = servers[:5]
+	}
+
+	// Create a queue containing all of the missing event IDs that we want
+	// to retrieve.
+	pending := make(chan string, missingCount)
 	for missingEventID := range missing {
+		pending <- missingEventID
+	}
+	close(pending)
+
+	// Define how many workers we should start to do this.
+	if missingCount < concurrentRequests {
+		concurrentRequests = missingCount
+	}
+
+	// Create the wait group.
+	var fetchgroup sync.WaitGroup
+	fetchgroup.Add(concurrentRequests)
+
+	// This is the only place where we'll write to t.haveEvents from
+	// multiple goroutines, and everywhere else is blocked on this
+	// synchronous function anyway.
+	var haveEventsMutex sync.Mutex
+
+	// Define what we'll do in order to fetch the missing event ID.
+	fetch := func(missingEventID string) {
 		var h *gomatrixserverlib.HeaderedEvent
-		h, err = t.lookupEvent(ctx, roomVersion, missingEventID, false)
+		h, err = t.lookupEvent(ctx, roomVersion, missingEventID, false, servers)
 		switch err.(type) {
 		case verifySigError:
-			continue
+			return
 		case nil:
-			// do nothing
+			break
 		default:
-			return nil, err
+			util.GetLogger(ctx).WithFields(logrus.Fields{
+				"event_id": missingEventID,
+				"room_id":  roomID,
+			}).Info("Failed to fetch missing event")
+			return
 		}
+		haveEventsMutex.Lock()
 		t.haveEvents[h.EventID()] = h
+		haveEventsMutex.Unlock()
 	}
+
+	// Create the worker.
+	worker := func(ch <-chan string) {
+		defer fetchgroup.Done()
+		for missingEventID := range ch {
+			fetch(missingEventID)
+		}
+	}
+
+	// Start the workers.
+	for i := 0; i < concurrentRequests; i++ {
+		go worker(pending)
+	}
+
+	// Wait for the workers to finish.
+	fetchgroup.Wait()
 	resp, err := t.createRespStateFromStateIDs(stateIDs)
 	return resp, err
 }
 
 func (t *txnReq) createRespStateFromStateIDs(stateIDs gomatrixserverlib.RespStateIDs) (
-	*gomatrixserverlib.RespState, error) {
+	*gomatrixserverlib.RespState, error) { // nolint:unparam
 	// create a RespState response using the response to /state_ids as a guide
-	respState := gomatrixserverlib.RespState{
-		AuthEvents:  make([]gomatrixserverlib.Event, len(stateIDs.AuthEventIDs)),
-		StateEvents: make([]gomatrixserverlib.Event, len(stateIDs.StateEventIDs)),
-	}
+	respState := gomatrixserverlib.RespState{}
 
 	for i := range stateIDs.StateEventIDs {
 		ev, ok := t.haveEvents[stateIDs.StateEventIDs[i]]
 		if !ok {
-			return nil, fmt.Errorf("missing state event %s", stateIDs.StateEventIDs[i])
+			logrus.Warnf("Missing state event in createRespStateFromStateIDs: %s", stateIDs.StateEventIDs[i])
+			continue
 		}
-		respState.StateEvents[i] = ev.Unwrap()
+		respState.StateEvents = append(respState.StateEvents, ev.Unwrap())
 	}
 	for i := range stateIDs.AuthEventIDs {
 		ev, ok := t.haveEvents[stateIDs.AuthEventIDs[i]]
 		if !ok {
-			return nil, fmt.Errorf("missing auth event %s", stateIDs.AuthEventIDs[i])
+			logrus.Warnf("Missing auth event in createRespStateFromStateIDs: %s", stateIDs.AuthEventIDs[i])
+			continue
 		}
-		respState.AuthEvents[i] = ev.Unwrap()
+		respState.AuthEvents = append(respState.AuthEvents, ev.Unwrap())
 	}
 	// We purposefully do not do auth checks on the returned events, as they will still
 	// be processed in the exact same way, just as a 'rejected' event
@@ -815,7 +1049,7 @@ func (t *txnReq) createRespStateFromStateIDs(stateIDs gomatrixserverlib.RespStat
 	return &respState, nil
 }
 
-func (t *txnReq) lookupEvent(ctx context.Context, roomVersion gomatrixserverlib.RoomVersion, missingEventID string, localFirst bool) (*gomatrixserverlib.HeaderedEvent, error) {
+func (t *txnReq) lookupEvent(ctx context.Context, roomVersion gomatrixserverlib.RoomVersion, missingEventID string, localFirst bool, servers []gomatrixserverlib.ServerName) (*gomatrixserverlib.HeaderedEvent, error) {
 	if localFirst {
 		// fetch from the roomserver
 		queryReq := api.QueryEventsByIDRequest{
@@ -828,19 +1062,27 @@ func (t *txnReq) lookupEvent(ctx context.Context, roomVersion gomatrixserverlib.
 			return &queryRes.Events[0], nil
 		}
 	}
-	txn, err := t.federation.GetEvent(ctx, t.Origin, missingEventID)
-	if err != nil || len(txn.PDUs) == 0 {
-		util.GetLogger(ctx).WithError(err).WithField("event_id", missingEventID).Warn("failed to get missing /event for event ID")
-		return nil, err
-	}
-	pdu := txn.PDUs[0]
 	var event gomatrixserverlib.Event
-	event, err = gomatrixserverlib.NewEventFromUntrustedJSON(pdu, roomVersion)
-	if err != nil {
-		util.GetLogger(ctx).WithError(err).Warnf("Transaction: Failed to parse event JSON of event %q", event.EventID())
-		return nil, unmarshalError{err}
+	found := false
+	for _, serverName := range servers {
+		txn, err := t.federation.GetEvent(ctx, serverName, missingEventID)
+		if err != nil || len(txn.PDUs) == 0 {
+			util.GetLogger(ctx).WithError(err).WithField("event_id", missingEventID).Warn("Failed to get missing /event for event ID")
+			continue
+		}
+		event, err = gomatrixserverlib.NewEventFromUntrustedJSON(txn.PDUs[0], roomVersion)
+		if err != nil {
+			util.GetLogger(ctx).WithError(err).WithField("event_id", missingEventID).Warnf("Transaction: Failed to parse event JSON of event")
+			continue
+		}
+		found = true
+		break
 	}
-	if err = gomatrixserverlib.VerifyAllEventSignatures(ctx, []gomatrixserverlib.Event{event}, t.keys); err != nil {
+	if !found {
+		util.GetLogger(ctx).WithField("event_id", missingEventID).Warnf("Failed to get missing /event for event ID from %d server(s)", len(servers))
+		return nil, fmt.Errorf("wasn't able to find event via %d server(s)", len(servers))
+	}
+	if err := gomatrixserverlib.VerifyAllEventSignatures(ctx, []gomatrixserverlib.Event{event}, t.keys); err != nil {
 		util.GetLogger(ctx).WithError(err).Warnf("Transaction: Couldn't validate signature of event %q", event.EventID())
 		return nil, verifySigError{event.EventID(), err}
 	}

federationapi/routing/send_test.go

@@ -77,10 +77,11 @@ func (p *testEDUProducer) InputSendToDeviceEvent(
 }
 
 type testRoomserverAPI struct {
-	inputRoomEvents           []api.InputRoomEvent
-	queryStateAfterEvents     func(*api.QueryStateAfterEventsRequest) api.QueryStateAfterEventsResponse
-	queryEventsByID           func(req *api.QueryEventsByIDRequest) api.QueryEventsByIDResponse
-	queryLatestEventsAndState func(*api.QueryLatestEventsAndStateRequest) api.QueryLatestEventsAndStateResponse
+	inputRoomEvents            []api.InputRoomEvent
+	queryMissingAuthPrevEvents func(*api.QueryMissingAuthPrevEventsRequest) api.QueryMissingAuthPrevEventsResponse
+	queryStateAfterEvents      func(*api.QueryStateAfterEventsRequest) api.QueryStateAfterEventsResponse
+	queryEventsByID            func(req *api.QueryEventsByIDRequest) api.QueryEventsByIDResponse
+	queryLatestEventsAndState  func(*api.QueryLatestEventsAndStateRequest) api.QueryLatestEventsAndStateResponse
 }
 
 func (t *testRoomserverAPI) SetFederationSenderAPI(fsAPI fsAPI.FederationSenderInternalAPI) {}
@@ -162,6 +163,20 @@ func (t *testRoomserverAPI) QueryStateAfterEvents(
 	return nil
 }
 
+// Query the state after a list of events in a room from the room server.
+func (t *testRoomserverAPI) QueryMissingAuthPrevEvents(
+	ctx context.Context,
+	request *api.QueryMissingAuthPrevEventsRequest,
+	response *api.QueryMissingAuthPrevEventsResponse,
+) error {
+	response.RoomVersion = testRoomVersion
+	res := t.queryMissingAuthPrevEvents(request)
+	response.RoomExists = res.RoomExists
+	response.MissingAuthEventIDs = res.MissingAuthEventIDs
+	response.MissingPrevEventIDs = res.MissingPrevEventIDs
+	return nil
+}
+
 // Query a list of events by event ID.
 func (t *testRoomserverAPI) QueryEventsByID(
 	ctx context.Context,
@@ -453,11 +468,11 @@ func assertInputRoomEvents(t *testing.T, got []api.InputRoomEvent, want []gomatr
 // to the roomserver. It's the most basic test possible.
 func TestBasicTransaction(t *testing.T) {
 	rsAPI := &testRoomserverAPI{
-		queryStateAfterEvents: func(req *api.QueryStateAfterEventsRequest) api.QueryStateAfterEventsResponse {
-			return api.QueryStateAfterEventsResponse{
-				PrevEventsExist: true,
-				RoomExists:      true,
-				StateEvents:     fromStateTuples(req.StateToFetch, nil),
+		queryMissingAuthPrevEvents: func(req *api.QueryMissingAuthPrevEventsRequest) api.QueryMissingAuthPrevEventsResponse {
+			return api.QueryMissingAuthPrevEventsResponse{
+				RoomExists:          true,
+				MissingAuthEventIDs: []string{},
+				MissingPrevEventIDs: []string{},
 			}
 		},
 	}
@@ -473,14 +488,11 @@ func TestBasicTransaction(t *testing.T) {
 // as it does the auth check.
 func TestTransactionFailAuthChecks(t *testing.T) {
 	rsAPI := &testRoomserverAPI{
-		queryStateAfterEvents: func(req *api.QueryStateAfterEventsRequest) api.QueryStateAfterEventsResponse {
-			return api.QueryStateAfterEventsResponse{
-				PrevEventsExist: true,
-				RoomExists:      true,
-				// omit the create event so auth checks fail
-				StateEvents: fromStateTuples(req.StateToFetch, []gomatrixserverlib.StateKeyTuple{
-					{EventType: gomatrixserverlib.MRoomCreate, StateKey: ""},
-				}),
+		queryMissingAuthPrevEvents: func(req *api.QueryMissingAuthPrevEventsRequest) api.QueryMissingAuthPrevEventsResponse {
+			return api.QueryMissingAuthPrevEventsResponse{
+				RoomExists:          true,
+				MissingAuthEventIDs: []string{},
+				MissingPrevEventIDs: []string{},
 			}
 		},
 	}
@@ -504,28 +516,41 @@ func TestTransactionFetchMissingPrevEvents(t *testing.T) {
 
 	var rsAPI *testRoomserverAPI // ref here so we can refer to inputRoomEvents inside these functions
 	rsAPI = &testRoomserverAPI{
+		queryEventsByID: func(req *api.QueryEventsByIDRequest) api.QueryEventsByIDResponse {
+			res := api.QueryEventsByIDResponse{}
+			for _, ev := range testEvents {
+				for _, id := range req.EventIDs {
+					if ev.EventID() == id {
+						res.Events = append(res.Events, ev)
+					}
+				}
+			}
+			return res
+		},
 		queryStateAfterEvents: func(req *api.QueryStateAfterEventsRequest) api.QueryStateAfterEventsResponse {
-			// we expect this to be called three times:
-			// - first with input event to realise there's a gap
-			// - second with the prevEvent to realise there is no gap
-			// - third with the input event to realise there is no longer a gap
-			prevEventsExist := false
+			return api.QueryStateAfterEventsResponse{
+				PrevEventsExist: true,
+				StateEvents:     testEvents[:5],
+			}
+		},
+		queryMissingAuthPrevEvents: func(req *api.QueryMissingAuthPrevEventsRequest) api.QueryMissingAuthPrevEventsResponse {
+			missingPrevEvent := []string{"missing_prev_event"}
 			if len(req.PrevEventIDs) == 1 {
 				switch req.PrevEventIDs[0] {
 				case haveEvent.EventID():
-					prevEventsExist = true
+					missingPrevEvent = []string{}
 				case prevEvent.EventID():
 					// we only have this event if we've been send prevEvent
 					if len(rsAPI.inputRoomEvents) == 1 && rsAPI.inputRoomEvents[0].Event.EventID() == prevEvent.EventID() {
-						prevEventsExist = true
+						missingPrevEvent = []string{}
 					}
 				}
 			}
 
-			return api.QueryStateAfterEventsResponse{
-				PrevEventsExist: prevEventsExist,
-				RoomExists:      true,
-				StateEvents:     fromStateTuples(req.StateToFetch, nil),
+			return api.QueryMissingAuthPrevEventsResponse{
+				RoomExists:          true,
+				MissingAuthEventIDs: []string{},
+				MissingPrevEventIDs: missingPrevEvent,
 			}
 		},
 		queryLatestEventsAndState: func(req *api.QueryLatestEventsAndStateRequest) api.QueryLatestEventsAndStateResponse {
@@ -626,6 +651,38 @@ func TestTransactionFetchMissingStateByStateIDs(t *testing.T) {
 				StateEvents:     stateEvents,
 			}
 		},
+
+		queryMissingAuthPrevEvents: func(req *api.QueryMissingAuthPrevEventsRequest) api.QueryMissingAuthPrevEventsResponse {
+			askingForEvent := req.PrevEventIDs[0]
+			haveEventB := false
+			haveEventC := false
+			for _, ev := range rsAPI.inputRoomEvents {
+				switch ev.Event.EventID() {
+				case eventB.EventID():
+					haveEventB = true
+				case eventC.EventID():
+					haveEventC = true
+				}
+			}
+			prevEventExists := false
+			if askingForEvent == eventC.EventID() {
+				prevEventExists = haveEventC
+			} else if askingForEvent == eventB.EventID() {
+				prevEventExists = haveEventB
+			}
+
+			var missingPrevEvent []string
+			if !prevEventExists {
+				missingPrevEvent = []string{"test"}
+			}
+
+			return api.QueryMissingAuthPrevEventsResponse{
+				RoomExists:          true,
+				MissingAuthEventIDs: []string{},
+				MissingPrevEventIDs: missingPrevEvent,
+			}
+		},
+
 		queryLatestEventsAndState: func(req *api.QueryLatestEventsAndStateRequest) api.QueryLatestEventsAndStateResponse {
 			omitTuples := []gomatrixserverlib.StateKeyTuple{
 				{EventType: gomatrixserverlib.MRoomPowerLevels, StateKey: ""},

federationapi/routing/threepid.go

@@ -89,7 +89,7 @@ func CreateInvitesFrom3PIDInvites(
 	}
 
 	// Send all the events
-	if err := api.SendEvents(req.Context(), rsAPI, evs, cfg.Matrix.ServerName, nil); err != nil {
+	if err := api.SendEvents(req.Context(), rsAPI, api.KindNew, evs, cfg.Matrix.ServerName, nil); err != nil {
 		util.GetLogger(req.Context()).WithError(err).Error("SendEvents failed")
 		return jsonerror.InternalServerError()
 	}
@@ -174,6 +174,7 @@ func ExchangeThirdPartyInvite(
 	// Send the event to the roomserver
 	if err = api.SendEvents(
 		httpReq.Context(), rsAPI,
+		api.KindNew,
 		[]gomatrixserverlib.HeaderedEvent{
 			signedEvent.Event.Headered(verRes.RoomVersion),
 		},

federationsender/federationsender.go

@@ -24,6 +24,7 @@ import (
 	"github.com/matrix-org/dendrite/federationsender/statistics"
 	"github.com/matrix-org/dendrite/federationsender/storage"
 	"github.com/matrix-org/dendrite/internal/setup"
+	"github.com/matrix-org/dendrite/internal/setup/kafka"
 	roomserverAPI "github.com/matrix-org/dendrite/roomserver/api"
 	"github.com/matrix-org/gomatrixserverlib"
 	"github.com/sirupsen/logrus"
@@ -55,6 +56,8 @@ func NewInternalAPI(
 		FailuresUntilBlacklist: cfg.FederationMaxRetries,
 	}
 
+	consumer, _ := kafka.SetupConsumerProducer(&cfg.Matrix.Kafka)
+
 	queues := queue.NewOutgoingQueues(
 		federationSenderDB, cfg.Matrix.ServerName, federation,
 		rsAPI, stats,
@@ -66,7 +69,7 @@ func NewInternalAPI(
 	)
 
 	rsConsumer := consumers.NewOutputRoomEventConsumer(
-		cfg, base.KafkaConsumer, queues,
+		cfg, consumer, queues,
 		federationSenderDB, rsAPI,
 	)
 	if err = rsConsumer.Start(); err != nil {
@@ -74,13 +77,13 @@ func NewInternalAPI(
 	}
 
 	tsConsumer := consumers.NewOutputEDUConsumer(
-		cfg, base.KafkaConsumer, queues, federationSenderDB,
+		cfg, consumer, queues, federationSenderDB,
 	)
 	if err := tsConsumer.Start(); err != nil {
 		logrus.WithError(err).Panic("failed to start typing server consumer")
 	}
 	keyConsumer := consumers.NewKeyChangeConsumer(
-		&base.Cfg.KeyServer, base.KafkaConsumer, queues, federationSenderDB, rsAPI,
+		&base.Cfg.KeyServer, consumer, queues, federationSenderDB, rsAPI,
 	)
 	if err := keyConsumer.Start(); err != nil {
 		logrus.WithError(err).Panic("failed to start key server consumer")

federationsender/internal/api.go

@@ -109,6 +109,8 @@ func (a *FederationSenderInternalAPI) doRequest(
 func (a *FederationSenderInternalAPI) GetUserDevices(
 	ctx context.Context, s gomatrixserverlib.ServerName, userID string,
 ) (gomatrixserverlib.RespUserDevices, error) {
+	ctx, cancel := context.WithTimeout(ctx, time.Second*30)
+	defer cancel()
 	ires, err := a.doRequest(s, func() (interface{}, error) {
 		return a.federation.GetUserDevices(ctx, s, userID)
 	})
@@ -121,6 +123,8 @@ func (a *FederationSenderInternalAPI) GetUserDevices(
 func (a *FederationSenderInternalAPI) ClaimKeys(
 	ctx context.Context, s gomatrixserverlib.ServerName, oneTimeKeys map[string]map[string]string,
 ) (gomatrixserverlib.RespClaimKeys, error) {
+	ctx, cancel := context.WithTimeout(ctx, time.Second*30)
+	defer cancel()
 	ires, err := a.doRequest(s, func() (interface{}, error) {
 		return a.federation.ClaimKeys(ctx, s, oneTimeKeys)
 	})
@@ -145,6 +149,8 @@ func (a *FederationSenderInternalAPI) QueryKeys(
 func (a *FederationSenderInternalAPI) Backfill(
 	ctx context.Context, s gomatrixserverlib.ServerName, roomID string, limit int, eventIDs []string,
 ) (res gomatrixserverlib.Transaction, err error) {
+	ctx, cancel := context.WithTimeout(ctx, time.Second*30)
+	defer cancel()
 	ires, err := a.doRequest(s, func() (interface{}, error) {
 		return a.federation.Backfill(ctx, s, roomID, limit, eventIDs)
 	})
@@ -157,6 +163,8 @@ func (a *FederationSenderInternalAPI) Backfill(
 func (a *FederationSenderInternalAPI) LookupState(
 	ctx context.Context, s gomatrixserverlib.ServerName, roomID, eventID string, roomVersion gomatrixserverlib.RoomVersion,
 ) (res gomatrixserverlib.RespState, err error) {
+	ctx, cancel := context.WithTimeout(ctx, time.Second*30)
+	defer cancel()
 	ires, err := a.doRequest(s, func() (interface{}, error) {
 		return a.federation.LookupState(ctx, s, roomID, eventID, roomVersion)
 	})
@@ -169,6 +177,8 @@ func (a *FederationSenderInternalAPI) LookupState(
 func (a *FederationSenderInternalAPI) LookupStateIDs(
 	ctx context.Context, s gomatrixserverlib.ServerName, roomID, eventID string,
 ) (res gomatrixserverlib.RespStateIDs, err error) {
+	ctx, cancel := context.WithTimeout(ctx, time.Second*30)
+	defer cancel()
 	ires, err := a.doRequest(s, func() (interface{}, error) {
 		return a.federation.LookupStateIDs(ctx, s, roomID, eventID)
 	})
@@ -181,6 +191,8 @@ func (a *FederationSenderInternalAPI) LookupStateIDs(
 func (a *FederationSenderInternalAPI) GetEvent(
 	ctx context.Context, s gomatrixserverlib.ServerName, eventID string,
 ) (res gomatrixserverlib.Transaction, err error) {
+	ctx, cancel := context.WithTimeout(ctx, time.Second*30)
+	defer cancel()
 	ires, err := a.doRequest(s, func() (interface{}, error) {
 		return a.federation.GetEvent(ctx, s, eventID)
 	})
@@ -193,6 +205,8 @@ func (a *FederationSenderInternalAPI) GetEvent(
 func (a *FederationSenderInternalAPI) GetServerKeys(
 	ctx context.Context, s gomatrixserverlib.ServerName,
 ) (gomatrixserverlib.ServerKeys, error) {
+	ctx, cancel := context.WithTimeout(ctx, time.Second*30)
+	defer cancel()
 	ires, err := a.doRequest(s, func() (interface{}, error) {
 		return a.federation.GetServerKeys(ctx, s)
 	})
@@ -205,6 +219,8 @@ func (a *FederationSenderInternalAPI) GetServerKeys(
 func (a *FederationSenderInternalAPI) LookupServerKeys(
 	ctx context.Context, s gomatrixserverlib.ServerName, keyRequests map[gomatrixserverlib.PublicKeyLookupRequest]gomatrixserverlib.Timestamp,
 ) ([]gomatrixserverlib.ServerKeys, error) {
+	ctx, cancel := context.WithTimeout(ctx, time.Minute)
+	defer cancel()
 	ires, err := a.doRequest(s, func() (interface{}, error) {
 		return a.federation.LookupServerKeys(ctx, s, keyRequests)
 	})

federationsender/internal/perform.go

@@ -2,6 +2,7 @@ package internal
 
 import (
 	"context"
+	"encoding/json"
 	"errors"
 	"fmt"
 	"time"
@@ -42,7 +43,7 @@ type federatedJoin struct {
 	RoomID string
 }
 
-// PerformJoinRequest implements api.FederationSenderInternalAPI
+// PerformJoin implements api.FederationSenderInternalAPI
 func (r *FederationSenderInternalAPI) PerformJoin(
 	ctx context.Context,
 	request *api.PerformJoinRequest,
@@ -174,8 +175,10 @@ func (r *FederationSenderInternalAPI) performJoinUsingServer(
 
 	// Work out if we support the room version that has been supplied in
 	// the make_join response.
+	// "If not provided, the room version is assumed to be either "1" or "2"."
+	// https://matrix.org/docs/spec/server_server/unstable#get-matrix-federation-v1-make-join-roomid-userid
 	if respMakeJoin.RoomVersion == "" {
-		respMakeJoin.RoomVersion = gomatrixserverlib.RoomVersionV1
+		respMakeJoin.RoomVersion = setDefaultRoomVersionFromJoinEvent(respMakeJoin.JoinEvent)
 	}
 	if _, err = respMakeJoin.RoomVersion.EventFormat(); err != nil {
 		return fmt.Errorf("respMakeJoin.RoomVersion.EventFormat: %w", err)
@@ -193,6 +196,11 @@ func (r *FederationSenderInternalAPI) performJoinUsingServer(
 		return fmt.Errorf("respMakeJoin.JoinEvent.Build: %w", err)
 	}
 
+	// No longer reuse the request context from this point forward.
+	// We don't want the client timing out to interrupt the join.
+	var cancel context.CancelFunc
+	ctx, cancel = context.WithCancel(context.Background())
+
 	// Try to perform a send_join using the newly built event.
 	respSendJoin, err := r.federation.SendJoin(
 		ctx,
@@ -202,17 +210,23 @@ func (r *FederationSenderInternalAPI) performJoinUsingServer(
 	)
 	if err != nil {
 		r.statistics.ForServer(serverName).Failure()
+		cancel()
 		return fmt.Errorf("r.federation.SendJoin: %w", err)
 	}
 	r.statistics.ForServer(serverName).Success()
 
+	// Sanity-check the join response to ensure that it has a create
+	// event, that the room version is known, etc.
+	if err := sanityCheckSendJoinResponse(respSendJoin); err != nil {
+		cancel()
+		return fmt.Errorf("sanityCheckSendJoinResponse: %w", err)
+	}
+
 	// Process the join response in a goroutine. The idea here is
 	// that we'll try and wait for as long as possible for the work
 	// to complete, but if the client does give up waiting, we'll
 	// still continue to process the join anyway so that we don't
 	// waste the effort.
-	var cancel context.CancelFunc
-	ctx, cancel = context.WithCancel(context.Background())
 	go func() {
 		defer cancel()
 
@@ -232,8 +246,9 @@ func (r *FederationSenderInternalAPI) performJoinUsingServer(
 		// If we successfully performed a send_join above then the other
 		// server now thinks we're a part of the room. Send the newly
 		// returned state to the roomserver to update our local view.
-		if err = roomserverAPI.SendEventWithRewrite(
+		if err = roomserverAPI.SendEventWithState(
 			ctx, r.rsAPI,
+			roomserverAPI.KindNew,
 			respState,
 			event.Headered(respMakeJoin.RoomVersion),
 			nil,
@@ -579,3 +594,53 @@ func (r *FederationSenderInternalAPI) PerformBroadcastEDU(
 
 	return nil
 }
+
+func sanityCheckSendJoinResponse(respSendJoin gomatrixserverlib.RespSendJoin) error {
+	// sanity check we have a create event and it has a known room version
+	for _, ev := range respSendJoin.AuthEvents {
+		if ev.Type() == gomatrixserverlib.MRoomCreate && ev.StateKeyEquals("") {
+			// make sure the room version is known
+			content := ev.Content()
+			verBody := struct {
+				Version string `json:"room_version"`
+			}{}
+			err := json.Unmarshal(content, &verBody)
+			if err != nil {
+				return err
+			}
+			if verBody.Version == "" {
+				// https://matrix.org/docs/spec/client_server/r0.6.0#m-room-create
+				// The version of the room. Defaults to "1" if the key does not exist.
+				verBody.Version = "1"
+			}
+			knownVersions := gomatrixserverlib.RoomVersions()
+			if _, ok := knownVersions[gomatrixserverlib.RoomVersion(verBody.Version)]; !ok {
+				return fmt.Errorf("send_join m.room.create event has an unknown room version: %s", verBody.Version)
+			}
+			return nil
+		}
+	}
+	return fmt.Errorf("send_join response is missing m.room.create event")
+}
+
+func setDefaultRoomVersionFromJoinEvent(joinEvent gomatrixserverlib.EventBuilder) gomatrixserverlib.RoomVersion {
+	// if auth events are not event references we know it must be v3+
+	// we have to do these shenanigans to satisfy sytest, specifically for:
+	// "Outbound federation rejects m.room.create events with an unknown room version"
+	hasEventRefs := true
+	authEvents, ok := joinEvent.AuthEvents.([]interface{})
+	if ok {
+		if len(authEvents) > 0 {
+			_, ok = authEvents[0].(string)
+			if ok {
+				// event refs are objects, not strings, so we know we must be dealing with a v3+ room.
+				hasEventRefs = false
+			}
+		}
+	}
+
+	if hasEventRefs {
+		return gomatrixserverlib.RoomVersionV1
+	}
+	return gomatrixserverlib.RoomVersionV4
+}

federationsender/queue/destinationqueue.go

@@ -224,7 +224,7 @@ func (oq *destinationQueue) backgroundSend() {
 			// The worker is idle so stop the goroutine. It'll get
 			// restarted automatically the next time we have an event to
 			// send.
-			log.Debugf("Queue %q has been idle for %s, going to sleep", oq.destination, queueIdleTimeout)
+			log.Tracef("Queue %q has been idle for %s, going to sleep", oq.destination, queueIdleTimeout)
 			return
 		}
 
@@ -349,7 +349,7 @@ func (oq *destinationQueue) nextTransaction() (bool, error) {
 	// TODO: we should check for 500-ish fails vs 400-ish here,
 	// since we shouldn't queue things indefinitely in response
 	// to a 400-ish error
-	ctx, cancel = context.WithTimeout(context.Background(), time.Second*30)
+	ctx, cancel = context.WithTimeout(context.Background(), time.Minute*5)
 	defer cancel()
 	_, err = oq.client.SendTransaction(ctx, t)
 	switch err.(type) {

go.mod

@@ -22,7 +22,7 @@ require (
 	github.com/matrix-org/go-http-js-libp2p v0.0.0-20200518170932-783164aeeda4
 	github.com/matrix-org/go-sqlite3-js v0.0.0-20200522092705-bc8506ccbcf3
 	github.com/matrix-org/gomatrix v0.0.0-20200827122206-7dd5e2a05bcd
-	github.com/matrix-org/gomatrixserverlib v0.0.0-20200925165243-b9780a852681
+	github.com/matrix-org/gomatrixserverlib v0.0.0-20201020160945-66a6d4926351
 	github.com/matrix-org/naffka v0.0.0-20200901083833-bcdd62999a91
 	github.com/matrix-org/util v0.0.0-20200807132607-55161520e1d4
 	github.com/mattn/go-sqlite3 v1.14.2
@@ -37,9 +37,10 @@ require (
 	github.com/tidwall/sjson v1.1.1
 	github.com/uber/jaeger-client-go v2.25.0+incompatible
 	github.com/uber/jaeger-lib v2.2.0+incompatible
-	github.com/yggdrasil-network/yggdrasil-go v0.3.15-0.20200806125501-cd4685a3b4de
+	github.com/yggdrasil-network/yggdrasil-go v0.3.15-0.20201006093556-760d9a7fd5ee
 	go.uber.org/atomic v1.6.0
 	golang.org/x/crypto v0.0.0-20200820211705-5c72a883971a
+	golang.org/x/net v0.0.0-20200528225125-3c3fba18258b
 	gopkg.in/h2non/bimg.v1 v1.1.4
 	gopkg.in/yaml.v2 v2.3.0
 )

go.sum

@@ -569,8 +569,8 @@ github.com/matrix-org/gomatrix v0.0.0-20190528120928-7df988a63f26 h1:Hr3zjRsq2bh
 github.com/matrix-org/gomatrix v0.0.0-20190528120928-7df988a63f26/go.mod h1:3fxX6gUjWyI/2Bt7J1OLhpCzOfO/bB3AiX0cJtEKud0=
 github.com/matrix-org/gomatrix v0.0.0-20200827122206-7dd5e2a05bcd h1:xVrqJK3xHREMNjwjljkAUaadalWc0rRbmVuQatzmgwg=
 github.com/matrix-org/gomatrix v0.0.0-20200827122206-7dd5e2a05bcd/go.mod h1:/gBX06Kw0exX1HrwmoBibFA98yBk/jxKpGVeyQbff+s=
-github.com/matrix-org/gomatrixserverlib v0.0.0-20200925165243-b9780a852681 h1:75fM7vPHiFGt+XxktT17LJD972XMtJ1n7FU1MpC08Zc=
-github.com/matrix-org/gomatrixserverlib v0.0.0-20200925165243-b9780a852681/go.mod h1:JsAzE1Ll3+gDWS9JSUHPJiiyAksvOOnGWF2nXdg4ZzU=
+github.com/matrix-org/gomatrixserverlib v0.0.0-20201020160945-66a6d4926351 h1:Yg+fvSEvDLjMFkkE4W0e1yBTtCMVmH3tztw/RhTLPcA=
+github.com/matrix-org/gomatrixserverlib v0.0.0-20201020160945-66a6d4926351/go.mod h1:JsAzE1Ll3+gDWS9JSUHPJiiyAksvOOnGWF2nXdg4ZzU=
 github.com/matrix-org/naffka v0.0.0-20200901083833-bcdd62999a91 h1:HJ6U3S3ljJqNffYMcIeAncp5qT/i+ZMiJ2JC2F0aXP4=
 github.com/matrix-org/naffka v0.0.0-20200901083833-bcdd62999a91/go.mod h1:sjyPyRxKM5uw1nD2cJ6O2OxI6GOqyVBfNXqKjBZTBZE=
 github.com/matrix-org/util v0.0.0-20190711121626-527ce5ddefc7 h1:ntrLa/8xVzeSs8vHFHK25k0C+NV74sYMJnNSg5NoSRo=
@@ -851,8 +851,8 @@ github.com/xdg/scram v0.0.0-20180814205039-7eeb5667e42c/go.mod h1:lB8K/P019DLNhe
 github.com/xdg/stringprep v1.0.0/go.mod h1:Jhud4/sHMO4oL310DaZAKk9ZaJ08SJfe+sJh0HrGL1Y=
 github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=
 github.com/yggdrasil-network/yggdrasil-extras v0.0.0-20200525205615-6c8a4a2e8855/go.mod h1:xQdsh08Io6nV4WRnOVTe6gI8/2iTvfLDQ0CYa5aMt+I=
-github.com/yggdrasil-network/yggdrasil-go v0.3.15-0.20200806125501-cd4685a3b4de h1:p91aw0Mvol825U+5bvV9BBPl+HQxIczj7wxIOxZs70M=
-github.com/yggdrasil-network/yggdrasil-go v0.3.15-0.20200806125501-cd4685a3b4de/go.mod h1:d+Nz6SPeG6kmeSPFL0cvfWfgwEql75fUnZiAONgvyBE=
+github.com/yggdrasil-network/yggdrasil-go v0.3.15-0.20201006093556-760d9a7fd5ee h1:Kot820OfxWfYrk5di5f4S5s0jXXrQj8w8BG5826HAv4=
+github.com/yggdrasil-network/yggdrasil-go v0.3.15-0.20201006093556-760d9a7fd5ee/go.mod h1:d+Nz6SPeG6kmeSPFL0cvfWfgwEql75fUnZiAONgvyBE=
 go.opencensus.io v0.18.0/go.mod h1:vKdFvxhtzZ9onBp9VKHK8z/sRpBMnKAsufL7wlDrCOA=
 go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
 go.opencensus.io v0.22.1/go.mod h1:Ap50jQcDJrx6rB6VgeeFPtuPIf3wMRvRfrfYDO6+BmA=

internal/config/config.go

@@ -16,7 +16,6 @@ package config
 
 import (
 	"bytes"
-	"crypto/sha256"
 	"encoding/pem"
 	"fmt"
 	"io"
@@ -63,7 +62,7 @@ type Dendrite struct {
 	KeyServer        KeyServer        `yaml:"key_server"`
 	MediaAPI         MediaAPI         `yaml:"media_api"`
 	RoomServer       RoomServer       `yaml:"room_server"`
-	ServerKeyAPI     ServerKeyAPI     `yaml:"server_key_api"`
+	SigningKeyServer SigningKeyServer `yaml:"signing_key_server"`
 	SyncAPI          SyncAPI          `yaml:"sync_api"`
 	UserAPI          UserAPI          `yaml:"user_api"`
 
@@ -252,20 +251,6 @@ func loadConfig(
 		c.Global.OldVerifyKeys[i].KeyID, c.Global.OldVerifyKeys[i].PrivateKey = keyID, privateKey
 	}
 
-	for _, certPath := range c.FederationAPI.FederationCertificatePaths {
-		absCertPath := absPath(basePath, certPath)
-		var pemData []byte
-		pemData, err = readFile(absCertPath)
-		if err != nil {
-			return nil, err
-		}
-		fingerprint := fingerprintPEM(pemData)
-		if fingerprint == nil {
-			return nil, fmt.Errorf("no certificate PEM data in %q", absCertPath)
-		}
-		c.FederationAPI.TLSFingerPrints = append(c.FederationAPI.TLSFingerPrints, *fingerprint)
-	}
-
 	c.MediaAPI.AbsBasePath = Path(absPath(basePath, c.MediaAPI.BasePath))
 
 	// Generate data from config options
@@ -317,7 +302,7 @@ func (c *Dendrite) Defaults() {
 	c.KeyServer.Defaults()
 	c.MediaAPI.Defaults()
 	c.RoomServer.Defaults()
-	c.ServerKeyAPI.Defaults()
+	c.SigningKeyServer.Defaults()
 	c.SyncAPI.Defaults()
 	c.UserAPI.Defaults()
 	c.AppServiceAPI.Defaults()
@@ -333,7 +318,7 @@ func (c *Dendrite) Verify(configErrs *ConfigErrors, isMonolith bool) {
 		&c.Global, &c.ClientAPI,
 		&c.EDUServer, &c.FederationAPI, &c.FederationSender,
 		&c.KeyServer, &c.MediaAPI, &c.RoomServer,
-		&c.ServerKeyAPI, &c.SyncAPI, &c.UserAPI,
+		&c.SigningKeyServer, &c.SyncAPI, &c.UserAPI,
 		&c.AppServiceAPI,
 	} {
 		c.Verify(configErrs, isMonolith)
@@ -348,7 +333,7 @@ func (c *Dendrite) Wiring() {
 	c.KeyServer.Matrix = &c.Global
 	c.MediaAPI.Matrix = &c.Global
 	c.RoomServer.Matrix = &c.Global
-	c.ServerKeyAPI.Matrix = &c.Global
+	c.SigningKeyServer.Matrix = &c.Global
 	c.SyncAPI.Matrix = &c.Global
 	c.UserAPI.Matrix = &c.Global
 	c.AppServiceAPI.Matrix = &c.Global
@@ -494,20 +479,6 @@ func readKeyPEM(path string, data []byte, enforceKeyIDFormat bool) (gomatrixserv
 	}
 }
 
-func fingerprintPEM(data []byte) *gomatrixserverlib.TLSFingerprint {
-	for {
-		var certDERBlock *pem.Block
-		certDERBlock, data = pem.Decode(data)
-		if data == nil {
-			return nil
-		}
-		if certDERBlock.Type == "CERTIFICATE" {
-			digest := sha256.Sum256(certDERBlock.Bytes)
-			return &gomatrixserverlib.TLSFingerprint{SHA256: digest[:]}
-		}
-	}
-}
-
 // AppServiceURL returns a HTTP URL for where the appservice component is listening.
 func (config *Dendrite) AppServiceURL() string {
 	// Hard code the appservice server to talk HTTP for now.
@@ -553,13 +524,13 @@ func (config *Dendrite) FederationSenderURL() string {
 	return string(config.FederationSender.InternalAPI.Connect)
 }
 
-// ServerKeyAPIURL returns an HTTP URL for where the server key API is listening.
-func (config *Dendrite) ServerKeyAPIURL() string {
-	// Hard code the server key API server to talk HTTP for now.
+// SigningKeyServerURL returns an HTTP URL for where the signing key server is listening.
+func (config *Dendrite) SigningKeyServerURL() string {
+	// Hard code the signing key server to talk HTTP for now.
 	// If we support HTTPS we need to think of a practical way to do certificate validation.
 	// People setting up servers shouldn't need to get a certificate valid for the public
 	// internet for an internal API.
-	return string(config.ServerKeyAPI.InternalAPI.Connect)
+	return string(config.SigningKeyServer.InternalAPI.Connect)
 }
 
 // KeyServerURL returns an HTTP URL for where the key server is listening.

internal/config/config_federationapi.go

@@ -1,7 +1,5 @@
 package config
 
-import "github.com/matrix-org/gomatrixserverlib"
-
 type FederationAPI struct {
 	Matrix *Global `yaml:"-"`
 
@@ -14,10 +12,6 @@ type FederationAPI struct {
 	// to match one of these certificates.
 	// The certificates should be in PEM format.
 	FederationCertificatePaths []Path `yaml:"federation_certificates"`
-
-	// A list of SHA256 TLS fingerprints for the X509 certificates used by the
-	// federation listener for this server.
-	TLSFingerPrints []gomatrixserverlib.TLSFingerprint `yaml:"-"`
 }
 
 func (c *FederationAPI) Defaults() {

internal/config/config_signingkeyserver.go

@@ -2,31 +2,34 @@ package config
 
 import "github.com/matrix-org/gomatrixserverlib"
 
-type ServerKeyAPI struct {
+type SigningKeyServer struct {
 	Matrix *Global `yaml:"-"`
 
 	InternalAPI InternalAPIOptions `yaml:"internal_api"`
 
-	// The ServerKey database caches the public keys of remote servers.
+	// The SigningKeyServer database caches the public keys of remote servers.
 	// It may be accessed by the FederationAPI, the ClientAPI, and the MediaAPI.
 	Database DatabaseOptions `yaml:"database"`
 
 	// Perspective keyservers, to use as a backup when direct key fetch
 	// requests don't succeed
 	KeyPerspectives KeyPerspectives `yaml:"key_perspectives"`
+
+	// Should we prefer direct key fetches over perspective ones?
+	PreferDirectFetch bool `yaml:"prefer_direct_fetch"`
 }
 
-func (c *ServerKeyAPI) Defaults() {
+func (c *SigningKeyServer) Defaults() {
 	c.InternalAPI.Listen = "http://localhost:7780"
 	c.InternalAPI.Connect = "http://localhost:7780"
 	c.Database.Defaults()
-	c.Database.ConnectionString = "file:serverkeyapi.db"
+	c.Database.ConnectionString = "file:signingkeyserver.db"
 }
 
-func (c *ServerKeyAPI) Verify(configErrs *ConfigErrors, isMonolith bool) {
-	checkURL(configErrs, "server_key_api.internal_api.listen", string(c.InternalAPI.Listen))
-	checkURL(configErrs, "server_key_api.internal_api.bind", string(c.InternalAPI.Connect))
-	checkNotEmpty(configErrs, "server_key_api.database.connection_string", string(c.Database.ConnectionString))
+func (c *SigningKeyServer) Verify(configErrs *ConfigErrors, isMonolith bool) {
+	checkURL(configErrs, "signing_key_server.internal_api.listen", string(c.InternalAPI.Listen))
+	checkURL(configErrs, "signing_key_server.internal_api.bind", string(c.InternalAPI.Connect))
+	checkNotEmpty(configErrs, "signing_key_server.database.connection_string", string(c.Database.ConnectionString))
 }
 
 // KeyPerspectives are used to configure perspective key servers for

internal/config/config_test.go

@@ -253,19 +253,6 @@ Key-ID: ` + testKeyID + `
 -----END MATRIX PRIVATE KEY-----
 `
 
-func TestFingerprintPEM(t *testing.T) {
-	got := fingerprintPEM([]byte(testCert))
-	if got == nil {
-		t.Error("failed to calculate fingerprint")
-	}
-	if string(got.SHA256) != testCertFingerprint {
-		t.Errorf("bad fingerprint: wanted %q got %q", got, testCertFingerprint)
-	}
-
-}
-
-const testCertFingerprint = "56.\\SPQxE\xd4\x95\xfb\xf6\xd5\x04\x91\xcb/\x07\xb1^\x88\x08\xe3\xc1p\xdfY\x04\x19w\xcb"
-
 const testCert = `
 -----BEGIN CERTIFICATE-----
 MIIE0zCCArugAwIBAgIJAPype3u24LJeMA0GCSqGSIb3DQEBCwUAMAAwHhcNMTcw

internal/setup/base.go

@@ -15,8 +15,10 @@
 package setup
 
 import (
+	"crypto/tls"
 	"fmt"
 	"io"
+	"net"
 	"net/http"
 	"net/url"
 	"time"
@@ -25,14 +27,12 @@ import (
 	"github.com/matrix-org/dendrite/internal/httputil"
 	"github.com/matrix-org/gomatrixserverlib"
 	"github.com/prometheus/client_golang/prometheus/promhttp"
-
-	"github.com/matrix-org/naffka"
-	naffkaStorage "github.com/matrix-org/naffka/storage"
+	"golang.org/x/net/http2"
+	"golang.org/x/net/http2/h2c"
 
 	"github.com/matrix-org/dendrite/internal"
 	"github.com/matrix-org/dendrite/userapi/storage/accounts"
 
-	"github.com/Shopify/sarama"
 	"github.com/gorilla/mux"
 
 	appserviceAPI "github.com/matrix-org/dendrite/appservice/api"
@@ -46,8 +46,8 @@ import (
 	keyinthttp "github.com/matrix-org/dendrite/keyserver/inthttp"
 	roomserverAPI "github.com/matrix-org/dendrite/roomserver/api"
 	rsinthttp "github.com/matrix-org/dendrite/roomserver/inthttp"
-	serverKeyAPI "github.com/matrix-org/dendrite/serverkeyapi/api"
-	skinthttp "github.com/matrix-org/dendrite/serverkeyapi/inthttp"
+	skapi "github.com/matrix-org/dendrite/signingkeyserver/api"
+	skinthttp "github.com/matrix-org/dendrite/signingkeyserver/inthttp"
 	userapi "github.com/matrix-org/dendrite/userapi/api"
 	userapiinthttp "github.com/matrix-org/dendrite/userapi/inthttp"
 	"github.com/sirupsen/logrus"
@@ -69,17 +69,18 @@ type BaseDendrite struct {
 	PublicMediaAPIMux      *mux.Router
 	InternalAPIMux         *mux.Router
 	UseHTTPAPIs            bool
+	apiHttpClient          *http.Client
 	httpClient             *http.Client
 	Cfg                    *config.Dendrite
 	Caches                 *caching.Caches
-	KafkaConsumer          sarama.Consumer
-	KafkaProducer          sarama.SyncProducer
+	//	KafkaConsumer          sarama.Consumer
+	//	KafkaProducer          sarama.SyncProducer
 }
 
 const HTTPServerTimeout = time.Minute * 5
 const HTTPClientTimeout = time.Second * 30
 
-const NoExternalListener = ""
+const NoListener = ""
 
 // NewBaseDendrite creates a new instance to be used by a component.
 // The componentName is used for logging purposes, and should be a friendly name
@@ -105,19 +106,27 @@ func NewBaseDendrite(cfg *config.Dendrite, componentName string, useHTTPAPIs boo
 		logrus.WithError(err).Panicf("failed to start opentracing")
 	}
 
-	var kafkaConsumer sarama.Consumer
-	var kafkaProducer sarama.SyncProducer
-	if cfg.Global.Kafka.UseNaffka {
-		kafkaConsumer, kafkaProducer = setupNaffka(cfg)
-	} else {
-		kafkaConsumer, kafkaProducer = setupKafka(cfg)
-	}
-
 	cache, err := caching.NewInMemoryLRUCache(true)
 	if err != nil {
 		logrus.WithError(err).Warnf("Failed to create cache")
 	}
 
+	apiClient := http.Client{
+		Timeout: time.Minute * 10,
+		Transport: &http2.Transport{
+			AllowHTTP: true,
+			DialTLS: func(network, addr string, _ *tls.Config) (net.Conn, error) {
+				// Ordinarily HTTP/2 would expect TLS, but the remote listener is
+				// H2C-enabled (HTTP/2 without encryption). Overriding the DialTLS
+				// function with a plain Dial allows us to trick the HTTP client
+				// into establishing a HTTP/2 connection without TLS.
+				// TODO: Eventually we will want to look at authenticating and
+				// encrypting these internal HTTP APIs, at which point we will have
+				// to reconsider H2C and change all this anyway.
+				return net.Dial(network, addr)
+			},
+		},
+	}
 	client := http.Client{Timeout: HTTPClientTimeout}
 	if cfg.FederationSender.Proxy.Enabled {
 		client.Transport = &http.Transport{Proxy: http.ProxyURL(&url.URL{
@@ -148,9 +157,8 @@ func NewBaseDendrite(cfg *config.Dendrite, componentName string, useHTTPAPIs boo
 		PublicKeyAPIMux:        mux.NewRouter().SkipClean(true).PathPrefix(httputil.PublicKeyPathPrefix).Subrouter().UseEncodedPath(),
 		PublicMediaAPIMux:      mux.NewRouter().SkipClean(true).PathPrefix(httputil.PublicMediaPathPrefix).Subrouter().UseEncodedPath(),
 		InternalAPIMux:         mux.NewRouter().SkipClean(true).PathPrefix(httputil.InternalPathPrefix).Subrouter().UseEncodedPath(),
+		apiHttpClient:          &apiClient,
 		httpClient:             &client,
-		KafkaConsumer:          kafkaConsumer,
-		KafkaProducer:          kafkaProducer,
 	}
 }
 
@@ -161,7 +169,7 @@ func (b *BaseDendrite) Close() error {
 
 // AppserviceHTTPClient returns the AppServiceQueryAPI for hitting the appservice component over HTTP.
 func (b *BaseDendrite) AppserviceHTTPClient() appserviceAPI.AppServiceQueryAPI {
-	a, err := asinthttp.NewAppserviceClient(b.Cfg.AppServiceURL(), b.httpClient)
+	a, err := asinthttp.NewAppserviceClient(b.Cfg.AppServiceURL(), b.apiHttpClient)
 	if err != nil {
 		logrus.WithError(err).Panic("CreateHTTPAppServiceAPIs failed")
 	}
@@ -170,27 +178,27 @@ func (b *BaseDendrite) AppserviceHTTPClient() appserviceAPI.AppServiceQueryAPI {
 
 // RoomserverHTTPClient returns RoomserverInternalAPI for hitting the roomserver over HTTP.
 func (b *BaseDendrite) RoomserverHTTPClient() roomserverAPI.RoomserverInternalAPI {
-	rsAPI, err := rsinthttp.NewRoomserverClient(b.Cfg.RoomServerURL(), b.httpClient, b.Caches)
+	rsAPI, err := rsinthttp.NewRoomserverClient(b.Cfg.RoomServerURL(), b.apiHttpClient, b.Caches)
 	if err != nil {
-		logrus.WithError(err).Panic("RoomserverHTTPClient failed", b.httpClient)
+		logrus.WithError(err).Panic("RoomserverHTTPClient failed", b.apiHttpClient)
 	}
 	return rsAPI
 }
 
 // UserAPIClient returns UserInternalAPI for hitting the userapi over HTTP.
 func (b *BaseDendrite) UserAPIClient() userapi.UserInternalAPI {
-	userAPI, err := userapiinthttp.NewUserAPIClient(b.Cfg.UserAPIURL(), b.httpClient)
+	userAPI, err := userapiinthttp.NewUserAPIClient(b.Cfg.UserAPIURL(), b.apiHttpClient)
 	if err != nil {
-		logrus.WithError(err).Panic("UserAPIClient failed", b.httpClient)
+		logrus.WithError(err).Panic("UserAPIClient failed", b.apiHttpClient)
 	}
 	return userAPI
 }
 
 // EDUServerClient returns EDUServerInputAPI for hitting the EDU server over HTTP
 func (b *BaseDendrite) EDUServerClient() eduServerAPI.EDUServerInputAPI {
-	e, err := eduinthttp.NewEDUServerClient(b.Cfg.EDUServerURL(), b.httpClient)
+	e, err := eduinthttp.NewEDUServerClient(b.Cfg.EDUServerURL(), b.apiHttpClient)
 	if err != nil {
-		logrus.WithError(err).Panic("EDUServerClient failed", b.httpClient)
+		logrus.WithError(err).Panic("EDUServerClient failed", b.apiHttpClient)
 	}
 	return e
 }
@@ -198,31 +206,31 @@ func (b *BaseDendrite) EDUServerClient() eduServerAPI.EDUServerInputAPI {
 // FederationSenderHTTPClient returns FederationSenderInternalAPI for hitting
 // the federation sender over HTTP
 func (b *BaseDendrite) FederationSenderHTTPClient() federationSenderAPI.FederationSenderInternalAPI {
-	f, err := fsinthttp.NewFederationSenderClient(b.Cfg.FederationSenderURL(), b.httpClient)
+	f, err := fsinthttp.NewFederationSenderClient(b.Cfg.FederationSenderURL(), b.apiHttpClient)
 	if err != nil {
-		logrus.WithError(err).Panic("FederationSenderHTTPClient failed", b.httpClient)
+		logrus.WithError(err).Panic("FederationSenderHTTPClient failed", b.apiHttpClient)
 	}
 	return f
 }
 
-// ServerKeyAPIClient returns ServerKeyInternalAPI for hitting the server key API over HTTP
-func (b *BaseDendrite) ServerKeyAPIClient() serverKeyAPI.ServerKeyInternalAPI {
-	f, err := skinthttp.NewServerKeyClient(
-		b.Cfg.ServerKeyAPIURL(),
-		b.httpClient,
+// SigningKeyServerHTTPClient returns SigningKeyServer for hitting the signing key server over HTTP
+func (b *BaseDendrite) SigningKeyServerHTTPClient() skapi.SigningKeyServerAPI {
+	f, err := skinthttp.NewSigningKeyServerClient(
+		b.Cfg.SigningKeyServerURL(),
+		b.apiHttpClient,
 		b.Caches,
 	)
 	if err != nil {
-		logrus.WithError(err).Panic("NewServerKeyInternalAPIHTTP failed", b.httpClient)
+		logrus.WithError(err).Panic("SigningKeyServerHTTPClient failed", b.httpClient)
 	}
 	return f
 }
 
 // KeyServerHTTPClient returns KeyInternalAPI for hitting the key server over HTTP
 func (b *BaseDendrite) KeyServerHTTPClient() keyserverAPI.KeyInternalAPI {
-	f, err := keyinthttp.NewKeyServerClient(b.Cfg.KeyServerURL(), b.httpClient)
+	f, err := keyinthttp.NewKeyServerClient(b.Cfg.KeyServerURL(), b.apiHttpClient)
 	if err != nil {
-		logrus.WithError(err).Panic("KeyServerHTTPClient failed", b.httpClient)
+		logrus.WithError(err).Panic("KeyServerHTTPClient failed", b.apiHttpClient)
 	}
 	return f
 }
@@ -238,13 +246,25 @@ func (b *BaseDendrite) CreateAccountsDB() accounts.Database {
 	return db
 }
 
+// CreateClient creates a new client (normally used for media fetch requests).
+// Should only be called once per component.
+func (b *BaseDendrite) CreateClient() *gomatrixserverlib.Client {
+	client := gomatrixserverlib.NewClient(
+		b.Cfg.FederationSender.DisableTLSValidation,
+	)
+	client.SetUserAgent(fmt.Sprintf("Dendrite/%s", internal.VersionString()))
+	return client
+}
+
 // CreateFederationClient creates a new federation client. Should only be called
 // once per component.
 func (b *BaseDendrite) CreateFederationClient() *gomatrixserverlib.FederationClient {
-	return gomatrixserverlib.NewFederationClient(
+	client := gomatrixserverlib.NewFederationClientWithTimeout(
 		b.Cfg.Global.ServerName, b.Cfg.Global.KeyID, b.Cfg.Global.PrivateKey,
-		b.Cfg.FederationSender.DisableTLSValidation,
+		b.Cfg.FederationSender.DisableTLSValidation, time.Minute*5,
 	)
+	client.SetUserAgent(fmt.Sprintf("Dendrite/%s", internal.VersionString()))
+	return client
 }
 
 // SetupAndServeHTTP sets up the HTTP server to serve endpoints registered on
@@ -257,22 +277,28 @@ func (b *BaseDendrite) SetupAndServeHTTP(
 	internalAddr, _ := internalHTTPAddr.Address()
 	externalAddr, _ := externalHTTPAddr.Address()
 
-	internalRouter := mux.NewRouter().SkipClean(true).UseEncodedPath()
-	externalRouter := internalRouter
+	externalRouter := mux.NewRouter().SkipClean(true).UseEncodedPath()
+	internalRouter := externalRouter
 
-	internalServ := &http.Server{
-		Addr:         string(internalAddr),
+	externalServ := &http.Server{
+		Addr:         string(externalAddr),
 		WriteTimeout: HTTPServerTimeout,
-		Handler:      internalRouter,
+		Handler:      externalRouter,
 	}
-	externalServ := internalServ
+	internalServ := externalServ
 
-	if externalAddr != NoExternalListener && externalAddr != internalAddr {
-		externalRouter = mux.NewRouter().SkipClean(true).UseEncodedPath()
-		externalServ = &http.Server{
-			Addr:         string(externalAddr),
-			WriteTimeout: HTTPServerTimeout,
-			Handler:      externalRouter,
+	if internalAddr != NoListener && externalAddr != internalAddr {
+		// H2C allows us to accept HTTP/2 connections without TLS
+		// encryption. Since we don't currently require any form of
+		// authentication or encryption on these internal HTTP APIs,
+		// H2C gives us all of the advantages of HTTP/2 (such as
+		// stream multiplexing and avoiding head-of-line blocking)
+		// without enabling TLS.
+		internalH2S := &http2.Server{}
+		internalRouter = mux.NewRouter().SkipClean(true).UseEncodedPath()
+		internalServ = &http.Server{
+			Addr:    string(internalAddr),
+			Handler: h2c.NewHandler(internalRouter, internalH2S),
 		}
 	}
 
@@ -286,23 +312,25 @@ func (b *BaseDendrite) SetupAndServeHTTP(
 	externalRouter.PathPrefix(httputil.PublicFederationPathPrefix).Handler(b.PublicFederationAPIMux)
 	externalRouter.PathPrefix(httputil.PublicMediaPathPrefix).Handler(b.PublicMediaAPIMux)
 
-	go func() {
-		logrus.Infof("Starting %s listener on %s", b.componentName, internalServ.Addr)
-		if certFile != nil && keyFile != nil {
-			if err := internalServ.ListenAndServeTLS(*certFile, *keyFile); err != nil {
-				logrus.WithError(err).Fatal("failed to serve HTTPS")
-			}
-		} else {
-			if err := internalServ.ListenAndServe(); err != nil {
-				logrus.WithError(err).Fatal("failed to serve HTTP")
-			}
-		}
-		logrus.Infof("Stopped %s listener on %s", b.componentName, internalServ.Addr)
-	}()
-
-	if externalAddr != NoExternalListener && internalAddr != externalAddr {
+	if internalAddr != NoListener && internalAddr != externalAddr {
 		go func() {
-			logrus.Infof("Starting %s listener on %s", b.componentName, externalServ.Addr)
+			logrus.Infof("Starting internal %s listener on %s", b.componentName, internalServ.Addr)
+			if certFile != nil && keyFile != nil {
+				if err := internalServ.ListenAndServeTLS(*certFile, *keyFile); err != nil {
+					logrus.WithError(err).Fatal("failed to serve HTTPS")
+				}
+			} else {
+				if err := internalServ.ListenAndServe(); err != nil {
+					logrus.WithError(err).Fatal("failed to serve HTTP")
+				}
+			}
+			logrus.Infof("Stopped internal %s listener on %s", b.componentName, internalServ.Addr)
+		}()
+	}
+
+	if externalAddr != NoListener {
+		go func() {
+			logrus.Infof("Starting external %s listener on %s", b.componentName, externalServ.Addr)
 			if certFile != nil && keyFile != nil {
 				if err := externalServ.ListenAndServeTLS(*certFile, *keyFile); err != nil {
 					logrus.WithError(err).Fatal("failed to serve HTTPS")
@@ -312,37 +340,9 @@ func (b *BaseDendrite) SetupAndServeHTTP(
 					logrus.WithError(err).Fatal("failed to serve HTTP")
 				}
 			}
-			logrus.Infof("Stopped %s listener on %s", b.componentName, externalServ.Addr)
+			logrus.Infof("Stopped external %s listener on %s", b.componentName, externalServ.Addr)
 		}()
 	}
 
 	select {}
 }
-
-// setupKafka creates kafka consumer/producer pair from the config.
-func setupKafka(cfg *config.Dendrite) (sarama.Consumer, sarama.SyncProducer) {
-	consumer, err := sarama.NewConsumer(cfg.Global.Kafka.Addresses, nil)
-	if err != nil {
-		logrus.WithError(err).Panic("failed to start kafka consumer")
-	}
-
-	producer, err := sarama.NewSyncProducer(cfg.Global.Kafka.Addresses, nil)
-	if err != nil {
-		logrus.WithError(err).Panic("failed to setup kafka producers")
-	}
-
-	return consumer, producer
-}
-
-// setupNaffka creates kafka consumer/producer pair from the config.
-func setupNaffka(cfg *config.Dendrite) (sarama.Consumer, sarama.SyncProducer) {
-	naffkaDB, err := naffkaStorage.NewDatabase(string(cfg.Global.Kafka.Database.ConnectionString))
-	if err != nil {
-		logrus.WithError(err).Panic("Failed to setup naffka database")
-	}
-	naff, err := naffka.New(naffkaDB)
-	if err != nil {
-		logrus.WithError(err).Panic("Failed to setup naffka")
-	}
-	return naff, naff
-}

internal/setup/kafka/kafka.go

@@ -0,0 +1,53 @@
+package kafka
+
+import (
+	"github.com/Shopify/sarama"
+	"github.com/matrix-org/dendrite/internal/config"
+	"github.com/matrix-org/naffka"
+	naffkaStorage "github.com/matrix-org/naffka/storage"
+	"github.com/sirupsen/logrus"
+)
+
+func SetupConsumerProducer(cfg *config.Kafka) (sarama.Consumer, sarama.SyncProducer) {
+	if cfg.UseNaffka {
+		return setupNaffka(cfg)
+	}
+	return setupKafka(cfg)
+}
+
+// setupKafka creates kafka consumer/producer pair from the config.
+func setupKafka(cfg *config.Kafka) (sarama.Consumer, sarama.SyncProducer) {
+	consumer, err := sarama.NewConsumer(cfg.Addresses, nil)
+	if err != nil {
+		logrus.WithError(err).Panic("failed to start kafka consumer")
+	}
+
+	producer, err := sarama.NewSyncProducer(cfg.Addresses, nil)
+	if err != nil {
+		logrus.WithError(err).Panic("failed to setup kafka producers")
+	}
+
+	return consumer, producer
+}
+
+// In monolith mode with Naffka, we don't have the same constraints about
+// consuming the same topic from more than one place like we do with Kafka.
+// Therefore, we will only open one Naffka connection in case Naffka is
+// running on SQLite.
+var naffkaInstance *naffka.Naffka
+
+// setupNaffka creates kafka consumer/producer pair from the config.
+func setupNaffka(cfg *config.Kafka) (sarama.Consumer, sarama.SyncProducer) {
+	if naffkaInstance != nil {
+		return naffkaInstance, naffkaInstance
+	}
+	naffkaDB, err := naffkaStorage.NewDatabase(string(cfg.Database.ConnectionString))
+	if err != nil {
+		logrus.WithError(err).Panic("Failed to setup naffka database")
+	}
+	naffkaInstance, err = naffka.New(naffkaDB)
+	if err != nil {
+		logrus.WithError(err).Panic("Failed to setup naffka")
+	}
+	return naffkaInstance, naffkaInstance
+}

internal/setup/monolith.go

@@ -15,7 +15,6 @@
 package setup
 
 import (
-	"github.com/Shopify/sarama"
 	"github.com/gorilla/mux"
 	appserviceAPI "github.com/matrix-org/dendrite/appservice/api"
 	"github.com/matrix-org/dendrite/clientapi"
@@ -28,7 +27,7 @@ import (
 	keyAPI "github.com/matrix-org/dendrite/keyserver/api"
 	"github.com/matrix-org/dendrite/mediaapi"
 	roomserverAPI "github.com/matrix-org/dendrite/roomserver/api"
-	serverKeyAPI "github.com/matrix-org/dendrite/serverkeyapi/api"
+	serverKeyAPI "github.com/matrix-org/dendrite/signingkeyserver/api"
 	"github.com/matrix-org/dendrite/syncapi"
 	userapi "github.com/matrix-org/dendrite/userapi/api"
 	"github.com/matrix-org/dendrite/userapi/storage/accounts"
@@ -38,19 +37,17 @@ import (
 // Monolith represents an instantiation of all dependencies required to build
 // all components of Dendrite, for use in monolith mode.
 type Monolith struct {
-	Config        *config.Dendrite
-	AccountDB     accounts.Database
-	KeyRing       *gomatrixserverlib.KeyRing
-	Client        *gomatrixserverlib.Client
-	FedClient     *gomatrixserverlib.FederationClient
-	KafkaConsumer sarama.Consumer
-	KafkaProducer sarama.SyncProducer
+	Config    *config.Dendrite
+	AccountDB accounts.Database
+	KeyRing   *gomatrixserverlib.KeyRing
+	Client    *gomatrixserverlib.Client
+	FedClient *gomatrixserverlib.FederationClient
 
 	AppserviceAPI       appserviceAPI.AppServiceQueryAPI
 	EDUInternalAPI      eduServerAPI.EDUServerInputAPI
 	FederationSenderAPI federationSenderAPI.FederationSenderInternalAPI
 	RoomserverAPI       roomserverAPI.RoomserverInternalAPI
-	ServerKeyAPI        serverKeyAPI.ServerKeyInternalAPI
+	ServerKeyAPI        serverKeyAPI.SigningKeyServerAPI
 	UserAPI             userapi.UserInternalAPI
 	KeyAPI              keyAPI.KeyInternalAPI
 
@@ -61,7 +58,7 @@ type Monolith struct {
 // AddAllPublicRoutes attaches all public paths to the given router
 func (m *Monolith) AddAllPublicRoutes(csMux, ssMux, keyMux, mediaMux *mux.Router) {
 	clientapi.AddPublicRoutes(
-		csMux, &m.Config.ClientAPI, m.KafkaProducer, m.AccountDB,
+		csMux, &m.Config.ClientAPI, m.AccountDB,
 		m.FedClient, m.RoomserverAPI,
 		m.EDUInternalAPI, m.AppserviceAPI, transactions.New(),
 		m.FederationSenderAPI, m.UserAPI, m.KeyAPI, m.ExtPublicRoomsProvider,
@@ -73,7 +70,7 @@ func (m *Monolith) AddAllPublicRoutes(csMux, ssMux, keyMux, mediaMux *mux.Router
 	)
 	mediaapi.AddPublicRoutes(mediaMux, &m.Config.MediaAPI, m.UserAPI, m.Client)
 	syncapi.AddPublicRoutes(
-		csMux, m.KafkaConsumer, m.UserAPI, m.RoomserverAPI,
+		csMux, m.UserAPI, m.RoomserverAPI,
 		m.KeyAPI, m.FedClient, &m.Config.SyncAPI,
 	)
 }

internal/sqlutil/migrate.go

@@ -0,0 +1,130 @@
+package sqlutil
+
+import (
+	"database/sql"
+	"fmt"
+	"runtime"
+	"sort"
+
+	"github.com/matrix-org/dendrite/internal/config"
+	"github.com/pressly/goose"
+)
+
+type Migrations struct {
+	registeredGoMigrations map[int64]*goose.Migration
+}
+
+func NewMigrations() *Migrations {
+	return &Migrations{
+		registeredGoMigrations: make(map[int64]*goose.Migration),
+	}
+}
+
+// Copy-pasted from goose directly to store migrations into a map we control
+
+// AddMigration adds a migration.
+func (m *Migrations) AddMigration(up func(*sql.Tx) error, down func(*sql.Tx) error) {
+	_, filename, _, _ := runtime.Caller(1)
+	m.AddNamedMigration(filename, up, down)
+}
+
+// AddNamedMigration : Add a named migration.
+func (m *Migrations) AddNamedMigration(filename string, up func(*sql.Tx) error, down func(*sql.Tx) error) {
+	v, _ := goose.NumericComponent(filename)
+	migration := &goose.Migration{Version: v, Next: -1, Previous: -1, Registered: true, UpFn: up, DownFn: down, Source: filename}
+
+	if existing, ok := m.registeredGoMigrations[v]; ok {
+		panic(fmt.Sprintf("failed to add migration %q: version conflicts with %q", filename, existing.Source))
+	}
+
+	m.registeredGoMigrations[v] = migration
+}
+
+// RunDeltas up to the latest version.
+func (m *Migrations) RunDeltas(db *sql.DB, props *config.DatabaseOptions) error {
+	maxVer := goose.MaxVersion
+	minVer := int64(0)
+	migrations, err := m.collect(minVer, maxVer)
+	if err != nil {
+		return fmt.Errorf("RunDeltas: Failed to collect migrations: %w", err)
+	}
+	if props.ConnectionString.IsPostgres() {
+		if err = goose.SetDialect("postgres"); err != nil {
+			return err
+		}
+	} else if props.ConnectionString.IsSQLite() {
+		if err = goose.SetDialect("sqlite3"); err != nil {
+			return err
+		}
+	} else {
+		return fmt.Errorf("Unknown connection string: %s", props.ConnectionString)
+	}
+	for {
+		current, err := goose.EnsureDBVersion(db)
+		if err != nil {
+			return fmt.Errorf("RunDeltas: Failed to EnsureDBVersion: %w", err)
+		}
+
+		next, err := migrations.Next(current)
+		if err != nil {
+			if err == goose.ErrNoNextVersion {
+				return nil
+			}
+
+			return fmt.Errorf("RunDeltas: Failed to load next migration to %+v : %w", next, err)
+		}
+
+		if err = next.Up(db); err != nil {
+			return fmt.Errorf("RunDeltas: Failed run migration: %w", err)
+		}
+	}
+}
+
+func (m *Migrations) collect(current, target int64) (goose.Migrations, error) {
+	var migrations goose.Migrations
+
+	// Go migrations registered via goose.AddMigration().
+	for _, migration := range m.registeredGoMigrations {
+		v, err := goose.NumericComponent(migration.Source)
+		if err != nil {
+			return nil, err
+		}
+		if versionFilter(v, current, target) {
+			migrations = append(migrations, migration)
+		}
+	}
+
+	migrations = sortAndConnectMigrations(migrations)
+
+	return migrations, nil
+}
+
+func sortAndConnectMigrations(migrations goose.Migrations) goose.Migrations {
+	sort.Sort(migrations)
+
+	// now that we're sorted in the appropriate direction,
+	// populate next and previous for each migration
+	for i, m := range migrations {
+		prev := int64(-1)
+		if i > 0 {
+			prev = migrations[i-1].Version
+			migrations[i-1].Next = m.Version
+		}
+		migrations[i].Previous = prev
+	}
+
+	return migrations
+}
+
+func versionFilter(v, current, target int64) bool {
+
+	if target > current {
+		return v > current && v <= target
+	}
+
+	if target < current {
+		return v <= current && v > target
+	}
+
+	return false
+}

internal/sqlutil/trace.go

@@ -93,7 +93,7 @@ func trackGoID(query string) {
 	if strings.HasPrefix(q, "SELECT") {
 		return // SELECTs can go on other goroutines
 	}
-	logrus.Warnf("unsafe goid: SQL executed not on an ExclusiveWriter: %s", q)
+	logrus.Warnf("unsafe goid %d: SQL executed not on an ExclusiveWriter: %s", thisGoID, q)
 }
 
 // Open opens a database specified by its database driver name and a driver-specific data source name,

internal/test/config.go

@@ -92,7 +92,7 @@ func MakeConfig(configDir, kafkaURI, database, host string, startPort int) (*con
 	cfg.KeyServer.Database.ConnectionString = config.DataSource(database)
 	cfg.MediaAPI.Database.ConnectionString = config.DataSource(database)
 	cfg.RoomServer.Database.ConnectionString = config.DataSource(database)
-	cfg.ServerKeyAPI.Database.ConnectionString = config.DataSource(database)
+	cfg.SigningKeyServer.Database.ConnectionString = config.DataSource(database)
 	cfg.SyncAPI.Database.ConnectionString = config.DataSource(database)
 	cfg.UserAPI.AccountDatabase.ConnectionString = config.DataSource(database)
 	cfg.UserAPI.DeviceDatabase.ConnectionString = config.DataSource(database)
@@ -104,7 +104,7 @@ func MakeConfig(configDir, kafkaURI, database, host string, startPort int) (*con
 	cfg.KeyServer.InternalAPI.Listen = assignAddress()
 	cfg.MediaAPI.InternalAPI.Listen = assignAddress()
 	cfg.RoomServer.InternalAPI.Listen = assignAddress()
-	cfg.ServerKeyAPI.InternalAPI.Listen = assignAddress()
+	cfg.SigningKeyServer.InternalAPI.Listen = assignAddress()
 	cfg.SyncAPI.InternalAPI.Listen = assignAddress()
 	cfg.UserAPI.InternalAPI.Listen = assignAddress()
 
@@ -115,7 +115,7 @@ func MakeConfig(configDir, kafkaURI, database, host string, startPort int) (*con
 	cfg.KeyServer.InternalAPI.Connect = cfg.KeyServer.InternalAPI.Listen
 	cfg.MediaAPI.InternalAPI.Connect = cfg.MediaAPI.InternalAPI.Listen
 	cfg.RoomServer.InternalAPI.Connect = cfg.RoomServer.InternalAPI.Listen
-	cfg.ServerKeyAPI.InternalAPI.Connect = cfg.ServerKeyAPI.InternalAPI.Listen
+	cfg.SigningKeyServer.InternalAPI.Connect = cfg.SigningKeyServer.InternalAPI.Listen
 	cfg.SyncAPI.InternalAPI.Connect = cfg.SyncAPI.InternalAPI.Listen
 	cfg.UserAPI.InternalAPI.Connect = cfg.UserAPI.InternalAPI.Listen
 

internal/version.go

@@ -1,6 +1,12 @@
 package internal
 
-import "fmt"
+import (
+	"fmt"
+	"strings"
+)
+
+// the final version string
+var version string
 
 // -ldflags "-X github.com/matrix-org/dendrite/internal.branch=master"
 var branch string
@@ -10,17 +16,28 @@ var build string
 
 const (
 	VersionMajor = 0
-	VersionMinor = 0
+	VersionMinor = 2
 	VersionPatch = 0
+	VersionTag   = "" // example: "rc1"
 )
 
 func VersionString() string {
-	version := fmt.Sprintf("%d.%d.%d", VersionMajor, VersionMinor, VersionPatch)
-	if branch != "" {
-		version += fmt.Sprintf("-%s", branch)
-	}
-	if build != "" {
-		version += fmt.Sprintf("+%s", build)
-	}
 	return version
 }
+
+func init() {
+	version = fmt.Sprintf("%d.%d.%d", VersionMajor, VersionMinor, VersionPatch)
+	if VersionTag != "" {
+		version += "-" + VersionTag
+	}
+	parts := []string{}
+	if build != "" {
+		parts = append(parts, build)
+	}
+	if branch != "" {
+		parts = append(parts, branch)
+	}
+	if len(parts) > 0 {
+		version += "+" + strings.Join(parts, ".")
+	}
+}

keyserver/internal/device_list_update_test.go

@@ -108,7 +108,7 @@ func newFedClient(tripper func(*http.Request) (*http.Response, error)) *gomatrix
 	fedClient := gomatrixserverlib.NewFederationClient(
 		gomatrixserverlib.ServerName("example.test"), gomatrixserverlib.KeyID("ed25519:test"), pkey, true,
 	)
-	fedClient.Client = *gomatrixserverlib.NewClientWithTransport(true, &roundTripper{tripper})
+	fedClient.Client = *gomatrixserverlib.NewClientWithTransport(&roundTripper{tripper})
 	return fedClient
 }
 

keyserver/keyserver.go

@@ -15,10 +15,10 @@
 package keyserver
 
 import (
-	"github.com/Shopify/sarama"
 	"github.com/gorilla/mux"
 	fedsenderapi "github.com/matrix-org/dendrite/federationsender/api"
 	"github.com/matrix-org/dendrite/internal/config"
+	"github.com/matrix-org/dendrite/internal/setup/kafka"
 	"github.com/matrix-org/dendrite/keyserver/api"
 	"github.com/matrix-org/dendrite/keyserver/internal"
 	"github.com/matrix-org/dendrite/keyserver/inthttp"
@@ -36,8 +36,10 @@ func AddInternalRoutes(router *mux.Router, intAPI api.KeyInternalAPI) {
 // NewInternalAPI returns a concerete implementation of the internal API. Callers
 // can call functions directly on the returned API or via an HTTP interface using AddInternalRoutes.
 func NewInternalAPI(
-	cfg *config.KeyServer, fedClient fedsenderapi.FederationClient, producer sarama.SyncProducer,
+	cfg *config.KeyServer, fedClient fedsenderapi.FederationClient,
 ) api.KeyInternalAPI {
+	_, producer := kafka.SetupConsumerProducer(&cfg.Matrix.Kafka)
+
 	db, err := storage.NewDatabase(&cfg.Database)
 	if err != nil {
 		logrus.WithError(err).Panicf("failed to connect to key server database")

roomserver/api/api.go

@@ -74,6 +74,13 @@ type RoomserverInternalAPI interface {
 		response *QueryStateAfterEventsResponse,
 	) error
 
+	// Query whether the roomserver is missing any auth or prev events.
+	QueryMissingAuthPrevEvents(
+		ctx context.Context,
+		request *QueryMissingAuthPrevEventsRequest,
+		response *QueryMissingAuthPrevEventsResponse,
+	) error
+
 	// Query a list of events by event ID.
 	QueryEventsByID(
 		ctx context.Context,

roomserver/api/api_trace.go

@@ -114,6 +114,16 @@ func (t *RoomserverInternalAPITrace) QueryStateAfterEvents(
 	return err
 }
 
+func (t *RoomserverInternalAPITrace) QueryMissingAuthPrevEvents(
+	ctx context.Context,
+	req *QueryMissingAuthPrevEventsRequest,
+	res *QueryMissingAuthPrevEventsResponse,
+) error {
+	err := t.Impl.QueryMissingAuthPrevEvents(ctx, req, res)
+	util.GetLogger(ctx).WithError(err).Infof("QueryMissingAuthPrevEvents req=%+v res=%+v", js(req), js(res))
+	return err
+}
+
 func (t *RoomserverInternalAPITrace) QueryEventsByID(
 	ctx context.Context,
 	req *QueryEventsByIDRequest,