Merge branch 'master' into 610-implement-account-deactivation

.github/ISSUE_TEMPLATE/BUG_REPORT.md

@@ -0,0 +1,49 @@
+---
+name: Bug report
+about: Create a report to help us improve
+
+---
+
+<!-- All bug reports must provide the following background information -->
+
+### Background information
+
+- **Dendrite version or git SHA**: 
+- **Monolith or Polylith?**: 
+- **SQLite3 or Postgres?**: 
+- **Running in Docker?**: 
+- **`go version`**: 
+
+<!--
+
+This is a bug report template. By following the instructions below and
+filling out the sections with your information, you will help the us to get all
+the necessary data to fix your issue.
+
+You can also preview your report before submitting it. You may remove sections
+that aren't relevant to your particular case.
+
+Text between <!-- and --​> marks will be invisible in the report.
+
+-->
+
+### Description
+
+<!-- Describe here the problem that you are experiencing -->
+
+### Steps to reproduce
+
+- list the steps
+- that reproduce the bug
+- using hyphens as bullet points
+
+<!--
+Describe how what happens differs from what you expected.
+
+If you can identify any relevant log snippets from server logs, please include
+those (please be careful to remove any personal or private data). Please surround them with
+``` (three backticks, on a line on their own), so that they are formatted legibly.
+
+Alternatively, please send logs to @kegan:matrix.org or @neilalexander:matrix.org
+with a link to the respective Github issue, thanks!
+-->

.github/ISSUE_TEMPLATE/FEATURE_REQUEST.md

@@ -0,0 +1,14 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+
+---
+
+<!--
+Please do not open feature requests for missing parts of the Matrix specification.
+We are tracking those features under https://github.com/matrix-org/dendrite/issues?q=is%3Aissue+is%3Aopen+label%3Aare-we-synapse-yet
+-->
+
+**Description:**
+
+<!-- Describe here the feature you are requesting. -->

.github/PULL_REQUEST_TEMPLATE.md

@@ -2,5 +2,5 @@
 
 <!-- Please read CONTRIBUTING.md before submitting your pull request -->
 
-* [ ] I have added any new tests that need to pass to `testfile` as specified in [docs/sytest.md](https://github.com/matrix-org/dendrite/blob/master/docs/sytest.md)
+* [ ] I have added any new tests that need to pass to `sytest-whitelist` as specified in [docs/sytest.md](https://github.com/matrix-org/dendrite/blob/master/docs/sytest.md)
 * [ ] Pull request includes a [sign off](https://github.com/matrix-org/dendrite/blob/master/docs/CONTRIBUTING.md#sign-off)

.github/workflows/codeql-analysis.yml

@@ -0,0 +1,34 @@
+name: "CodeQL"
+
+on:
+  push:
+    branches: [master]
+  pull_request:
+    branches: [master]
+
+jobs:
+  analyze:
+    name: Analyze
+    runs-on: ubuntu-latest
+
+    strategy:
+      fail-fast: false
+      matrix:
+        language: ['go']
+
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v2
+      with:
+        fetch-depth: 2
+
+    - run: git checkout HEAD^2
+      if: ${{ github.event_name == 'pull_request' }}
+
+    - name: Initialize CodeQL
+      uses: github/codeql-action/init@v1
+      with:
+        languages: ${{ matrix.language }}
+
+    - name: Perform CodeQL Analysis
+      uses: github/codeql-action/analyze@v1

README.md

@@ -1,11 +1,14 @@
-# Dendrite [![Build Status](https://badge.buildkite.com/4be40938ab19f2bbc4a6c6724517353ee3ec1422e279faf374.svg?branch=master)](https://buildkite.com/matrix-dot-org/dendrite) [![Dendrite Dev on Matrix](https://img.shields.io/matrix/dendrite-dev:matrix.org.svg?label=%23dendrite-dev%3Amatrix.org&logo=matrix&server_fqdn=matrix.org)](https://matrix.to/#/#dendrite-dev:matrix.org) [![Dendrite on Matrix](https://img.shields.io/matrix/dendrite:matrix.org.svg?label=%23dendrite%3Amatrix.org&logo=matrix&server_fqdn=matrix.org)](https://matrix.to/#/#dendrite:matrix.org)
+# Dendrite [![Build Status](https://badge.buildkite.com/4be40938ab19f2bbc4a6c6724517353ee3ec1422e279faf374.svg?branch=master)](https://buildkite.com/matrix-dot-org/dendrite) [![Dendrite](https://img.shields.io/matrix/dendrite:matrix.org.svg?label=%23dendrite%3Amatrix.org&logo=matrix&server_fqdn=matrix.org)](https://matrix.to/#/#dendrite:matrix.org) [![Dendrite Dev](https://img.shields.io/matrix/dendrite-dev:matrix.org.svg?label=%23dendrite-dev%3Amatrix.org&logo=matrix&server_fqdn=matrix.org)](https://matrix.to/#/#dendrite-dev:matrix.org)
 
-Dendrite is a second-generation Matrix homeserver written in Go. It is not recommended to use Dendrite as
+Dendrite is a second-generation Matrix homeserver written in Go!
-a production homeserver at this time as there is no stable release.
 
-Dendrite will start to receive versioned releases stable enough to run [once we enter beta](https://github.com/matrix-org/dendrite/milestone/8).
+Join us in:
 
-# Quick start
+- **[#dendrite:matrix.org](https://matrix.to/#/#dendrite:matrix.org)** - General chat about the Dendrite project, for users and server admins alike
+- **[#dendrite-dev:matrix.org](https://matrix.to/#/#dendrite-dev:matrix.org)** - The place for developers, where all Dendrite development discussion happens
+- **[#dendrite-alerts:matrix.org](https://matrix.to/#/#dendrite-alerts:matrix.org)** - Release notifications and important info, highly recommended for all Dendrite server admins
+
+## Quick start
 
 Requires Go 1.13+ and SQLite3 (Postgres is also supported):
 
@@ -30,7 +33,7 @@ $ ./dendrite-monolith-server --tls-cert server.crt --tls-key server.key --config
 Then point your favourite Matrix client at `http://localhost:8008`. For full installation information, see
 [INSTALL.md](docs/INSTALL.md). For running in Docker, see [build/docker](build/docker).
 
-# Progress
+## Progress
 
 We use a script called Are We Synapse Yet which checks Sytest compliance rates. Sytest is a black-box homeserver
 test rig with around 900 tests. The script works out how many of these tests are passing on Dendrite and it
@@ -59,7 +62,7 @@ This means Dendrite supports amongst others:
  - E2E keys and device lists
 
 
-# Contributing
+## Contributing
 
 We would be grateful for any help on issues marked as
 [Are We Synapse Yet](https://github.com/matrix-org/dendrite/labels/are-we-synapse-yet). These issues
@@ -101,7 +104,7 @@ look for [Good First Issues](https://github.com/matrix-org/dendrite/labels/good%
 familiar with the project, look for [Help Wanted](https://github.com/matrix-org/dendrite/labels/help-wanted)
 issues.
 
-# Hardware requirements
+## Hardware requirements
 
 Dendrite in Monolith + SQLite works in a range of environments including iOS and in-browser via WASM.
 
@@ -112,12 +115,3 @@ encrypted rooms:
  - CPU: Brief spikes when processing events, typically idles at 1% CPU.
 
 This means Dendrite should comfortably work on things like Raspberry Pis.
-
-# Discussion
-
-For questions about Dendrite we have a dedicated room on Matrix
-[#dendrite:matrix.org](https://matrix.to/#/#dendrite:matrix.org). Development
-discussion should happen in
-[#dendrite-dev:matrix.org](https://matrix.to/#/#dendrite-dev:matrix.org).
-
-

build.sh

@@ -3,10 +3,16 @@
 # Put installed packages into ./bin
 export GOBIN=$PWD/`dirname $0`/bin
 
-export BRANCH=`(git symbolic-ref --short HEAD | cut -d'/' -f 3 )|| ""`
+if [ -d ".git" ] 
-export BUILD=`git rev-parse --short HEAD || ""`
+then
+    export BUILD=`git rev-parse --short HEAD || ""`
+    export BRANCH=`(git symbolic-ref --short HEAD | tr -d \/ ) || ""`
+    [[ $BRANCH == "master" ]] && export BRANCH=""
 
-export FLAGS="-X github.com/matrix-org/dendrite/internal.branch=$BRANCH -X github.com/matrix-org/dendrite/internal.build=$BUILD"
+    export FLAGS="-X github.com/matrix-org/dendrite/internal.branch=$BRANCH -X github.com/matrix-org/dendrite/internal.build=$BUILD"
+else
+    export FLAGS=""
+fi
 
 go install -trimpath -ldflags "$FLAGS" -v $PWD/`dirname $0`/cmd/...
 

build/docker/config/dendrite-config.yaml

@@ -38,8 +38,13 @@ global:
   # The path to the signing private key file, used to sign requests and events.
   private_key: matrix_key.pem
 
-  # A unique identifier for this private key. Must start with the prefix "ed25519:".
+  # The paths and expiry timestamps (as a UNIX timestamp in millisecond precision)
-  key_id: ed25519:auto
+  # to old signing private keys that were formerly in use on this domain. These
+  # keys will not be used for federation request or event signing, but will be
+  # provided to any other homeserver that asks when trying to verify old events.
+  # old_private_keys:
+  # - private_key: old_matrix_key.pem
+  #   expired_at: 1601024554498
 
   # How long a remote server can cache our server signing key before requesting it
   # again. Increasing this number will reduce the number of requests made by other
@@ -133,6 +138,14 @@ client_api:
     turn_username: ""
     turn_password: ""
 
+  # Settings for rate-limited endpoints. Rate limiting will kick in after the
+  # threshold number of "slots" have been taken by requests from a specific 
+  # host. Each "slot" will be released after the cooloff time in milliseconds.
+  rate_limiting:
+    enabled: true
+    threshold: 5
+    cooloff_ms: 500
+
 # Configuration for the EDU server.
 edu_server:
   internal_api:
@@ -261,6 +274,11 @@ server_key_api:
     - key_id: ed25519:a_RXGa
       public_key: l8Hft5qXKn1vfHrg3p4+W8gELQVo8N13JkluMfmn2sQ
       
+  # This option will control whether Dendrite will prefer to look up keys directly
+  # or whether it should try perspective servers first, using direct fetches as a
+  # last resort.
+  prefer_direct_fetch: false
+
 # Configuration for the Sync API.
 sync_api:
   internal_api:
@@ -291,6 +309,8 @@ user_api:
     conn_max_lifetime: -1
 
 # Configuration for Opentracing.
+# See https://github.com/matrix-org/dendrite/tree/master/docs/tracing for information on
+# how this works and how to set it up.
 tracing:
   enabled: false
   jaeger:

build/gobind/monolith.go

@@ -88,16 +88,16 @@ func (m *DendriteMonolith) Start() {
 	cfg.Global.PrivateKey = ygg.SigningPrivateKey()
 	cfg.Global.KeyID = gomatrixserverlib.KeyID(signing.KeyID)
 	cfg.Global.Kafka.UseNaffka = true
-	cfg.Global.Kafka.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-naffka.db", m.StorageDirectory))
+	cfg.Global.Kafka.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-p2p-naffka.db", m.StorageDirectory))
-	cfg.UserAPI.AccountDatabase.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-account.db", m.StorageDirectory))
+	cfg.UserAPI.AccountDatabase.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-p2p-account.db", m.StorageDirectory))
-	cfg.UserAPI.DeviceDatabase.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-device.db", m.StorageDirectory))
+	cfg.UserAPI.DeviceDatabase.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-p2p-device.db", m.StorageDirectory))
-	cfg.MediaAPI.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-mediaapi.db", m.StorageDirectory))
+	cfg.MediaAPI.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-p2p-mediaapi.db", m.StorageDirectory))
-	cfg.SyncAPI.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-syncapi.db", m.StorageDirectory))
+	cfg.SyncAPI.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-p2p-syncapi.db", m.StorageDirectory))
-	cfg.RoomServer.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-roomserver.db", m.StorageDirectory))
+	cfg.RoomServer.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-p2p-roomserver.db", m.StorageDirectory))
-	cfg.ServerKeyAPI.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-serverkey.db", m.StorageDirectory))
+	cfg.ServerKeyAPI.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-p2p-serverkey.db", m.StorageDirectory))
-	cfg.KeyServer.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-keyserver.db", m.StorageDirectory))
+	cfg.KeyServer.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-p2p-keyserver.db", m.StorageDirectory))
-	cfg.FederationSender.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-federationsender.db", m.StorageDirectory))
+	cfg.FederationSender.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-p2p-federationsender.db", m.StorageDirectory))
-	cfg.AppServiceAPI.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-appservice.db", m.StorageDirectory))
+	cfg.AppServiceAPI.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-p2p-appservice.db", m.StorageDirectory))
 	cfg.MediaAPI.BasePath = config.Path(fmt.Sprintf("%s/tmp", m.StorageDirectory))
 	cfg.MediaAPI.AbsBasePath = config.Path(fmt.Sprintf("%s/tmp", m.StorageDirectory))
 	cfg.FederationSender.FederationMaxRetries = 8

cmd/dendrite-media-api-server/main.go

@@ -17,7 +17,6 @@ package main
 import (
 	"github.com/matrix-org/dendrite/internal/setup"
 	"github.com/matrix-org/dendrite/mediaapi"
-	"github.com/matrix-org/gomatrixserverlib"
 )
 
 func main() {
@@ -26,7 +25,7 @@ func main() {
 	defer base.Close() // nolint: errcheck
 
 	userAPI := base.UserAPIClient()
-	client := gomatrixserverlib.NewClient(cfg.FederationSender.DisableTLSValidation)
+	client := base.CreateClient()
 
 	mediaapi.AddPublicRoutes(base.PublicMediaAPIMux, &base.Cfg.MediaAPI, userAPI, client)
 

cmd/dendrite-monolith-server/main.go

@@ -29,7 +29,6 @@ import (
 	"github.com/matrix-org/dendrite/roomserver/api"
 	"github.com/matrix-org/dendrite/serverkeyapi"
 	"github.com/matrix-org/dendrite/userapi"
-	"github.com/matrix-org/gomatrixserverlib"
 )
 
 var (
@@ -125,7 +124,7 @@ func main() {
 	monolith := setup.Monolith{
 		Config:        base.Cfg,
 		AccountDB:     accountDB,
-		Client:        gomatrixserverlib.NewClient(cfg.FederationSender.DisableTLSValidation),
+		Client:        base.CreateClient(),
 		FedClient:     federation,
 		KeyRing:       keyRing,
 		KafkaConsumer: base.KafkaConsumer,

docs/nginx/monolith-sample.conf

@@ -0,0 +1,24 @@
+server {
+    listen 443 ssl;
+    server_name my.hostname.com;
+
+    ssl_certificate /path/to/fullchain.pem;
+    ssl_certificate_key /path/to/privkey.pem;
+    ssl_dhparam /path/to/ssl-dhparams.pem;
+
+    proxy_set_header Host      $host;
+    proxy_set_header X-Real-IP $remote_addr;
+    proxy_read_timeout         600;
+
+    location /.well-known/matrix/server {
+        return 200 '{ "m.server": "my.hostname.com:443" }';
+    }
+
+    location /.well-known/matrix/client {
+        return 200 '{ "m.homeserver": { "base_url": "https://my.hostname.com" } }';
+    }
+
+    location /_matrix {
+        proxy_pass http://monolith:8008;
+    }
+}

docs/nginx/polylith-sample.conf

@@ -0,0 +1,36 @@
+server {
+    listen 443 ssl;
+    server_name my.hostname.com;
+
+    ssl_certificate /path/to/fullchain.pem;
+    ssl_certificate_key /path/to/privkey.pem;
+    ssl_dhparam /path/to/ssl-dhparams.pem;
+
+    proxy_set_header Host      $host;
+    proxy_set_header X-Real-IP $remote_addr;
+    proxy_read_timeout         600;
+
+    location /.well-known/matrix/server {
+        return 200 '{ "m.server": "my.hostname.com:443" }';
+    }
+
+    location /.well-known/matrix/client {
+        return 200 '{ "m.homeserver": { "base_url": "https://my.hostname.com" } }';
+    }
+
+    location /_matrix/client {
+        proxy_pass http://client_api:8071;
+    }
+
+    location /_matrix/federation {
+        proxy_pass http://federation_api:8072;
+    }
+
+    location /_matrix/key {
+        proxy_pass http://federation_api:8072;
+    }
+
+    location /_matrix/media {
+        proxy_pass http://media_api:8074;
+    }
+}

federationapi/routing/join.go

@@ -192,9 +192,7 @@ func SendJoin(
 	if event.StateKey() == nil || event.StateKeyEquals("") {
 		return util.JSONResponse{
 			Code: http.StatusBadRequest,
-			JSON: jsonerror.BadJSON(
+			JSON: jsonerror.BadJSON("No state key was provided in the join event."),
-				fmt.Sprintf("No state key was provided in the join event."),
-			),
 		}
 	}
 

federationapi/routing/send.go

@@ -872,7 +872,7 @@ func (t *txnReq) lookupMissingStateViaStateIDs(ctx context.Context, roomID, even
 		h, err = t.lookupEvent(ctx, roomVersion, missingEventID, false)
 		switch err.(type) {
 		case verifySigError:
-			break
+			return
 		case nil:
 			break
 		default:

go.mod

@@ -22,7 +22,7 @@ require (
 	github.com/matrix-org/go-http-js-libp2p v0.0.0-20200518170932-783164aeeda4
 	github.com/matrix-org/go-sqlite3-js v0.0.0-20200522092705-bc8506ccbcf3
 	github.com/matrix-org/gomatrix v0.0.0-20200827122206-7dd5e2a05bcd
-	github.com/matrix-org/gomatrixserverlib v0.0.0-20200929155210-32fc5888d26a
+	github.com/matrix-org/gomatrixserverlib v0.0.0-20201002084023-8bcafefa3290
 	github.com/matrix-org/naffka v0.0.0-20200901083833-bcdd62999a91
 	github.com/matrix-org/util v0.0.0-20200807132607-55161520e1d4
 	github.com/mattn/go-sqlite3 v1.14.2

go.sum

@@ -569,14 +569,8 @@ github.com/matrix-org/gomatrix v0.0.0-20190528120928-7df988a63f26 h1:Hr3zjRsq2bh
 github.com/matrix-org/gomatrix v0.0.0-20190528120928-7df988a63f26/go.mod h1:3fxX6gUjWyI/2Bt7J1OLhpCzOfO/bB3AiX0cJtEKud0=
 github.com/matrix-org/gomatrix v0.0.0-20200827122206-7dd5e2a05bcd h1:xVrqJK3xHREMNjwjljkAUaadalWc0rRbmVuQatzmgwg=
 github.com/matrix-org/gomatrix v0.0.0-20200827122206-7dd5e2a05bcd/go.mod h1:/gBX06Kw0exX1HrwmoBibFA98yBk/jxKpGVeyQbff+s=
-github.com/matrix-org/gomatrixserverlib v0.0.0-20200929152221-6fe6457127ad h1:n0P/Oy8ZqqTPzum6FEayAjamsmvJTuIcA10WQ8GcK70=
+github.com/matrix-org/gomatrixserverlib v0.0.0-20201002084023-8bcafefa3290 h1:ilT9QNIh2KXfvzIALtAe31IvLVZH7mVjVtOOTxdd0tY=
-github.com/matrix-org/gomatrixserverlib v0.0.0-20200929152221-6fe6457127ad/go.mod h1:JsAzE1Ll3+gDWS9JSUHPJiiyAksvOOnGWF2nXdg4ZzU=
+github.com/matrix-org/gomatrixserverlib v0.0.0-20201002084023-8bcafefa3290/go.mod h1:JsAzE1Ll3+gDWS9JSUHPJiiyAksvOOnGWF2nXdg4ZzU=
-github.com/matrix-org/gomatrixserverlib v0.0.0-20200929154026-a52e7a5f0553 h1:tiel2c3I9xr0SRS05g3UvOjj6Sgg1I3Yn2/oGA1GgLk=
-github.com/matrix-org/gomatrixserverlib v0.0.0-20200929154026-a52e7a5f0553/go.mod h1:JsAzE1Ll3+gDWS9JSUHPJiiyAksvOOnGWF2nXdg4ZzU=
-github.com/matrix-org/gomatrixserverlib v0.0.0-20200929154241-9414c4d0b5f2 h1:a07U5eFT521mFiUtA/A8NwiZp5vfRU59/QKs+pa3Fkc=
-github.com/matrix-org/gomatrixserverlib v0.0.0-20200929154241-9414c4d0b5f2/go.mod h1:JsAzE1Ll3+gDWS9JSUHPJiiyAksvOOnGWF2nXdg4ZzU=
-github.com/matrix-org/gomatrixserverlib v0.0.0-20200929155210-32fc5888d26a h1:kIwbS7eY7P/MX0oN4wRHGkoc4eTTnwOcdCawBZ3SrJI=
-github.com/matrix-org/gomatrixserverlib v0.0.0-20200929155210-32fc5888d26a/go.mod h1:JsAzE1Ll3+gDWS9JSUHPJiiyAksvOOnGWF2nXdg4ZzU=
 github.com/matrix-org/naffka v0.0.0-20200901083833-bcdd62999a91 h1:HJ6U3S3ljJqNffYMcIeAncp5qT/i+ZMiJ2JC2F0aXP4=
 github.com/matrix-org/naffka v0.0.0-20200901083833-bcdd62999a91/go.mod h1:sjyPyRxKM5uw1nD2cJ6O2OxI6GOqyVBfNXqKjBZTBZE=
 github.com/matrix-org/util v0.0.0-20190711121626-527ce5ddefc7 h1:ntrLa/8xVzeSs8vHFHK25k0C+NV74sYMJnNSg5NoSRo=

internal/setup/base.go

@@ -69,6 +69,7 @@ type BaseDendrite struct {
 	PublicMediaAPIMux      *mux.Router
 	InternalAPIMux         *mux.Router
 	UseHTTPAPIs            bool
+	apiHttpClient          *http.Client
 	httpClient             *http.Client
 	Cfg                    *config.Dendrite
 	Caches                 *caching.Caches
@@ -118,6 +119,7 @@ func NewBaseDendrite(cfg *config.Dendrite, componentName string, useHTTPAPIs boo
 		logrus.WithError(err).Warnf("Failed to create cache")
 	}
 
+	apiClient := http.Client{Timeout: time.Minute * 10}
 	client := http.Client{Timeout: HTTPClientTimeout}
 	if cfg.FederationSender.Proxy.Enabled {
 		client.Transport = &http.Transport{Proxy: http.ProxyURL(&url.URL{
@@ -148,6 +150,7 @@ func NewBaseDendrite(cfg *config.Dendrite, componentName string, useHTTPAPIs boo
 		PublicKeyAPIMux:        mux.NewRouter().SkipClean(true).PathPrefix(httputil.PublicKeyPathPrefix).Subrouter().UseEncodedPath(),
 		PublicMediaAPIMux:      mux.NewRouter().SkipClean(true).PathPrefix(httputil.PublicMediaPathPrefix).Subrouter().UseEncodedPath(),
 		InternalAPIMux:         mux.NewRouter().SkipClean(true).PathPrefix(httputil.InternalPathPrefix).Subrouter().UseEncodedPath(),
+		apiHttpClient:          &apiClient,
 		httpClient:             &client,
 		KafkaConsumer:          kafkaConsumer,
 		KafkaProducer:          kafkaProducer,
@@ -161,7 +164,7 @@ func (b *BaseDendrite) Close() error {
 
 // AppserviceHTTPClient returns the AppServiceQueryAPI for hitting the appservice component over HTTP.
 func (b *BaseDendrite) AppserviceHTTPClient() appserviceAPI.AppServiceQueryAPI {
-	a, err := asinthttp.NewAppserviceClient(b.Cfg.AppServiceURL(), b.httpClient)
+	a, err := asinthttp.NewAppserviceClient(b.Cfg.AppServiceURL(), b.apiHttpClient)
 	if err != nil {
 		logrus.WithError(err).Panic("CreateHTTPAppServiceAPIs failed")
 	}
@@ -170,27 +173,27 @@ func (b *BaseDendrite) AppserviceHTTPClient() appserviceAPI.AppServiceQueryAPI {
 
 // RoomserverHTTPClient returns RoomserverInternalAPI for hitting the roomserver over HTTP.
 func (b *BaseDendrite) RoomserverHTTPClient() roomserverAPI.RoomserverInternalAPI {
-	rsAPI, err := rsinthttp.NewRoomserverClient(b.Cfg.RoomServerURL(), b.httpClient, b.Caches)
+	rsAPI, err := rsinthttp.NewRoomserverClient(b.Cfg.RoomServerURL(), b.apiHttpClient, b.Caches)
 	if err != nil {
-		logrus.WithError(err).Panic("RoomserverHTTPClient failed", b.httpClient)
+		logrus.WithError(err).Panic("RoomserverHTTPClient failed", b.apiHttpClient)
 	}
 	return rsAPI
 }
 
 // UserAPIClient returns UserInternalAPI for hitting the userapi over HTTP.
 func (b *BaseDendrite) UserAPIClient() userapi.UserInternalAPI {
-	userAPI, err := userapiinthttp.NewUserAPIClient(b.Cfg.UserAPIURL(), b.httpClient)
+	userAPI, err := userapiinthttp.NewUserAPIClient(b.Cfg.UserAPIURL(), b.apiHttpClient)
 	if err != nil {
-		logrus.WithError(err).Panic("UserAPIClient failed", b.httpClient)
+		logrus.WithError(err).Panic("UserAPIClient failed", b.apiHttpClient)
 	}
 	return userAPI
 }
 
 // EDUServerClient returns EDUServerInputAPI for hitting the EDU server over HTTP
 func (b *BaseDendrite) EDUServerClient() eduServerAPI.EDUServerInputAPI {
-	e, err := eduinthttp.NewEDUServerClient(b.Cfg.EDUServerURL(), b.httpClient)
+	e, err := eduinthttp.NewEDUServerClient(b.Cfg.EDUServerURL(), b.apiHttpClient)
 	if err != nil {
-		logrus.WithError(err).Panic("EDUServerClient failed", b.httpClient)
+		logrus.WithError(err).Panic("EDUServerClient failed", b.apiHttpClient)
 	}
 	return e
 }
@@ -198,9 +201,9 @@ func (b *BaseDendrite) EDUServerClient() eduServerAPI.EDUServerInputAPI {
 // FederationSenderHTTPClient returns FederationSenderInternalAPI for hitting
 // the federation sender over HTTP
 func (b *BaseDendrite) FederationSenderHTTPClient() federationSenderAPI.FederationSenderInternalAPI {
-	f, err := fsinthttp.NewFederationSenderClient(b.Cfg.FederationSenderURL(), b.httpClient)
+	f, err := fsinthttp.NewFederationSenderClient(b.Cfg.FederationSenderURL(), b.apiHttpClient)
 	if err != nil {
-		logrus.WithError(err).Panic("FederationSenderHTTPClient failed", b.httpClient)
+		logrus.WithError(err).Panic("FederationSenderHTTPClient failed", b.apiHttpClient)
 	}
 	return f
 }
@@ -209,7 +212,7 @@ func (b *BaseDendrite) FederationSenderHTTPClient() federationSenderAPI.Federati
 func (b *BaseDendrite) ServerKeyAPIClient() serverKeyAPI.ServerKeyInternalAPI {
 	f, err := skinthttp.NewServerKeyClient(
 		b.Cfg.ServerKeyAPIURL(),
-		b.httpClient,
+		b.apiHttpClient,
 		b.Caches,
 	)
 	if err != nil {
@@ -220,9 +223,9 @@ func (b *BaseDendrite) ServerKeyAPIClient() serverKeyAPI.ServerKeyInternalAPI {
 
 // KeyServerHTTPClient returns KeyInternalAPI for hitting the key server over HTTP
 func (b *BaseDendrite) KeyServerHTTPClient() keyserverAPI.KeyInternalAPI {
-	f, err := keyinthttp.NewKeyServerClient(b.Cfg.KeyServerURL(), b.httpClient)
+	f, err := keyinthttp.NewKeyServerClient(b.Cfg.KeyServerURL(), b.apiHttpClient)
 	if err != nil {
-		logrus.WithError(err).Panic("KeyServerHTTPClient failed", b.httpClient)
+		logrus.WithError(err).Panic("KeyServerHTTPClient failed", b.apiHttpClient)
 	}
 	return f
 }
@@ -238,13 +241,25 @@ func (b *BaseDendrite) CreateAccountsDB() accounts.Database {
 	return db
 }
 
+// CreateClient creates a new client (normally used for media fetch requests).
+// Should only be called once per component.
+func (b *BaseDendrite) CreateClient() *gomatrixserverlib.Client {
+	client := gomatrixserverlib.NewClient(
+		b.Cfg.FederationSender.DisableTLSValidation,
+	)
+	client.SetUserAgent(fmt.Sprintf("Dendrite/%s", internal.VersionString()))
+	return client
+}
+
 // CreateFederationClient creates a new federation client. Should only be called
 // once per component.
 func (b *BaseDendrite) CreateFederationClient() *gomatrixserverlib.FederationClient {
-	return gomatrixserverlib.NewFederationClient(
+	client := gomatrixserverlib.NewFederationClient(
 		b.Cfg.Global.ServerName, b.Cfg.Global.KeyID, b.Cfg.Global.PrivateKey,
 		b.Cfg.FederationSender.DisableTLSValidation,
 	)
+	client.SetUserAgent(fmt.Sprintf("Dendrite/%s", internal.VersionString()))
+	return client
 }
 
 // SetupAndServeHTTP sets up the HTTP server to serve endpoints registered on

internal/version.go

@@ -12,10 +12,11 @@ const (
 	VersionMajor = 0
 	VersionMinor = 0
 	VersionPatch = 0
+	VersionTag   = "" // example: "rc1"
 )
 
 func VersionString() string {
-	version := fmt.Sprintf("%d.%d.%d", VersionMajor, VersionMinor, VersionPatch)
+	version := fmt.Sprintf("%d.%d.%d%s", VersionMajor, VersionMinor, VersionPatch, VersionTag)
 	if branch != "" {
 		version += fmt.Sprintf("-%s", branch)
 	}

roomserver/state/state.go

@@ -717,11 +717,7 @@ func ResolveConflictsAdhoc(
 		// Append the events if there is already a conflicted list for
 		// this tuple key, create it if not.
 		tuple := stateKeyTuple{event.Type(), *event.StateKey()}
-		if _, ok := eventMap[tuple]; ok {
+		eventMap[tuple] = append(eventMap[tuple], event)
-			eventMap[tuple] = append(eventMap[tuple], event)
-		} else {
-			eventMap[tuple] = []gomatrixserverlib.Event{event}
-		}
 	}
 
 	// Split out the events in the map into conflicted and unconflicted

roomserver/storage/sqlite3/previous_events_table.go

@@ -98,7 +98,7 @@ func (s *previousEventStatements) InsertPreviousEvent(
 	eventNIDAsString := fmt.Sprintf("%d", eventNID)
 	selectStmt := sqlutil.TxStmt(txn, s.selectPreviousEventExistsStmt)
 	err := selectStmt.QueryRowContext(ctx, previousEventID, previousEventReferenceSHA256).Scan(&eventNIDs)
-	if err != sql.ErrNoRows {
+	if err != nil && err != sql.ErrNoRows {
 		return fmt.Errorf("selectStmt.QueryRowContext.Scan: %w", err)
 	}
 	var nids []string

serverkeyapi/internal/api.go

@@ -6,6 +6,7 @@ import (
 	"fmt"
 	"time"
 
+	"github.com/matrix-org/dendrite/internal/config"
 	"github.com/matrix-org/dendrite/serverkeyapi/api"
 	"github.com/matrix-org/gomatrixserverlib"
 	"github.com/sirupsen/logrus"
@@ -18,6 +19,7 @@ type ServerKeyAPI struct {
 	ServerPublicKey   ed25519.PublicKey
 	ServerKeyID       gomatrixserverlib.KeyID
 	ServerKeyValidity time.Duration
+	OldServerKeys     []config.OldVerifyKeys
 
 	OurKeyRing gomatrixserverlib.KeyRing
 	FedClient  gomatrixserverlib.KeyClient
@@ -112,14 +114,17 @@ func (s *ServerKeyAPI) FetcherName() string {
 }
 
 // handleLocalKeys handles cases where the key request contains
-// a request for our own server keys.
+// a request for our own server keys, either current or old.
 func (s *ServerKeyAPI) handleLocalKeys(
 	_ context.Context,
 	requests map[gomatrixserverlib.PublicKeyLookupRequest]gomatrixserverlib.Timestamp,
 	results map[gomatrixserverlib.PublicKeyLookupRequest]gomatrixserverlib.PublicKeyLookupResult,
 ) {
 	for req := range requests {
-		if req.ServerName == s.ServerName {
+		if req.ServerName != s.ServerName {
+			continue
+		}
+		if req.KeyID == s.ServerKeyID {
 			// We found a key request that is supposed to be for our own
 			// keys. Remove it from the request list so we don't hit the
 			// database or the fetchers for it.
@@ -133,6 +138,28 @@ func (s *ServerKeyAPI) handleLocalKeys(
 				ExpiredTS:    gomatrixserverlib.PublicKeyNotExpired,
 				ValidUntilTS: gomatrixserverlib.AsTimestamp(time.Now().Add(s.ServerKeyValidity)),
 			}
+		} else {
+			// The key request doesn't match our current key. Let's see
+			// if it matches any of our old verify keys.
+			for _, oldVerifyKey := range s.OldServerKeys {
+				if req.KeyID == oldVerifyKey.KeyID {
+					// We found a key request that is supposed to be an expired
+					// key.
+					delete(requests, req)
+
+					// Insert our own key into the response.
+					results[req] = gomatrixserverlib.PublicKeyLookupResult{
+						VerifyKey: gomatrixserverlib.VerifyKey{
+							Key: gomatrixserverlib.Base64Bytes(oldVerifyKey.PrivateKey.Public().(ed25519.PublicKey)),
+						},
+						ExpiredTS:    oldVerifyKey.ExpiredAt,
+						ValidUntilTS: gomatrixserverlib.PublicKeyNotValid,
+					}
+
+					// No need to look at the other keys.
+					break
+				}
+			}
 		}
 	}
 }
@@ -175,7 +202,7 @@ func (s *ServerKeyAPI) handleDatabaseKeys(
 // the remaining requests.
 func (s *ServerKeyAPI) handleFetcherKeys(
 	ctx context.Context,
-	now gomatrixserverlib.Timestamp,
+	_ gomatrixserverlib.Timestamp,
 	fetcher gomatrixserverlib.KeyFetcher,
 	requests map[gomatrixserverlib.PublicKeyLookupRequest]gomatrixserverlib.Timestamp,
 	results map[gomatrixserverlib.PublicKeyLookupRequest]gomatrixserverlib.PublicKeyLookupResult,

serverkeyapi/serverkeyapi.go

@@ -49,6 +49,7 @@ func NewInternalAPI(
 		ServerPublicKey:   cfg.Matrix.PrivateKey.Public().(ed25519.PublicKey),
 		ServerKeyID:       cfg.Matrix.KeyID,
 		ServerKeyValidity: cfg.Matrix.KeyValidityPeriod,
+		OldServerKeys:     cfg.Matrix.OldVerifyKeys,
 		FedClient:         fedClient,
 		OurKeyRing: gomatrixserverlib.KeyRing{
 			KeyFetchers: []gomatrixserverlib.KeyFetcher{},

syncapi/storage/storage_test.go

@@ -81,7 +81,7 @@ func SimpleRoom(t *testing.T, roomID, userA, userB string) (msgs []gomatrixserve
 	}))
 	state = append(state, events[len(events)-1])
 	events = append(events, MustCreateEvent(t, roomID, []gomatrixserverlib.HeaderedEvent{events[len(events)-1]}, &gomatrixserverlib.EventBuilder{
-		Content:  []byte(fmt.Sprintf(`{"membership":"join"}`)),
+		Content:  []byte(`{"membership":"join"}`),
 		Type:     "m.room.member",
 		StateKey: &userA,
 		Sender:   userA,
@@ -97,7 +97,7 @@ func SimpleRoom(t *testing.T, roomID, userA, userB string) (msgs []gomatrixserve
 		}))
 	}
 	events = append(events, MustCreateEvent(t, roomID, []gomatrixserverlib.HeaderedEvent{events[len(events)-1]}, &gomatrixserverlib.EventBuilder{
-		Content:  []byte(fmt.Sprintf(`{"membership":"join"}`)),
+		Content:  []byte(`{"membership":"join"}`),
 		Type:     "m.room.member",
 		StateKey: &userB,
 		Sender:   userB,
@@ -348,7 +348,7 @@ func TestGetEventsInRangeWithEventsSameDepth(t *testing.T) {
 		Depth:    int64(len(events) + 1),
 	}))
 	events = append(events, MustCreateEvent(t, testRoomID, []gomatrixserverlib.HeaderedEvent{events[len(events)-1]}, &gomatrixserverlib.EventBuilder{
-		Content:  []byte(fmt.Sprintf(`{"membership":"join"}`)),
+		Content:  []byte(`{"membership":"join"}`),
 		Type:     "m.room.member",
 		StateKey: &testUserIDA,
 		Sender:   testUserIDA,
@@ -367,7 +367,7 @@ func TestGetEventsInRangeWithEventsSameDepth(t *testing.T) {
 	}
 	// merge the fork, prev_events are all 3 messages, depth is increased by 1.
 	events = append(events, MustCreateEvent(t, testRoomID, events[len(events)-3:], &gomatrixserverlib.EventBuilder{
-		Content: []byte(fmt.Sprintf(`{"body":"Message merge"}`)),
+		Content: []byte(`{"body":"Message merge"}`),
 		Type:    "m.room.message",
 		Sender:  testUserIDA,
 		Depth:   depth + 1,
@@ -438,7 +438,7 @@ func TestGetEventsInTopologicalRangeMultiRoom(t *testing.T) {
 			Depth:    int64(len(events) + 1),
 		}))
 		events = append(events, MustCreateEvent(t, roomID, []gomatrixserverlib.HeaderedEvent{events[len(events)-1]}, &gomatrixserverlib.EventBuilder{
-			Content:  []byte(fmt.Sprintf(`{"membership":"join"}`)),
+			Content:  []byte(`{"membership":"join"}`),
 			Type:     "m.room.member",
 			StateKey: &testUserIDA,
 			Sender:   testUserIDA,
@@ -484,7 +484,7 @@ func TestGetEventsInRangeWithEventsInsertedLikeBackfill(t *testing.T) {
 	// "federation" join
 	userC := fmt.Sprintf("@radiance:%s", testOrigin)
 	joinEvent := MustCreateEvent(t, testRoomID, []gomatrixserverlib.HeaderedEvent{events[len(events)-1]}, &gomatrixserverlib.EventBuilder{
-		Content:  []byte(fmt.Sprintf(`{"membership":"join"}`)),
+		Content:  []byte(`{"membership":"join"}`),
 		Type:     "m.room.member",
 		StateKey: &userC,
 		Sender:   userC,
@@ -615,14 +615,14 @@ func TestInviteBehaviour(t *testing.T) {
 	db := MustCreateDatabase(t)
 	inviteRoom1 := "!inviteRoom1:somewhere"
 	inviteEvent1 := MustCreateEvent(t, inviteRoom1, nil, &gomatrixserverlib.EventBuilder{
-		Content:  []byte(fmt.Sprintf(`{"membership":"invite"}`)),
+		Content:  []byte(`{"membership":"invite"}`),
 		Type:     "m.room.member",
 		StateKey: &testUserIDA,
 		Sender:   "@inviteUser1:somewhere",
 	})
 	inviteRoom2 := "!inviteRoom2:somewhere"
 	inviteEvent2 := MustCreateEvent(t, inviteRoom2, nil, &gomatrixserverlib.EventBuilder{
-		Content:  []byte(fmt.Sprintf(`{"membership":"invite"}`)),
+		Content:  []byte(`{"membership":"invite"}`),
 		Type:     "m.room.member",
 		StateKey: &testUserIDA,
 		Sender:   "@inviteUser2:somewhere",