Merge branch 'master' into neilalexander/userapilocks

README.md

@@ -1,22 +1,68 @@
 # Dendrite [![Build Status](https://badge.buildkite.com/4be40938ab19f2bbc4a6c6724517353ee3ec1422e279faf374.svg?branch=master)](https://buildkite.com/matrix-dot-org/dendrite) [![Dendrite Dev on Matrix](https://img.shields.io/matrix/dendrite-dev:matrix.org.svg?label=%23dendrite-dev%3Amatrix.org&logo=matrix&server_fqdn=matrix.org)](https://matrix.to/#/#dendrite-dev:matrix.org) [![Dendrite on Matrix](https://img.shields.io/matrix/dendrite:matrix.org.svg?label=%23dendrite%3Amatrix.org&logo=matrix&server_fqdn=matrix.org)](https://matrix.to/#/#dendrite:matrix.org)
 
-Dendrite will be a second-generation Matrix homeserver written in Go.
+Dendrite is a second-generation Matrix homeserver written in Go. It is not recommended to use Dendrite as
+a production homeserver at this time as there is no stable release. An overview of the design can be found
+in [DESIGN.md](docs/DESIGN.md).
 
-It's still very much a work in progress, but installation instructions can be
+# Quick start
-found in [INSTALL.md](docs/INSTALL.md). It is not recommended to use Dendrite as a
+
-production homeserver at this time.
+Requires Go 1.13+ and SQLite3 (Postgres is also supported):
+
+```bash
+$ git clone https://github.com/matrix-org/dendrite
+$ cd dendrite
+
+# generate self-signed certificate and an event signing key for federation
+$ go build ./cmd/generate-keys
+$ ./generate-keys --private-key matrix_key.pem --tls-cert server.crt --tls-key server.key
+
+# Copy and modify the config file:
+# you'll need to set a server name and paths to the keys at the very least, along with setting
+# up the database filenames
+$ cp dendrite-config.yaml dendrite.yaml
+
+# build and run the server
+$ go build ./cmd/dendrite-monolith-server
+$ ./dendrite-monolith-server --tls-cert server.crt --tls-key server.key --config dendrite.yaml
+```
+
+Then point your favourite Matrix client at `http://localhost:8008`. For full installation information, see
+[INSTALL.md](docs/INSTALL.md). For running in Docker, see [build/docker](build/docker).
+
+# Progress
+
+We use a script called Are We Synapse Yet which checks Sytest compliance rates. Sytest is a black-box homeserver
+test rig with around 900 tests. The script works out how many of these tests are passing on Dendrite and it
+updates with CI. As of July 2020 we're at around 48% CS API coverage and 50% Federation coverage, though check
+CI for the latest numbers. In practice, this means you can communicate locally and via federation with Synapse
+servers such as matrix.org reasonably well. There's a long list of features that are not implemented, notably:
+ - Receipts
+ - Push
+ - Search and Context
+ - User Directory
+ - Presence
+ - Guests
+ - E2E keys and device lists
+
+We are prioritising features that will benefit single-user homeservers first (e.g Receipts, E2E) rather
+than features that massive deployments may be interested in (User Directory, OpenID, Guests, Admin APIs, AS API).
+This means Dendrite supports amongst others:
+ - Core room functionality (creating rooms, invites, auth rules)
+ - Federation in rooms v1-v6
+ - Backfilling locally and via federation
+ - Accounts, Profiles and Devices
+ - Published room lists
+ - Typing
+ - Media APIs
+ - Redaction
+ - Tagging
 
-An overview of the design can be found in [DESIGN.md](docs/DESIGN.md).
 
 # Contributing
 
 Everyone is welcome to help out and contribute! See
 [CONTRIBUTING.md](docs/CONTRIBUTING.md) to get started!
 
-Please note that, as of February 2020, Dendrite now only targets Go 1.13 or
-later. Please ensure that you are using at least Go 1.13 when developing for
-Dendrite.
-
 # Discussion
 
 For questions about Dendrite we have a dedicated room on Matrix
@@ -24,8 +70,4 @@ For questions about Dendrite we have a dedicated room on Matrix
 discussion should happen in
 [#dendrite-dev:matrix.org](https://matrix.to/#/#dendrite-dev:matrix.org).
 
-# Progress
-
-There's plenty still to do to make Dendrite usable! We're tracking progress in a
-[project board](https://github.com/matrix-org/dendrite/projects/2).
 

docs/CODE_STYLE.md

@@ -1,12 +1,7 @@
 # Code Style
 
-We follow the standard Go style using goimports, but with a few extra
+In addition to standard Go code style (`gofmt`, `goimports`), we use `golangci-lint`
-considerations.
+to run a number of linters, the exact list can be found under linters in [.golangci.yml](.golangci.yml).
-
-## Linters
-
-We use `golangci-lint` to run a number of linters, the exact list can be found
-under linters in [.golangci.yml](.golangci.yml).
 [Installation](https://github.com/golangci/golangci-lint#install) and [Editor
 Integration](https://github.com/golangci/golangci-lint#editor-integration) for
 it can be found in the readme of golangci-lint.
@@ -22,37 +17,10 @@ The linters can be run using [scripts/find-lint.sh](scripts/find-lint.sh)
 [scripts/build-test-lint.sh](scripts/build-test-lint.sh).
 
 
-## HTTP Error Handling
+## Labels
-
-Unfortunately, converting errors into HTTP responses with the correct status
-code and message can be done in a number of ways in golang:
-
-1. Having functions return `JSONResponse` directly, which can then either set
-   it to an error response or a `200 OK`.
-2. Have the HTTP handler try and cast error values to types that are handled
-   differently.
-3. Have the HTTP handler call functions whose errors can only be interpreted
-   one way, for example if a `validate(...)` call returns an error then handler
-   knows to respond with a `400 Bad Request`.
-
-We attempt to always use option #3, as it more naturally fits with the way that
-golang generally does error handling. In particular, option #1 effectively
-requires reinventing a new error handling scheme just for HTTP handlers.
-
-
-## Line length
-
-We strive for a line length of roughly 80 characters, though less than 100 is
-acceptable if necessary. Longer lines are fine if there is nothing of interest
-after the first 80-100 characters (e.g. long string literals).
-
-
-## TODOs and FIXMEs
-
-The majority of TODOs and FIXMEs should have an associated tracking issue on
-github. These can be added just before merging of the PR to master, and the
-issue number should be added to the comment, e.g. `// TODO(#324): ...`
 
+In addition to `TODO` and `FIXME` we also use `NOTSPEC` to identify deviations
+from the Matrix specification.
 
 ## Logging
 

docs/INSTALL.md

@@ -3,20 +3,21 @@
 Dendrite can be run in one of two configurations:
 
 * **Polylith mode**: A cluster of individual components, dealing with different
-  aspects of the Matrix protocol (see [WIRING.md](WIRING.md)). Components communicate with each other using internal HTTP APIs and [Apache Kafka](https://kafka.apache.org). This will almost certainly be the preferred model
+  aspects of the Matrix protocol (see [WIRING.md](WIRING-Current.md)). Components communicate
-  for large-scale deployments.
+  with each other using internal HTTP APIs and [Apache Kafka](https://kafka.apache.org).
+  This will almost certainly be the preferred model for large-scale deployments.
 
 * **Monolith mode**: All components run in the same process. In this mode,
    Kafka is completely optional and can instead be replaced with an in-process
-   lightweight implementation called [Naffka](https://github.com/matrix-org/naffka). This will usually be the preferred model for low-volume, low-user
+   lightweight implementation called [Naffka](https://github.com/matrix-org/naffka). This
-   or experimental deployments.
+   will usually be the preferred model for low-volume, low-user or experimental deployments.
 
-Regardless of whether you are running in polylith or monolith mode, each Dendrite component that requires storage has its own database. Both Postgres
+Regardless of whether you are running in polylith or monolith mode, each Dendrite component that
-and SQLite are supported and can be mixed-and-matched across components as
+requires storage has its own database. Both Postgres and SQLite are supported and can be
-needed in the configuration file.
+mixed-and-matched across components as needed in the configuration file.
 
-Be advised that Dendrite is still developmental and it's not recommended for
+Be advised that Dendrite is still in development and it's not recommended for
-use in production environments yet!
+use in production environments just yet!
 
 ## Requirements
 
@@ -119,16 +120,10 @@ Assuming that Postgres 9.5 (or later) is installed:
 
 Each Dendrite server requires unique server keys.
 
-Generate the self-signed SSL certificate for federation:
+Generate the self-signed SSL certificate for federation and the server signing key:
 
 ```bash
-test -f server.key || openssl req -x509 -newkey rsa:4096 -keyout server.key -out server.crt -days 3650 -nodes -subj /CN=localhost
+./bin/generate-keys --private-key matrix_key.pem --tls-cert server.crt --tls-key server.key
-```
-
-Generate the server signing key:
-
-```
-test -f matrix_key.pem || ./bin/generate-keys -private-key matrix_key.pem
 ```
 
 ### Configuration file
@@ -152,7 +147,8 @@ public keys for dead homeservers from somewhere else.
 ## Starting a monolith server
 
 It is possible to use Naffka as an in-process replacement to Kafka when using
-the monolith server. To do this, set `use_naffka: true` in your `dendrite.yaml` configuration and uncomment the relevant Naffka line in the `database` section.
+the monolith server. To do this, set `use_naffka: true` in your `dendrite.yaml`
+configuration and uncomment the relevant Naffka line in the `database` section.
 Be sure to update the database username and password if needed.
 
 The monolith server can be started as shown below. By default it listens for
@@ -166,60 +162,7 @@ as shown below, it will also listen for HTTPS connections on port 8448.
 
 ## Starting a polylith deployment
 
-The following contains scripts which will run all the required processes in order to point a Matrix client at Dendrite. Conceptually, you are wiring together to form the following diagram:
+The following contains scripts which will run all the required processes in order to point a Matrix client at Dendrite.
-
-```
-
-                                         /media   +---------------------------+
-                      +----------->+------------->| dendrite-media-api-server |
-                      ^            ^              +---------------------------+
-                      |            |            :7774
-                      |            |
-                      |            |
-                      |            |   
-                      |            |   
-                      |            |   
-                      |            |  
-                      |            |  
-                      |            |  
-                      |            |     /sync    +--------------------------+                 
-                      |            |   +--------->| dendrite-sync-api-server |<================++
-                      |            |   |          +--------------------------+                 ||
-                      |            |   |        :7773    |         ^^                          ||
-Matrix      +------------------+   |   |                 |         ||    client_data           ||
-Clients --->| client-api-proxy |-------+    +<-----------+         ++=============++           ||
-            +------------------+   |   |    |                                     ||           ||
-          :8008                    |   | CS API   +----------------------------+  ||           ||
-                                   |   +--------->| dendrite-client-api-server |==++           ||
-                                   |        |     +----------------------------+               ||
-                                   |        |   :7771    |                                     ||
-                                   |        |            |                                     ||
-                                   |        +<-----------+                                     ||
-                                   |        |                                                  ||
-                                   |        |                                                  ||
-                                   |        |           +----------------------+    room_event ||
-                                   |        +---------->| dendrite-room-server |===============++
-                                   |        |           +----------------------+               ||
-                                   |        |         :7770                                    ||
-                                   |        |                      ++==========================++
-                                   |        +<------------+        ||
-                                   |        |             |        VV
-                                   |        |     +-----------------------------------+              Matrix
-                                   |        |     | dendrite-federation-sender-server |------------> Servers
-                                   |        |     +-----------------------------------+
-                                   |        |   :7776
-                                   |        |
-                       +---------->+        +<-----------+
-                       |                                 |
-Matrix      +----------------------+  SS API  +--------------------------------+
-Servers --->| federation-api-proxy |--------->| dendrite-federation-api-server |
-            +----------------------+          +--------------------------------+
-          :8448                             :7772
-
-
-   A --> B  = HTTP requests (A = client, B = server)
-   A ==> B  = Kafka (A = producer, B = consumer)
-```
 
 ### Client proxy
 
@@ -248,21 +191,13 @@ to support federation.
 
 ### Client API server
 
-This is what implements message sending. Clients talk to this via the proxy in
+This is what implements CS API endpoints. Clients talk to this via the proxy in
-order to send messages.
+order to send messages, create and join rooms, etc.
 
 ```bash
 ./bin/dendrite-client-api-server --config=dendrite.yaml
 ```
 
-### Room server
-
-This is what implements the room DAG. Clients do not talk to this.
-
-```bash
-./bin/dendrite-room-server --config=dendrite.yaml
-```
-
 ### Sync server
 
 This is what implements `/sync` requests. Clients talk to this via the proxy
@@ -283,7 +218,7 @@ order to upload and retrieve media.
 
 ### Federation API server
 
-This implements federation requests. Servers talk to this via the proxy in
+This implements the federation API. Servers talk to this via the proxy in
 order to send transactions.  This is only required if you want to support
 federation.
 
@@ -291,7 +226,30 @@ federation.
 ./bin/dendrite-federation-api-server --config dendrite.yaml
 ```
 
-### Federation sender
+### Internal components
+
+This refers to components that are not directly spoken to by clients. They are only
+contacted by other components. This includes the following components.
+
+#### Room server
+
+This is what implements the room DAG. Clients do not talk to this.
+
+```bash
+./bin/dendrite-room-server --config=dendrite.yaml
+```
+
+#### Current state server
+
+This tracks the current state of rooms which various components need to know. For example,
+`/publicRooms` implemented by client API asks this server for the room names, joined member
+counts, etc.
+
+```bash
+./bin/dendrite-current-state-server --config=dendrite.yaml
+```
+
+#### Federation sender
 
 This sends events from our users to other servers.  This is only required if
 you want to support federation.
@@ -300,7 +258,7 @@ you want to support federation.
 ./bin/dendrite-federation-sender-server --config dendrite.yaml
 ```
 
-### Appservice server
+#### Appservice server
 
 This sends events from the network to [application
 services](https://matrix.org/docs/spec/application_service/unstable.html)
@@ -311,16 +269,31 @@ application services on your homeserver.
 ./bin/dendrite-appservice-server --config dendrite.yaml
 ```
 
-### Key server
+#### Key server
 
-This manages end-to-end encryption keys (or rather, it will do when it's
+This manages end-to-end encryption keys for users.
-finished).
 
 ```bash
 ./bin/dendrite-key-server --config dendrite.yaml
 ```
 
-### User server
+#### Server Key server
+
+This manages signing keys for servers.
+
+```bash
+./bin/dendrite-server-key-api-server --config dendrite.yaml
+```
+
+#### EDU server
+
+This manages processing EDUs such as typing, send-to-device events and presence. Clients do not talk to
+
+```bash
+./bin/dendrite-edu-server --config dendrite.yaml
+```
+
+#### User server
 
 This manages user accounts, device access tokens and user account data,
 amongst other things.

keyserver/api/api.go

@@ -14,7 +14,10 @@
 
 package api
 
-import "context"
+import (
+	"context"
+	"encoding/json"
+)
 
 type KeyInternalAPI interface {
 	PerformUploadKeys(ctx context.Context, req *PerformUploadKeysRequest, res *PerformUploadKeysResponse)
@@ -27,11 +30,62 @@ type KeyError struct {
 	Error string
 }
 
-type PerformUploadKeysRequest struct {
+// DeviceKeys represents a set of device keys for a single device
+// https://matrix.org/docs/spec/client_server/r0.6.1#post-matrix-client-r0-keys-upload
+type DeviceKeys struct {
+	// The user who owns this device
+	UserID string
+	// The device ID of this device
+	DeviceID string
+	// The raw device key JSON
+	KeyJSON []byte
 }
 
+// OneTimeKeys represents a set of one-time keys for a single device
+// https://matrix.org/docs/spec/client_server/r0.6.1#post-matrix-client-r0-keys-upload
+type OneTimeKeys struct {
+	// The user who owns this device
+	UserID string
+	// The device ID of this device
+	DeviceID string
+	// A map of algorithm:key_id => key JSON
+	KeyJSON map[string]json.RawMessage
+}
+
+// OneTimeKeysCount represents the counts of one-time keys for a single device
+type OneTimeKeysCount struct {
+	// The user who owns this device
+	UserID string
+	// The device ID of this device
+	DeviceID string
+	// algorithm to count e.g:
+	// {
+	//   "curve25519": 10,
+	//   "signed_curve25519": 20
+	// }
+	KeyCount map[string]int
+}
+
+// PerformUploadKeysRequest is the request to PerformUploadKeys
+type PerformUploadKeysRequest struct {
+	DeviceKeys  []DeviceKeys
+	OneTimeKeys []OneTimeKeys
+}
+
+// PerformUploadKeysResponse is the response to PerformUploadKeys
 type PerformUploadKeysResponse struct {
 	Error *KeyError
+	// A map of user_id -> device_id -> Error for tracking failures.
+	KeyErrors        map[string]map[string]*KeyError
+	OneTimeKeyCounts []OneTimeKeysCount
+}
+
+// KeyError sets a key error field on KeyErrors
+func (r *PerformUploadKeysResponse) KeyError(userID, deviceID string, err *KeyError) {
+	if r.KeyErrors[userID] == nil {
+		r.KeyErrors[userID] = make(map[string]*KeyError)
+	}
+	r.KeyErrors[userID][deviceID] = err
 }
 
 type PerformClaimKeysRequest struct {

keyserver/internal/internal.go

@@ -1,15 +1,37 @@
+// Copyright 2020 The Matrix.org Foundation C.I.C.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
 package internal
 
 import (
+	"bytes"
 	"context"
+	"fmt"
 
 	"github.com/matrix-org/dendrite/keyserver/api"
+	"github.com/matrix-org/dendrite/keyserver/storage"
+	"github.com/tidwall/gjson"
 )
 
-type KeyInternalAPI struct{}
+type KeyInternalAPI struct {
+	db storage.Database
+}
 
 func (a *KeyInternalAPI) PerformUploadKeys(ctx context.Context, req *api.PerformUploadKeysRequest, res *api.PerformUploadKeysResponse) {
-
+	res.KeyErrors = make(map[string]map[string]*api.KeyError)
+	a.uploadDeviceKeys(ctx, req, res)
+	a.uploadOneTimeKeys(ctx, req, res)
 }
 func (a *KeyInternalAPI) PerformClaimKeys(ctx context.Context, req *api.PerformClaimKeysRequest, res *api.PerformClaimKeysResponse) {
 
@@ -17,3 +39,84 @@ func (a *KeyInternalAPI) PerformClaimKeys(ctx context.Context, req *api.PerformC
 func (a *KeyInternalAPI) QueryKeys(ctx context.Context, req *api.QueryKeysRequest, res *api.QueryKeysResponse) {
 
 }
+
+func (a *KeyInternalAPI) uploadDeviceKeys(ctx context.Context, req *api.PerformUploadKeysRequest, res *api.PerformUploadKeysResponse) {
+	var keysToStore []api.DeviceKeys
+	// assert that the user ID / device ID are not lying for each key
+	for _, key := range req.DeviceKeys {
+		gotUserID := gjson.GetBytes(key.KeyJSON, "user_id").Str
+		gotDeviceID := gjson.GetBytes(key.KeyJSON, "device_id").Str
+		if gotUserID == key.UserID && gotDeviceID == key.DeviceID {
+			keysToStore = append(keysToStore, key)
+			continue
+		}
+
+		res.KeyError(key.UserID, key.DeviceID, &api.KeyError{
+			Error: fmt.Sprintf(
+				"user_id or device_id mismatch: users: %s - %s, devices: %s - %s",
+				gotUserID, key.UserID, gotDeviceID, key.DeviceID,
+			),
+		})
+	}
+	// get existing device keys so we can check for changes
+	existingKeys := make([]api.DeviceKeys, len(keysToStore))
+	for i := range keysToStore {
+		existingKeys[i] = api.DeviceKeys{
+			UserID:   keysToStore[i].UserID,
+			DeviceID: keysToStore[i].DeviceID,
+		}
+	}
+	if err := a.db.DeviceKeysJSON(ctx, existingKeys); err != nil {
+		res.Error = &api.KeyError{
+			Error: fmt.Sprintf("failed to query existing device keys: %s", err.Error()),
+		}
+		return
+	}
+	// store the device keys and emit changes
+	if err := a.db.StoreDeviceKeys(ctx, keysToStore); err != nil {
+		res.Error = &api.KeyError{
+			Error: fmt.Sprintf("failed to store device keys: %s", err.Error()),
+		}
+		return
+	}
+	a.emitDeviceKeyChanges(existingKeys, keysToStore)
+}
+
+func (a *KeyInternalAPI) uploadOneTimeKeys(ctx context.Context, req *api.PerformUploadKeysRequest, res *api.PerformUploadKeysResponse) {
+	for _, key := range req.OneTimeKeys {
+		// grab existing keys based on (user/device/algorithm/key ID)
+		keyIDsWithAlgorithms := make([]string, len(key.KeyJSON))
+		i := 0
+		for keyIDWithAlgo := range key.KeyJSON {
+			keyIDsWithAlgorithms[i] = keyIDWithAlgo
+			i++
+		}
+		existingKeys, err := a.db.ExistingOneTimeKeys(ctx, key.UserID, key.DeviceID, keyIDsWithAlgorithms)
+		if err != nil {
+			res.KeyError(key.UserID, key.DeviceID, &api.KeyError{
+				Error: "failed to query existing one-time keys: " + err.Error(),
+			})
+			continue
+		}
+		for keyIDWithAlgo := range existingKeys {
+			// if keys exist and the JSON doesn't match, error out as the key already exists
+			if !bytes.Equal(existingKeys[keyIDWithAlgo], key.KeyJSON[keyIDWithAlgo]) {
+				res.KeyError(key.UserID, key.DeviceID, &api.KeyError{
+					Error: fmt.Sprintf("%s device %s: algorithm / key ID %s one-time key already exists", key.UserID, key.DeviceID, keyIDWithAlgo),
+				})
+				continue
+			}
+		}
+		// store one-time keys
+		if err := a.db.StoreOneTimeKeys(ctx, key); err != nil {
+			res.KeyError(key.UserID, key.DeviceID, &api.KeyError{
+				Error: fmt.Sprintf("%s device %s : failed to store one-time keys: %s", key.UserID, key.DeviceID, err.Error()),
+			})
+		}
+	}
+
+}
+
+func (a *KeyInternalAPI) emitDeviceKeyChanges(existing, new []api.DeviceKeys) {
+	// TODO
+}

keyserver/storage/interface.go

@@ -0,0 +1,38 @@
+// Copyright 2020 The Matrix.org Foundation C.I.C.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package storage
+
+import (
+	"context"
+	"encoding/json"
+
+	"github.com/matrix-org/dendrite/keyserver/api"
+)
+
+type Database interface {
+	// ExistingOneTimeKeys returns a map of keyIDWithAlgorithm to key JSON for the given parameters. If no keys exist with this combination
+	// of user/device/key/algorithm 4-uple then it is omitted from the map. Returns an error when failing to communicate with the database.
+	ExistingOneTimeKeys(ctx context.Context, userID, deviceID string, keyIDsWithAlgorithms []string) (map[string]json.RawMessage, error)
+
+	// StoreOneTimeKeys persists the given one-time keys.
+	StoreOneTimeKeys(ctx context.Context, keys api.OneTimeKeys) error
+
+	// DeviceKeysJSON populates the KeyJSON for the given keys. If any proided `keys` have a `KeyJSON` already then it will be replaced.
+	DeviceKeysJSON(ctx context.Context, keys []api.DeviceKeys) error
+
+	// StoreDeviceKeys persists the given keys. Keys with the same user ID and device ID will be replaced.
+	// Returns an error if there was a problem storing the keys.
+	StoreDeviceKeys(ctx context.Context, keys []api.DeviceKeys) error
+}