From 77b1fb823ecd6f332ba725cb3e038ec656ab2ac8 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Tue, 14 Jul 2020 11:52:09 +0100 Subject: [PATCH] Update docs --- README.md | 44 ++++++++++--- docs/CODE_STYLE.md | 42 ++----------- docs/INSTALL.md | 151 +++++++++++++++++++-------------------------- 3 files changed, 101 insertions(+), 136 deletions(-) diff --git a/README.md b/README.md index 07cf9d926..168a864a4 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,8 @@ # Dendrite [![Build Status](https://badge.buildkite.com/4be40938ab19f2bbc4a6c6724517353ee3ec1422e279faf374.svg?branch=master)](https://buildkite.com/matrix-dot-org/dendrite) [![Dendrite Dev on Matrix](https://img.shields.io/matrix/dendrite-dev:matrix.org.svg?label=%23dendrite-dev%3Amatrix.org&logo=matrix&server_fqdn=matrix.org)](https://matrix.to/#/#dendrite-dev:matrix.org) [![Dendrite on Matrix](https://img.shields.io/matrix/dendrite:matrix.org.svg?label=%23dendrite%3Amatrix.org&logo=matrix&server_fqdn=matrix.org)](https://matrix.to/#/#dendrite:matrix.org) -Dendrite is a second-generation Matrix homeserver written in Go. It is not recommended to use Dendrite as a production homeserver at this time as there is no stable release. An overview of the design can be found in [DESIGN.md](docs/DESIGN.md). +Dendrite is a second-generation Matrix homeserver written in Go. It is not recommended to use Dendrite as +a production homeserver at this time as there is no stable release. An overview of the design can be found +in [DESIGN.md](docs/DESIGN.md). # Quick start @@ -11,7 +13,7 @@ $ git clone https://github.com/matrix-org/dendrite $ cd dendrite # generate self-signed certificate and an event signing key for federation -$ go build -o bin/generate-keys ./cmd/generate-keys +$ go build ./cmd/generate-keys $ ./generate-keys --private-key matrix_key.pem --tls-cert server.crt --tls-key server.key # Copy and modify the config file: @@ -24,7 +26,36 @@ $ go build ./cmd/dendrite-monolith-server $ ./dendrite-monolith-server --tls-cert server.crt --tls-key server.key --config dendrite.yaml ``` -For full installation information, see [INSTALL.md](docs/INSTALL.md). +Then point your favourite Matrix client at `http://localhost:8008`. For full installation information, see +[INSTALL.md](docs/INSTALL.md). For running in Docker, see [build/docker](build/docker). + +# Progress + +We use a script called Are We Synapse Yet which checks Sytest compliance rates. Sytest is a black-box homeserver +test rig with around 900 tests. The script works out how many of these tests are passing on Dendrite and it +updates with CI. As of July 2020 we're at around 48% CS API coverage and 50% Federation coverage, though check +CI for the latest numbers. In practice, this means you can communicate locally and via federation with Synapse +servers such as matrix.org reasonably well. There's a long list of features that are not implemented, notably: + - Receipts + - Push + - Search and Context + - User Directory + - Presence + - Guests + - E2E keys and device lists + +We are prioritising features that will benefit single-user homeservers first (e.g Receipts, E2E) rather +than features that massive deployments may be interested in (User Directory, OpenID, Guests, Admin APIs, AS API). +This means Dendrite supports amongst others: + - Core room functionality (creating rooms, invites, auth rules) + - Federation in rooms v1-v6 + - Backfilling locally and via federation + - Accounts, Profiles and Devices + - Published room lists + - Typing + - Media APIs + - Redaction + - Tagging # Contributing @@ -32,10 +63,6 @@ For full installation information, see [INSTALL.md](docs/INSTALL.md). Everyone is welcome to help out and contribute! See [CONTRIBUTING.md](docs/CONTRIBUTING.md) to get started! -Please note that, as of February 2020, Dendrite now only targets Go 1.13 or -later. Please ensure that you are using at least Go 1.13 when developing for -Dendrite. - # Discussion For questions about Dendrite we have a dedicated room on Matrix @@ -43,7 +70,4 @@ For questions about Dendrite we have a dedicated room on Matrix discussion should happen in [#dendrite-dev:matrix.org](https://matrix.to/#/#dendrite-dev:matrix.org). -# Progress - -We use a script called Are We Synapse Yet which checks Sytest compliance rates. Sytest is a black-box homeserver test rig with around 900 tests. The script works out how many of these tests are passing on Dendrite and it updates with CI. As of July 2020 we're at around 46% CS API coverage and 50% Federation coverage, though check CI for the latest numbers. diff --git a/docs/CODE_STYLE.md b/docs/CODE_STYLE.md index 1f40c76fc..248d76107 100644 --- a/docs/CODE_STYLE.md +++ b/docs/CODE_STYLE.md @@ -1,12 +1,7 @@ # Code Style -We follow the standard Go style using goimports, but with a few extra -considerations. - -## Linters - -We use `golangci-lint` to run a number of linters, the exact list can be found -under linters in [.golangci.yml](.golangci.yml). +In addition to standard Go code style (`gofmt`, `goimports`), we use `golangci-lint` +to run a number of linters, the exact list can be found under linters in [.golangci.yml](.golangci.yml). [Installation](https://github.com/golangci/golangci-lint#install) and [Editor Integration](https://github.com/golangci/golangci-lint#editor-integration) for it can be found in the readme of golangci-lint. @@ -22,37 +17,10 @@ The linters can be run using [scripts/find-lint.sh](scripts/find-lint.sh) [scripts/build-test-lint.sh](scripts/build-test-lint.sh). -## HTTP Error Handling - -Unfortunately, converting errors into HTTP responses with the correct status -code and message can be done in a number of ways in golang: - -1. Having functions return `JSONResponse` directly, which can then either set - it to an error response or a `200 OK`. -2. Have the HTTP handler try and cast error values to types that are handled - differently. -3. Have the HTTP handler call functions whose errors can only be interpreted - one way, for example if a `validate(...)` call returns an error then handler - knows to respond with a `400 Bad Request`. - -We attempt to always use option #3, as it more naturally fits with the way that -golang generally does error handling. In particular, option #1 effectively -requires reinventing a new error handling scheme just for HTTP handlers. - - -## Line length - -We strive for a line length of roughly 80 characters, though less than 100 is -acceptable if necessary. Longer lines are fine if there is nothing of interest -after the first 80-100 characters (e.g. long string literals). - - -## TODOs and FIXMEs - -The majority of TODOs and FIXMEs should have an associated tracking issue on -github. These can be added just before merging of the PR to master, and the -issue number should be added to the comment, e.g. `// TODO(#324): ...` +## Labels +In addition to `TODO` and `FIXME` we also use `NOTSPEC` to identify deviations +from the Matrix specification. ## Logging diff --git a/docs/INSTALL.md b/docs/INSTALL.md index c97351809..cf8adee81 100644 --- a/docs/INSTALL.md +++ b/docs/INSTALL.md @@ -3,20 +3,21 @@ Dendrite can be run in one of two configurations: * **Polylith mode**: A cluster of individual components, dealing with different - aspects of the Matrix protocol (see [WIRING.md](WIRING.md)). Components communicate with each other using internal HTTP APIs and [Apache Kafka](https://kafka.apache.org). This will almost certainly be the preferred model - for large-scale deployments. + aspects of the Matrix protocol (see [WIRING.md](WIRING-Current.md)). Components communicate + with each other using internal HTTP APIs and [Apache Kafka](https://kafka.apache.org). + This will almost certainly be the preferred model for large-scale deployments. * **Monolith mode**: All components run in the same process. In this mode, Kafka is completely optional and can instead be replaced with an in-process - lightweight implementation called [Naffka](https://github.com/matrix-org/naffka). This will usually be the preferred model for low-volume, low-user - or experimental deployments. + lightweight implementation called [Naffka](https://github.com/matrix-org/naffka). This + will usually be the preferred model for low-volume, low-user or experimental deployments. -Regardless of whether you are running in polylith or monolith mode, each Dendrite component that requires storage has its own database. Both Postgres -and SQLite are supported and can be mixed-and-matched across components as -needed in the configuration file. +Regardless of whether you are running in polylith or monolith mode, each Dendrite component that +requires storage has its own database. Both Postgres and SQLite are supported and can be +mixed-and-matched across components as needed in the configuration file. -Be advised that Dendrite is still developmental and it's not recommended for -use in production environments yet! +Be advised that Dendrite is still in development and it's not recommended for +use in production environments just yet! ## Requirements @@ -119,16 +120,10 @@ Assuming that Postgres 9.5 (or later) is installed: Each Dendrite server requires unique server keys. -Generate the self-signed SSL certificate for federation: +Generate the self-signed SSL certificate for federation and the server signing key: ```bash -test -f server.key || openssl req -x509 -newkey rsa:4096 -keyout server.key -out server.crt -days 3650 -nodes -subj /CN=localhost -``` - -Generate the server signing key: - -``` -test -f matrix_key.pem || ./bin/generate-keys -private-key matrix_key.pem +./bin/generate-keys --private-key matrix_key.pem --tls-cert server.crt --tls-key server.key ``` ### Configuration file @@ -152,7 +147,8 @@ public keys for dead homeservers from somewhere else. ## Starting a monolith server It is possible to use Naffka as an in-process replacement to Kafka when using -the monolith server. To do this, set `use_naffka: true` in your `dendrite.yaml` configuration and uncomment the relevant Naffka line in the `database` section. +the monolith server. To do this, set `use_naffka: true` in your `dendrite.yaml` +configuration and uncomment the relevant Naffka line in the `database` section. Be sure to update the database username and password if needed. The monolith server can be started as shown below. By default it listens for @@ -166,60 +162,7 @@ as shown below, it will also listen for HTTPS connections on port 8448. ## Starting a polylith deployment -The following contains scripts which will run all the required processes in order to point a Matrix client at Dendrite. Conceptually, you are wiring together to form the following diagram: - -``` - - /media +---------------------------+ - +----------->+------------->| dendrite-media-api-server | - ^ ^ +---------------------------+ - | | :7774 - | | - | | - | | - | | - | | - | | - | | - | | - | | /sync +--------------------------+ - | | +--------->| dendrite-sync-api-server |<================++ - | | | +--------------------------+ || - | | | :7773 | ^^ || -Matrix +------------------+ | | | || client_data || -Clients --->| client-api-proxy |-------+ +<-----------+ ++=============++ || - +------------------+ | | | || || - :8008 | | CS API +----------------------------+ || || - | +--------->| dendrite-client-api-server |==++ || - | | +----------------------------+ || - | | :7771 | || - | | | || - | +<-----------+ || - | | || - | | || - | | +----------------------+ room_event || - | +---------->| dendrite-room-server |===============++ - | | +----------------------+ || - | | :7770 || - | | ++==========================++ - | +<------------+ || - | | | VV - | | +-----------------------------------+ Matrix - | | | dendrite-federation-sender-server |------------> Servers - | | +-----------------------------------+ - | | :7776 - | | - +---------->+ +<-----------+ - | | -Matrix +----------------------+ SS API +--------------------------------+ -Servers --->| federation-api-proxy |--------->| dendrite-federation-api-server | - +----------------------+ +--------------------------------+ - :8448 :7772 - - - A --> B = HTTP requests (A = client, B = server) - A ==> B = Kafka (A = producer, B = consumer) -``` +The following contains scripts which will run all the required processes in order to point a Matrix client at Dendrite. ### Client proxy @@ -248,21 +191,13 @@ to support federation. ### Client API server -This is what implements message sending. Clients talk to this via the proxy in -order to send messages. +This is what implements CS API endpoints. Clients talk to this via the proxy in +order to send messages, create and join rooms, etc. ```bash ./bin/dendrite-client-api-server --config=dendrite.yaml ``` -### Room server - -This is what implements the room DAG. Clients do not talk to this. - -```bash -./bin/dendrite-room-server --config=dendrite.yaml -``` - ### Sync server This is what implements `/sync` requests. Clients talk to this via the proxy @@ -283,7 +218,7 @@ order to upload and retrieve media. ### Federation API server -This implements federation requests. Servers talk to this via the proxy in +This implements the federation API. Servers talk to this via the proxy in order to send transactions. This is only required if you want to support federation. @@ -291,7 +226,30 @@ federation. ./bin/dendrite-federation-api-server --config dendrite.yaml ``` -### Federation sender +### Internal components + +This refers to components that are not directly spoken to by clients. They are only +contacted by other components. This includes the following components. + +#### Room server + +This is what implements the room DAG. Clients do not talk to this. + +```bash +./bin/dendrite-room-server --config=dendrite.yaml +``` + +#### Current state server + +This tracks the current state of rooms which various components need to know. For example, +`/publicRooms` implemented by client API asks this server for the room names, joined member +counts, etc. + +```bash +./bin/dendrite-current-state-server --config=dendrite.yaml +``` + +#### Federation sender This sends events from our users to other servers. This is only required if you want to support federation. @@ -300,7 +258,7 @@ you want to support federation. ./bin/dendrite-federation-sender-server --config dendrite.yaml ``` -### Appservice server +#### Appservice server This sends events from the network to [application services](https://matrix.org/docs/spec/application_service/unstable.html) @@ -311,16 +269,31 @@ application services on your homeserver. ./bin/dendrite-appservice-server --config dendrite.yaml ``` -### Key server +#### Key server -This manages end-to-end encryption keys (or rather, it will do when it's -finished). +This manages end-to-end encryption keys for users. ```bash ./bin/dendrite-key-server --config dendrite.yaml ``` -### User server +#### Server Key server + +This manages signing keys for servers. + +```bash +./bin/dendrite-server-key-api-server --config dendrite.yaml +``` + +#### EDU server + +This manages processing EDUs such as typing, send-to-device events and presence. Clients do not talk to + +```bash +./bin/dendrite-edu-server --config dendrite.yaml +``` + +#### User server This manages user accounts, device access tokens and user account data, amongst other things.