Update docs

README.md

@@ -1,6 +1,8 @@
 # Dendrite [![Build Status](https://badge.buildkite.com/4be40938ab19f2bbc4a6c6724517353ee3ec1422e279faf374.svg?branch=master)](https://buildkite.com/matrix-dot-org/dendrite) [![Dendrite Dev on Matrix](https://img.shields.io/matrix/dendrite-dev:matrix.org.svg?label=%23dendrite-dev%3Amatrix.org&logo=matrix&server_fqdn=matrix.org)](https://matrix.to/#/#dendrite-dev:matrix.org) [![Dendrite on Matrix](https://img.shields.io/matrix/dendrite:matrix.org.svg?label=%23dendrite%3Amatrix.org&logo=matrix&server_fqdn=matrix.org)](https://matrix.to/#/#dendrite:matrix.org)
 
-Dendrite is a second-generation Matrix homeserver written in Go. It is not recommended to use Dendrite as a production homeserver at this time as there is no stable release. An overview of the design can be found in [DESIGN.md](docs/DESIGN.md).
+Dendrite is a second-generation Matrix homeserver written in Go. It is not recommended to use Dendrite as
+a production homeserver at this time as there is no stable release. An overview of the design can be found
+in [DESIGN.md](docs/DESIGN.md).
 
 # Quick start
 
@@ -11,7 +13,7 @@ $ git clone https://github.com/matrix-org/dendrite
 $ cd dendrite
 
 # generate self-signed certificate and an event signing key for federation
-$ go build -o bin/generate-keys ./cmd/generate-keys
+$ go build ./cmd/generate-keys
 $ ./generate-keys --private-key matrix_key.pem --tls-cert server.crt --tls-key server.key
 
 # Copy and modify the config file:
@@ -24,7 +26,36 @@ $ go build ./cmd/dendrite-monolith-server
 $ ./dendrite-monolith-server --tls-cert server.crt --tls-key server.key --config dendrite.yaml
 ```
 
-For full installation information, see [INSTALL.md](docs/INSTALL.md).
+Then point your favourite Matrix client at `http://localhost:8008`. For full installation information, see
+[INSTALL.md](docs/INSTALL.md). For running in Docker, see [build/docker](build/docker).
+
+# Progress
+
+We use a script called Are We Synapse Yet which checks Sytest compliance rates. Sytest is a black-box homeserver
+test rig with around 900 tests. The script works out how many of these tests are passing on Dendrite and it
+updates with CI. As of July 2020 we're at around 48% CS API coverage and 50% Federation coverage, though check
+CI for the latest numbers. In practice, this means you can communicate locally and via federation with Synapse
+servers such as matrix.org reasonably well. There's a long list of features that are not implemented, notably:
+ - Receipts
+ - Push
+ - Search and Context
+ - User Directory
+ - Presence
+ - Guests
+ - E2E keys and device lists
+
+We are prioritising features that will benefit single-user homeservers first (e.g Receipts, E2E) rather
+than features that massive deployments may be interested in (User Directory, OpenID, Guests, Admin APIs, AS API).
+This means Dendrite supports amongst others:
+ - Core room functionality (creating rooms, invites, auth rules)
+ - Federation in rooms v1-v6
+ - Backfilling locally and via federation
+ - Accounts, Profiles and Devices
+ - Published room lists
+ - Typing
+ - Media APIs
+ - Redaction
+ - Tagging
 
 
 # Contributing
@@ -32,10 +63,6 @@ For full installation information, see [INSTALL.md](docs/INSTALL.md).
 Everyone is welcome to help out and contribute! See
 [CONTRIBUTING.md](docs/CONTRIBUTING.md) to get started!
 
-Please note that, as of February 2020, Dendrite now only targets Go 1.13 or
-later. Please ensure that you are using at least Go 1.13 when developing for
-Dendrite.
-
 # Discussion
 
 For questions about Dendrite we have a dedicated room on Matrix
@@ -43,7 +70,4 @@ For questions about Dendrite we have a dedicated room on Matrix
 discussion should happen in
 [#dendrite-dev:matrix.org](https://matrix.to/#/#dendrite-dev:matrix.org).
 
-# Progress
-
-We use a script called Are We Synapse Yet which checks Sytest compliance rates. Sytest is a black-box homeserver test rig with around 900 tests. The script works out how many of these tests are passing on Dendrite and it updates with CI. As of July 2020 we're at around 46% CS API coverage and 50% Federation coverage, though check CI for the latest numbers.
 

docs/CODE_STYLE.md

@@ -1,12 +1,7 @@
 # Code Style
 
-We follow the standard Go style using goimports, but with a few extra
+In addition to standard Go code style (`gofmt`, `goimports`), we use `golangci-lint`
-considerations.
+to run a number of linters, the exact list can be found under linters in [.golangci.yml](.golangci.yml).
-
-## Linters
-
-We use `golangci-lint` to run a number of linters, the exact list can be found
-under linters in [.golangci.yml](.golangci.yml).
 [Installation](https://github.com/golangci/golangci-lint#install) and [Editor
 Integration](https://github.com/golangci/golangci-lint#editor-integration) for
 it can be found in the readme of golangci-lint.
@@ -22,37 +17,10 @@ The linters can be run using [scripts/find-lint.sh](scripts/find-lint.sh)
 [scripts/build-test-lint.sh](scripts/build-test-lint.sh).
 
 
-## HTTP Error Handling
+## Labels
-
-Unfortunately, converting errors into HTTP responses with the correct status
-code and message can be done in a number of ways in golang:
-
-1. Having functions return `JSONResponse` directly, which can then either set
-   it to an error response or a `200 OK`.
-2. Have the HTTP handler try and cast error values to types that are handled
-   differently.
-3. Have the HTTP handler call functions whose errors can only be interpreted
-   one way, for example if a `validate(...)` call returns an error then handler
-   knows to respond with a `400 Bad Request`.
-
-We attempt to always use option #3, as it more naturally fits with the way that
-golang generally does error handling. In particular, option #1 effectively
-requires reinventing a new error handling scheme just for HTTP handlers.
-
-
-## Line length
-
-We strive for a line length of roughly 80 characters, though less than 100 is
-acceptable if necessary. Longer lines are fine if there is nothing of interest
-after the first 80-100 characters (e.g. long string literals).
-
-
-## TODOs and FIXMEs
-
-The majority of TODOs and FIXMEs should have an associated tracking issue on
-github. These can be added just before merging of the PR to master, and the
-issue number should be added to the comment, e.g. `// TODO(#324): ...`
 
+In addition to `TODO` and `FIXME` we also use `NOTSPEC` to identify deviations
+from the Matrix specification.
 
 ## Logging
 

docs/INSTALL.md

@@ -3,20 +3,21 @@
 Dendrite can be run in one of two configurations:
 
 * **Polylith mode**: A cluster of individual components, dealing with different
-  aspects of the Matrix protocol (see [WIRING.md](WIRING.md)). Components communicate with each other using internal HTTP APIs and [Apache Kafka](https://kafka.apache.org). This will almost certainly be the preferred model
+  aspects of the Matrix protocol (see [WIRING.md](WIRING-Current.md)). Components communicate
-  for large-scale deployments.
+  with each other using internal HTTP APIs and [Apache Kafka](https://kafka.apache.org).
+  This will almost certainly be the preferred model for large-scale deployments.
 
 * **Monolith mode**: All components run in the same process. In this mode,
    Kafka is completely optional and can instead be replaced with an in-process
-   lightweight implementation called [Naffka](https://github.com/matrix-org/naffka). This will usually be the preferred model for low-volume, low-user
+   lightweight implementation called [Naffka](https://github.com/matrix-org/naffka). This
-   or experimental deployments.
+   will usually be the preferred model for low-volume, low-user or experimental deployments.
 
-Regardless of whether you are running in polylith or monolith mode, each Dendrite component that requires storage has its own database. Both Postgres
+Regardless of whether you are running in polylith or monolith mode, each Dendrite component that
-and SQLite are supported and can be mixed-and-matched across components as
+requires storage has its own database. Both Postgres and SQLite are supported and can be
-needed in the configuration file.
+mixed-and-matched across components as needed in the configuration file.
 
-Be advised that Dendrite is still developmental and it's not recommended for
+Be advised that Dendrite is still in development and it's not recommended for
-use in production environments yet!
+use in production environments just yet!
 
 ## Requirements
 
@@ -119,16 +120,10 @@ Assuming that Postgres 9.5 (or later) is installed:
 
 Each Dendrite server requires unique server keys.
 
-Generate the self-signed SSL certificate for federation:
+Generate the self-signed SSL certificate for federation and the server signing key:
 
 ```bash
-test -f server.key || openssl req -x509 -newkey rsa:4096 -keyout server.key -out server.crt -days 3650 -nodes -subj /CN=localhost
+./bin/generate-keys --private-key matrix_key.pem --tls-cert server.crt --tls-key server.key
-```
-
-Generate the server signing key:
-
-```
-test -f matrix_key.pem || ./bin/generate-keys -private-key matrix_key.pem
 ```
 
 ### Configuration file
@@ -152,7 +147,8 @@ public keys for dead homeservers from somewhere else.
 ## Starting a monolith server
 
 It is possible to use Naffka as an in-process replacement to Kafka when using
-the monolith server. To do this, set `use_naffka: true` in your `dendrite.yaml` configuration and uncomment the relevant Naffka line in the `database` section.
+the monolith server. To do this, set `use_naffka: true` in your `dendrite.yaml`
+configuration and uncomment the relevant Naffka line in the `database` section.
 Be sure to update the database username and password if needed.
 
 The monolith server can be started as shown below. By default it listens for
@@ -166,60 +162,7 @@ as shown below, it will also listen for HTTPS connections on port 8448.
 
 ## Starting a polylith deployment
 
-The following contains scripts which will run all the required processes in order to point a Matrix client at Dendrite. Conceptually, you are wiring together to form the following diagram:
+The following contains scripts which will run all the required processes in order to point a Matrix client at Dendrite.
-
-```
-
-                                         /media   +---------------------------+
-                      +----------->+------------->| dendrite-media-api-server |
-                      ^            ^              +---------------------------+
-                      |            |            :7774
-                      |            |
-                      |            |
-                      |            |   
-                      |            |   
-                      |            |   
-                      |            |  
-                      |            |  
-                      |            |  
-                      |            |     /sync    +--------------------------+                 
-                      |            |   +--------->| dendrite-sync-api-server |<================++
-                      |            |   |          +--------------------------+                 ||
-                      |            |   |        :7773    |         ^^                          ||
-Matrix      +------------------+   |   |                 |         ||    client_data           ||
-Clients --->| client-api-proxy |-------+    +<-----------+         ++=============++           ||
-            +------------------+   |   |    |                                     ||           ||
-          :8008                    |   | CS API   +----------------------------+  ||           ||
-                                   |   +--------->| dendrite-client-api-server |==++           ||
-                                   |        |     +----------------------------+               ||
-                                   |        |   :7771    |                                     ||
-                                   |        |            |                                     ||
-                                   |        +<-----------+                                     ||
-                                   |        |                                                  ||
-                                   |        |                                                  ||
-                                   |        |           +----------------------+    room_event ||
-                                   |        +---------->| dendrite-room-server |===============++
-                                   |        |           +----------------------+               ||
-                                   |        |         :7770                                    ||
-                                   |        |                      ++==========================++
-                                   |        +<------------+        ||
-                                   |        |             |        VV
-                                   |        |     +-----------------------------------+              Matrix
-                                   |        |     | dendrite-federation-sender-server |------------> Servers
-                                   |        |     +-----------------------------------+
-                                   |        |   :7776
-                                   |        |
-                       +---------->+        +<-----------+
-                       |                                 |
-Matrix      +----------------------+  SS API  +--------------------------------+
-Servers --->| federation-api-proxy |--------->| dendrite-federation-api-server |
-            +----------------------+          +--------------------------------+
-          :8448                             :7772
-
-
-   A --> B  = HTTP requests (A = client, B = server)
-   A ==> B  = Kafka (A = producer, B = consumer)
-```
 
 ### Client proxy
 
@@ -248,21 +191,13 @@ to support federation.
 
 ### Client API server
 
-This is what implements message sending. Clients talk to this via the proxy in
+This is what implements CS API endpoints. Clients talk to this via the proxy in
-order to send messages.
+order to send messages, create and join rooms, etc.
 
 ```bash
 ./bin/dendrite-client-api-server --config=dendrite.yaml
 ```
 
-### Room server
-
-This is what implements the room DAG. Clients do not talk to this.
-
-```bash
-./bin/dendrite-room-server --config=dendrite.yaml
-```
-
 ### Sync server
 
 This is what implements `/sync` requests. Clients talk to this via the proxy
@@ -283,7 +218,7 @@ order to upload and retrieve media.
 
 ### Federation API server
 
-This implements federation requests. Servers talk to this via the proxy in
+This implements the federation API. Servers talk to this via the proxy in
 order to send transactions.  This is only required if you want to support
 federation.
 
@@ -291,7 +226,30 @@ federation.
 ./bin/dendrite-federation-api-server --config dendrite.yaml
 ```
 
-### Federation sender
+### Internal components
+
+This refers to components that are not directly spoken to by clients. They are only
+contacted by other components. This includes the following components.
+
+#### Room server
+
+This is what implements the room DAG. Clients do not talk to this.
+
+```bash
+./bin/dendrite-room-server --config=dendrite.yaml
+```
+
+#### Current state server
+
+This tracks the current state of rooms which various components need to know. For example,
+`/publicRooms` implemented by client API asks this server for the room names, joined member
+counts, etc.
+
+```bash
+./bin/dendrite-current-state-server --config=dendrite.yaml
+```
+
+#### Federation sender
 
 This sends events from our users to other servers.  This is only required if
 you want to support federation.
@@ -300,7 +258,7 @@ you want to support federation.
 ./bin/dendrite-federation-sender-server --config dendrite.yaml
 ```
 
-### Appservice server
+#### Appservice server
 
 This sends events from the network to [application
 services](https://matrix.org/docs/spec/application_service/unstable.html)
@@ -311,16 +269,31 @@ application services on your homeserver.
 ./bin/dendrite-appservice-server --config dendrite.yaml
 ```
 
-### Key server
+#### Key server
 
-This manages end-to-end encryption keys (or rather, it will do when it's
+This manages end-to-end encryption keys for users.
-finished).
 
 ```bash
 ./bin/dendrite-key-server --config dendrite.yaml
 ```
 
-### User server
+#### Server Key server
+
+This manages signing keys for servers.
+
+```bash
+./bin/dendrite-server-key-api-server --config dendrite.yaml
+```
+
+#### EDU server
+
+This manages processing EDUs such as typing, send-to-device events and presence. Clients do not talk to
+
+```bash
+./bin/dendrite-edu-server --config dendrite.yaml
+```
+
+#### User server
 
 This manages user accounts, device access tokens and user account data,
 amongst other things.