Merge branch 'master' into master

.travis.yml

@@ -24,8 +24,8 @@ before_script:
  - openssl req -x509 -newkey rsa:4096 -keyout server.key -out server.crt -days 365 -nodes -subj /CN=localhost
 
 script:
- - ./travis-install-kafka.sh
+ - ./scripts/install-local-kafka.sh
- - ./travis-test.sh
+ - ./scripts/travis-test.sh
 
 notifications:
     webhooks:

DESIGN.md

@@ -0,0 +1,140 @@
+# Design
+
+## Log Based Architecture
+
+### Decomposition and Decoupling
+
+A matrix homeserver can be built around append-only event logs built from the
+messages, receipts, presence, typing notifications, device messages and other
+events sent by users on the homeservers or by other homeservers.
+
+The server would then decompose into two categories: writers that add new
+entries to the logs and readers that read those entries.
+
+The event logs then serve to decouple the two components, the writers and
+readers need only agree on the format of the entries in the event log.
+This format could be largely derived from the wire format of the events used
+in the client and federation protocols:
+
+
+     C-S API   +---------+    Event Log    +---------+   C-S API
+    ---------> |         |+  (e.g. kafka)  |         |+ --------->
+               | Writers || =============> | Readers ||
+    ---------> |         ||                |         || --------->
+     S-S API   +---------+|                +---------+|   S-S API
+                +---------+                 +---------+
+
+However the way matrix handles state events in a room creates a few
+complications for this model.
+
+ 1) Writers require the room state at an event to check if it is allowed.
+ 2) Readers require the room state at an event to determine the users and
+    servers that are allowed to see the event.
+ 3) A client can query the current state of the room from a reader.
+
+The writers and readers cannot extract the necessary information directly from
+the event logs because it would take too long to extract the information as the
+state is built up by collecting individual state events from the event history.
+
+The writers and readers therefore need access to something that stores copies
+of the event state in a form that can be efficiently queried. One possibility
+would be for the readers and writers to maintain copies of the current state
+in local databases. A second possibility would be to add a dedicated component
+that maintained the state of the room and exposed an API that the readers and
+writers could query to get the state. The second has the advantage that the
+state is calculated and stored in a single location.
+
+
+     C-S API   +---------+    Log   +--------+   Log   +---------+   C-S API
+    ---------> |         |+ ======> |        | ======> |         |+ --------->
+               | Writers ||         |  Room  |         | Readers ||
+    ---------> |         || <------ | Server | ------> |         || --------->
+     S-S API   +---------+|  Query  |        |  Query  +---------+|  S-S API
+                +---------+         +--------+          +---------+
+
+
+The room server can annotate the events it logs to the readers with room state
+so that the readers can avoid querying the room server unnecessarily.
+
+[This architecture can be extended to cover most of the APIs.](WIRING.md)
+
+## How things are supposed to work.
+
+### Local client sends an event in an existing room.
+
+  0) The client sends a PUT `/_matrix/client/r0/rooms/{roomId}/send` request
+    and an HTTP loadbalancer routes the request to a ClientAPI.
+
+  1) The ClientAPI:
+
+    * Authenticates the local user using the `access_token` sent in the HTTP
+      request.
+    * Checks if it has already processed or is processing a request with the
+      same `txnID`.
+    * Calculates which state events are needed to auth the request.
+    * Queries the necessary state events and the latest events in the room
+      from the RoomServer.
+    * Confirms that the room exists and checks whether the event is allowed by
+      the auth checks.
+    * Builds and signs the events.
+    * Writes the event to a "InputRoomEvent" kafka topic.
+    * Send a `200 OK` response to the client.
+
+  2) The RoomServer reads the event from "InputRoomEvent" kafka topic:
+
+    * Checks if it has already has a copy of the event.
+    * Checks if the event is allowed by the auth checks using the auth events
+      at the event.
+    * Calculates the room state at the event.
+    * Works out what the latest events in the room after processing this event
+      are.
+    * Calculate how the changes in the latest events affect the current state
+      of the room.
+    * TODO: Workout what events determine the visibility of this event to other
+      users
+    * Writes the event along with the changes in current state to an
+      "OutputRoomEvent" kafka topic. It writes all the events for a room to
+      the same kafka partition.
+
+  3a) The ClientSync reads the event from the "OutputRoomEvent" kafka topic:
+
+    * Updates its copy of the current state for the room.
+    * Works out which users need to be notified about the event.
+    * Wakes up any pending `/_matrix/client/r0/sync` requests for those users.
+    * Adds the event to the recent timeline events for the room.
+
+  3b) The FederationSender reads the event from the "OutputRoomEvent" kafka topic:
+
+    * Updates its copy of the current state for the room.
+    * Works out which remote servers need to be notified about the event.
+    * Sends a `/_matrix/federation/v1/send` request to those servers.
+    * Or if there is a request in progress then add the event to a queue to be
+      sent when the previous request finishes.
+
+### Remote server sends an event in an existing room.
+
+  0) The remote server sends a `PUT /_matrix/federation/v1/send` request and an
+    HTTP loadbalancer routes the request to a FederationReceiver.
+
+  1) The FederationReceiver:
+
+    * Authenticates the remote server using the "X-Matrix" authorisation header.
+    * Checks if it has already processed or is processing a request with the
+      same `txnID`.
+    * Checks the signatures for the events.
+      Fetches the ed25519 keys for the event senders if necessary.
+    * Queries the RoomServer for a copy of the state of the room at each event.
+    * If the RoomServer doesn't know the state of the room at an event then
+      query the state of the room at the event from the remote server using
+      `GET /_matrix/federation/v1/state_ids` falling back to
+      `GET /_matrix/federation/v1/state` if necessary.
+    * Once the state at each event is known check whether the events are
+      allowed by the auth checks against the state at each event.
+    * For each event that is allowed write the event to the "InputRoomEvent"
+      kafka topic.
+    * Send a 200 OK response to the remote server listing which events were
+      successfully processed and which events failed
+
+  2) The RoomServer processes the event the same as it would a local event.
+
+  3a) The ClientSync processes the event the same as it would a local event.

INSTALL.md

@@ -17,7 +17,7 @@ Dendrite can be run in one of two configurations:
  - For Kafka (optional if using the monolith server):
    - Unix-based system (https://kafka.apache.org/documentation/#os)
    - JDK 1.8+ / OpenJDK 1.8+
-   - Apache Kafka 0.10.2+ (see [travis-install-kafka.sh](travis-install-kafka.sh) for up-to-date version numbers)
+   - Apache Kafka 0.10.2+ (see [scripts/install-local-kafka.sh](scripts/install-local-kafka.sh) for up-to-date version numbers)
 
 
 ## Setting up a development environment
@@ -34,7 +34,7 @@ go get github.com/constabulary/gb/...
 gb build
 ```
 
-If using Kafka, install and start it (c.f. [travis-install-kafka.sh](travis-install-kafka.sh)):
+If using Kafka, install and start it (c.f. [scripts/install-local-kafka.sh](scripts/install-local-kafka.sh)):
 ```bash
 MIRROR=http://apache.mirror.anlx.net/kafka/0.10.2.0/kafka_2.11-0.10.2.0.tgz
 

README.md

@@ -2,153 +2,24 @@
 
 Dendrite will be a matrix homeserver written in go.
 
-# Install
+It's still very much a work in progress, but installation instructions can
+be found in [INSTALL.md](INSTALL.md)
 
-Dendrite is still very much a work in progress, but those wishing to work on it
+An overview of the design can be found in [DESIGN.md](DESIGN.md)
-may be interested in the installation instructions in [INSTALL.md](INSTALL.md).
 
-# Design
+# Contributing
 
-## Log Based Architecture
+Everyone is welcome to help out and contribute! See [CONTRIBUTING.md](CONTRIBUTING.md)
+to get started!
 
-### Decomposition and Decoupling
+We aim to try and make it as easy as possible to jump in.
 
-A matrix homeserver can be built around append-only event logs built from the
+# Discussion
-messages, receipts, presence, typing notifications, device messages and other
-events sent by users on the homeservers or by other homeservers.
 
-The server would then decompose into two categories: writers that add new
+For questions about Dendrite we have a dedicated room on Matrix
-entries to the logs and readers that read those entries.
+[#dendrite:matrix.org](https://riot.im/develop/#/room/#dendrite:matrix.org).
 
-The event logs then serve to decouple the two components, the writers and
+# Progress
-readers need only agree on the format of the entries in the event log.
-This format could be largely derived from the wire format of the events used
-in the client and federation protocols:
-
-
-     C-S API   +---------+    Event Log    +---------+   C-S API
-    ---------> |         |+  (e.g. kafka)  |         |+ --------->
-               | Writers || =============> | Readers ||
-    ---------> |         ||                |         || --------->
-     S-S API   +---------+|                +---------+|   S-S API
-                +---------+                 +---------+
-
-However the way matrix handles state events in a room creates a few
-complications for this model.
-
- 1) Writers require the room state at an event to check if it is allowed.
- 2) Readers require the room state at an event to determine the users and
-    servers that are allowed to see the event.
- 3) A client can query the current state of the room from a reader.
-
-The writers and readers cannot extract the necessary information directly from
-the event logs because it would take too long to extract the information as the
-state is built up by collecting individual state events from the event history.
-
-The writers and readers therefore need access to something that stores copies
-of the event state in a form that can be efficiently queried. One possibility
-would be for the readers and writers to maintain copies of the current state
-in local databases. A second possibility would be to add a dedicated component
-that maintained the state of the room and exposed an API that the readers and
-writers could query to get the state. The second has the advantage that the
-state is calculated and stored in a single location.
-
-
-     C-S API   +---------+    Log   +--------+   Log   +---------+   C-S API
-    ---------> |         |+ ======> |        | ======> |         |+ --------->
-               | Writers ||         |  Room  |         | Readers ||
-    ---------> |         || <------ | Server | ------> |         || --------->
-     S-S API   +---------+|  Query  |        |  Query  +---------+|  S-S API
-                +---------+         +--------+          +---------+
-
-
-The room server can annotate the events it logs to the readers with room state
-so that the readers can avoid querying the room server unnecessarily.
-
-[This architecture can be extended to cover most of the APIs.](WIRING.md)
-
-## How things are supposed to work.
-
-### Local client sends an event in an existing room.
-
-  0) The client sends a PUT `/_matrix/client/r0/rooms/{roomId}/send` request
-    and an HTTP loadbalancer routes the request to a ClientAPI.
-
-  1) The ClientAPI:
-
-    * Authenticates the local user using the `access_token` sent in the HTTP
-      request.
-    * Checks if it has already processed or is processing a request with the
-      same `txnID`.
-    * Calculates which state events are needed to auth the request.
-    * Queries the necessary state events and the latest events in the room
-      from the RoomServer.
-    * Confirms that the room exists and checks whether the event is allowed by
-      the auth checks.
-    * Builds and signs the events.
-    * Writes the event to a "InputRoomEvent" kafka topic.
-    * Send a `200 OK` response to the client.
-
-  2) The RoomServer reads the event from "InputRoomEvent" kafka topic:
-
-    * Checks if it has already has a copy of the event.
-    * Checks if the event is allowed by the auth checks using the auth events
-      at the event.
-    * Calculates the room state at the event.
-    * Works out what the latest events in the room after processing this event
-      are.
-    * Calculate how the changes in the latest events affect the current state
-      of the room.
-    * TODO: Workout what events determine the visibility of this event to other
-      users
-    * Writes the event along with the changes in current state to an
-      "OutputRoomEvent" kafka topic. It writes all the events for a room to
-      the same kafka partition.
-
-  3a) The ClientSync reads the event from the "OutputRoomEvent" kafka topic:
-
-    * Updates its copy of the current state for the room.
-    * Works out which users need to be notified about the event.
-    * Wakes up any pending `/_matrix/client/r0/sync` requests for those users.
-    * Adds the event to the recent timeline events for the room.
-
-  3b) The FederationSender reads the event from the "OutputRoomEvent" kafka topic:
-
-    * Updates its copy of the current state for the room.
-    * Works out which remote servers need to be notified about the event.
-    * Sends a `/_matrix/federation/v1/send` request to those servers.
-    * Or if there is a request in progress then add the event to a queue to be
-      sent when the previous request finishes.
-
-### Remote server sends an event in an existing room.
-
-  0) The remote server sends a `PUT /_matrix/federation/v1/send` request and an
-    HTTP loadbalancer routes the request to a FederationReceiver.
-
-  1) The FederationReceiver:
-
-    * Authenticates the remote server using the "X-Matrix" authorisation header.
-    * Checks if it has already processed or is processing a request with the
-      same `txnID`.
-    * Checks the signatures for the events.
-      Fetches the ed25519 keys for the event senders if necessary.
-    * Queries the RoomServer for a copy of the state of the room at each event.
-    * If the RoomServer doesn't know the state of the room at an event then
-      query the state of the room at the event from the remote server using
-      `GET /_matrix/federation/v1/state_ids` falling back to
-      `GET /_matrix/federation/v1/state` if necessary.
-    * Once the state at each event is known check whether the events are
-      allowed by the auth checks against the state at each event.
-    * For each event that is allowed write the event to the "InputRoomEvent"
-      kafka topic.
-    * Send a 200 OK response to the remote server listing which events were
-      successfully processed and which events failed
-
-  2) The RoomServer processes the event the same as it would a local event.
-
-  3a) The ClientSync processes the event the same as it would a local event.
-
-# TODO
 
 There's plenty still to do to make Dendrite usable! We're tracking progress in
 a [spreadsheet](https://docs.google.com/spreadsheets/d/1tkMNpIpPjvuDJWjPFbw_xzNzOHBA-Hp50Rkpcr43xTw).

hooks/pre-commit

@@ -2,33 +2,4 @@
 
 set -eu
 
-# Tune the GC to use more memory to reduce the number of garbage collections
+./scripts/build-test-lint.sh
-export GOGC=400
-export GOPATH="$(pwd):$(pwd)/vendor"
-export PATH="$PATH:$(pwd)/vendor/bin:$(pwd)/bin"
-
-echo "Checking that it builds"
-gb build
-
-# Check that all the packages can build.
-# When `go build` is given multiple packages it won't output anything, and just
-# checks that everything builds. This seems to do a better job of handling
-# missing imports than `gb build` does.
-echo "Double checking it builds..."
-go build github.com/matrix-org/dendrite/cmd/...
-
-echo "Installing lint search engine..."
-go install github.com/alecthomas/gometalinter/
-gometalinter --config=linter.json ./... --install
-
-echo "Looking for lint..."
-gometalinter --config=linter.json ./... --enable-gc
-
-echo "Double checking spelling..."
-misspell -error src *.md
-
-echo "Testing..."
-gb test
-
-
-echo "Done!"

scripts/README.md

@@ -0,0 +1,7 @@
+# Dev Scripts
+
+These are a collection of scripts that should be helpful for those developing
+on dendrite.
+
+See `find-lint.sh` for environment variables that control linter resource
+usage.

scripts/build-test-lint.sh

@@ -0,0 +1,26 @@
+#! /bin/bash
+
+# Builds, tests and lints dendrite, and should be run before pushing commits
+
+set -eu
+
+export GOPATH="$(pwd):$(pwd)/vendor"
+export PATH="$PATH:$(pwd)/vendor/bin:$(pwd)/bin"
+
+echo "Checking that it builds"
+gb build
+
+# Check that all the packages can build.
+# When `go build` is given multiple packages it won't output anything, and just
+# checks that everything builds. This seems to do a better job of handling
+# missing imports than `gb build` does.
+echo "Double checking it builds..."
+go build github.com/matrix-org/dendrite/cmd/...
+
+./scripts/find-lint.sh
+
+echo "Double checking spelling..."
+misspell -error src *.md
+
+echo "Testing..."
+gb test

scripts/find-lint.sh

@@ -0,0 +1,41 @@
+#! /bin/bash
+
+# Runs the linters against dendrite
+
+# The linters can take a lot of resources and are slow, so they can be
+# configured using two environment variables:
+#
+# - `DENDRITE_LINT_CONCURRENCY` - number of concurrent linters to run,
+#   gometalinter defaults this to 8
+# - `DENDRITE_LINT_DISABLE_GC` - if set then the the go gc will be disabled
+#   when running the linters, speeding them up but using much more memory.
+
+
+set -eu
+
+export GOPATH="$(pwd):$(pwd)/vendor"
+export PATH="$PATH:$(pwd)/vendor/bin:$(pwd)/bin"
+
+args=""
+if [ ${1:-""} = "fast" ]
+then args="--config=linter-fast.json"
+else args="--config=linter.json"
+fi
+
+if [ -n "${DENDRITE_LINT_CONCURRENCY:-}" ]
+then args="$args --concurrency=$DENDRITE_LINT_CONCURRENCY"
+fi
+
+if [ -z "${DENDRITE_LINT_DISABLE_GC:-}" ]
+then args="$args --enable-gc"
+fi
+
+echo "Installing lint search engine..."
+go install github.com/alecthomas/gometalinter/
+gometalinter --config=linter.json ./... --install
+
+echo "Looking for lint..."
+gometalinter ./... $args
+
+echo "Double checking spelling..."
+misspell -error src *.md

scripts/install-local-kafka.sh

@@ -1,5 +1,7 @@
 # /bin/bash
 
+# Downloads, installs and runs a kafka instance
+
 set -eu
 
 # The mirror to download kafka from is picked from the list of mirrors at

scripts/travis-test.sh

@@ -1,7 +1,13 @@
 #! /bin/bash
 
+# The entry point for travis tests
+
 set -eu
 
+# Tune the GC to use more memory to reduce the number of garbage collections
+export GOGC=400
+export DENDRITE_LINT_DISABLE_GC=1
+
 # Check that the servers build (this is done explicitly because `gb build` can silently fail (exit 0) and then we'd test a stale binary)
 gb build github.com/matrix-org/dendrite/cmd/dendrite-room-server
 gb build github.com/matrix-org/dendrite/cmd/roomserver-integration-tests
@@ -12,8 +18,8 @@ gb build github.com/matrix-org/dendrite/cmd/dendrite-media-api-server
 gb build github.com/matrix-org/dendrite/cmd/mediaapi-integration-tests
 gb build github.com/matrix-org/dendrite/cmd/client-api-proxy
 
-# Run the pre commit hooks
+# Run unit tests and linters
-./hooks/pre-commit
+./scripts/build-test-lint.sh
 
 # Run the integration tests
 bin/roomserver-integration-tests

src/github.com/matrix-org/dendrite/clientapi/auth/auth.go

@@ -29,17 +29,13 @@ import (
 	"github.com/matrix-org/util"
 )
 
-// UnknownDeviceID is the default device id if one is not specified.
-// This deviates from Synapse which generates a new device ID if one is not specified.
-// It's preferable to not amass a huge list of valid access tokens for an account,
-// so limiting it to 1 unknown device for now limits the number of valid tokens.
-// Clients should be giving us device IDs.
-var UnknownDeviceID = "unknown-device"
-
 // OWASP recommends at least 128 bits of entropy for tokens: https://www.owasp.org/index.php/Insufficient_Session-ID_Length
 // 32 bytes => 256 bits
 var tokenByteLength = 32
 
+// The length of generated device IDs
+var deviceIDByteLength = 6
+
 // DeviceDatabase represents a device database.
 type DeviceDatabase interface {
 	// Look up the device matching the given access token.
@@ -62,8 +58,8 @@ func VerifyAccessToken(req *http.Request, deviceDB DeviceDatabase) (device *auth
 	if err != nil {
 		if err == sql.ErrNoRows {
 			resErr = &util.JSONResponse{
-				Code: 403,
+				Code: 401,
-				JSON: jsonerror.Forbidden("Invalid access token"),
+				JSON: jsonerror.UnknownToken("Unknown token"),
 			}
 		} else {
 			resErr = &util.JSONResponse{
@@ -86,6 +82,18 @@ func GenerateAccessToken() (string, error) {
 	return base64.RawURLEncoding.EncodeToString(b), nil
 }
 
+// GenerateDeviceID creates a new device id. Returns an error if failed to generate
+// random bytes.
+func GenerateDeviceID() (string, error) {
+	b := make([]byte, deviceIDByteLength)
+	_, err := rand.Read(b)
+	if err != nil {
+		return "", err
+	}
+	// url-safe no padding
+	return base64.RawURLEncoding.EncodeToString(b), nil
+}
+
 // extractAccessToken from a request, or return an error detailing what went wrong. The
 // error message MUST be human-readable and comprehensible to the client.
 func extractAccessToken(req *http.Request) (string, error) {

src/github.com/matrix-org/dendrite/clientapi/auth/storage/accounts/storage.go

@@ -338,4 +338,13 @@ func (d *Database) PutFilter(
 		return "", err
 	}
 	return d.filter.findMaxID(ctx, localpart)
+
+// CheckAccountAvailability checks if the username/localpart is already present in the database.
+// If the DB returns sql.ErrNoRows the Localpart isn't taken.
+func (d *Database) CheckAccountAvailability(ctx context.Context, localpart string) (bool, error) {
+	_, err := d.accounts.selectAccountByLocalpart(ctx, localpart)
+	if err == sql.ErrNoRows {
+		return true, nil
 	}
+	return false, err
+

src/github.com/matrix-org/dendrite/clientapi/auth/storage/devices/storage.go

@@ -18,6 +18,7 @@ import (
 	"context"
 	"database/sql"
 
+	"github.com/matrix-org/dendrite/clientapi/auth"
 	"github.com/matrix-org/dendrite/clientapi/auth/authtypes"
 	"github.com/matrix-org/dendrite/common"
 	"github.com/matrix-org/gomatrixserverlib"
@@ -55,20 +56,42 @@ func (d *Database) GetDeviceByAccessToken(
 // If there is already a device with the same device ID for this user, that access token will be revoked
 // and replaced with the given accessToken. If the given accessToken is already in use for another device,
 // an error will be returned.
+// If no device ID is given one is generated.
 // Returns the device on success.
 func (d *Database) CreateDevice(
-	ctx context.Context, localpart, deviceID, accessToken string,
+	ctx context.Context, localpart string, deviceID *string, accessToken string,
 ) (dev *authtypes.Device, returnErr error) {
+	if deviceID != nil {
 		returnErr = common.WithTransaction(d.db, func(txn *sql.Tx) error {
 			var err error
 			// Revoke existing token for this device
-		if err = d.devices.deleteDevice(ctx, txn, deviceID, localpart); err != nil {
+			if err = d.devices.deleteDevice(ctx, txn, *deviceID, localpart); err != nil {
 				return err
 			}
 
-		dev, err = d.devices.insertDevice(ctx, txn, deviceID, localpart, accessToken)
+			dev, err = d.devices.insertDevice(ctx, txn, *deviceID, localpart, accessToken)
 			return err
 		})
+	} else {
+		// We generate device IDs in a loop in case its already taken.
+		// We cap this at going round 5 times to ensure we don't spin forever
+		var newDeviceID string
+		for i := 1; i <= 5; i++ {
+			newDeviceID, returnErr = auth.GenerateDeviceID()
+			if returnErr != nil {
+				return
+			}
+
+			returnErr = common.WithTransaction(d.db, func(txn *sql.Tx) error {
+				var err error
+				dev, err = d.devices.insertDevice(ctx, txn, newDeviceID, localpart, accessToken)
+				return err
+			})
+			if returnErr == nil {
+				return
+			}
+		}
+	}
 	return
 }
 

src/github.com/matrix-org/dendrite/clientapi/readers/login.go

@@ -46,6 +46,7 @@ type loginResponse struct {
 	UserID      string                       `json:"user_id"`
 	AccessToken string                       `json:"access_token"`
 	HomeServer  gomatrixserverlib.ServerName `json:"home_server"`
+	DeviceID    string                       `json:"device_id"`
 }
 
 func passwordLogin() loginFlows {
@@ -113,15 +114,12 @@ func Login(
 
 		token, err := auth.GenerateAccessToken()
 		if err != nil {
-			return util.JSONResponse{
+			httputil.LogThenError(req, err)
-				Code: 500,
-				JSON: jsonerror.Unknown("Failed to generate access token"),
-			}
 		}
 
 		// TODO: Use the device ID in the request
 		dev, err := deviceDB.CreateDevice(
-			req.Context(), acc.Localpart, auth.UnknownDeviceID, token,
+			req.Context(), acc.Localpart, nil, token,
 		)
 		if err != nil {
 			return util.JSONResponse{
@@ -136,6 +134,7 @@ func Login(
 				UserID:      dev.UserID,
 				AccessToken: dev.AccessToken,
 				HomeServer:  cfg.Matrix.ServerName,
+				DeviceID:    dev.ID,
 			},
 		}
 	}

src/github.com/matrix-org/dendrite/clientapi/routing/routing.go

@@ -131,6 +131,10 @@ func Setup(
 		return writers.LegacyRegister(req, accountDB, deviceDB, &cfg)
 	})).Methods("POST", "OPTIONS")
 
+	r0mux.Handle("/register/available", common.MakeExternalAPI("registerAvailable", func(req *http.Request) util.JSONResponse {
+		return writers.RegisterAvailable(req, accountDB)
+	})).Methods("GET")
+
 	r0mux.Handle("/directory/room/{roomAlias}",
 		common.MakeAuthAPI("directory_room", deviceDB, func(req *http.Request, device *authtypes.Device) util.JSONResponse {
 			vars := mux.Vars(req)

src/github.com/matrix-org/dendrite/clientapi/writers/register.go

@@ -91,24 +91,14 @@ type registerResponse struct {
 	DeviceID    string                       `json:"device_id"`
 }
 
-// Validate returns an error response if the username/password are invalid
+// validateUserName returns an error response if the username is invalid
-func validate(username, password string) *util.JSONResponse {
+func validateUserName(username string) *util.JSONResponse {
 	// https://github.com/matrix-org/synapse/blob/v0.20.0/synapse/rest/client/v2_alpha/register.py#L161
-	if len(password) > maxPasswordLength {
+	if len(username) > maxUsernameLength {
-		return &util.JSONResponse{
-			Code: 400,
-			JSON: jsonerror.BadJSON(fmt.Sprintf("'password' >%d characters", maxPasswordLength)),
-		}
-	} else if len(username) > maxUsernameLength {
 		return &util.JSONResponse{
 			Code: 400,
 			JSON: jsonerror.BadJSON(fmt.Sprintf("'username' >%d characters", maxUsernameLength)),
 		}
-	} else if len(password) > 0 && len(password) < minPasswordLength {
-		return &util.JSONResponse{
-			Code: 400,
-			JSON: jsonerror.WeakPassword(fmt.Sprintf("password too weak: min %d chars", minPasswordLength)),
-		}
 	} else if !validUsernameRegex.MatchString(username) {
 		return &util.JSONResponse{
 			Code: 400,
@@ -123,6 +113,23 @@ func validate(username, password string) *util.JSONResponse {
 	return nil
 }
 
+// validatePassword returns an error response if the password is invalid
+func validatePassword(password string) *util.JSONResponse {
+	// https://github.com/matrix-org/synapse/blob/v0.20.0/synapse/rest/client/v2_alpha/register.py#L161
+	if len(password) > maxPasswordLength {
+		return &util.JSONResponse{
+			Code: 400,
+			JSON: jsonerror.BadJSON(fmt.Sprintf("'password' >%d characters", maxPasswordLength)),
+		}
+	} else if len(password) > 0 && len(password) < minPasswordLength {
+		return &util.JSONResponse{
+			Code: 400,
+			JSON: jsonerror.WeakPassword(fmt.Sprintf("password too weak: min %d chars", minPasswordLength)),
+		}
+	}
+	return nil
+}
+
 // Register processes a /register request. http://matrix.org/speculator/spec/HEAD/client_server/unstable.html#post-matrix-client-unstable-register
 func Register(
 	req *http.Request,
@@ -149,7 +156,10 @@ func Register(
 		}
 	}
 
-	if resErr = validate(r.Username, r.Password); resErr != nil {
+	if resErr = validateUserName(r.Username); resErr != nil {
+		return *resErr
+	}
+	if resErr = validatePassword(r.Password); resErr != nil {
 		return *resErr
 	}
 
@@ -209,7 +219,10 @@ func LegacyRegister(
 	if resErr != nil {
 		return *resErr
 	}
-	if resErr = validate(r.Username, r.Password); resErr != nil {
+	if resErr = validateUserName(r.Username); resErr != nil {
+		return *resErr
+	}
+	if resErr = validatePassword(r.Password); resErr != nil {
 		return *resErr
 	}
 
@@ -290,7 +303,7 @@ func completeRegistration(
 	}
 
 	// // TODO: Use the device ID in the request.
-	dev, err := deviceDB.CreateDevice(ctx, username, auth.UnknownDeviceID, token)
+	dev, err := deviceDB.CreateDevice(ctx, username, nil, token)
 	if err != nil {
 		return util.JSONResponse{
 			Code: 500,
@@ -344,3 +357,40 @@ func isValidMacLogin(
 
 	return hmac.Equal(givenMac, expectedMAC), nil
 }
+
+type availableResponse struct {
+	Available bool `json:"available"`
+}
+
+// RegisterAvailable checks if the username is already taken or invalid
+func RegisterAvailable(
+	req *http.Request,
+	accountDB *accounts.Database,
+) util.JSONResponse {
+	username := req.URL.Query().Get("username")
+
+	if err := validateUserName(username); err != nil {
+		return *err
+	}
+
+	availability, availabilityErr := accountDB.CheckAccountAvailability(req.Context(), username)
+	if availabilityErr != nil {
+		return util.JSONResponse{
+			Code: 500,
+			JSON: jsonerror.Unknown("failed to check availability: " + availabilityErr.Error()),
+		}
+	}
+	if !availability {
+		return util.JSONResponse{
+			Code: 400,
+			JSON: jsonerror.InvalidUsername("A different user ID has already been registered for this session"),
+		}
+	}
+
+	return util.JSONResponse{
+		Code: 200,
+		JSON: availableResponse{
+			Available: true,
+		},
+	}
+}

src/github.com/matrix-org/dendrite/cmd/create-account/main.go

@@ -87,7 +87,7 @@ func main() {
 	}
 
 	device, err := deviceDB.CreateDevice(
-		context.Background(), *username, "create-account-script", *accessToken,
+		context.Background(), *username, nil, *accessToken,
 	)
 	if err != nil {
 		fmt.Println(err.Error())