Merge branch 'master' into sql-rows-close-err

CONTRIBUTING.md

@@ -20,34 +20,40 @@ should pick up any unit test and run it). There are also [scripts](scripts) for
 [linting](scripts/find-lint.sh) and doing a [build/test/lint
 run](scripts/build-test-lint.sh).
 
+As of February 2020, we are deprecating support for Go 1.11 and Go 1.12 and are
+now targeting Go 1.13 or later. Please ensure that you are using at least Go
+1.13 when developing for Dendrite - our CI will lint and run tests against this
+version.
+
 ## Continuous Integration
 
 When a Pull Request is submitted, continuous integration jobs are run
-automatically to ensure the code builds and is relatively well-written. The
+automatically to ensure the code builds and is relatively well-written. The jobs
-jobs are run on [Buildkite](https://buildkite.com/matrix-dot-org/dendrite/),
+are run on [Buildkite](https://buildkite.com/matrix-dot-org/dendrite/), and the
-and the Buildkite pipeline configuration can be found in Matrix.org's
+Buildkite pipeline configuration can be found in Matrix.org's [pipelines
-[pipelines repository](https://github.com/matrix-org/pipelines).
+repository](https://github.com/matrix-org/pipelines).
 
 If a job fails, click the "details" button and you should be taken to the job's
 logs.
 
-![Click the details button on the failing build step](docs/images/details-button-location.jpg)
+![Click the details button on the failing build
+step](docs/images/details-button-location.jpg)
 
-Scroll down to the failing step and you should see some log output. Scan
+Scroll down to the failing step and you should see some log output. Scan the
-the logs until you find what it's complaining about, fix it, submit a new
+logs until you find what it's complaining about, fix it, submit a new commit,
-commit, then rinse and repeat until CI passes.
+then rinse and repeat until CI passes.
 
 ### Running CI Tests Locally
 
 To save waiting for CI to finish after every commit, it is ideal to run the
-checks locally before pushing, fixing errors first. This also saves other
+checks locally before pushing, fixing errors first. This also saves other people
-people time as only so many PRs can be tested at a given time.
+time as only so many PRs can be tested at a given time.
 
-To execute what Buildkite tests, first run `./scripts/build-test-lint.sh`;
+To execute what Buildkite tests, first run `./scripts/build-test-lint.sh`; this
-this script will build the code, lint it, and run `go test ./...` with race
+script will build the code, lint it, and run `go test ./...` with race condition
-condition checking enabled. If something needs to be changed, fix it and then
+checking enabled. If something needs to be changed, fix it and then run the
-run the script again until it no longer complains. Be warned that the linting
+script again until it no longer complains. Be warned that the linting can take a
-can take a significant amount of CPU and RAM.
+significant amount of CPU and RAM.
 
 Once the code builds, run [Sytest](https://github.com/matrix-org/sytest)
 according to the guide in
@@ -61,16 +67,18 @@ tests.
 
 ## Picking Things To Do
 
-If you're new then feel free to pick up an issue labelled [good first issue](https://github.com/matrix-org/dendrite/labels/good%20first%20issue).
+If you're new then feel free to pick up an issue labelled [good first
+issue](https://github.com/matrix-org/dendrite/labels/good%20first%20issue).
 These should be well-contained, small pieces of work that can be picked up to
 help you get familiar with the code base.
 
 Once you're comfortable with hacking on Dendrite there are issues lablled as
-[help wanted](https://github.com/matrix-org/dendrite/labels/help%20wanted), these
+[help wanted](https://github.com/matrix-org/dendrite/labels/help%20wanted),
-are often slightly larger or more complicated pieces of work but are hopefully
+these are often slightly larger or more complicated pieces of work but are
-nonetheless fairly well-contained.
+hopefully nonetheless fairly well-contained.
 
-We ask people who are familiar with Dendrite to leave the [good first issue](https://github.com/matrix-org/dendrite/labels/good%20first%20issue)
+We ask people who are familiar with Dendrite to leave the [good first
+issue](https://github.com/matrix-org/dendrite/labels/good%20first%20issue)
 issues so that there is always a way for new people to come and get involved.
 
 ## Getting Help
@@ -79,9 +87,11 @@ For questions related to developing on Dendrite we have a dedicated room on
 Matrix [#dendrite-dev:matrix.org](https://matrix.to/#/#dendrite-dev:matrix.org)
 where we're happy to help.
 
-For more general questions please use [#dendrite:matrix.org](https://matrix.to/#/#dendrite:matrix.org).
+For more general questions please use
+[#dendrite:matrix.org](https://matrix.to/#/#dendrite:matrix.org).
 
 ## Sign off
 
 We ask that everyone who contributes to the project signs off their
-contributions, in accordance with the [DCO](https://github.com/matrix-org/matrix-doc/blob/master/CONTRIBUTING.rst#sign-off).
+contributions, in accordance with the
+[DCO](https://github.com/matrix-org/matrix-doc/blob/master/CONTRIBUTING.rst#sign-off).

INSTALL.md

@@ -12,7 +12,7 @@ Dendrite can be run in one of two configurations:
 
 ## Requirements
 
- - Go 1.11+
+ - Go 1.13+
  - Postgres 9.5+
  - For Kafka (optional if using the monolith server):
    - Unix-based system (https://kafka.apache.org/documentation/#os)
@@ -22,7 +22,7 @@ Dendrite can be run in one of two configurations:
 
 ## Setting up a development environment
 
-Assumes Go 1.10+ and JDK 1.8+ are already installed and are on PATH.
+Assumes Go 1.13+ and JDK 1.8+ are already installed and are on PATH.
 
 ```bash
 # Get the code
@@ -101,7 +101,7 @@ Create config file, based on `dendrite-config.yaml`. Call it `dendrite.yaml`. Th
 
 It is possible to use 'naffka' as an in-process replacement to Kafka when using
 the monolith server. To do this, set `use_naffka: true` in `dendrite.yaml` and uncomment
-the necessary line related to naffka in the `database` section. Be sure to update the 
+the necessary line related to naffka in the `database` section. Be sure to update the
 database username and password if needed.
 
 The monolith server can be started as shown below. By default it listens for
@@ -255,7 +255,7 @@ you want to support federation.
 ./bin/dendrite-federation-sender-server --config dendrite.yaml
 ```
 
-### Run an appservice server 
+### Run an appservice server
 
 This sends events from the network to [application
 services](https://matrix.org/docs/spec/application_service/unstable.html)

README.md

@@ -1,27 +1,30 @@
 # Dendrite [![Build Status](https://badge.buildkite.com/4be40938ab19f2bbc4a6c6724517353ee3ec1422e279faf374.svg?branch=master)](https://buildkite.com/matrix-dot-org/dendrite) [![Dendrite Dev on Matrix](https://img.shields.io/matrix/dendrite-dev:matrix.org.svg?label=%23dendrite-dev%3Amatrix.org&logo=matrix&server_fqdn=matrix.org)](https://matrix.to/#/#dendrite-dev:matrix.org) [![Dendrite on Matrix](https://img.shields.io/matrix/dendrite:matrix.org.svg?label=%23dendrite%3Amatrix.org&logo=matrix&server_fqdn=matrix.org)](https://matrix.to/#/#dendrite:matrix.org)
 
-Dendrite will be a matrix homeserver written in go.
+Dendrite will be a second-generation Matrix homeserver written in Go.
 
-It's still very much a work in progress, but installation instructions can
+It's still very much a work in progress, but installation instructions can be
-be found in [INSTALL.md](INSTALL.md)
+found in [INSTALL.md](INSTALL.md). It is not recommended to use Dendrite as a
+production homeserver at this time.
 
-An overview of the design can be found in [DESIGN.md](DESIGN.md)
+An overview of the design can be found in [DESIGN.md](DESIGN.md).
 
 # Contributing
 
-Everyone is welcome to help out and contribute! See [CONTRIBUTING.md](CONTRIBUTING.md)
+Everyone is welcome to help out and contribute! See
-to get started!
+[CONTRIBUTING.md](CONTRIBUTING.md) to get started!
 
-We aim to try and make it as easy as possible to jump in.
+Please note that, as of February 2020, Dendrite now only targets Go 1.13 or
+later. Please ensure that you are using at least Go 1.13 when developing for
+Dendrite.
 
 # Discussion
 
 For questions about Dendrite we have a dedicated room on Matrix
-[#dendrite:matrix.org](https://matrix.to/#/#dendrite:matrix.org).
+[#dendrite:matrix.org](https://matrix.to/#/#dendrite:matrix.org). Development
-Development discussion should happen in
+discussion should happen in
 [#dendrite-dev:matrix.org](https://matrix.to/#/#dendrite-dev:matrix.org).
 
 # Progress
 
-There's plenty still to do to make Dendrite usable! We're tracking progress in
+There's plenty still to do to make Dendrite usable! We're tracking progress in a
-a [project board](https://github.com/matrix-org/dendrite/projects/2).
+[project board](https://github.com/matrix-org/dendrite/projects/2).

appservice/appservice.go

@@ -100,7 +100,7 @@ func SetupAppServiceAPIComponent(
 
 	// Set up HTTP Endpoints
 	routing.Setup(
-		base.APIMux, *base.Cfg, roomserverQueryAPI, roomserverAliasAPI,
+		base.APIMux, base.Cfg, roomserverQueryAPI, roomserverAliasAPI,
 		accountsDB, federation, transactionsCache,
 	)
 

appservice/routing/routing.go

@@ -36,7 +36,7 @@ const pathPrefixApp = "/_matrix/app/v1"
 // applied:
 // nolint: gocyclo
 func Setup(
-	apiMux *mux.Router, cfg config.Dendrite, // nolint: unparam
+	apiMux *mux.Router, cfg *config.Dendrite, // nolint: unparam
 	queryAPI api.RoomserverQueryAPI, aliasAPI api.RoomserverAliasAPI, // nolint: unparam
 	accountDB *accounts.Database, // nolint: unparam
 	federation *gomatrixserverlib.FederationClient, // nolint: unparam

clientapi/auth/storage/devices/devices_table.go

@@ -19,10 +19,10 @@ import (
 	"database/sql"
 	"time"
 
-	"github.com/matrix-org/dendrite/common"
+	"github.com/lib/pq"
-
 	"github.com/matrix-org/dendrite/clientapi/auth/authtypes"
 	"github.com/matrix-org/dendrite/clientapi/userutil"
+	"github.com/matrix-org/dendrite/common"
 	"github.com/matrix-org/gomatrixserverlib"
 )
 
@@ -80,6 +80,9 @@ const deleteDeviceSQL = "" +
 const deleteDevicesByLocalpartSQL = "" +
 	"DELETE FROM device_devices WHERE localpart = $1"
 
+const deleteDevicesSQL = "" +
+	"DELETE FROM device_devices WHERE localpart = $1 AND device_id = ANY($2)"
+
 type devicesStatements struct {
 	insertDeviceStmt             *sql.Stmt
 	selectDeviceByTokenStmt      *sql.Stmt
@@ -88,6 +91,7 @@ type devicesStatements struct {
 	updateDeviceNameStmt         *sql.Stmt
 	deleteDeviceStmt             *sql.Stmt
 	deleteDevicesByLocalpartStmt *sql.Stmt
+	deleteDevicesStmt            *sql.Stmt
 	serverName                   gomatrixserverlib.ServerName
 }
 
@@ -117,6 +121,9 @@ func (s *devicesStatements) prepare(db *sql.DB, server gomatrixserverlib.ServerN
 	if s.deleteDevicesByLocalpartStmt, err = db.Prepare(deleteDevicesByLocalpartSQL); err != nil {
 		return
 	}
+	if s.deleteDevicesStmt, err = db.Prepare(deleteDevicesSQL); err != nil {
+		return
+	}
 	s.serverName = server
 	return
 }
@@ -142,6 +149,7 @@ func (s *devicesStatements) insertDevice(
 	}, nil
 }
 
+// deleteDevice removes a single device by id and user localpart.
 func (s *devicesStatements) deleteDevice(
 	ctx context.Context, txn *sql.Tx, id, localpart string,
 ) error {
@@ -150,6 +158,18 @@ func (s *devicesStatements) deleteDevice(
 	return err
 }
 
+// deleteDevices removes a single or multiple devices by ids and user localpart.
+// Returns an error if the execution failed.
+func (s *devicesStatements) deleteDevices(
+	ctx context.Context, txn *sql.Tx, localpart string, devices []string,
+) error {
+	stmt := common.TxStmt(txn, s.deleteDevicesStmt)
+	_, err := stmt.ExecContext(ctx, localpart, pq.Array(devices))
+	return err
+}
+
+// deleteDevicesByLocalpart removes all devices for the
+// given user localpart.
 func (s *devicesStatements) deleteDevicesByLocalpart(
 	ctx context.Context, txn *sql.Tx, localpart string,
 ) error {

clientapi/auth/storage/devices/storage.go

@@ -152,6 +152,21 @@ func (d *Database) RemoveDevice(
 	})
 }
 
+// RemoveDevices revokes one or more devices by deleting the entry in the database
+// matching with the given device IDs and user ID localpart.
+// If the devices don't exist, it will not return an error
+// If something went wrong during the deletion, it will return the SQL error.
+func (d *Database) RemoveDevices(
+	ctx context.Context, localpart string, devices []string,
+) error {
+	return common.WithTransaction(d.db, func(txn *sql.Tx) error {
+		if err := d.devices.deleteDevices(ctx, txn, localpart, devices); err != sql.ErrNoRows {
+			return err
+		}
+		return nil
+	})
+}
+
 // RemoveAllDevices revokes devices by deleting the entry in the
 // database matching the given user ID localpart.
 // If something went wrong during the deletion, it will return the SQL error.

clientapi/clientapi.go

@@ -67,7 +67,7 @@ func SetupClientAPIComponent(
 	}
 
 	routing.Setup(
-		base.APIMux, *base.Cfg, roomserverProducer, queryAPI, aliasAPI, asAPI,
+		base.APIMux, base.Cfg, roomserverProducer, queryAPI, aliasAPI, asAPI,
 		accountsDB, deviceDB, federation, *keyRing, userUpdateProducer,
 		syncProducer, typingProducer, transactionsCache, fedSenderAPI,
 	)

clientapi/routing/auth_fallback.go

@@ -102,7 +102,7 @@ func serveTemplate(w http.ResponseWriter, templateHTML string, data map[string]s
 // AuthFallback implements GET and POST /auth/{authType}/fallback/web?session={sessionID}
 func AuthFallback(
 	w http.ResponseWriter, req *http.Request, authType string,
-	cfg config.Dendrite,
+	cfg *config.Dendrite,
 ) *util.JSONResponse {
 	sessionID := req.URL.Query().Get("session")
 
@@ -130,7 +130,7 @@ func AuthFallback(
 	if req.Method == http.MethodGet {
 		// Handle Recaptcha
 		if authType == authtypes.LoginTypeRecaptcha {
-			if err := checkRecaptchaEnabled(&cfg, w, req); err != nil {
+			if err := checkRecaptchaEnabled(cfg, w, req); err != nil {
 				return err
 			}
 
@@ -144,7 +144,7 @@ func AuthFallback(
 	} else if req.Method == http.MethodPost {
 		// Handle Recaptcha
 		if authType == authtypes.LoginTypeRecaptcha {
-			if err := checkRecaptchaEnabled(&cfg, w, req); err != nil {
+			if err := checkRecaptchaEnabled(cfg, w, req); err != nil {
 				return err
 			}
 
@@ -156,7 +156,7 @@ func AuthFallback(
 			}
 
 			response := req.Form.Get("g-recaptcha-response")
-			if err := validateRecaptcha(&cfg, response, clientIP); err != nil {
+			if err := validateRecaptcha(cfg, response, clientIP); err != nil {
 				util.GetLogger(req.Context()).Error(err)
 				return err
 			}

clientapi/routing/createroom.go

@@ -134,7 +134,7 @@ type fledglingEvent struct {
 // CreateRoom implements /createRoom
 func CreateRoom(
 	req *http.Request, device *authtypes.Device,
-	cfg config.Dendrite, producer *producers.RoomserverProducer,
+	cfg *config.Dendrite, producer *producers.RoomserverProducer,
 	accountDB *accounts.Database, aliasAPI roomserverAPI.RoomserverAliasAPI,
 	asAPI appserviceAPI.AppServiceQueryAPI,
 ) util.JSONResponse {
@@ -148,7 +148,7 @@ func CreateRoom(
 // nolint: gocyclo
 func createRoom(
 	req *http.Request, device *authtypes.Device,
-	cfg config.Dendrite, roomID string, producer *producers.RoomserverProducer,
+	cfg *config.Dendrite, roomID string, producer *producers.RoomserverProducer,
 	accountDB *accounts.Database, aliasAPI roomserverAPI.RoomserverAliasAPI,
 	asAPI appserviceAPI.AppServiceQueryAPI,
 ) util.JSONResponse {
@@ -344,7 +344,7 @@ func createRoom(
 func buildEvent(
 	builder *gomatrixserverlib.EventBuilder,
 	provider gomatrixserverlib.AuthEventProvider,
-	cfg config.Dendrite,
+	cfg *config.Dendrite,
 	evTime time.Time,
 ) (*gomatrixserverlib.Event, error) {
 	eventsNeeded, err := gomatrixserverlib.StateNeededForEventBuilder(builder)

clientapi/routing/device.go

@@ -40,6 +40,10 @@ type deviceUpdateJSON struct {
 	DisplayName *string `json:"display_name"`
 }
 
+type devicesDeleteJSON struct {
+	Devices []string `json:"devices"`
+}
+
 // GetDeviceByID handles /devices/{deviceID}
 func GetDeviceByID(
 	req *http.Request, deviceDB *devices.Database, device *authtypes.Device,
@@ -146,3 +150,54 @@ func UpdateDeviceByID(
 		JSON: struct{}{},
 	}
 }
+
+// DeleteDeviceById handles DELETE requests to /devices/{deviceId}
+func DeleteDeviceById(
+	req *http.Request, deviceDB *devices.Database, device *authtypes.Device,
+	deviceID string,
+) util.JSONResponse {
+	localpart, _, err := gomatrixserverlib.SplitID('@', device.UserID)
+	if err != nil {
+		return httputil.LogThenError(req, err)
+	}
+	ctx := req.Context()
+
+	defer req.Body.Close() // nolint: errcheck
+
+	if err := deviceDB.RemoveDevice(ctx, deviceID, localpart); err != nil {
+		return httputil.LogThenError(req, err)
+	}
+
+	return util.JSONResponse{
+		Code: http.StatusOK,
+		JSON: struct{}{},
+	}
+}
+
+// DeleteDevices handles POST requests to /delete_devices
+func DeleteDevices(
+	req *http.Request, deviceDB *devices.Database, device *authtypes.Device,
+) util.JSONResponse {
+	localpart, _, err := gomatrixserverlib.SplitID('@', device.UserID)
+	if err != nil {
+		return httputil.LogThenError(req, err)
+	}
+
+	ctx := req.Context()
+	payload := devicesDeleteJSON{}
+
+	if err := json.NewDecoder(req.Body).Decode(&payload); err != nil {
+		return httputil.LogThenError(req, err)
+	}
+
+	defer req.Body.Close() // nolint: errcheck
+
+	if err := deviceDB.RemoveDevices(ctx, localpart, payload.Devices); err != nil {
+		return httputil.LogThenError(req, err)
+	}
+
+	return util.JSONResponse{
+		Code: http.StatusOK,
+		JSON: struct{}{},
+	}
+}

clientapi/routing/getevent.go

@@ -31,7 +31,7 @@ type getEventRequest struct {
 	device         *authtypes.Device
 	roomID         string
 	eventID        string
-	cfg            config.Dendrite
+	cfg            *config.Dendrite
 	federation     *gomatrixserverlib.FederationClient
 	keyRing        gomatrixserverlib.KeyRing
 	requestedEvent gomatrixserverlib.Event
@@ -44,7 +44,7 @@ func GetEvent(
 	device *authtypes.Device,
 	roomID string,
 	eventID string,
-	cfg config.Dendrite,
+	cfg *config.Dendrite,
 	queryAPI api.RoomserverQueryAPI,
 	federation *gomatrixserverlib.FederationClient,
 	keyRing gomatrixserverlib.KeyRing,

clientapi/routing/joinroom.go

@@ -39,7 +39,7 @@ func JoinRoomByIDOrAlias(
 	req *http.Request,
 	device *authtypes.Device,
 	roomIDOrAlias string,
-	cfg config.Dendrite,
+	cfg *config.Dendrite,
 	federation *gomatrixserverlib.FederationClient,
 	producer *producers.RoomserverProducer,
 	queryAPI roomserverAPI.RoomserverQueryAPI,
@@ -98,7 +98,7 @@ type joinRoomReq struct {
 	evTime     time.Time
 	content    map[string]interface{}
 	userID     string
-	cfg        config.Dendrite
+	cfg        *config.Dendrite
 	federation *gomatrixserverlib.FederationClient
 	producer   *producers.RoomserverProducer
 	queryAPI   roomserverAPI.RoomserverQueryAPI

clientapi/routing/login.go

@@ -71,7 +71,7 @@ func passwordLogin() loginFlows {
 // Login implements GET and POST /login
 func Login(
 	req *http.Request, accountDB *accounts.Database, deviceDB *devices.Database,
-	cfg config.Dendrite,
+	cfg *config.Dendrite,
 ) util.JSONResponse {
 	if req.Method == http.MethodGet { // TODO: support other forms of login other than password, depending on config options
 		return util.JSONResponse{

clientapi/routing/membership.go

@@ -41,7 +41,7 @@ var errMissingUserID = errors.New("'user_id' must be supplied")
 // by building a m.room.member event then sending it to the room server
 func SendMembership(
 	req *http.Request, accountDB *accounts.Database, device *authtypes.Device,
-	roomID string, membership string, cfg config.Dendrite,
+	roomID string, membership string, cfg *config.Dendrite,
 	queryAPI roomserverAPI.RoomserverQueryAPI, asAPI appserviceAPI.AppServiceQueryAPI,
 	producer *producers.RoomserverProducer,
 ) util.JSONResponse {
@@ -119,7 +119,7 @@ func buildMembershipEvent(
 	body threepid.MembershipRequest, accountDB *accounts.Database,
 	device *authtypes.Device,
 	membership, roomID string,
-	cfg config.Dendrite, evTime time.Time,
+	cfg *config.Dendrite, evTime time.Time,
 	queryAPI roomserverAPI.RoomserverQueryAPI, asAPI appserviceAPI.AppServiceQueryAPI,
 ) (*gomatrixserverlib.Event, error) {
 	stateKey, reason, err := getMembershipStateKey(body, device, membership)
@@ -165,7 +165,7 @@ func buildMembershipEvent(
 func loadProfile(
 	ctx context.Context,
 	userID string,
-	cfg config.Dendrite,
+	cfg *config.Dendrite,
 	accountDB *accounts.Database,
 	asAPI appserviceAPI.AppServiceQueryAPI,
 ) (*authtypes.Profile, error) {
@@ -214,7 +214,7 @@ func checkAndProcessThreepid(
 	req *http.Request,
 	device *authtypes.Device,
 	body *threepid.MembershipRequest,
-	cfg config.Dendrite,
+	cfg *config.Dendrite,
 	queryAPI roomserverAPI.RoomserverQueryAPI,
 	accountDB *accounts.Database,
 	producer *producers.RoomserverProducer,

clientapi/routing/memberships.go

@@ -33,7 +33,7 @@ type response struct {
 // GetMemberships implements GET /rooms/{roomId}/members
 func GetMemberships(
 	req *http.Request, device *authtypes.Device, roomID string, joinedOnly bool,
-	_ config.Dendrite,
+	_ *config.Dendrite,
 	queryAPI api.RoomserverQueryAPI,
 ) util.JSONResponse {
 	queryReq := api.QueryMembershipsForRoomRequest{

clientapi/routing/profile.go

@@ -343,7 +343,7 @@ func buildMembershipEvents(
 			return nil, err
 		}
 
-		event, err := common.BuildEvent(ctx, &builder, *cfg, evTime, queryAPI, nil)
+		event, err := common.BuildEvent(ctx, &builder, cfg, evTime, queryAPI, nil)
 		if err != nil {
 			return nil, err
 		}

clientapi/routing/register.go

@@ -991,7 +991,7 @@ type availableResponse struct {
 // RegisterAvailable checks if the username is already taken or invalid.
 func RegisterAvailable(
 	req *http.Request,
-	cfg config.Dendrite,
+	cfg *config.Dendrite,
 	accountDB *accounts.Database,
 ) util.JSONResponse {
 	username := req.URL.Query().Get("username")

clientapi/routing/routing.go

@@ -47,7 +47,7 @@ const pathPrefixUnstable = "/_matrix/client/unstable"
 // applied:
 // nolint: gocyclo
 func Setup(
-	apiMux *mux.Router, cfg config.Dendrite,
+	apiMux *mux.Router, cfg *config.Dendrite,
 	producer *producers.RoomserverProducer,
 	queryAPI roomserverAPI.RoomserverQueryAPI,
 	aliasAPI roomserverAPI.RoomserverAliasAPI,
@@ -170,11 +170,11 @@ func Setup(
 	).Methods(http.MethodPut, http.MethodOptions)
 
 	r0mux.Handle("/register", common.MakeExternalAPI("register", func(req *http.Request) util.JSONResponse {
-		return Register(req, accountDB, deviceDB, &cfg)
+		return Register(req, accountDB, deviceDB, cfg)
 	})).Methods(http.MethodPost, http.MethodOptions)
 
 	v1mux.Handle("/register", common.MakeExternalAPI("register", func(req *http.Request) util.JSONResponse {
-		return LegacyRegister(req, accountDB, deviceDB, &cfg)
+		return LegacyRegister(req, accountDB, deviceDB, cfg)
 	})).Methods(http.MethodPost, http.MethodOptions)
 
 	r0mux.Handle("/register/available", common.MakeExternalAPI("registerAvailable", func(req *http.Request) util.JSONResponse {
@@ -187,7 +187,7 @@ func Setup(
 			if err != nil {
 				return util.ErrorResponse(err)
 			}
-			return DirectoryRoom(req, vars["roomAlias"], federation, &cfg, aliasAPI, federationSender)
+			return DirectoryRoom(req, vars["roomAlias"], federation, cfg, aliasAPI, federationSender)
 		}),
 	).Methods(http.MethodGet, http.MethodOptions)
 
@@ -197,7 +197,7 @@ func Setup(
 			if err != nil {
 				return util.ErrorResponse(err)
 			}
-			return SetLocalAlias(req, device, vars["roomAlias"], &cfg, aliasAPI)
+			return SetLocalAlias(req, device, vars["roomAlias"], cfg, aliasAPI)
 		}),
 	).Methods(http.MethodPut, http.MethodOptions)
 
@@ -301,7 +301,7 @@ func Setup(
 			if err != nil {
 				return util.ErrorResponse(err)
 			}
-			return GetProfile(req, accountDB, &cfg, vars["userID"], asAPI, federation)
+			return GetProfile(req, accountDB, cfg, vars["userID"], asAPI, federation)
 		}),
 	).Methods(http.MethodGet, http.MethodOptions)
 
@@ -311,7 +311,7 @@ func Setup(
 			if err != nil {
 				return util.ErrorResponse(err)
 			}
-			return GetAvatarURL(req, accountDB, &cfg, vars["userID"], asAPI, federation)
+			return GetAvatarURL(req, accountDB, cfg, vars["userID"], asAPI, federation)
 		}),
 	).Methods(http.MethodGet, http.MethodOptions)
 
@@ -321,7 +321,7 @@ func Setup(
 			if err != nil {
 				return util.ErrorResponse(err)
 			}
-			return SetAvatarURL(req, accountDB, device, vars["userID"], userUpdateProducer, &cfg, producer, queryAPI)
+			return SetAvatarURL(req, accountDB, device, vars["userID"], userUpdateProducer, cfg, producer, queryAPI)
 		}),
 	).Methods(http.MethodPut, http.MethodOptions)
 	// Browsers use the OPTIONS HTTP method to check if the CORS policy allows
@@ -333,7 +333,7 @@ func Setup(
 			if err != nil {
 				return util.ErrorResponse(err)
 			}
-			return GetDisplayName(req, accountDB, &cfg, vars["userID"], asAPI, federation)
+			return GetDisplayName(req, accountDB, cfg, vars["userID"], asAPI, federation)
 		}),
 	).Methods(http.MethodGet, http.MethodOptions)
 
@@ -343,7 +343,7 @@ func Setup(
 			if err != nil {
 				return util.ErrorResponse(err)
 			}
-			return SetDisplayName(req, accountDB, device, vars["userID"], userUpdateProducer, &cfg, producer, queryAPI)
+			return SetDisplayName(req, accountDB, device, vars["userID"], userUpdateProducer, cfg, producer, queryAPI)
 		}),
 	).Methods(http.MethodPut, http.MethodOptions)
 	// Browsers use the OPTIONS HTTP method to check if the CORS policy allows
@@ -503,6 +503,22 @@ func Setup(
 		}),
 	).Methods(http.MethodPut, http.MethodOptions)
 
+	r0mux.Handle("/devices/{deviceID}",
+		common.MakeAuthAPI("delete_device", authData, func(req *http.Request, device *authtypes.Device) util.JSONResponse {
+			vars, err := common.URLDecodeMapValues(mux.Vars(req))
+			if err != nil {
+				return util.ErrorResponse(err)
+			}
+			return DeleteDeviceById(req, deviceDB, device, vars["deviceID"])
+		}),
+	).Methods(http.MethodDelete, http.MethodOptions)
+
+	r0mux.Handle("/delete_devices",
+		common.MakeAuthAPI("delete_devices", authData, func(req *http.Request, device *authtypes.Device) util.JSONResponse {
+			return DeleteDevices(req, deviceDB, device)
+		}),
+	).Methods(http.MethodPost, http.MethodOptions)
+
 	// Stub implementations for sytest
 	r0mux.Handle("/events",
 		common.MakeExternalAPI("events", func(req *http.Request) util.JSONResponse {

clientapi/routing/sendevent.go

@@ -43,7 +43,7 @@ func SendEvent(
 	req *http.Request,
 	device *authtypes.Device,
 	roomID, eventType string, txnID, stateKey *string,
-	cfg config.Dendrite,
+	cfg *config.Dendrite,
 	queryAPI api.RoomserverQueryAPI,
 	producer *producers.RoomserverProducer,
 	txnCache *transactions.Cache,
@@ -93,7 +93,7 @@ func generateSendEvent(
 	req *http.Request,
 	device *authtypes.Device,
 	roomID, eventType string, stateKey *string,
-	cfg config.Dendrite,
+	cfg *config.Dendrite,
 	queryAPI api.RoomserverQueryAPI,
 ) (*gomatrixserverlib.Event, *util.JSONResponse) {
 	// parse the incoming http request

clientapi/routing/threepid.go

@@ -39,7 +39,7 @@ type threePIDsResponse struct {
 // RequestEmailToken implements:
 //     POST /account/3pid/email/requestToken
 //     POST /register/email/requestToken
-func RequestEmailToken(req *http.Request, accountDB *accounts.Database, cfg config.Dendrite) util.JSONResponse {
+func RequestEmailToken(req *http.Request, accountDB *accounts.Database, cfg *config.Dendrite) util.JSONResponse {
 	var body threepid.EmailAssociationRequest
 	if reqErr := httputil.UnmarshalJSONRequest(req, &body); reqErr != nil {
 		return *reqErr
@@ -83,7 +83,7 @@ func RequestEmailToken(req *http.Request, accountDB *accounts.Database, cfg conf
 // CheckAndSave3PIDAssociation implements POST /account/3pid
 func CheckAndSave3PIDAssociation(
 	req *http.Request, accountDB *accounts.Database, device *authtypes.Device,
-	cfg config.Dendrite,
+	cfg *config.Dendrite,
 ) util.JSONResponse {
 	var body threepid.EmailAssociationCheckRequest
 	if reqErr := httputil.UnmarshalJSONRequest(req, &body); reqErr != nil {

clientapi/routing/voip.go

@@ -31,7 +31,7 @@ import (
 
 // RequestTurnServer implements:
 //     GET /voip/turnServer
-func RequestTurnServer(req *http.Request, device *authtypes.Device, cfg config.Dendrite) util.JSONResponse {
+func RequestTurnServer(req *http.Request, device *authtypes.Device, cfg *config.Dendrite) util.JSONResponse {
 	turnConfig := cfg.TURN
 
 	// TODO Guest Support

clientapi/threepid/invites.go

@@ -86,7 +86,7 @@ var (
 // can be emitted.
 func CheckAndProcessInvite(
 	ctx context.Context,
-	device *authtypes.Device, body *MembershipRequest, cfg config.Dendrite,
+	device *authtypes.Device, body *MembershipRequest, cfg *config.Dendrite,
 	queryAPI api.RoomserverQueryAPI, db *accounts.Database,
 	producer *producers.RoomserverProducer, membership string, roomID string,
 	evTime time.Time,
@@ -137,7 +137,7 @@ func CheckAndProcessInvite(
 // Returns an error if a check or a request failed.
 func queryIDServer(
 	ctx context.Context,
-	db *accounts.Database, cfg config.Dendrite, device *authtypes.Device,
+	db *accounts.Database, cfg *config.Dendrite, device *authtypes.Device,
 	body *MembershipRequest, roomID string,
 ) (lookupRes *idServerLookupResponse, storeInviteRes *idServerStoreInviteResponse, err error) {
 	if err = isTrusted(body.IDServer, cfg); err != nil {
@@ -206,7 +206,7 @@ func queryIDServerLookup(ctx context.Context, body *MembershipRequest) (*idServe
 // Returns an error if the request failed to send or if the response couldn't be parsed.
 func queryIDServerStoreInvite(
 	ctx context.Context,
-	db *accounts.Database, cfg config.Dendrite, device *authtypes.Device,
+	db *accounts.Database, cfg *config.Dendrite, device *authtypes.Device,
 	body *MembershipRequest, roomID string,
 ) (*idServerStoreInviteResponse, error) {
 	// Retrieve the sender's profile to get their display name
@@ -330,7 +330,7 @@ func checkIDServerSignatures(
 func emit3PIDInviteEvent(
 	ctx context.Context,
 	body *MembershipRequest, res *idServerStoreInviteResponse,
-	device *authtypes.Device, roomID string, cfg config.Dendrite,
+	device *authtypes.Device, roomID string, cfg *config.Dendrite,
 	queryAPI api.RoomserverQueryAPI, producer *producers.RoomserverProducer,
 	evTime time.Time,
 ) error {

clientapi/threepid/threepid.go

@@ -53,7 +53,7 @@ type Credentials struct {
 // Returns an error if there was a problem sending the request or decoding the
 // response, or if the identity server responded with a non-OK status.
 func CreateSession(
-	ctx context.Context, req EmailAssociationRequest, cfg config.Dendrite,
+	ctx context.Context, req EmailAssociationRequest, cfg *config.Dendrite,
 ) (string, error) {
 	if err := isTrusted(req.IDServer, cfg); err != nil {
 		return "", err
@@ -101,7 +101,7 @@ func CreateSession(
 // Returns an error if there was a problem sending the request or decoding the
 // response, or if the identity server responded with a non-OK status.
 func CheckAssociation(
-	ctx context.Context, creds Credentials, cfg config.Dendrite,
+	ctx context.Context, creds Credentials, cfg *config.Dendrite,
 ) (bool, string, string, error) {
 	if err := isTrusted(creds.IDServer, cfg); err != nil {
 		return false, "", "", err
@@ -142,7 +142,7 @@ func CheckAssociation(
 // identifier and a Matrix ID.
 // Returns an error if there was a problem sending the request or decoding the
 // response, or if the identity server responded with a non-OK status.
-func PublishAssociation(creds Credentials, userID string, cfg config.Dendrite) error {
+func PublishAssociation(creds Credentials, userID string, cfg *config.Dendrite) error {
 	if err := isTrusted(creds.IDServer, cfg); err != nil {
 		return err
 	}
@@ -177,7 +177,7 @@ func PublishAssociation(creds Credentials, userID string, cfg config.Dendrite) e
 // isTrusted checks if a given identity server is part of the list of trusted
 // identity servers in the configuration file.
 // Returns an error if the server isn't trusted.
-func isTrusted(idServer string, cfg config.Dendrite) error {
+func isTrusted(idServer string, cfg *config.Dendrite) error {
 	for _, server := range cfg.Matrix.TrustedIDServers {
 		if idServer == server {
 			return nil

common/events.go

@@ -39,7 +39,7 @@ var ErrRoomNoExists = errors.New("Room does not exist")
 // Returns an error if something else went wrong
 func BuildEvent(
 	ctx context.Context,
-	builder *gomatrixserverlib.EventBuilder, cfg config.Dendrite, evTime time.Time,
+	builder *gomatrixserverlib.EventBuilder, cfg *config.Dendrite, evTime time.Time,
 	queryAPI api.RoomserverQueryAPI, queryRes *api.QueryLatestEventsAndStateResponse,
 ) (*gomatrixserverlib.Event, error) {
 	err := AddPrevEventsToEvent(ctx, builder, queryAPI, queryRes)

common/test/config.go

@@ -111,6 +111,7 @@ func MakeConfig(configDir, kafkaURI, database, host string, startPort int) (*con
 	// Bind to the same address as the listen address
 	// All microservices are run on the same host in testing
 	cfg.Bind.ClientAPI = cfg.Listen.ClientAPI
+	cfg.Bind.AppServiceAPI = cfg.Listen.AppServiceAPI
 	cfg.Bind.FederationAPI = cfg.Listen.FederationAPI
 	cfg.Bind.MediaAPI = cfg.Listen.MediaAPI
 	cfg.Bind.RoomServer = cfg.Listen.RoomServer

docs/sytest.md

@@ -44,6 +44,7 @@ args:
     user: dendrite
     database: dendrite
     host: 127.0.0.1
+    sslmode: disable
 type: pg
 EOF
 ```

federationapi/federationapi.go

@@ -45,8 +45,8 @@ func SetupFederationAPIComponent(
 	roomserverProducer := producers.NewRoomserverProducer(inputAPI)
 
 	routing.Setup(
-		base.APIMux, *base.Cfg, queryAPI, aliasAPI, asAPI,
+		base.APIMux, base.Cfg, queryAPI, aliasAPI, asAPI,
-		roomserverProducer, federationSenderAPI, *keyRing, federation, accountsDB,
+		roomserverProducer, federationSenderAPI, *keyRing,
-		deviceDB,
+		federation, accountsDB, deviceDB,
 	)
 }

federationapi/routing/backfill.go

@@ -34,7 +34,7 @@ func Backfill(
 	request *gomatrixserverlib.FederationRequest,
 	query api.RoomserverQueryAPI,
 	roomID string,
-	cfg config.Dendrite,
+	cfg *config.Dendrite,
 ) util.JSONResponse {
 	var res api.QueryBackfillResponse
 	var eIDs []string

federationapi/routing/invite.go

@@ -32,7 +32,7 @@ func Invite(
 	request *gomatrixserverlib.FederationRequest,
 	roomID string,
 	eventID string,
-	cfg config.Dendrite,
+	cfg *config.Dendrite,
 	producer *producers.RoomserverProducer,
 	keys gomatrixserverlib.KeyRing,
 ) util.JSONResponse {

federationapi/routing/join.go

@@ -33,7 +33,7 @@ import (
 func MakeJoin(
 	httpReq *http.Request,
 	request *gomatrixserverlib.FederationRequest,
-	cfg config.Dendrite,
+	cfg *config.Dendrite,
 	query api.RoomserverQueryAPI,
 	roomID, userID string,
 ) util.JSONResponse {
@@ -97,7 +97,7 @@ func MakeJoin(
 func SendJoin(
 	httpReq *http.Request,
 	request *gomatrixserverlib.FederationRequest,
-	cfg config.Dendrite,
+	cfg *config.Dendrite,
 	query api.RoomserverQueryAPI,
 	producer *producers.RoomserverProducer,
 	keys gomatrixserverlib.KeyRing,

federationapi/routing/keys.go

@@ -27,7 +27,7 @@ import (
 
 // LocalKeys returns the local keys for the server.
 // See https://matrix.org/docs/spec/server_server/unstable.html#publishing-keys
-func LocalKeys(cfg config.Dendrite) util.JSONResponse {
+func LocalKeys(cfg *config.Dendrite) util.JSONResponse {
 	keys, err := localKeys(cfg, time.Now().Add(cfg.Matrix.KeyValidityPeriod))
 	if err != nil {
 		return util.ErrorResponse(err)
@@ -35,7 +35,7 @@ func LocalKeys(cfg config.Dendrite) util.JSONResponse {
 	return util.JSONResponse{Code: http.StatusOK, JSON: keys}
 }
 
-func localKeys(cfg config.Dendrite, validUntil time.Time) (*gomatrixserverlib.ServerKeys, error) {
+func localKeys(cfg *config.Dendrite, validUntil time.Time) (*gomatrixserverlib.ServerKeys, error) {
 	var keys gomatrixserverlib.ServerKeys
 
 	keys.ServerName = cfg.Matrix.ServerName

federationapi/routing/leave.go

@@ -31,7 +31,7 @@ import (
 func MakeLeave(
 	httpReq *http.Request,
 	request *gomatrixserverlib.FederationRequest,
-	cfg config.Dendrite,
+	cfg *config.Dendrite,
 	query api.RoomserverQueryAPI,
 	roomID, userID string,
 ) util.JSONResponse {
@@ -95,7 +95,7 @@ func MakeLeave(
 func SendLeave(
 	httpReq *http.Request,
 	request *gomatrixserverlib.FederationRequest,
-	cfg config.Dendrite,
+	cfg *config.Dendrite,
 	producer *producers.RoomserverProducer,
 	keys gomatrixserverlib.KeyRing,
 	roomID, eventID string,

federationapi/routing/profile.go

@@ -31,7 +31,7 @@ import (
 func GetProfile(
 	httpReq *http.Request,
 	accountDB *accounts.Database,
-	cfg config.Dendrite,
+	cfg *config.Dendrite,
 	asAPI appserviceAPI.AppServiceQueryAPI,
 ) util.JSONResponse {
 	userID, field := httpReq.FormValue("user_id"), httpReq.FormValue("field")

federationapi/routing/query.go

@@ -32,7 +32,7 @@ import (
 func RoomAliasToID(
 	httpReq *http.Request,
 	federation *gomatrixserverlib.FederationClient,
-	cfg config.Dendrite,
+	cfg *config.Dendrite,
 	aliasAPI roomserverAPI.RoomserverAliasAPI,
 	senderAPI federationSenderAPI.FederationSenderQueryAPI,
 ) util.JSONResponse {

federationapi/routing/routing.go

@@ -43,7 +43,7 @@ const (
 // nolint: gocyclo
 func Setup(
 	apiMux *mux.Router,
-	cfg config.Dendrite,
+	cfg *config.Dendrite,
 	query roomserverAPI.RoomserverQueryAPI,
 	aliasAPI roomserverAPI.RoomserverAliasAPI,
 	asAPI appserviceAPI.AppServiceQueryAPI,

federationapi/routing/send.go

@@ -34,7 +34,7 @@ func Send(
 	httpReq *http.Request,
 	request *gomatrixserverlib.FederationRequest,
 	txnID gomatrixserverlib.TransactionID,
-	cfg config.Dendrite,
+	cfg *config.Dendrite,
 	query api.RoomserverQueryAPI,
 	producer *producers.RoomserverProducer,
 	keys gomatrixserverlib.KeyRing,

federationapi/routing/threepid.go

@@ -59,7 +59,7 @@ var (
 // CreateInvitesFrom3PIDInvites implements POST /_matrix/federation/v1/3pid/onbind
 func CreateInvitesFrom3PIDInvites(
 	req *http.Request, queryAPI roomserverAPI.RoomserverQueryAPI,
-	asAPI appserviceAPI.AppServiceQueryAPI, cfg config.Dendrite,
+	asAPI appserviceAPI.AppServiceQueryAPI, cfg *config.Dendrite,
 	producer *producers.RoomserverProducer, federation *gomatrixserverlib.FederationClient,
 	accountDB *accounts.Database,
 ) util.JSONResponse {
@@ -98,7 +98,7 @@ func ExchangeThirdPartyInvite(
 	request *gomatrixserverlib.FederationRequest,
 	roomID string,
 	queryAPI roomserverAPI.RoomserverQueryAPI,
-	cfg config.Dendrite,
+	cfg *config.Dendrite,
 	federation *gomatrixserverlib.FederationClient,
 	producer *producers.RoomserverProducer,
 ) util.JSONResponse {
@@ -172,7 +172,7 @@ func ExchangeThirdPartyInvite(
 // necessary data to do so.
 func createInviteFrom3PIDInvite(
 	ctx context.Context, queryAPI roomserverAPI.RoomserverQueryAPI,
-	asAPI appserviceAPI.AppServiceQueryAPI, cfg config.Dendrite,
+	asAPI appserviceAPI.AppServiceQueryAPI, cfg *config.Dendrite,
 	inv invite, federation *gomatrixserverlib.FederationClient,
 	accountDB *accounts.Database,
 ) (*gomatrixserverlib.Event, error) {
@@ -230,7 +230,7 @@ func createInviteFrom3PIDInvite(
 func buildMembershipEvent(
 	ctx context.Context,
 	builder *gomatrixserverlib.EventBuilder, queryAPI roomserverAPI.RoomserverQueryAPI,
-	cfg config.Dendrite,
+	cfg *config.Dendrite,
 ) (*gomatrixserverlib.Event, error) {
 	eventsNeeded, err := gomatrixserverlib.StateNeededForEventBuilder(builder)
 	if err != nil {
@@ -290,7 +290,7 @@ func buildMembershipEvent(
 // them responded with an error.
 func sendToRemoteServer(
 	ctx context.Context, inv invite,
-	federation *gomatrixserverlib.FederationClient, _ config.Dendrite,
+	federation *gomatrixserverlib.FederationClient, _ *config.Dendrite,
 	builder gomatrixserverlib.EventBuilder,
 ) (err error) {
 	remoteServers := make([]gomatrixserverlib.ServerName, 2)

sytest-blacklist

@@ -22,3 +22,9 @@ Real non-joined users can get individual state for world_readable rooms after le
 
 # Blacklisted until matrix-org/dendrite#862 is reverted due to Riot bug
 Latest account data appears in v2 /sync
+
+# Blacklisted due to flakiness
+Outbound federation can backfill events
+
+# Blacklisted due to alias work on Synapse
+Alias creators can delete canonical alias with no ops