mirrors/dendrite
1
0
Fork 0
mirror of https://github.com/matrix-org/dendrite.git synced 2025-12-29 01:33:10 -06:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into fix-failing-ban-tests

Browse source
This commit is contained in:
kegsay 2021-07-19 18:03:40 +01:00 committed by GitHub
parent 56a2a5843a 5094bc89bf
commit c3ee147598
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
82 changed files with 3063 additions and 2624 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

41
CHANGES.md
View file

@ -1,5 +1,46 @@
# Changelog
## Dendrite 0.4.0 (2021-07-12)
### Features
* All-new state storage in the roomserver, which dramatically reduces disk space utilisation
* State snapshots and blocks are now aggressively deduplicated and reused wherever possible, with state blocks being reduced by up to 15x and snapshot references being reduced up to 2x
* Dendrite will upgrade to the new state storage automatically on the first run after upgrade, although this may take some time depending on the size of the state storage
* Appservice support has been improved significantly, with many bridges now working correctly with Dendrite
* Events are now correctly sent to appservices based on room memberships
* Aliases and namespaces are now handled correctly, calling the appservice to query for aliases as needed
* Appservice user registrations are no longer being subject to incorrect validation checks
* Shared secret registration has now been implemented correctly
* The roomserver input API implements a new queuing system to reduce backpressure across rooms
* Checking if the local server is in a room has been optimised substantially, reducing CPU usage
* State resolution v2 has been optimised further by improving the power level checks, reducing CPU usage
* The federation API `/send` endpoint now deduplicates missing auth and prev events more aggressively to reduce memory usage
* The federation API `/send` endpoint now uses workers to reduce backpressure across rooms
* The bcrypt cost for password storage is now configurable with the `user_api.bcrypt_cost` option
* The federation API will now use significantly less memory when calling `/get_missing_events`
* MSC2946 Spaces endpoints have been updated to stable endpoint naming
* The media API can now be configured without a maximum file size
* A new `dendrite-upgrade-test` test has been added for verifying database schema upgrades across versions
* Added Prometheus metrics for roomserver backpressure, excessive device list updates and federation API event processing summaries
* Sentry support has been added for error reporting
### Fixes
* Removed the legacy `/v1` register endpoint. Dendrite only implements `/r0` of the CS API, and the legacy `/v1` endpoint had implementation errors which made it possible to bypass shared secret registration (thanks to Jakob Varmose Bentzen for reporting this)
* Attempting to register an account that already exists now returns a sensible error code rather than a HTTP 500
* Dendrite will no longer attempt to `/make_join` with itself if listed in the request `server_names`
* `/sync` will no longer return immediately if there is nothing to sync, which happened particularly with new accounts, causing high CPU usage
* Malicious media uploads can no longer exhaust all available memory (contributed by [S7evinK](https://github.com/S7evinK))
* Selecting one-time keys from the database has been optimised (contributed by [S7evinK](https://github.com/S7evinK))
* The return code when trying to fetch missing account data has been fixed (contributed by [adamgreig](https://github.com/adamgreig))
* Dendrite will no longer attempt to use `/make_leave` over federation when rejecting a local invite
* A panic has been fixed in `QueryMembershipsForRoom`
* A panic on duplicate membership events has been fixed in the federation sender
* A panic has been fixed in in `IsInterestedInRoomID` (contributed by [bodqhrohro](https://github.com/bodqhrohro))
* A panic in the roomserver has been fixed when handling empty state sets
* A panic in the federation API has been fixed when handling cached events
## Dendrite 0.3.11 (2021-03-02)
### Fixes

30
are-we-synapse-yet.list
View file

@ -17,6 +17,13 @@ reg POST /register rejects registration of usernames with '£'
reg POST /register rejects registration of usernames with 'é'
reg POST /register rejects registration of usernames with '\n'
reg POST /register rejects registration of usernames with '''
reg POST /register allows registration of usernames with 'q'
reg POST /register allows registration of usernames with '3'
reg POST /register allows registration of usernames with '.'
reg POST /register allows registration of usernames with '_'
reg POST /register allows registration of usernames with '='
reg POST /register allows registration of usernames with '-'
reg POST /register allows registration of usernames with '/'
reg POST /r0/admin/register with shared secret
reg POST /r0/admin/register admin with shared secret
reg POST /r0/admin/register with shared secret downcases capitals
@ -76,7 +83,7 @@ rst GET /rooms/:room_id/state/m.room.topic gets topic
rst GET /rooms/:room_id/state fetches entire room state
crm POST /createRoom with creation content
ali PUT /directory/room/:room_alias creates alias
nsp GET /rooms/:room_id/aliases lists aliases
ali GET /rooms/:room_id/aliases lists aliases
jon POST /rooms/:room_id/join can join a room
jon POST /join/:room_alias can join a room
jon POST /join/:room_id can join a room
@ -95,6 +102,7 @@ typ Typing notifications don't leak (3 subtests)
rst GET /rooms/:room_id/state/m.room.power_levels can fetch levels
rst PUT /rooms/:room_id/state/m.room.power_levels can set levels
rst PUT power_levels should not explode if the old power levels were empty
rst Users cannot set notifications powerlevel higher than their own (2 subtests)
rst Both GET and PUT work
rct POST /rooms/:room_id/receipt can create receipts
red POST /rooms/:room_id/read_markers can create read marker
@ -175,7 +183,7 @@ ali Users with sufficient power-level can delete other's aliases
ali Can delete canonical alias
ali Alias creators can delete alias with no ops
ali Alias creators can delete canonical alias with no ops
ali Only room members can list aliases of a room
msc Only room members can list aliases of a room
inv Can invite users to invite-only rooms
inv Uninvited users cannot join the room
inv Invited user can reject invite
@ -353,6 +361,7 @@ syn Syncing a new room with a large timeline limit isn't limited
syn A full_state incremental update returns only recent timeline
syn A prev_batch token can be used in the v1 messages API
syn A next_batch token can be used in the v1 messages API
syn A prev_batch token from incremental sync can be used in the v1 messages API
syn User sees their own presence in a sync
syn User is offline if they set_presence=offline in their sync
syn User sees updates to presence from other users in the incremental sync.
@ -574,6 +583,7 @@ fqu Outbound federation can query profile data
fqu Inbound federation can query profile data
fqu Outbound federation can query room alias directory
fqu Inbound federation can query room alias directory
fsj Membership event with an invalid displayname in the send_join response should not cause room join to fail
fsj Outbound federation can query v1 /send_join
fsj Outbound federation can query v2 /send_join
fmj Outbound federation passes make_join failures through to the client
@ -596,7 +606,7 @@ fsj Inbound: send_join rejects invalid JSON for room version 6
fed Outbound federation can send events
fed Inbound federation can receive events
fed Inbound federation can receive redacted events
fed Ephemeral messages received from servers are correctly expired
msc Ephemeral messages received from servers are correctly expired
fed Events whose auth_events are in the wrong room do not mess up the room state
fed Inbound federation can return events
fed Inbound federation redacts events from erased users
@ -743,6 +753,10 @@ nsp Set group joinable and join it
nsp Group is not joinable by default
nsp Group is joinable over federation
nsp Room is transitioned on local and remote groups upon room upgrade
nsp POST /_synapse/admin/v1/register with shared secret
nsp POST /_synapse/admin/v1/register admin with shared secret
nsp POST /_synapse/admin/v1/register with shared secret downcases capitals
nsp POST /_synapse/admin/v1/register with shared secret disallows symbols
3pd Can bind 3PID via home server
3pd Can bind and unbind 3PID via homeserver
3pd Can unbind 3PID via homeserver when bound out of band
@ -859,8 +873,14 @@ jso Invalid JSON special values
inv Can invite users to invite-only rooms (2 subtests)
plv setting 'm.room.name' respects room powerlevel (2 subtests)
psh Messages that notify from another user increment notification_count
psh Messages that org.matrix.msc2625.mark_unread from another user increment org.matrix.msc2625.unread_count
msc Messages that org.matrix.msc2625.mark_unread from another user increment org.matrix.msc2625.unread_count
dvk Can claim one time key using POST (2 subtests)
fdk Can query remote device keys using POST (1 subtests)
fdk Can claim remote one time key using POST (2 subtests)
fmj Inbound /make_join rejects attempts to join rooms where all users have left
fmj Inbound /make_join rejects attempts to join rooms where all users have left
msc Local users can peek into world_readable rooms by room ID
msc We can't peek into rooms with shared history_visibility
msc We can't peek into rooms with invited history_visibility
msc We can't peek into rooms with joined history_visibility
msc Local users can peek by room alias
msc Peeked rooms only turn up in the sync for the device who peeked them

4
are-we-synapse-yet.py
View file

@ -35,6 +35,7 @@ test_mappings = {
"nsp": "Non-Spec API",
"unk": "Unknown API (no group specified)",
"app": "Application Services API",
"msc": "MSCs",
"f": "Federation", # flag to mark test involves federation
"federation_apis": {
@ -223,6 +224,7 @@ def main(results_tap_path, verbose):
},
"nonspec": {
"nsp": {},
"msc": {},
"unk": {}
},
}
@ -237,6 +239,8 @@ def main(results_tap_path, verbose):
summary["nonspec"]["unk"][name] = test_result["ok"]
if group_id == "nsp":
summary["nonspec"]["nsp"][name] = test_result["ok"]
elif group_id == "msc":
summary["nonspec"]["msc"][name] = test_result["ok"]
elif group_id == "app":
summary["appservice"]["app"][name] = test_result["ok"]
elif group_id in test_mappings["federation_apis"]:

14
build/docker/README.md
View file

@ -28,8 +28,8 @@ There are three sample `docker-compose` files:
The `docker-compose` files refer to the `/etc/dendrite` volume as where the
runtime config should come from. The mounted folder must contain:
- `dendrite.yaml` configuration file (based on the sample `dendrite-config.yaml`
in the `docker/config` folder in the [Dendrite repository](https://github.com/matrix-org/dendrite)
- `dendrite.yaml` configuration file (based on the [`dendrite-config.yaml`](https://raw.githubusercontent.com/matrix-org/dendrite/master/dendrite-config.yaml)
sample in the `build/docker/config` folder of this repository.)
- `matrix_key.pem` server key, as generated using `cmd/generate-keys`
- `server.crt` certificate file
- `server.key` private key file for the above certificate
@ -50,8 +50,7 @@ The key files will now exist in your current working directory, and can be mount
## Starting Dendrite as a monolith deployment
Create your config based on the `dendrite.yaml` configuration file in the `docker/config`
folder in the [Dendrite repository](https://github.com/matrix-org/dendrite).
Create your config based on the [`dendrite-config.yaml`](https://raw.githubusercontent.com/matrix-org/dendrite/master/dendrite-config.yaml) configuration file in the `build/docker/config` folder of this repository. And rename the config file to `dendrite.yml` (and put it in your `config` directory).
Once in place, start the PostgreSQL dependency:
@ -67,8 +66,7 @@ docker-compose -f docker-compose.monolith.yml up
## Starting Dendrite as a polylith deployment
Create your config based on the `dendrite.yaml` configuration file in the `docker/config`
folder in the [Dendrite repository](https://github.com/matrix-org/dendrite).
Create your config based on the [`dendrite-config.yaml`](https://raw.githubusercontent.com/matrix-org/dendrite/master/dendrite-config.yaml) configuration file in the `build/docker/config` folder of this repository. And rename the config file to `dendrite.yml` (and put it in your `config` directory).
Once in place, start all the dependencies:
@ -84,10 +82,10 @@ docker-compose -f docker-compose.polylith.yml up
## Building the images
The `docker/images-build.sh` script will build the base image, followed by
The `build/docker/images-build.sh` script will build the base image, followed by
all of the component images.
The `docker/images-push.sh` script will push them to Docker Hub (subject
The `build/docker/images-push.sh` script will push them to Docker Hub (subject
to permissions).
If you wish to build and push your own images, rename `matrixdotorg/dendrite` to

1
build/gobind-pinecone/monolith.go
View file

@ -334,6 +334,7 @@ func (m *DendriteMonolith) Start() {
base.PublicFederationAPIMux,
base.PublicKeyAPIMux,
base.PublicMediaAPIMux,
base.SynapseAdminMux,
)
httpRouter := mux.NewRouter().SkipClean(true).UseEncodedPath()

21
build/gobind-yggdrasil/monolith.go
View file

@ -45,10 +45,6 @@ func (m *DendriteMonolith) PeerCount() int {
return m.YggdrasilNode.PeerCount()
}
func (m *DendriteMonolith) SessionCount() int {
return m.YggdrasilNode.SessionCount()
}
func (m *DendriteMonolith) SetMulticastEnabled(enabled bool) {
m.YggdrasilNode.SetMulticastEnabled(enabled)
}
@ -78,7 +74,7 @@ func (m *DendriteMonolith) Start() {
panic(err)
}
ygg, err := yggconn.Setup("dendrite", m.StorageDirectory)
ygg, err := yggconn.Setup("dendrite", m.StorageDirectory, "")
if err != nil {
panic(err)
}
@ -87,7 +83,7 @@ func (m *DendriteMonolith) Start() {
cfg := &config.Dendrite{}
cfg.Defaults()
cfg.Global.ServerName = gomatrixserverlib.ServerName(ygg.DerivedServerName())
cfg.Global.PrivateKey = ygg.SigningPrivateKey()
cfg.Global.PrivateKey = ygg.PrivateKey()
cfg.Global.KeyID = gomatrixserverlib.KeyID(signing.KeyID)
cfg.Global.Kafka.UseNaffka = true
cfg.Global.Kafka.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-p2p-naffka.db", m.StorageDirectory))
@ -134,18 +130,6 @@ func (m *DendriteMonolith) Start() {
asAPI := appservice.NewInternalAPI(base, userAPI, rsAPI)
rsAPI.SetAppserviceAPI(asAPI)
ygg.SetSessionFunc(func(address string) {
req := &api.PerformServersAliveRequest{
Servers: []gomatrixserverlib.ServerName{
gomatrixserverlib.ServerName(address),
},
}
res := &api.PerformServersAliveResponse{}
if err := fsAPI.PerformServersAlive(context.TODO(), req, res); err != nil {
logrus.WithError(err).Error("Failed to send wake-up message to newly connected node")
}
})
// The underlying roomserver implementation needs to be able to call the fedsender.
// This is different to rsAPI which can be the http client which doesn't need this dependency
rsAPI.SetFederationSenderAPI(fsAPI)
@ -173,6 +157,7 @@ func (m *DendriteMonolith) Start() {
base.PublicFederationAPIMux,
base.PublicKeyAPIMux,
base.PublicMediaAPIMux,
base.SynapseAdminMux,
)
httpRouter := mux.NewRouter()

3
clientapi/clientapi.go
View file

@ -35,6 +35,7 @@ import (
// AddPublicRoutes sets up and registers HTTP handlers for the ClientAPI component.
func AddPublicRoutes(
router *mux.Router,
synapseAdminRouter *mux.Router,
cfg *config.ClientAPI,
accountsDB accounts.Database,
federation *gomatrixserverlib.FederationClient,
@ -56,7 +57,7 @@ func AddPublicRoutes(
}
routing.Setup(
router, cfg, eduInputAPI, rsAPI, asAPI,
router, synapseAdminRouter, cfg, eduInputAPI, rsAPI, asAPI,
accountsDB, userAPI, federation,
syncProducer, transactionsCache, fsAPI, keyAPI, extRoomsProvider, mscCfg,
)

92
clientapi/routing/register.go
View file

@ -17,10 +17,7 @@ package routing
import (
"context"
"crypto/hmac"
"crypto/sha1"
"encoding/json"
"errors"
"fmt"
"io/ioutil"
"net/http"
@ -594,7 +591,6 @@ func handleRegistrationFlow(
accessToken string,
accessTokenErr error,
) util.JSONResponse {
// TODO: Shared secret registration (create new user scripts)
// TODO: Enable registration config flag
// TODO: Guest account upgrading
@ -643,20 +639,6 @@ func handleRegistrationFlow(
// Add Recaptcha to the list of completed registration stages
AddCompletedSessionStage(sessionID, authtypes.LoginTypeRecaptcha)
case authtypes.LoginTypeSharedSecret:
// Check shared secret against config
valid, err := isValidMacLogin(cfg, r.Username, r.Password, r.Admin, r.Auth.Mac)
if err != nil {
util.GetLogger(req.Context()).WithError(err).Error("isValidMacLogin failed")
return jsonerror.InternalServerError()
} else if !valid {
return util.MessageResponse(http.StatusForbidden, "HMAC incorrect")
}
// Add SharedSecret to the list of completed registration stages
AddCompletedSessionStage(sessionID, authtypes.LoginTypeSharedSecret)
case authtypes.LoginTypeDummy:
// there is nothing to do
// Add Dummy to the list of completed registration stages
@ -849,49 +831,6 @@ func completeRegistration(
}
}
// Used for shared secret registration.
// Checks if the username, password and isAdmin flag matches the given mac.
func isValidMacLogin(
cfg *config.ClientAPI,
username, password string,
isAdmin bool,
givenMac []byte,
) (bool, error) {
sharedSecret := cfg.RegistrationSharedSecret
// Check that shared secret registration isn't disabled.
if cfg.RegistrationSharedSecret == "" {
return false, errors.New("Shared secret registration is disabled")
}
// Double check that username/password don't contain the HMAC delimiters. We should have
// already checked this.
if strings.Contains(username, "\x00") {
return false, errors.New("Username contains invalid character")
}
if strings.Contains(password, "\x00") {
return false, errors.New("Password contains invalid character")
}
if sharedSecret == "" {
return false, errors.New("Shared secret registration is disabled")
}
adminString := "notadmin"
if isAdmin {
adminString = "admin"
}
joined := strings.Join([]string{username, password, adminString}, "\x00")
mac := hmac.New(sha1.New, []byte(sharedSecret))
_, err := mac.Write([]byte(joined))
if err != nil {
return false, err
}
expectedMAC := mac.Sum(nil)
return hmac.Equal(givenMac, expectedMAC), nil
}
// checkFlows checks a single completed flow against another required one. If
// one contains at least all of the stages that the other does, checkFlows
// returns true.
@ -995,3 +934,34 @@ func RegisterAvailable(
},
}
}
func handleSharedSecretRegistration(userAPI userapi.UserInternalAPI, sr *SharedSecretRegistration, req *http.Request) util.JSONResponse {
ssrr, err := NewSharedSecretRegistrationRequest(req.Body)
if err != nil {
return util.JSONResponse{
Code: 400,
JSON: jsonerror.BadJSON(fmt.Sprintf("malformed json: %s", err)),
}
}
valid, err := sr.IsValidMacLogin(ssrr.Nonce, ssrr.User, ssrr.Password, ssrr.Admin, ssrr.MacBytes)
if err != nil {
return util.ErrorResponse(err)
}
if !valid {
return util.JSONResponse{
Code: 403,
JSON: jsonerror.Forbidden("bad mac"),
}
}
// downcase capitals
ssrr.User = strings.ToLower(ssrr.User)
if resErr := validateUsername(ssrr.User); resErr != nil {
return *resErr
}
if resErr := validatePassword(ssrr.Password); resErr != nil {
return *resErr
}
deviceID := "shared_secret_registration"
return completeRegistration(req.Context(), userAPI, ssrr.User, ssrr.Password, "", req.RemoteAddr, req.UserAgent(), false, &ssrr.User, &deviceID)
}

99
clientapi/routing/register_secret.go Normal file
View file

@ -0,0 +1,99 @@
package routing
import (
"context"
"crypto/hmac"
"crypto/sha1"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"strings"
"time"
"github.com/matrix-org/dendrite/internal"
"github.com/matrix-org/util"
cache "github.com/patrickmn/go-cache"
)
type SharedSecretRegistrationRequest struct {
User string `json:"username"`
Password string `json:"password"`
Nonce string `json:"nonce"`
MacBytes []byte
MacStr string `json:"mac"`
Admin bool `json:"admin"`
}
func NewSharedSecretRegistrationRequest(reader io.ReadCloser) (*SharedSecretRegistrationRequest, error) {
defer internal.CloseAndLogIfError(context.Background(), reader, "NewSharedSecretRegistrationRequest: failed to close request body")
var ssrr SharedSecretRegistrationRequest
err := json.NewDecoder(reader).Decode(&ssrr)
if err != nil {
return nil, err
}
ssrr.MacBytes, err = hex.DecodeString(ssrr.MacStr)
return &ssrr, err
}
type SharedSecretRegistration struct {
sharedSecret string
nonces *cache.Cache
}
func NewSharedSecretRegistration(sharedSecret string) *SharedSecretRegistration {
return &SharedSecretRegistration{
sharedSecret: sharedSecret,
// nonces live for 5mins, purge every 10mins
nonces: cache.New(5*time.Minute, 10*time.Minute),
}
}
func (r *SharedSecretRegistration) GenerateNonce() string {
nonce := util.RandomString(16)
r.nonces.Set(nonce, true, cache.DefaultExpiration)
return nonce
}
func (r *SharedSecretRegistration) validNonce(nonce string) bool {
_, exists := r.nonces.Get(nonce)
return exists
}
func (r *SharedSecretRegistration) IsValidMacLogin(
nonce, username, password string,
isAdmin bool,
givenMac []byte,
) (bool, error) {
// Check that shared secret registration isn't disabled.
if r.sharedSecret == "" {
return false, errors.New("Shared secret registration is disabled")
}
if !r.validNonce(nonce) {
return false, fmt.Errorf("Incorrect or expired nonce: %s", nonce)
}
// Check that username/password don't contain the HMAC delimiters.
if strings.Contains(username, "\x00") {
return false, errors.New("Username contains invalid character")
}
if strings.Contains(password, "\x00") {
return false, errors.New("Password contains invalid character")
}
adminString := "notadmin"
if isAdmin {
adminString = "admin"
}
joined := strings.Join([]string{nonce, username, password, adminString}, "\x00")
mac := hmac.New(sha1.New, []byte(r.sharedSecret))
_, err := mac.Write([]byte(joined))
if err != nil {
return false, err
}
expectedMAC := mac.Sum(nil)
return hmac.Equal(givenMac, expectedMAC), nil
}

43
clientapi/routing/register_secret_test.go Normal file
View file

@ -0,0 +1,43 @@
package routing
import (
"bytes"
"io/ioutil"
"testing"
"github.com/patrickmn/go-cache"
)
func TestSharedSecretRegister(t *testing.T) {
// these values have come from a local synapse instance to ensure compatibility
jsonStr := []byte(`{"admin":false,"mac":"f1ba8d37123866fd659b40de4bad9b0f8965c565","nonce":"759f047f312b99ff428b21d581256f8592b8976e58bc1b543972dc6147e529a79657605b52d7becd160ff5137f3de11975684319187e06901955f79e5a6c5a79","password":"wonderland","username":"alice"}`)
sharedSecret := "dendritetest"
req, err := NewSharedSecretRegistrationRequest(ioutil.NopCloser(bytes.NewBuffer(jsonStr)))
if err != nil {
t.Fatalf("failed to read request: %s", err)
}
r := NewSharedSecretRegistration(sharedSecret)
// force the nonce to be known
r.nonces.Set(req.Nonce, true, cache.DefaultExpiration)
valid, err := r.IsValidMacLogin(req.Nonce, req.User, req.Password, req.Admin, req.MacBytes)
if err != nil {
t.Fatalf("failed to check for valid mac: %s", err)
}
if !valid {
t.Errorf("mac login failed, wanted success")
}
// modify the mac so it fails
req.MacBytes[0] = 0xff
valid, err = r.IsValidMacLogin(req.Nonce, req.User, req.Password, req.Admin, req.MacBytes)
if err != nil {
t.Fatalf("failed to check for valid mac: %s", err)
}
if valid {
t.Errorf("mac login succeeded, wanted failure")
}
}

29
clientapi/routing/routing.go
View file

@ -37,6 +37,7 @@ import (
"github.com/matrix-org/dendrite/userapi/storage/accounts"
"github.com/matrix-org/gomatrixserverlib"
"github.com/matrix-org/util"
"github.com/sirupsen/logrus"
)
// Setup registers HTTP handlers with the given ServeMux. It also supplies the given http.Client
@ -46,7 +47,7 @@ import (
// applied:
// nolint: gocyclo
func Setup(
publicAPIMux *mux.Router, cfg *config.ClientAPI,
publicAPIMux, synapseAdminRouter *mux.Router, cfg *config.ClientAPI,
eduAPI eduServerAPI.EDUServerInputAPI,
rsAPI roomserverAPI.RoomserverInternalAPI,
asAPI appserviceAPI.AppServiceQueryAPI,
@ -88,6 +89,32 @@ func Setup(
}),
).Methods(http.MethodGet, http.MethodOptions)
if cfg.RegistrationSharedSecret != "" {
logrus.Info("Enabling shared secret registration at /_synapse/admin/v1/register")
sr := NewSharedSecretRegistration(cfg.RegistrationSharedSecret)
synapseAdminRouter.Handle("/admin/v1/register",
httputil.MakeExternalAPI("shared_secret_registration", func(req *http.Request) util.JSONResponse {
if req.Method == http.MethodGet {
return util.JSONResponse{
Code: 200,
JSON: struct {
Nonce string `json:"nonce"`
}{
Nonce: sr.GenerateNonce(),
},
}
}
if req.Method == http.MethodPost {
return handleSharedSecretRegistration(userAPI, sr, req)
}
return util.JSONResponse{
Code: http.StatusMethodNotAllowed,
JSON: jsonerror.NotFound("unknown method"),
}
}),
).Methods(http.MethodGet, http.MethodPost, http.MethodOptions)
}
r0mux := publicAPIMux.PathPrefix("/r0").Subrouter()
unstableMux := publicAPIMux.PathPrefix("/unstable").Subrouter()

147
cmd/create-room-events/main.go
View file

@ -1,147 +0,0 @@
// Copyright 2017 Vector Creations Ltd
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Generate a list of matrix room events for load testing.
// Writes the events to stdout by default.
package main
import (
"encoding/base64"
"encoding/json"
"flag"
"fmt"
"os"
"strings"
"time"
"github.com/matrix-org/dendrite/roomserver/api"
"github.com/matrix-org/gomatrixserverlib"
"golang.org/x/crypto/ed25519"
)
const usage = `Usage: %s
Generate a list of matrix room events for load testing.
Writes the events to stdout separated by new lines
Arguments:
`
var (
serverName = flag.String("server-name", "localhost", "The name of the matrix server to generate events for")
keyID = flag.String("key-id", "ed25519:auto", "The ID of the key used to sign the events")
privateKeyString = flag.String("private-key", defaultKey, "Base64 encoded private key to sign events with")
roomID = flag.String("room-id", "!roomid:$SERVER_NAME", "The room ID to generate events in")
userID = flag.String("user-id", "@userid:$SERVER_NAME", "The user ID to use as the event sender")
messageCount = flag.Int("message-count", 10, "The number of m.room.messsage events to generate")
format = flag.String("Format", "InputRoomEvent", "The output format to use for the messages: InputRoomEvent or Event")
ver = flag.String("version", string(gomatrixserverlib.RoomVersionV1), "Room version to generate events as")
)
// By default we use a private key of 0.
const defaultKey = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
var privateKey ed25519.PrivateKey
var emptyString = ""
var now time.Time
var b gomatrixserverlib.EventBuilder
var eventID int
func main() {
flag.Usage = func() {
fmt.Fprintf(os.Stderr, usage, os.Args[0])
flag.PrintDefaults()
}
flag.Parse()
*userID = strings.Replace(*userID, "$SERVER_NAME", *serverName, 1)
*roomID = strings.Replace(*roomID, "$SERVER_NAME", *serverName, 1)
// Decode the ed25519 private key.
privateKeyBytes, err := base64.RawStdEncoding.DecodeString(*privateKeyString)
if err != nil {
panic(err)
}
privateKey = ed25519.PrivateKey(privateKeyBytes)
// Build a m.room.create event.
b.Sender = *userID
b.RoomID = *roomID
b.Type = "m.room.create"
b.StateKey = &emptyString
b.SetContent(map[string]string{"creator": *userID}) // nolint: errcheck
create := buildAndOutput()
// Build a m.room.member event.
b.Type = "m.room.member"
b.StateKey = userID
b.SetContent(map[string]string{"membership": gomatrixserverlib.Join}) // nolint: errcheck
b.AuthEvents = []gomatrixserverlib.EventReference{create}
member := buildAndOutput()
// Build a number of m.room.message events.
b.Type = "m.room.message"
b.StateKey = nil
b.SetContent(map[string]string{"body": "Test Message"}) // nolint: errcheck
b.AuthEvents = []gomatrixserverlib.EventReference{create, member}
for i := 0; i < *messageCount; i++ {
buildAndOutput()
}
}
// Build an event and write the event to the output.
func buildAndOutput() gomatrixserverlib.EventReference {
eventID++
now = time.Unix(0, 0)
name := gomatrixserverlib.ServerName(*serverName)
key := gomatrixserverlib.KeyID(*keyID)
event, err := b.Build(
now, name, key, privateKey,
gomatrixserverlib.RoomVersion(*ver),
)
if err != nil {
panic(err)
}
writeEvent(event)
reference := event.EventReference()
b.PrevEvents = []gomatrixserverlib.EventReference{reference}
b.Depth++
return reference
}
// Write an event to the output.
func writeEvent(event *gomatrixserverlib.Event) {
encoder := json.NewEncoder(os.Stdout)
if *format == "InputRoomEvent" {
var ire api.InputRoomEvent
ire.Kind = api.KindNew
ire.Event = event.Headered(gomatrixserverlib.RoomVersion(*ver))
authEventIDs := []string{}
for _, ref := range b.AuthEvents.([]gomatrixserverlib.EventReference) {
authEventIDs = append(authEventIDs, ref.EventID)
}
ire.AuthEventIDs = authEventIDs
if err := encoder.Encode(ire); err != nil {
panic(err)
}
} else if *format == "Event" {
if err := encoder.Encode(event); err != nil {
panic(err)
}
} else {
panic(fmt.Errorf("Format %q is not valid, must be %q or %q", *format, "InputRoomEvent", "Event"))
}
}

1
cmd/dendrite-demo-libp2p/main.go
View file

@ -197,6 +197,7 @@ func main() {
base.Base.PublicFederationAPIMux,
base.Base.PublicKeyAPIMux,
base.Base.PublicMediaAPIMux,
base.Base.SynapseAdminMux,
)
if err := mscs.Enable(&base.Base, &monolith); err != nil {
logrus.WithError(err).Fatalf("Failed to enable MSCs")

1
cmd/dendrite-demo-pinecone/main.go
View file

@ -210,6 +210,7 @@ func main() {
base.PublicFederationAPIMux,
base.PublicKeyAPIMux,
base.PublicMediaAPIMux,
base.SynapseAdminMux,
)
wsUpgrader := websocket.Upgrader{

53
cmd/dendrite-demo-yggdrasil/convert/25519.go
View file

@ -1,53 +0,0 @@
// Copyright 2019 Google LLC
//
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file or at
// https://developers.google.com/open-source/licenses/bsd
//
// Original code from https://github.com/FiloSottile/age/blob/bbab440e198a4d67ba78591176c7853e62d29e04/internal/age/ssh.go
package convert
import (
"crypto/ed25519"
"crypto/sha512"
"math/big"
"golang.org/x/crypto/curve25519"
)
var curve25519P, _ = new(big.Int).SetString("57896044618658097711785492504343953926634992332820282019728792003956564819949", 10)
func Ed25519PrivateKeyToCurve25519(pk ed25519.PrivateKey) []byte {
h := sha512.New()
_, _ = h.Write(pk.Seed())
out := h.Sum(nil)
return out[:curve25519.ScalarSize]
}
func Ed25519PublicKeyToCurve25519(pk ed25519.PublicKey) []byte {
// ed25519.PublicKey is a little endian representation of the y-coordinate,
// with the most significant bit set based on the sign of the x-coordinate.
bigEndianY := make([]byte, ed25519.PublicKeySize)
for i, b := range pk {
bigEndianY[ed25519.PublicKeySize-i-1] = b
}
bigEndianY[0] &= 0b0111_1111
// The Montgomery u-coordinate is derived through the bilinear map
// u = (1 + y) / (1 - y)
// See https://blog.filippo.io/using-ed25519-keys-for-encryption.
y := new(big.Int).SetBytes(bigEndianY)
denom := big.NewInt(1)
denom.ModInverse(denom.Sub(denom, y), curve25519P) // 1 / (1 - y)
u := y.Mul(y.Add(y, big.NewInt(1)), denom)
u.Mod(u, curve25519P)
out := make([]byte, curve25519.PointSize)
uBytes := u.Bytes()
for i, b := range uBytes {
out[len(uBytes)-i-1] = b
}
return out
}

51
cmd/dendrite-demo-yggdrasil/convert/25519_test.go
View file

@ -1,51 +0,0 @@
// Copyright 2020 The Matrix.org Foundation C.I.C.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package convert
import (
"bytes"
"crypto/ed25519"
"encoding/hex"
"testing"
"golang.org/x/crypto/curve25519"
)
func TestKeyConversion(t *testing.T) {
edPub, edPriv, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
t.Log("Signing public:", hex.EncodeToString(edPub))
t.Log("Signing private:", hex.EncodeToString(edPriv))
cuPriv := Ed25519PrivateKeyToCurve25519(edPriv)
t.Log("Encryption private:", hex.EncodeToString(cuPriv))
cuPub := Ed25519PublicKeyToCurve25519(edPub)
t.Log("Converted encryption public:", hex.EncodeToString(cuPub))
var realPub, realPriv [32]byte
copy(realPriv[:32], cuPriv[:32])
curve25519.ScalarBaseMult(&realPub, &realPriv)
t.Log("Scalar-multed encryption public:", hex.EncodeToString(realPub[:]))
if !bytes.Equal(realPriv[:], cuPriv[:]) {
t.Fatal("Private keys should be equal (this means the test is broken)")
}
if !bytes.Equal(realPub[:], cuPub[:]) {
t.Fatal("Public keys should be equal")
}
}

29
cmd/dendrite-demo-yggdrasil/main.go
View file

@ -56,21 +56,23 @@ func main() {
flag.Parse()
internal.SetupPprof()
ygg, err := yggconn.Setup(*instanceName, ".")
ygg, err := yggconn.Setup(*instanceName, ".", *instancePeer)
if err != nil {
panic(err)
}
ygg.SetMulticastEnabled(true)
if instancePeer != nil && *instancePeer != "" {
if err = ygg.SetStaticPeer(*instancePeer); err != nil {
logrus.WithError(err).Error("Failed to set static peer")
/*
ygg.SetMulticastEnabled(true)
if instancePeer != nil && *instancePeer != "" {
if err = ygg.SetStaticPeer(*instancePeer); err != nil {
logrus.WithError(err).Error("Failed to set static peer")
}
}
}
*/
cfg := &config.Dendrite{}
cfg.Defaults()
cfg.Global.ServerName = gomatrixserverlib.ServerName(ygg.DerivedServerName())
cfg.Global.PrivateKey = ygg.SigningPrivateKey()
cfg.Global.PrivateKey = ygg.PrivateKey()
cfg.Global.KeyID = gomatrixserverlib.KeyID(signing.KeyID)
cfg.Global.Kafka.UseNaffka = true
cfg.UserAPI.AccountDatabase.ConnectionString = config.DataSource(fmt.Sprintf("file:%s-account.db", *instanceName))
@ -117,18 +119,6 @@ func main() {
base, federation, rsAPI, keyRing, true,
)
ygg.SetSessionFunc(func(address string) {
req := &api.PerformServersAliveRequest{
Servers: []gomatrixserverlib.ServerName{
gomatrixserverlib.ServerName(address),
},
}
res := &api.PerformServersAliveResponse{}
if err := fsAPI.PerformServersAlive(context.TODO(), req, res); err != nil {
logrus.WithError(err).Error("Failed to send wake-up message to newly connected node")
}
})
rsComponent.SetFederationSenderAPI(fsAPI)
monolith := setup.Monolith{
@ -154,6 +144,7 @@ func main() {
base.PublicFederationAPIMux,
base.PublicKeyAPIMux,
base.PublicMediaAPIMux,
base.SynapseAdminMux,
)
if err := mscs.Enable(base, &monolith); err != nil {
logrus.WithError(err).Fatalf("Failed to enable MSCs")

1
cmd/dendrite-demo-yggdrasil/yggconn/client.go
View file

@ -51,7 +51,6 @@ func (n *Node) CreateFederationClient(
ResponseHeaderTimeout: 10 * time.Second,
IdleConnTimeout: 30 * time.Second,
DialContext: n.DialerContext,
TLSClientConfig: n.tlsConfig,
},
},
)

206
cmd/dendrite-demo-yggdrasil/yggconn/node.go
View file

@ -17,7 +17,6 @@ package yggconn
import (
"context"
"crypto/ed25519"
"crypto/tls"
"encoding/hex"
"encoding/json"
"fmt"
@ -26,60 +25,48 @@ import (
"net"
"os"
"strings"
"sync"
"time"
"github.com/lucas-clemente/quic-go"
"github.com/matrix-org/dendrite/cmd/dendrite-demo-yggdrasil/convert"
"github.com/matrix-org/gomatrixserverlib"
"go.uber.org/atomic"
"github.com/neilalexander/utp"
ironwoodtypes "github.com/Arceliar/ironwood/types"
yggdrasilconfig "github.com/yggdrasil-network/yggdrasil-go/src/config"
yggdrasilcore "github.com/yggdrasil-network/yggdrasil-go/src/core"
yggdrasildefaults "github.com/yggdrasil-network/yggdrasil-go/src/defaults"
yggdrasilmulticast "github.com/yggdrasil-network/yggdrasil-go/src/multicast"
"github.com/yggdrasil-network/yggdrasil-go/src/yggdrasil"
gologme "github.com/gologme/log"
)
type Node struct {
core *yggdrasil.Core
config *yggdrasilconfig.NodeConfig
state *yggdrasilconfig.NodeState
multicast *yggdrasilmulticast.Multicast
log *gologme.Logger
listener quic.Listener
tlsConfig *tls.Config
quicConfig *quic.Config
sessions sync.Map // string -> *session
sessionCount atomic.Uint32
sessionFunc func(address string)
coords sync.Map // string -> yggdrasil.Coords
incoming chan QUICStream
NewSession func(remote gomatrixserverlib.ServerName)
core *yggdrasilcore.Core
config *yggdrasilconfig.NodeConfig
multicast *yggdrasilmulticast.Multicast
log *gologme.Logger
listener quic.Listener
utpSocket *utp.Socket
incoming chan net.Conn
}
func (n *Node) Dialer(_, address string) (net.Conn, error) {
func (n *Node) DialerContext(ctx context.Context, _, address string) (net.Conn, error) {
tokens := strings.Split(address, ":")
raw, err := hex.DecodeString(tokens[0])
if err != nil {
return nil, fmt.Errorf("hex.DecodeString: %w", err)
}
converted := convert.Ed25519PublicKeyToCurve25519(ed25519.PublicKey(raw))
convhex := hex.EncodeToString(converted)
return n.Dial("curve25519", convhex)
pk := make(ironwoodtypes.Addr, ed25519.PublicKeySize)
copy(pk, raw[:])
return n.utpSocket.DialAddrContext(ctx, pk)
}
func (n *Node) DialerContext(ctx context.Context, network, address string) (net.Conn, error) {
return n.Dialer(network, address)
}
func Setup(instanceName, storageDirectory string) (*Node, error) {
func Setup(instanceName, storageDirectory, peerURI string) (*Node, error) {
n := &Node{
core: &yggdrasil.Core{},
config: yggdrasilconfig.GenerateConfig(),
core: &yggdrasilcore.Core{},
config: yggdrasildefaults.GenerateConfig(),
multicast: &yggdrasilmulticast.Multicast{},
log: gologme.New(os.Stdout, "YGG ", log.Flags()),
incoming: make(chan QUICStream),
incoming: make(chan net.Conn),
}
yggfile := fmt.Sprintf("%s/%s-yggdrasil.conf", storageDirectory, instanceName)
@ -93,24 +80,11 @@ func Setup(instanceName, storageDirectory string) (*Node, error) {
}
}
n.core.SetCoordChangeCallback(func(old, new yggdrasil.Coords) {
fmt.Println("COORDINATE CHANGE!")
fmt.Println("Old:", old)
fmt.Println("New:", new)
n.sessions.Range(func(k, v interface{}) bool {
if s, ok := v.(*session); ok {
fmt.Println("Killing session", k)
s.kill()
}
return true
})
})
n.config.Peers = []string{}
if peerURI != "" {
n.config.Peers = append(n.config.Peers, peerURI)
}
n.config.AdminListen = "none"
n.config.MulticastInterfaces = []string{}
n.config.EncryptionPrivateKey = hex.EncodeToString(n.EncryptionPrivateKey())
n.config.EncryptionPublicKey = hex.EncodeToString(n.EncryptionPublicKey())
j, err := json.MarshalIndent(n.config, "", " ")
if err != nil {
@ -123,34 +97,22 @@ func Setup(instanceName, storageDirectory string) (*Node, error) {
n.log.EnableLevel("error")
n.log.EnableLevel("warn")
n.log.EnableLevel("info")
n.state, err = n.core.Start(n.config, n.log)
if err = n.core.Start(n.config, n.log); err != nil {
panic(err)
}
n.utpSocket, err = utp.NewSocketFromPacketConnNoClose(n.core)
if err != nil {
panic(err)
}
if err = n.multicast.Init(n.core, n.state, n.log, nil); err != nil {
if err = n.multicast.Init(n.core, n.config, n.log, nil); err != nil {
panic(err)
}
if err = n.multicast.Start(); err != nil {
panic(err)
}
n.tlsConfig = n.generateTLSConfig()
n.quicConfig = &quic.Config{
MaxIncomingStreams: 0,
MaxIncomingUniStreams: 0,
KeepAlive: true,
MaxIdleTimeout: time.Minute * 30,
HandshakeTimeout: time.Second * 15,
}
copy(n.quicConfig.StatelessResetKey, n.EncryptionPublicKey())
n.log.Println("Public curve25519:", n.core.EncryptionPublicKey())
n.log.Println("Public ed25519:", n.core.SigningPublicKey())
go func() {
time.Sleep(time.Second)
n.listenFromYgg()
}()
n.log.Println("Public key:", n.core.PublicKey())
go n.listenFromYgg()
return n, nil
}
@ -163,64 +125,33 @@ func (n *Node) Stop() {
}
func (n *Node) DerivedServerName() string {
return hex.EncodeToString(n.SigningPublicKey())
return hex.EncodeToString(n.PublicKey())
}
func (n *Node) DerivedSessionName() string {
return hex.EncodeToString(n.EncryptionPublicKey())
func (n *Node) PrivateKey() ed25519.PrivateKey {
sk := make(ed25519.PrivateKey, ed25519.PrivateKeySize)
sb, err := hex.DecodeString(n.config.PrivateKey)
if err == nil {
copy(sk, sb[:])
} else {
panic(err)
}
return sk
}
func (n *Node) EncryptionPublicKey() []byte {
edkey := n.SigningPublicKey()
return convert.Ed25519PublicKeyToCurve25519(edkey)
}
func (n *Node) EncryptionPrivateKey() []byte {
edkey := n.SigningPrivateKey()
return convert.Ed25519PrivateKeyToCurve25519(edkey)
}
func (n *Node) SigningPublicKey() ed25519.PublicKey {
pubBytes, _ := hex.DecodeString(n.config.SigningPublicKey)
return ed25519.PublicKey(pubBytes)
}
func (n *Node) SigningPrivateKey() ed25519.PrivateKey {
privBytes, _ := hex.DecodeString(n.config.SigningPrivateKey)
return ed25519.PrivateKey(privBytes)
}
func (n *Node) SetSessionFunc(f func(address string)) {
n.sessionFunc = f
func (n *Node) PublicKey() ed25519.PublicKey {
return n.core.PublicKey()
}
func (n *Node) PeerCount() int {
return len(n.core.GetPeers()) - 1
}
func (n *Node) SessionCount() int {
return int(n.sessionCount.Load())
return len(n.core.GetPeers())
}
func (n *Node) KnownNodes() []gomatrixserverlib.ServerName {
nodemap := map[string]struct{}{
//"b5ae50589e50991dd9dd7d59c5c5f7a4521e8da5b603b7f57076272abc58b374": {},
nodemap := map[string]struct{}{}
for _, peer := range n.core.GetPeers() {
nodemap[hex.EncodeToString(peer.Key)] = struct{}{}
}
for _, peer := range n.core.GetSwitchPeers() {
nodemap[hex.EncodeToString(peer.SigPublicKey[:])] = struct{}{}
}
n.sessions.Range(func(_, v interface{}) bool {
session, ok := v.(quic.Session)
if !ok {
return true
}
if len(session.ConnectionState().PeerCertificates) != 1 {
return true
}
subjectName := session.ConnectionState().PeerCertificates[0].Subject.CommonName
nodemap[subjectName] = struct{}{}
return true
})
var nodes []gomatrixserverlib.ServerName
for node := range nodemap {
nodes = append(nodes, gomatrixserverlib.ServerName(node))
@ -229,53 +160,22 @@ func (n *Node) KnownNodes() []gomatrixserverlib.ServerName {
}
func (n *Node) SetMulticastEnabled(enabled bool) {
if enabled {
n.config.MulticastInterfaces = []string{".*"}
} else {
n.config.MulticastInterfaces = []string{}
}
n.multicast.UpdateConfig(n.config)
if !enabled {
n.DisconnectMulticastPeers()
}
// TODO: There's no dynamic reconfiguration in Yggdrasil v0.4
// so we need a solution for this.
}
func (n *Node) DisconnectMulticastPeers() {
for _, sp := range n.core.GetSwitchPeers() {
if !strings.HasPrefix(sp.Endpoint, "fe80") {
continue
}
if err := n.core.DisconnectPeer(sp.Port); err != nil {
n.log.Printf("Failed to disconnect port %d: %s", sp.Port, err)
}
}
// TODO: There's no dynamic reconfiguration in Yggdrasil v0.4
// so we need a solution for this.
}
func (n *Node) DisconnectNonMulticastPeers() {
for _, sp := range n.core.GetSwitchPeers() {
if strings.HasPrefix(sp.Endpoint, "fe80") {
continue
}
if err := n.core.DisconnectPeer(sp.Port); err != nil {
n.log.Printf("Failed to disconnect port %d: %s", sp.Port, err)
}
}
// TODO: There's no dynamic reconfiguration in Yggdrasil v0.4
// so we need a solution for this.
}
func (n *Node) SetStaticPeer(uri string) error {
n.config.Peers = []string{}
n.core.UpdateConfig(n.config)
n.DisconnectNonMulticastPeers()
if uri != "" {
n.log.Infoln("Adding static peer", uri)
if err := n.core.AddPeer(uri, ""); err != nil {
n.log.Warnln("Adding static peer failed:", err)
return err
}
if err := n.core.CallPeer(uri, ""); err != nil {
n.log.Warnln("Calling static peer failed:", err)
return err
}
}
// TODO: There's no dynamic reconfiguration in Yggdrasil v0.4
// so we need a solution for this.
return nil
}

235
cmd/dendrite-demo-yggdrasil/yggconn/session.go
View file

@ -16,94 +16,17 @@ package yggconn
import (
"context"
"crypto/rand"
"crypto/rsa"
"crypto/tls"
"crypto/x509"
"crypto/x509/pkix"
"encoding/hex"
"encoding/pem"
"errors"
"fmt"
"math/big"
"net"
"time"
"github.com/lucas-clemente/quic-go"
"github.com/yggdrasil-network/yggdrasil-go/src/crypto"
"github.com/yggdrasil-network/yggdrasil-go/src/yggdrasil"
)
type session struct {
node *Node
session quic.Session
address string
context context.Context
cancel context.CancelFunc
}
func (n *Node) newSession(sess quic.Session, address string) *session {
ctx, cancel := context.WithCancel(context.TODO())
return &session{
node: n,
session: sess,
address: address,
context: ctx,
cancel: cancel,
}
}
func (s *session) kill() {
s.cancel()
}
func (n *Node) listenFromYgg() {
var err error
n.listener, err = quic.Listen(
n.core, // yggdrasil.PacketConn
n.tlsConfig, // TLS config
n.quicConfig, // QUIC config
)
if err != nil {
panic(err)
}
for {
n.log.Infoln("Waiting to accept QUIC sessions")
session, err := n.listener.Accept(context.TODO())
conn, err := n.utpSocket.Accept()
if err != nil {
n.log.Println("n.listener.Accept:", err)
n.log.Println("n.utpSocket.Accept:", err)
return
}
if len(session.ConnectionState().PeerCertificates) != 1 {
_ = session.CloseWithError(0, "expected a peer certificate")
continue
}
address := session.ConnectionState().PeerCertificates[0].DNSNames[0]
n.log.Infoln("Accepted connection from", address)
go n.newSession(session, address).listenFromQUIC()
go n.sessionFunc(address)
}
}
func (s *session) listenFromQUIC() {
if existing, ok := s.node.sessions.Load(s.address); ok {
if existingSession, ok := existing.(*session); ok {
fmt.Println("Killing existing session to replace", s.address)
existingSession.kill()
}
}
s.node.sessionCount.Inc()
s.node.sessions.Store(s.address, s)
defer s.node.sessions.Delete(s.address)
defer s.node.sessionCount.Dec()
for {
st, err := s.session.AcceptStream(s.context)
if err != nil {
s.node.log.Println("session.AcceptStream:", err)
return
}
s.node.incoming <- QUICStream{st, s.session}
n.incoming <- conn
}
}
@ -129,155 +52,5 @@ func (n *Node) Dial(network, address string) (net.Conn, error) {
// Implements http.Transport.DialContext
func (n *Node) DialContext(ctx context.Context, network, address string) (net.Conn, error) {
s, ok1 := n.sessions.Load(address)
session, ok2 := s.(*session)
if !ok1 || !ok2 {
// First of all, check if we think we know the coords of this
// node. If we do then we'll try to dial to it directly. This
// will either succeed or fail.
if v, ok := n.coords.Load(address); ok {
coords, ok := v.(yggdrasil.Coords)
if !ok {
n.coords.Delete(address)
return nil, errors.New("should have found yggdrasil.Coords but didn't")
}
n.log.Infof("Coords %s for %q cached, trying to dial", coords.String(), address)
var err error
// We think we know the coords. Try to dial the node.
if session, err = n.tryDial(address, coords); err != nil {
// We thought we knew the coords but it didn't result
// in a successful dial. Nuke them from the cache.
n.coords.Delete(address)
n.log.Infof("Cached coords %s for %q failed", coords.String(), address)
}
}
// We either don't know the coords for the node, or we failed
// to dial it before, in which case try to resolve the coords.
if _, ok := n.coords.Load(address); !ok {
var coords yggdrasil.Coords
var err error
// First look and see if the node is something that we already
// know about from our direct switch peers.
for _, peer := range n.core.GetSwitchPeers() {
if peer.PublicKey.String() == address {
coords = peer.Coords
n.log.Infof("%q is a direct peer, coords are %s", address, coords.String())
n.coords.Store(address, coords)
break
}
}
// If it isn' a node that we know directly then try to search
// the network.
if coords == nil {
n.log.Infof("Searching for coords for %q", address)
dest, derr := hex.DecodeString(address)
if derr != nil {
return nil, derr
}
if len(dest) != crypto.BoxPubKeyLen {
return nil, errors.New("invalid key length supplied")
}
var pubKey crypto.BoxPubKey
copy(pubKey[:], dest)
nodeID := crypto.GetNodeID(&pubKey)
nodeMask := &crypto.NodeID{}
for i := range nodeMask {
nodeMask[i] = 0xFF
}
fmt.Println("Resolving coords")
coords, err = n.core.Resolve(nodeID, nodeMask)
if err != nil {
return nil, fmt.Errorf("n.core.Resolve: %w", err)
}
fmt.Println("Found coords:", coords)
n.coords.Store(address, coords)
}
// We now know the coords in theory. Let's try dialling the
// node again.
if session, err = n.tryDial(address, coords); err != nil {
return nil, fmt.Errorf("n.tryDial: %w", err)
}
}
}
if session == nil {
return nil, fmt.Errorf("should have found session but didn't")
}
st, err := session.session.OpenStream()
if err != nil {
n.log.Println("session.OpenStream:", err)
_ = session.session.CloseWithError(0, "expected to be able to open session")
return nil, err
}
return QUICStream{st, session.session}, nil
}
func (n *Node) tryDial(address string, coords yggdrasil.Coords) (*session, error) {
quicSession, err := quic.Dial(
n.core, // yggdrasil.PacketConn
coords, // dial address
address, // dial SNI
n.tlsConfig, // TLS config
n.quicConfig, // QUIC config
)
if err != nil {
return nil, err
}
if len(quicSession.ConnectionState().PeerCertificates) != 1 {
_ = quicSession.CloseWithError(0, "expected a peer certificate")
return nil, errors.New("didn't receive a peer certificate")
}
if len(quicSession.ConnectionState().PeerCertificates[0].DNSNames) != 1 {
_ = quicSession.CloseWithError(0, "expected a DNS name")
return nil, errors.New("didn't receive a DNS name")
}
if gotAddress := quicSession.ConnectionState().PeerCertificates[0].DNSNames[0]; address != gotAddress {
_ = quicSession.CloseWithError(0, "you aren't the host I was hoping for")
return nil, fmt.Errorf("expected %q but dialled %q", address, gotAddress)
}
session := n.newSession(quicSession, address)
go session.listenFromQUIC()
go n.sessionFunc(address)
return session, nil
}
func (n *Node) generateTLSConfig() *tls.Config {
key, err := rsa.GenerateKey(rand.Reader, 1024)
if err != nil {
panic(err)
}
template := x509.Certificate{
Subject: pkix.Name{
CommonName: n.DerivedServerName(),
},
SerialNumber: big.NewInt(1),
NotAfter: time.Now().Add(time.Hour * 24 * 365),
DNSNames: []string{n.DerivedSessionName()},
}
certDER, err := x509.CreateCertificate(rand.Reader, &template, &template, &key.PublicKey, key)
if err != nil {
panic(err)
}
keyPEM := pem.EncodeToMemory(&pem.Block{Type: "RSA PRIVATE KEY", Bytes: x509.MarshalPKCS1PrivateKey(key)})
certPEM := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: certDER})
tlsCert, err := tls.X509KeyPair(certPEM, keyPEM)
if err != nil {
panic(err)
}
return &tls.Config{
Certificates: []tls.Certificate{tlsCert},
NextProtos: []string{"quic-matrix-ygg"},
InsecureSkipVerify: true,
ClientAuth: tls.RequireAnyClientCert,
GetClientCertificate: func(info *tls.CertificateRequestInfo) (*tls.Certificate, error) {
return &tlsCert, nil
},
}
return n.utpSocket.DialContext(ctx, network, address)
}

20
cmd/dendrite-demo-yggdrasil/yggconn/stream.go
View file

@ -1,20 +0,0 @@
package yggconn
import (
"net"
"github.com/lucas-clemente/quic-go"
)
type QUICStream struct {
quic.Stream
session quic.Session
}
func (s QUICStream) LocalAddr() net.Addr {
return s.session.LocalAddr()
}
func (s QUICStream) RemoteAddr() net.Addr {
return s.session.RemoteAddr()
}

1
cmd/dendrite-monolith-server/main.go
View file

@ -149,6 +149,7 @@ func main() {
base.PublicFederationAPIMux,
base.PublicKeyAPIMux,
base.PublicMediaAPIMux,
base.SynapseAdminMux,
)
if len(base.Cfg.MSCs.MSCs) > 0 {

2
cmd/dendrite-polylith-multi/personalities/clientapi.go
View file

@ -33,7 +33,7 @@ func ClientAPI(base *setup.BaseDendrite, cfg *config.Dendrite) {
keyAPI := base.KeyServerHTTPClient()
clientapi.AddPublicRoutes(
base.PublicClientAPIMux, &base.Cfg.ClientAPI, accountDB, federation,
base.PublicClientAPIMux, base.SynapseAdminMux, &base.Cfg.ClientAPI, accountDB, federation,
rsAPI, eduInputAPI, asQuery, transactions.New(), fsAPI, userAPI, keyAPI, nil,
&cfg.MSCs,
)

503
cmd/dendrite-upgrade-tests/main.go Normal file
View file

@ -0,0 +1,503 @@
package main
import (
"bytes"
"context"
"encoding/json"
"flag"
"fmt"
"io/ioutil"
"log"
"net/http"
"os"
"path"
"regexp"
"runtime"
"sort"
"strconv"
"strings"
"sync"
"time"
"github.com/Masterminds/semver/v3"
"github.com/codeclysm/extract"
"github.com/docker/docker/api/types"
"github.com/docker/docker/api/types/container"
"github.com/docker/docker/api/types/filters"
"github.com/docker/docker/api/types/mount"
"github.com/docker/docker/api/types/volume"
"github.com/docker/docker/client"
"github.com/docker/go-connections/nat"
)
var (
flagTempDir = flag.String("tmp", "tmp", "Path to temporary directory to dump tarballs to")
flagFrom = flag.String("from", "HEAD-1", "The version to start from e.g '0.3.1'. If 'HEAD-N' then starts N versions behind HEAD.")
flagTo = flag.String("to", "HEAD", "The version to end on e.g '0.3.3'.")
flagBuildConcurrency = flag.Int("build-concurrency", runtime.NumCPU(), "The amount of build concurrency when building images")
flagHead = flag.String("head", "", "Location to a dendrite repository to treat as HEAD instead of Github")
flagDockerHost = flag.String("docker-host", "localhost", "The hostname of the docker client. 'localhost' if running locally, 'host.docker.internal' if running in Docker.")
alphaNumerics = regexp.MustCompile("[^a-zA-Z0-9]+")
)
const HEAD = "HEAD"
// Embed the Dockerfile to use when building dendrite versions.
// We cannot use the dockerfile associated with the repo with each version sadly due to changes in
// Docker versions. Specifically, earlier Dendrite versions are incompatible with newer Docker clients
// due to the error:
// When using COPY with more than one source file, the destination must be a directory and end with a /
// We need to run a postgres anyway, so use the dockerfile associated with Complement instead.
const Dockerfile = `FROM golang:1.13-stretch as build
RUN apt-get update && apt-get install -y postgresql
WORKDIR /build
# Copy the build context to the repo as this is the right dendrite code. This is different to the
# Complement Dockerfile which wgets a branch.
COPY . .
RUN go build ./cmd/dendrite-monolith-server
RUN go build ./cmd/generate-keys
RUN go build ./cmd/generate-config
RUN ./generate-config --ci > dendrite.yaml
RUN ./generate-keys --private-key matrix_key.pem --tls-cert server.crt --tls-key server.key
# Replace the connection string with a single postgres DB, using user/db = 'postgres' and no password
RUN sed -i "s%connection_string:.*$%connection_string: postgresql://postgres@localhost/postgres?sslmode=disable%g" dendrite.yaml
# No password when connecting over localhost
RUN sed -i "s%127.0.0.1/32 md5%127.0.0.1/32 trust%g" /etc/postgresql/9.6/main/pg_hba.conf
# Bump up max conns for moar concurrency
RUN sed -i 's/max_connections = 100/max_connections = 2000/g' /etc/postgresql/9.6/main/postgresql.conf
RUN sed -i 's/max_open_conns:.*$/max_open_conns: 100/g' dendrite.yaml
# This entry script starts postgres, waits for it to be up then starts dendrite
RUN echo '\
#!/bin/bash -eu \n\
pg_lsclusters \n\
pg_ctlcluster 9.6 main start \n\
\n\
until pg_isready \n\
do \n\
echo "Waiting for postgres"; \n\
sleep 1; \n\
done \n\
\n\
sed -i "s/server_name: localhost/server_name: ${SERVER_NAME}/g" dendrite.yaml \n\
./dendrite-monolith-server --tls-cert server.crt --tls-key server.key --config dendrite.yaml \n\
' > run_dendrite.sh && chmod +x run_dendrite.sh
ENV SERVER_NAME=localhost
EXPOSE 8008 8448
CMD /build/run_dendrite.sh `
const dendriteUpgradeTestLabel = "dendrite_upgrade_test"
// downloadArchive downloads an arbitrary github archive of the form:
// https://github.com/matrix-org/dendrite/archive/v0.3.11.tar.gz
// and re-tarballs it without the top-level directory which contains branch information. It inserts
// the contents of `dockerfile` as a root file `Dockerfile` in the re-tarballed directory such that
// you can directly feed the retarballed archive to `ImageBuild` to have it run said dockerfile.
// Returns the tarball buffer on success.
func downloadArchive(cli *http.Client, tmpDir, archiveURL string, dockerfile []byte) (*bytes.Buffer, error) {
resp, err := cli.Get(archiveURL)
if err != nil {
return nil, err
}
// nolint:errcheck
defer resp.Body.Close()
if resp.StatusCode != 200 {
return nil, fmt.Errorf("got HTTP %d", resp.StatusCode)
}
_ = os.RemoveAll(tmpDir)
if err = os.Mkdir(tmpDir, os.ModePerm); err != nil {
return nil, fmt.Errorf("failed to make temporary directory: %s", err)
}
// nolint:errcheck
defer os.RemoveAll(tmpDir)
// dump the tarball temporarily, stripping the top-level directory
err = extract.Archive(context.Background(), resp.Body, tmpDir, func(inPath string) string {
// remove top level
segments := strings.Split(inPath, "/")
return strings.Join(segments[1:], "/")
})
if err != nil {
return nil, err
}
// add top level Dockerfile
err = ioutil.WriteFile(path.Join(tmpDir, "Dockerfile"), dockerfile, os.ModePerm)
if err != nil {
return nil, fmt.Errorf("failed to inject /Dockerfile: %w", err)
}
// now re-tarball it :/
var tarball bytes.Buffer
err = compress(tmpDir, &tarball)
if err != nil {
return nil, err
}
return &tarball, nil
}
// buildDendrite builds Dendrite on the branchOrTagName given. Returns the image ID or an error
func buildDendrite(httpClient *http.Client, dockerClient *client.Client, tmpDir, branchOrTagName string) (string, error) {
var tarball *bytes.Buffer
var err error
// If a custom HEAD location is given, use that, else pull from github. Mostly useful for CI
// where we want to use the working directory.
if branchOrTagName == HEAD && *flagHead != "" {
log.Printf("%s: Using %s as HEAD", branchOrTagName, *flagHead)
// add top level Dockerfile
err = ioutil.WriteFile(path.Join(*flagHead, "Dockerfile"), []byte(Dockerfile), os.ModePerm)
if err != nil {
return "", fmt.Errorf("Custom HEAD: failed to inject /Dockerfile: %w", err)
}
// now tarball it
var buffer bytes.Buffer
err = compress(*flagHead, &buffer)
if err != nil {
return "", fmt.Errorf("failed to tarball custom HEAD %s : %s", *flagHead, err)
}
tarball = &buffer
} else {
log.Printf("%s: Downloading version %s to %s\n", branchOrTagName, branchOrTagName, tmpDir)
// pull an archive, this contains a top-level directory which screws with the build context
// which we need to fix up post download
u := fmt.Sprintf("https://github.com/matrix-org/dendrite/archive/%s.tar.gz", branchOrTagName)
tarball, err = downloadArchive(httpClient, tmpDir, u, []byte(Dockerfile))
if err != nil {
return "", fmt.Errorf("failed to download archive %s: %w", u, err)
}
log.Printf("%s: %s => %d bytes\n", branchOrTagName, u, tarball.Len())
}
log.Printf("%s: Building version %s\n", branchOrTagName, branchOrTagName)
res, err := dockerClient.ImageBuild(context.Background(), tarball, types.ImageBuildOptions{
Tags: []string{"dendrite-upgrade"},
})
if err != nil {
return "", fmt.Errorf("failed to start building image: %s", err)
}
// nolint:errcheck
defer res.Body.Close()
decoder := json.NewDecoder(res.Body)
// {"aux":{"ID":"sha256:247082c717963bc2639fc2daed08838d67811ea12356cd4fda43e1ffef94f2eb"}}
var imageID string
for decoder.More() {
var dl struct {
Stream string `json:"stream"`
Aux map[string]interface{} `json:"aux"`
}
if err := decoder.Decode(&dl); err != nil {
return "", fmt.Errorf("failed to decode build image output line: %w", err)
}
log.Printf("%s: %s", branchOrTagName, dl.Stream)
if dl.Aux != nil {
imgID, ok := dl.Aux["ID"]
if ok {
imageID = imgID.(string)
}
}
}
return imageID, nil
}
func getAndSortVersionsFromGithub(httpClient *http.Client) (semVers []*semver.Version, err error) {
u := "https://api.github.com/repos/matrix-org/dendrite/tags"
res, err := httpClient.Get(u)
if err != nil {
return nil, err
}
if res.StatusCode != 200 {
return nil, fmt.Errorf("%s returned HTTP %d", u, res.StatusCode)
}
resp := []struct {
Name string `json:"name"`
}{}
if err = json.NewDecoder(res.Body).Decode(&resp); err != nil {
return nil, err
}
for _, r := range resp {
v, err := semver.NewVersion(r.Name)
if err != nil {
continue // not a semver, that's ok and isn't an error, we allow tags that aren't semvers
}
semVers = append(semVers, v)
}
sort.Sort(semver.Collection(semVers))
return semVers, nil
}
func calculateVersions(cli *http.Client, from, to string) []string {
semvers, err := getAndSortVersionsFromGithub(cli)
if err != nil {
log.Fatalf("failed to collect semvers from github: %s", err)
}
// snip the lower bound depending on --from
if from != "" {
if strings.HasPrefix(from, "HEAD-") {
var headN int
headN, err = strconv.Atoi(strings.TrimPrefix(from, "HEAD-"))
if err != nil {
log.Fatalf("invalid --from, try 'HEAD-1'")
}
if headN >= len(semvers) {
log.Fatalf("only have %d versions, but asked to go to HEAD-%d", len(semvers), headN)
}
if headN > 0 {
semvers = semvers[len(semvers)-headN:]
}
} else {
fromVer, err := semver.NewVersion(from)
if err != nil {
log.Fatalf("invalid --from: %s", err)
}
i := 0
for i = 0; i < len(semvers); i++ {
if semvers[i].LessThan(fromVer) {
continue
}
break
}
semvers = semvers[i:]
}
}
if to != "" && to != HEAD {
toVer, err := semver.NewVersion(to)
if err != nil {
log.Fatalf("invalid --to: %s", err)
}
var i int
for i = len(semvers) - 1; i >= 0; i-- {
if semvers[i].GreaterThan(toVer) {
continue
}
break
}
semvers = semvers[:i+1]
}
var versions []string
for _, sv := range semvers {
versions = append(versions, sv.Original())
}
if to == HEAD {
versions = append(versions, HEAD)
}
return versions
}
func buildDendriteImages(httpClient *http.Client, dockerClient *client.Client, baseTempDir string, concurrency int, branchOrTagNames []string) map[string]string {
// concurrently build all versions, this can be done in any order. The mutex protects the map
branchToImageID := make(map[string]string)
var mu sync.Mutex
var wg sync.WaitGroup
wg.Add(concurrency)
ch := make(chan string, len(branchOrTagNames))
for _, branchName := range branchOrTagNames {
ch <- branchName
}
close(ch)
for i := 0; i < concurrency; i++ {
go func() {
defer wg.Done()
for branchName := range ch {
tmpDir := baseTempDir + alphaNumerics.ReplaceAllString(branchName, "")
imgID, err := buildDendrite(httpClient, dockerClient, tmpDir, branchName)
if err != nil {
log.Fatalf("%s: failed to build dendrite image: %s", branchName, err)
}
mu.Lock()
branchToImageID[branchName] = imgID
mu.Unlock()
}
}()
}
wg.Wait()
return branchToImageID
}
func runImage(dockerClient *client.Client, volumeName, version, imageID string) (csAPIURL, containerID string, err error) {
log.Printf("%s: running image %s\n", version, imageID)
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
defer cancel()
body, err := dockerClient.ContainerCreate(ctx, &container.Config{
Image: imageID,
Env: []string{"SERVER_NAME=hs1"},
Labels: map[string]string{
dendriteUpgradeTestLabel: "yes",
},
}, &container.HostConfig{
PublishAllPorts: true,
Mounts: []mount.Mount{
{
Type: mount.TypeVolume,
Source: volumeName,
Target: "/var/lib/postgresql/9.6/main",
},
},
}, nil, nil, "dendrite_upgrade_test_"+version)
if err != nil {
return "", "", fmt.Errorf("failed to ContainerCreate: %s", err)
}
containerID = body.ID
err = dockerClient.ContainerStart(ctx, containerID, types.ContainerStartOptions{})
if err != nil {
return "", "", fmt.Errorf("failed to ContainerStart: %s", err)
}
inspect, err := dockerClient.ContainerInspect(ctx, containerID)
if err != nil {
return "", "", err
}
csapiPortInfo, ok := inspect.NetworkSettings.Ports[nat.Port("8008/tcp")]
if !ok {
return "", "", fmt.Errorf("port 8008 not exposed - exposed ports: %v", inspect.NetworkSettings.Ports)
}
baseURL := fmt.Sprintf("http://%s:%s", *flagDockerHost, csapiPortInfo[0].HostPort)
versionsURL := fmt.Sprintf("%s/_matrix/client/versions", baseURL)
// hit /versions to check it is up
var lastErr error
for i := 0; i < 500; i++ {
res, err := http.Get(versionsURL)
if err != nil {
lastErr = fmt.Errorf("GET %s => error: %s", versionsURL, err)
time.Sleep(50 * time.Millisecond)
continue
}
if res.StatusCode != 200 {
lastErr = fmt.Errorf("GET %s => HTTP %s", versionsURL, res.Status)
time.Sleep(50 * time.Millisecond)
continue
}
lastErr = nil
break
}
if lastErr != nil {
logs, err := dockerClient.ContainerLogs(context.Background(), containerID, types.ContainerLogsOptions{
ShowStdout: true,
ShowStderr: true,
})
// ignore errors when cannot get logs, it's just for debugging anyways
if err == nil {
logbody, err := ioutil.ReadAll(logs)
if err == nil {
log.Printf("Container logs:\n\n%s\n\n", string(logbody))
}
}
}
return baseURL, containerID, lastErr
}
func destroyContainer(dockerClient *client.Client, containerID string) {
err := dockerClient.ContainerRemove(context.TODO(), containerID, types.ContainerRemoveOptions{
Force: true,
})
if err != nil {
log.Printf("failed to remove container %s : %s", containerID, err)
}
}
func loadAndRunTests(dockerClient *client.Client, volumeName, v string, branchToImageID map[string]string) error {
csAPIURL, containerID, err := runImage(dockerClient, volumeName, v, branchToImageID[v])
if err != nil {
return fmt.Errorf("failed to run container for branch %v: %v", v, err)
}
defer destroyContainer(dockerClient, containerID)
log.Printf("URL %s -> %s \n", csAPIURL, containerID)
if err = runTests(csAPIURL, v); err != nil {
return fmt.Errorf("failed to run tests on version %s: %s", v, err)
}
return nil
}
func verifyTests(dockerClient *client.Client, volumeName string, versions []string, branchToImageID map[string]string) error {
lastVer := versions[len(versions)-1]
csAPIURL, containerID, err := runImage(dockerClient, volumeName, lastVer, branchToImageID[lastVer])
if err != nil {
return fmt.Errorf("failed to run container for branch %v: %v", lastVer, err)
}
defer destroyContainer(dockerClient, containerID)
return verifyTestsRan(csAPIURL, versions)
}
// cleanup old containers/volumes from a previous run
func cleanup(dockerClient *client.Client) {
// ignore all errors, we are just cleaning up and don't want to fail just because we fail to cleanup
containers, _ := dockerClient.ContainerList(context.Background(), types.ContainerListOptions{
Filters: label(dendriteUpgradeTestLabel),
})
for _, c := range containers {
s := time.Second
_ = dockerClient.ContainerStop(context.Background(), c.ID, &s)
_ = dockerClient.ContainerRemove(context.Background(), c.ID, types.ContainerRemoveOptions{
Force: true,
})
}
_ = dockerClient.VolumeRemove(context.Background(), "dendrite_upgrade_test", true)
}
func label(in string) filters.Args {
f := filters.NewArgs()
f.Add("label", in)
return f
}
func main() {
flag.Parse()
httpClient := &http.Client{
Timeout: 60 * time.Second,
}
dockerClient, err := client.NewClientWithOpts(client.FromEnv)
if err != nil {
log.Fatalf("failed to make docker client: %s", err)
}
if *flagFrom == "" {
flag.Usage()
os.Exit(1)
}
cleanup(dockerClient)
versions := calculateVersions(httpClient, *flagFrom, *flagTo)
log.Printf("Testing dendrite versions: %v\n", versions)
branchToImageID := buildDendriteImages(httpClient, dockerClient, *flagTempDir, *flagBuildConcurrency, versions)
// make a shared postgres volume
volume, err := dockerClient.VolumeCreate(context.Background(), volume.VolumeCreateBody{
Name: "dendrite_upgrade_test",
Labels: map[string]string{
dendriteUpgradeTestLabel: "yes",
},
})
if err != nil {
log.Fatalf("failed to make docker volume: %s", err)
}
failed := false
defer func() {
perr := recover()
log.Println("removing postgres volume")
verr := dockerClient.VolumeRemove(context.Background(), volume.Name, true)
if perr == nil {
perr = verr
}
if perr != nil {
panic(perr)
}
if failed {
os.Exit(1)
}
}()
// run through images sequentially
for _, v := range versions {
if err = loadAndRunTests(dockerClient, volume.Name, v, branchToImageID); err != nil {
log.Printf("failed to run tests for %v: %s\n", v, err)
failed = true
break
}
}
if err := verifyTests(dockerClient, volume.Name, versions, branchToImageID); err != nil {
log.Printf("failed to verify test results: %s", err)
failed = true
}
}

63
cmd/dendrite-upgrade-tests/tar.go Normal file
View file

@ -0,0 +1,63 @@
package main
import (
"archive/tar"
"compress/gzip"
"io"
"os"
"path/filepath"
"strings"
)
// From https://gist.github.com/mimoo/25fc9716e0f1353791f5908f94d6e726
// Modified to strip off top-level when compressing
func compress(src string, buf io.Writer) error {
// tar > gzip > buf
zr := gzip.NewWriter(buf)
tw := tar.NewWriter(zr)
// walk through every file in the folder
err := filepath.Walk(src, func(file string, fi os.FileInfo, e error) error {
// generate tar header
header, err := tar.FileInfoHeader(fi, file)
if err != nil {
return err
}
// must provide real name
// (see https://golang.org/src/archive/tar/common.go?#L626)
header.Name = strings.TrimPrefix(filepath.ToSlash(file), src+"/")
// write header
if err := tw.WriteHeader(header); err != nil {
return err
}
// if not a dir, write file content
if !fi.IsDir() {
data, err := os.Open(file)
if err != nil {
return err
}
if _, err = io.Copy(tw, data); err != nil {
return err
}
if err = data.Close(); err != nil {
return err
}
}
return nil
})
if err != nil {
return err
}
// produce tar
if err := tw.Close(); err != nil {
return err
}
// produce gzip
if err := zr.Close(); err != nil {
return err
}
//
return nil
}

192
cmd/dendrite-upgrade-tests/tests.go Normal file
View file

@ -0,0 +1,192 @@
package main
import (
"fmt"
"log"
"strings"
"github.com/matrix-org/gomatrix"
"github.com/matrix-org/gomatrixserverlib"
)
const userPassword = "this_is_a_long_password"
type user struct {
userID string
localpart string
client *gomatrix.Client
}
// runTests performs the following operations:
// - register alice and bob with branch name muxed into the localpart
// - create a DM room for the 2 users and exchange messages
// - create/join a public #global room and exchange messages
func runTests(baseURL, branchName string) error {
// register 2 users
users := []user{
{
localpart: "alice" + branchName,
},
{
localpart: "bob" + branchName,
},
}
for i, u := range users {
client, err := gomatrix.NewClient(baseURL, "", "")
if err != nil {
return err
}
resp, err := client.RegisterDummy(&gomatrix.ReqRegister{
Username: strings.ToLower(u.localpart),
Password: userPassword,
})
if err != nil {
return fmt.Errorf("failed to register %s: %s", u.localpart, err)
}
client, err = gomatrix.NewClient(baseURL, resp.UserID, resp.AccessToken)
if err != nil {
return err
}
users[i].client = client
users[i].userID = resp.UserID
}
// create DM room, join it and exchange messages
createRoomResp, err := users[0].client.CreateRoom(&gomatrix.ReqCreateRoom{
Preset: "trusted_private_chat",
Invite: []string{users[1].userID},
IsDirect: true,
})
if err != nil {
return fmt.Errorf("failed to create DM room: %s", err)
}
dmRoomID := createRoomResp.RoomID
if _, err = users[1].client.JoinRoom(dmRoomID, "", nil); err != nil {
return fmt.Errorf("failed to join DM room: %s", err)
}
msgs := []struct {
client *gomatrix.Client
text string
}{
{
client: users[0].client, text: "1: " + branchName,
},
{
client: users[1].client, text: "2: " + branchName,
},
{
client: users[0].client, text: "3: " + branchName,
},
{
client: users[1].client, text: "4: " + branchName,
},
}
for _, msg := range msgs {
_, err = msg.client.SendText(dmRoomID, msg.text)
if err != nil {
return fmt.Errorf("failed to send text in dm room: %s", err)
}
}
// attempt to create/join the shared public room
publicRoomID := ""
createRoomResp, err = users[0].client.CreateRoom(&gomatrix.ReqCreateRoom{
RoomAliasName: "global",
Preset: "public_chat",
})
if err != nil { // this is okay and expected if the room already exists and the aliases clash
// try to join it
_, domain, err2 := gomatrixserverlib.SplitID('@', users[0].userID)
if err2 != nil {
return fmt.Errorf("failed to split user ID: %s, %s", users[0].userID, err2)
}
joinRoomResp, err2 := users[0].client.JoinRoom(fmt.Sprintf("#global:%s", domain), "", nil)
if err2 != nil {
return fmt.Errorf("alice failed to join public room: %s", err2)
}
publicRoomID = joinRoomResp.RoomID
} else {
publicRoomID = createRoomResp.RoomID
}
if _, err = users[1].client.JoinRoom(publicRoomID, "", nil); err != nil {
return fmt.Errorf("bob failed to join public room: %s", err)
}
// send messages
for _, msg := range msgs {
_, err = msg.client.SendText(publicRoomID, "public "+msg.text)
if err != nil {
return fmt.Errorf("failed to send text in public room: %s", err)
}
}
log.Printf("OK! rooms(public=%s, dm=%s) users(%s, %s)\n", publicRoomID, dmRoomID, users[0].userID, users[1].userID)
return nil
}
// verifyTestsRan checks that the HS has the right rooms/messages
func verifyTestsRan(baseURL string, branchNames []string) error {
log.Println("Verifying tests....")
// check we can login as all users
var resp *gomatrix.RespLogin
for _, branchName := range branchNames {
client, err := gomatrix.NewClient(baseURL, "", "")
if err != nil {
return err
}
userLocalparts := []string{
"alice" + branchName,
"bob" + branchName,
}
for _, userLocalpart := range userLocalparts {
resp, err = client.Login(&gomatrix.ReqLogin{
Type: "m.login.password",
User: strings.ToLower(userLocalpart),
Password: userPassword,
})
if err != nil {
return fmt.Errorf("failed to login as %s: %s", userLocalpart, err)
}
if resp.AccessToken == "" {
return fmt.Errorf("failed to login, bad response: %+v", resp)
}
}
}
log.Println(" accounts exist: OK")
client, err := gomatrix.NewClient(baseURL, resp.UserID, resp.AccessToken)
if err != nil {
return err
}
_, domain, err := gomatrixserverlib.SplitID('@', client.UserID)
if err != nil {
return err
}
u := client.BuildURL("directory", "room", fmt.Sprintf("#global:%s", domain))
r := struct {
RoomID string `json:"room_id"`
}{}
err = client.MakeRequest("GET", u, nil, &r)
if err != nil {
return fmt.Errorf("failed to /directory: %s", err)
}
if r.RoomID == "" {
return fmt.Errorf("/directory lookup returned no room ID")
}
log.Println(" public room exists: OK")
history, err := client.Messages(r.RoomID, client.Store.LoadNextBatch(client.UserID), "", 'b', 100)
if err != nil {
return fmt.Errorf("failed to get /messages: %s", err)
}
// we expect 4 messages per version
msgCount := 0
for _, ev := range history.Chunk {
if ev.Type == "m.room.message" {
msgCount += 1
}
}
wantMsgCount := len(branchNames) * 4
if msgCount != wantMsgCount {
return fmt.Errorf("got %d messages in global room, want %d", msgCount, wantMsgCount)
}
log.Println(" messages exist: OK")
return nil
}

1
cmd/dendritejs-pinecone/main.go
View file

@ -215,6 +215,7 @@ func main() {
base.PublicFederationAPIMux,
base.PublicKeyAPIMux,
base.PublicMediaAPIMux,
base.SynapseAdminMux,
)
httpRouter := mux.NewRouter().SkipClean(true).UseEncodedPath()

1
cmd/dendritejs/main.go
View file

@ -236,6 +236,7 @@ func main() {
base.PublicFederationAPIMux,
base.PublicKeyAPIMux,
base.PublicMediaAPIMux,
base.SynapseAdminMux,
)
httpRouter := mux.NewRouter().SkipClean(true).UseEncodedPath()

93
cmd/kafka-producer/main.go
View file

@ -1,93 +0,0 @@
// Copyright 2017 Vector Creations Ltd
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
"bufio"
"flag"
"fmt"
"os"
"strings"
sarama "github.com/Shopify/sarama"
)
const usage = `Usage: %s
Reads a list of newline separated messages from stdin and writes them to a single partition in kafka.
Arguments:
`
var (
brokerList = flag.String("brokers", os.Getenv("KAFKA_PEERS"), "The comma separated list of brokers in the Kafka cluster. You can also set the KAFKA_PEERS environment variable")
topic = flag.String("topic", "", "REQUIRED: the topic to produce to")
partition = flag.Int("partition", 0, "The partition to produce to. All the messages will be written to this partition.")
)
func main() {
flag.Usage = func() {
fmt.Fprintf(os.Stderr, usage, os.Args[0])
flag.PrintDefaults()
}
flag.Parse()
if *brokerList == "" {
fmt.Fprintln(os.Stderr, "no -brokers specified. Alternatively, set the KAFKA_PEERS environment variable")
os.Exit(1)
}
if *topic == "" {
fmt.Fprintln(os.Stderr, "no -topic specified")
os.Exit(1)
}
config := sarama.NewConfig()
config.Producer.RequiredAcks = sarama.WaitForAll
config.Producer.Return.Successes = true
config.Producer.Partitioner = sarama.NewManualPartitioner
producer, err := sarama.NewSyncProducer(strings.Split(*brokerList, ","), config)
if err != nil {
fmt.Fprintln(os.Stderr, "Failed to open Kafka producer:", err)
os.Exit(1)
}
defer func() {
if err := producer.Close(); err != nil {
fmt.Fprintln(os.Stderr, "Failed to close Kafka producer cleanly:", err)
}
}()
scanner := bufio.NewScanner(os.Stdin)
for scanner.Scan() {
line := scanner.Bytes()
message := &sarama.ProducerMessage{
Topic: *topic,
Partition: int32(*partition),
Value: sarama.ByteEncoder(line),
}
if _, _, err := producer.SendMessage(message); err != nil {
fmt.Fprintln(os.Stderr, "Failed to send message:", err)
os.Exit(1)
}
}
if err := scanner.Err(); err != nil {
fmt.Fprintln(os.Stderr, "reading standard input:", err)
}
}

69
cmd/mediaapi-integration-tests/TESTS.md
View file

@ -1,69 +0,0 @@
# Media API Tests
## Implemented
* functional
* upload
* normal case
* download
* local file
* existing
* non-existing
* remote file
* existing
* thumbnail
* original file formats
* JPEG
* local file
* existing
* remote file
* existing
* cache
* cold
* hot
* pre-generation according to configuration
* scale
* crop
* dynamic generation
* cold cache
* larger than original
* scale
## TODO
* functional
* upload
* file too large
* 0-byte file?
* invalid filename
* invalid content-type
* download
* invalid origin
* invalid media id
* thumbnail
* original file formats
* GIF
* PNG
* BMP
* SVG
* PDF
* TIFF
* WEBP
* local file
* non-existing
* remote file
* non-existing
* pre-generation according to configuration
* manual verification + hash check for regressions?
* dynamic generation
* hot cache
* limit on dimensions?
* 0x0
* crop
* load
* 100 parallel requests
* same file
* different local files
* different remote files
* pre-generated thumbnails
* non-pre-generated thumbnails

269
cmd/mediaapi-integration-tests/main.go
View file

@ -1,269 +0,0 @@
// Copyright 2017 Vector Creations Ltd
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
"fmt"
"io/ioutil"
"net/http"
"os"
"os/exec"
"path"
"path/filepath"
"time"
"github.com/matrix-org/dendrite/internal/test"
"github.com/matrix-org/gomatrixserverlib"
"gopkg.in/yaml.v2"
)
var (
// How long to wait for the server to write the expected output messages.
// This needs to be high enough to account for the time it takes to create
// the postgres database tables which can take a while on travis.
timeoutString = test.Defaulting(os.Getenv("TIMEOUT"), "10s")
// The name of maintenance database to connect to in order to create the test database.
postgresDatabase = test.Defaulting(os.Getenv("POSTGRES_DATABASE"), "postgres")
// The name of the test database to create.
testDatabaseName = test.Defaulting(os.Getenv("DATABASE_NAME"), "mediaapi_test")
// Postgres docker container name (for running psql). If not set, psql must be in PATH.
postgresContainerName = os.Getenv("POSTGRES_CONTAINER")
// Test image to be uploaded/downloaded
testJPEG = test.Defaulting(os.Getenv("TEST_JPEG_PATH"), "cmd/mediaapi-integration-tests/totem.jpg")
kafkaURI = test.Defaulting(os.Getenv("KAFKA_URIS"), "localhost:9092")
)
var thumbnailSizes = (`
- width: 32
height: 32
method: crop
- width: 96
height: 96
method: crop
- width: 320
height: 240
method: scale
- width: 640
height: 480
method: scale
- width: 800
height: 600
method: scale
`)
const serverType = "media-api"
const testMediaID = "1VuVy8u_hmDllD8BrcY0deM34Bl7SPJeY9J6BkMmpx0"
const testContentType = "image/jpeg"
const testOrigin = "localhost:18001"
var testDatabaseTemplate = "dbname=%s sslmode=disable binary_parameters=yes"
var timeout time.Duration
var port = 10000
func startMediaAPI(suffix string, dynamicThumbnails bool) (*exec.Cmd, chan error, *exec.Cmd, string, string) {
dir, err := ioutil.TempDir("", serverType+"-server-test"+suffix)
if err != nil {
panic(err)
}
proxyAddr := "localhost:1800" + suffix
database := fmt.Sprintf(testDatabaseTemplate, testDatabaseName+suffix)
cfg, nextPort, err := test.MakeConfig(dir, kafkaURI, database, "localhost", port)
if err != nil {
panic(err)
}
cfg.Global.ServerName = gomatrixserverlib.ServerName(proxyAddr)
cfg.MediaAPI.DynamicThumbnails = dynamicThumbnails
if err = yaml.Unmarshal([]byte(thumbnailSizes), &cfg.MediaAPI.ThumbnailSizes); err != nil {
panic(err)
}
port = nextPort
if err = test.WriteConfig(cfg, dir); err != nil {
panic(err)
}
serverArgs := []string{
"--config", filepath.Join(dir, test.ConfigFile),
}
databases := []string{
testDatabaseName + suffix,
}
proxyCmd, _ := test.StartProxy(proxyAddr, cfg)
test.InitDatabase(
postgresDatabase,
postgresContainerName,
databases,
)
cmd, cmdChan := test.CreateBackgroundCommand(
filepath.Join(filepath.Dir(os.Args[0]), "dendrite-"+serverType+"-server"),
serverArgs,
)
fmt.Printf("==TESTSERVER== STARTED %v -> %v : %v\n", proxyAddr, cfg.MediaAPI.InternalAPI.Listen, dir)
return cmd, cmdChan, proxyCmd, proxyAddr, dir
}
func cleanUpServer(cmd *exec.Cmd, dir string) {
// ensure server is dead, only cleaning up so don't care about errors this returns
cmd.Process.Kill() // nolint: errcheck
if err := os.RemoveAll(dir); err != nil {
fmt.Printf("WARNING: Failed to remove temporary directory %v: %q\n", dir, err)
}
}
// Runs a battery of media API server tests
// The tests will pause at various points in this list to conduct tests on the HTTP responses before continuing.
func main() {
fmt.Println("==TESTING==", os.Args[0])
var err error
timeout, err = time.ParseDuration(timeoutString)
if err != nil {
fmt.Printf("ERROR: Invalid timeout string %v: %q\n", timeoutString, err)
return
}
// create server1 with only pre-generated thumbnails allowed
server1Cmd, server1CmdChan, server1ProxyCmd, server1ProxyAddr, server1Dir := startMediaAPI("1", false)
defer cleanUpServer(server1Cmd, server1Dir)
defer server1ProxyCmd.Process.Kill() // nolint: errcheck
testDownload(server1ProxyAddr, server1ProxyAddr, "doesnotexist", 404, server1CmdChan)
// upload a JPEG file
testUpload(
server1ProxyAddr, testJPEG,
)
// download that JPEG file
testDownload(server1ProxyAddr, testOrigin, testMediaID, 200, server1CmdChan)
// thumbnail that JPEG file
testThumbnail(64, 64, "crop", server1ProxyAddr, server1CmdChan)
// create server2 with dynamic thumbnail generation
server2Cmd, server2CmdChan, server2ProxyCmd, server2ProxyAddr, server2Dir := startMediaAPI("2", true)
defer cleanUpServer(server2Cmd, server2Dir)
defer server2ProxyCmd.Process.Kill() // nolint: errcheck
testDownload(server2ProxyAddr, server2ProxyAddr, "doesnotexist", 404, server2CmdChan)
// pre-generated thumbnail that JPEG file via server2
testThumbnail(800, 600, "scale", server2ProxyAddr, server2CmdChan)
// download that JPEG file via server2
testDownload(server2ProxyAddr, testOrigin, testMediaID, 200, server2CmdChan)
// dynamic thumbnail that JPEG file via server2
testThumbnail(1920, 1080, "scale", server2ProxyAddr, server2CmdChan)
// thumbnail that JPEG file via server2
testThumbnail(10000, 10000, "scale", server2ProxyAddr, server2CmdChan)
}
func getMediaURI(host, endpoint, query string, components []string) string {
pathComponents := []string{host, "_matrix/media/v1", endpoint}
pathComponents = append(pathComponents, components...)
return "https://" + path.Join(pathComponents...) + query
}
func testUpload(host, filePath string) {
fmt.Printf("==TESTING== upload %v to %v\n", filePath, host)
file, err := os.Open(filePath)
defer file.Close() // nolint: errcheck, staticcheck, megacheck
if err != nil {
panic(err)
}
filename := filepath.Base(filePath)
stat, err := file.Stat()
if os.IsNotExist(err) {
panic(err)
}
fileSize := stat.Size()
req, err := http.NewRequest(
"POST",
getMediaURI(host, "upload", "?filename="+filename, nil),
file,
)
if err != nil {
panic(err)
}
req.ContentLength = fileSize
req.Header.Set("Content-Type", testContentType)
wantedBody := `{"content_uri": "mxc://localhost:18001/` + testMediaID + `"}`
testReq := &test.Request{
Req: req,
WantedStatusCode: 200,
WantedBody: test.CanonicalJSONInput([]string{wantedBody})[0],
}
if err := testReq.Do(); err != nil {
panic(err)
}
fmt.Printf("==TESTING== upload %v to %v PASSED\n", filePath, host)
}
func testDownload(host, origin, mediaID string, wantedStatusCode int, serverCmdChan chan error) {
req, err := http.NewRequest(
"GET",
getMediaURI(host, "download", "", []string{
origin,
mediaID,
}),
nil,
)
if err != nil {
panic(err)
}
testReq := &test.Request{
Req: req,
WantedStatusCode: wantedStatusCode,
WantedBody: "",
}
testReq.Run(fmt.Sprintf("download mxc://%v/%v from %v", origin, mediaID, host), timeout, serverCmdChan)
}
func testThumbnail(width, height int, resizeMethod, host string, serverCmdChan chan error) {
query := fmt.Sprintf("?width=%v&height=%v", width, height)
if resizeMethod != "" {
query += "&method=" + resizeMethod
}
req, err := http.NewRequest(
"GET",
getMediaURI(host, "thumbnail", query, []string{
testOrigin,
testMediaID,
}),
nil,
)
if err != nil {
panic(err)
}
testReq := &test.Request{
Req: req,
WantedStatusCode: 200,
WantedBody: "",
}
testReq.Run(fmt.Sprintf("thumbnail mxc://%v/%v%v from %v", testOrigin, testMediaID, query, host), timeout, serverCmdChan)
}

BIN
cmd/mediaapi-integration-tests/totem.jpg
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 1.8 MiB

442
cmd/roomserver-integration-tests/main.go
View file

@ -1,442 +0,0 @@
// Copyright 2017 Vector Creations Ltd
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
"context"
"fmt"
"io/ioutil"
"os"
"os/exec"
"path/filepath"
"strings"
"time"
"encoding/json"
"net/http"
"github.com/matrix-org/dendrite/internal/caching"
"github.com/matrix-org/dendrite/internal/test"
"github.com/matrix-org/dendrite/roomserver/api"
"github.com/matrix-org/dendrite/roomserver/inthttp"
"github.com/matrix-org/dendrite/setup/config"
"github.com/matrix-org/gomatrixserverlib"
)
var (
// Path to where kafka is installed.
kafkaDir = defaulting(os.Getenv("KAFKA_DIR"), "kafka")
// The URI the kafka zookeeper is listening on.
zookeeperURI = defaulting(os.Getenv("ZOOKEEPER_URI"), "localhost:2181")
// The URI the kafka server is listening on.
kafkaURI = defaulting(os.Getenv("KAFKA_URIS"), "localhost:9092")
// How long to wait for the roomserver to write the expected output messages.
// This needs to be high enough to account for the time it takes to create
// the postgres database tables which can take a while on travis.
timeoutString = defaulting(os.Getenv("TIMEOUT"), "60s")
// Timeout for http client
timeoutHTTPClient = defaulting(os.Getenv("TIMEOUT_HTTP"), "30s")
// The name of maintenance database to connect to in order to create the test database.
postgresDatabase = defaulting(os.Getenv("POSTGRES_DATABASE"), "postgres")
// The name of the test database to create.
testDatabaseName = defaulting(os.Getenv("DATABASE_NAME"), "roomserver_test")
// The postgres connection config for connecting to the test database.
testDatabase = defaulting(os.Getenv("DATABASE"), fmt.Sprintf("dbname=%s binary_parameters=yes", testDatabaseName))
)
var exe = test.KafkaExecutor{
ZookeeperURI: zookeeperURI,
KafkaDirectory: kafkaDir,
KafkaURI: kafkaURI,
// Send stdout and stderr to our stderr so that we see error messages from
// the kafka process.
OutputWriter: os.Stderr,
}
func defaulting(value, defaultValue string) string {
if value == "" {
value = defaultValue
}
return value
}
var (
timeout time.Duration
timeoutHTTP time.Duration
)
func init() {
var err error
timeout, err = time.ParseDuration(timeoutString)
if err != nil {
panic(err)
}
timeoutHTTP, err = time.ParseDuration(timeoutHTTPClient)
if err != nil {
panic(err)
}
}
func createDatabase(database string) error {
cmd := exec.Command("psql", postgresDatabase)
cmd.Stdin = strings.NewReader(
fmt.Sprintf("DROP DATABASE IF EXISTS %s; CREATE DATABASE %s;", database, database),
)
// Send stdout and stderr to our stderr so that we see error messages from
// the psql process
cmd.Stdout = os.Stderr
cmd.Stderr = os.Stderr
return cmd.Run()
}
// runAndReadFromTopic runs a command and waits for a number of messages to be
// written to a kafka topic. It returns if the command exits, the number of
// messages is reached or after a timeout. It kills the command before it returns.
// It returns a list of the messages read from the command on success or an error
// on failure.
func runAndReadFromTopic(runCmd *exec.Cmd, readyURL string, doInput func(), topic string, count int, checkQueryAPI func()) ([]string, error) {
type result struct {
// data holds all of stdout on success.
data []byte
// err is set on failure.
err error
}
done := make(chan result)
readCmd := exec.Command(
filepath.Join(kafkaDir, "bin", "kafka-console-consumer.sh"),
"--bootstrap-server", kafkaURI,
"--topic", topic,
"--from-beginning",
"--max-messages", fmt.Sprintf("%d", count),
)
// Send stderr to our stderr so the user can see any error messages.
readCmd.Stderr = os.Stderr
// Kill both processes before we exit.
defer func() { runCmd.Process.Kill() }() // nolint: errcheck
defer func() { readCmd.Process.Kill() }() // nolint: errcheck
// Run the command, read the messages and wait for a timeout in parallel.
go func() {
// Read all of stdout.
defer func() {
if err := recover(); err != nil {
if errv, ok := err.(error); ok {
done <- result{nil, errv}
} else {
panic(err)
}
}
}()
data, err := readCmd.Output()
checkQueryAPI()
done <- result{data, err}
}()
go func() {
err := runCmd.Run()
done <- result{nil, err}
}()
go func() {
time.Sleep(timeout)
done <- result{nil, fmt.Errorf("Timeout reading %d messages from topic %q", count, topic)}
}()
// Poll the HTTP listener of the process waiting for it to be ready to receive requests.
ready := make(chan struct{})
go func() {
delay := 10 * time.Millisecond
for {
time.Sleep(delay)
if delay < 100*time.Millisecond {
delay *= 2
}
resp, err := http.Get(readyURL)
if err != nil {
continue
}
if resp.StatusCode == 200 {
break
}
}
ready <- struct{}{}
}()
// Wait for the roomserver to be ready to receive input or for it to crash.
select {
case <-ready:
case r := <-done:
return nil, r.err
}
// Write the input now that the server is running.
doInput()
// Wait for one of the tasks to finsh.
r := <-done
if r.err != nil {
return nil, r.err
}
// The kafka console consumer writes a newline character after each message.
// So we split on newline characters
lines := strings.Split(string(r.data), "\n")
if len(lines) > 0 {
// Remove the blank line at the end of the data.
lines = lines[:len(lines)-1]
}
return lines, nil
}
func writeToRoomServer(input []string, roomserverURL string) error {
var request api.InputRoomEventsRequest
var response api.InputRoomEventsResponse
var err error
request.InputRoomEvents = make([]api.InputRoomEvent, len(input))
for i := range input {
if err = json.Unmarshal([]byte(input[i]), &request.InputRoomEvents[i]); err != nil {
return err
}
}
x, err := inthttp.NewRoomserverClient(roomserverURL, &http.Client{Timeout: timeoutHTTP}, nil)
if err != nil {
return err
}
x.InputRoomEvents(context.Background(), &request, &response)
return response.Err()
}
// testRoomserver is used to run integration tests against a single roomserver.
// It creates new kafka topics for the input and output of the roomserver.
// It writes the input messages to the input kafka topic, formatting each message
// as canonical JSON so that it fits on a single line.
// It then runs the roomserver and waits for a number of messages to be written
// to the output topic.
// Once those messages have been written it runs the checkQueries function passing
// a api.RoomserverQueryAPI client. The caller can use this function to check the
// behaviour of the query API.
func testRoomserver(input []string, wantOutput []string, checkQueries func(api.RoomserverInternalAPI)) {
dir, err := ioutil.TempDir("", "room-server-test")
if err != nil {
panic(err)
}
cfg, _, err := test.MakeConfig(dir, kafkaURI, testDatabase, "localhost", 10000)
if err != nil {
panic(err)
}
if err = test.WriteConfig(cfg, dir); err != nil {
panic(err)
}
outputTopic := cfg.Global.Kafka.TopicFor(config.TopicOutputRoomEvent)
err = exe.DeleteTopic(outputTopic)
if err != nil {
panic(err)
}
if err = exe.CreateTopic(outputTopic); err != nil {
panic(err)
}
if err = createDatabase(testDatabaseName); err != nil {
panic(err)
}
cache, err := caching.NewInMemoryLRUCache(false)
if err != nil {
panic(err)
}
doInput := func() {
fmt.Printf("Roomserver is ready to receive input, sending %d events\n", len(input))
if err = writeToRoomServer(input, cfg.RoomServerURL()); err != nil {
panic(err)
}
}
cmd := exec.Command(filepath.Join(filepath.Dir(os.Args[0]), "dendrite-room-server"))
// Append the roomserver config to the existing environment.
// We append to the environment rather than replacing so that any additional
// postgres and golang environment variables such as PGHOST are passed to
// the roomserver process.
cmd.Stderr = os.Stderr
cmd.Args = []string{"dendrite-room-server", "--config", filepath.Join(dir, test.ConfigFile)}
gotOutput, err := runAndReadFromTopic(cmd, cfg.RoomServerURL()+"/metrics", doInput, outputTopic, len(wantOutput), func() {
queryAPI, _ := inthttp.NewRoomserverClient("http://"+string(cfg.RoomServer.InternalAPI.Connect), &http.Client{Timeout: timeoutHTTP}, cache)
checkQueries(queryAPI)
})
if err != nil {
panic(err)
}
if len(wantOutput) != len(gotOutput) {
panic(fmt.Errorf("Wanted %d lines of output got %d lines", len(wantOutput), len(gotOutput)))
}
for i := range wantOutput {
if !equalJSON(wantOutput[i], gotOutput[i]) {
panic(fmt.Errorf("Wanted %q at index %d got %q", wantOutput[i], i, gotOutput[i]))
}
}
}
func equalJSON(a, b string) bool {
canonicalA, err := gomatrixserverlib.CanonicalJSON([]byte(a))
if err != nil {
panic(err)
}
canonicalB, err := gomatrixserverlib.CanonicalJSON([]byte(b))
if err != nil {
panic(err)
}
return string(canonicalA) == string(canonicalB)
}
func main() {
fmt.Println("==TESTING==", os.Args[0])
input := []string{
`{
"auth_event_ids": [],
"kind": 1,
"event": {
"origin": "matrix.org",
"signatures": {
"matrix.org": {
"ed25519:auto": "3kXGwNtdj+zqEXlI8PWLiB76xtrQ7SxcvPuXAEVCTo+QPoBoUvLi1RkHs6O5mDz7UzIowK5bi1seAN4vOh0OBA"
}
},
"origin_server_ts": 1463671337837,
"sender": "@richvdh:matrix.org",
"event_id": "$1463671337126266wrSBX:matrix.org",
"prev_events": [],
"state_key": "",
"content": {"creator": "@richvdh:matrix.org"},
"depth": 1,
"prev_state": [],
"room_id": "!HCXfdvrfksxuYnIFiJ:matrix.org",
"auth_events": [],
"hashes": {"sha256": "Q05VLC8nztN2tguy+KnHxxhitI95wK9NelnsDaXRqeo"},
"type": "m.room.create"}
}`, `{
"auth_event_ids": ["$1463671337126266wrSBX:matrix.org"],
"kind": 2,
"state_event_ids": ["$1463671337126266wrSBX:matrix.org"],
"event": {
"origin": "matrix.org",
"signatures": {
"matrix.org": {
"ed25519:auto": "a2b3xXYVPPFeG1sHCU3hmZnAaKqZFgzGZozijRGblG5Y//ewRPAn1A2mCrI2UM5I+0zqr70cNpHgF8bmNFu4BA"
}
},
"origin_server_ts": 1463671339844,
"sender": "@richvdh:matrix.org",
"event_id": "$1463671339126270PnVwC:matrix.org",
"prev_events": [[
"$1463671337126266wrSBX:matrix.org", {"sha256": "h/VS07u8KlMwT3Ee8JhpkC7sa1WUs0Srgs+l3iBv6c0"}
]],
"membership": "join",
"state_key": "@richvdh:matrix.org",
"content": {
"membership": "join",
"avatar_url": "mxc://matrix.org/ZafPzsxMJtLaSaJXloBEKiws",
"displayname": "richvdh"
},
"depth": 2,
"prev_state": [],
"room_id": "!HCXfdvrfksxuYnIFiJ:matrix.org",
"auth_events": [[
"$1463671337126266wrSBX:matrix.org", {"sha256": "h/VS07u8KlMwT3Ee8JhpkC7sa1WUs0Srgs+l3iBv6c0"}
]],
"hashes": {"sha256": "t9t3sZV1Eu0P9Jyrs7pge6UTa1zuTbRdVxeUHnrQVH0"},
"type": "m.room.member"},
"has_state": true
}`,
}
want := []string{
`{"type":"new_room_event","new_room_event":{
"event":{
"auth_events":[[
"$1463671337126266wrSBX:matrix.org",{"sha256":"h/VS07u8KlMwT3Ee8JhpkC7sa1WUs0Srgs+l3iBv6c0"}
]],
"content":{
"avatar_url":"mxc://matrix.org/ZafPzsxMJtLaSaJXloBEKiws",
"displayname":"richvdh",
"membership":"join"
},
"depth": 2,
"event_id": "$1463671339126270PnVwC:matrix.org",
"hashes": {"sha256":"t9t3sZV1Eu0P9Jyrs7pge6UTa1zuTbRdVxeUHnrQVH0"},
"membership": "join",
"origin": "matrix.org",
"origin_server_ts": 1463671339844,
"prev_events": [[
"$1463671337126266wrSBX:matrix.org",{"sha256":"h/VS07u8KlMwT3Ee8JhpkC7sa1WUs0Srgs+l3iBv6c0"}
]],
"prev_state":[],
"room_id":"!HCXfdvrfksxuYnIFiJ:matrix.org",
"sender":"@richvdh:matrix.org",
"signatures":{
"matrix.org":{
"ed25519:auto":"a2b3xXYVPPFeG1sHCU3hmZnAaKqZFgzGZozijRGblG5Y//ewRPAn1A2mCrI2UM5I+0zqr70cNpHgF8bmNFu4BA"
}
},
"state_key":"@richvdh:matrix.org",
"type":"m.room.member"
},
"state_before_removes_event_ids":["$1463671339126270PnVwC:matrix.org"],
"state_before_adds_event_ids":null,
"latest_event_ids":["$1463671339126270PnVwC:matrix.org"],
"adds_state_event_ids":["$1463671337126266wrSBX:matrix.org", "$1463671339126270PnVwC:matrix.org"],
"removes_state_event_ids":null,
"last_sent_event_id":"",
"send_as_server":"",
"transaction_id": null
}}`,
}
testRoomserver(input, want, func(q api.RoomserverInternalAPI) {
var response api.QueryLatestEventsAndStateResponse
if err := q.QueryLatestEventsAndState(
context.Background(),
&api.QueryLatestEventsAndStateRequest{
RoomID: "!HCXfdvrfksxuYnIFiJ:matrix.org",
StateToFetch: []gomatrixserverlib.StateKeyTuple{
{EventType: "m.room.member", StateKey: "@richvdh:matrix.org"},
},
},
&response,
); err != nil {
panic(err)
}
if !response.RoomExists {
panic(fmt.Errorf(`Wanted room "!HCXfdvrfksxuYnIFiJ:matrix.org" to exist`))
}
if len(response.LatestEvents) != 1 || response.LatestEvents[0].EventID != "$1463671339126270PnVwC:matrix.org" {
panic(fmt.Errorf(`Wanted "$1463671339126270PnVwC:matrix.org" to be the latest event got %#v`, response.LatestEvents))
}
if len(response.StateEvents) != 1 || response.StateEvents[0].EventID() != "$1463671339126270PnVwC:matrix.org" {
panic(fmt.Errorf(`Wanted "$1463671339126270PnVwC:matrix.org" to be the state event got %#v`, response.StateEvents))
}
})
fmt.Println("==PASSED==", os.Args[0])
}

563
cmd/syncserver-integration-tests/main.go
View file

@ -1,563 +0,0 @@
// Copyright 2017 Vector Creations Ltd
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"os"
"os/exec"
"path/filepath"
"time"
"github.com/matrix-org/dendrite/internal/test"
"github.com/matrix-org/dendrite/roomserver/api"
"github.com/matrix-org/dendrite/setup/config"
"github.com/matrix-org/gomatrixserverlib"
)
var (
// Path to where kafka is installed.
kafkaDir = test.Defaulting(os.Getenv("KAFKA_DIR"), "kafka")
// The URI the kafka zookeeper is listening on.
zookeeperURI = test.Defaulting(os.Getenv("ZOOKEEPER_URI"), "localhost:2181")
// The URI the kafka server is listening on.
kafkaURI = test.Defaulting(os.Getenv("KAFKA_URIS"), "localhost:9092")
// The address the syncserver should listen on.
syncserverAddr = test.Defaulting(os.Getenv("SYNCSERVER_URI"), "localhost:9876")
// How long to wait for the syncserver to write the expected output messages.
// This needs to be high enough to account for the time it takes to create
// the postgres database tables which can take a while on travis.
timeoutString = test.Defaulting(os.Getenv("TIMEOUT"), "10s")
// The name of maintenance database to connect to in order to create the test database.
postgresDatabase = test.Defaulting(os.Getenv("POSTGRES_DATABASE"), "postgres")
// Postgres docker container name (for running psql). If not set, psql must be in PATH.
postgresContainerName = os.Getenv("POSTGRES_CONTAINER")
// The name of the test database to create.
testDatabaseName = test.Defaulting(os.Getenv("DATABASE_NAME"), "syncserver_test")
// The postgres connection config for connecting to the test database.
testDatabase = test.Defaulting(os.Getenv("DATABASE"), fmt.Sprintf("dbname=%s sslmode=disable binary_parameters=yes", testDatabaseName))
)
const inputTopic = "syncserverInput"
const clientTopic = "clientapiserverOutput"
var exe = test.KafkaExecutor{
ZookeeperURI: zookeeperURI,
KafkaDirectory: kafkaDir,
KafkaURI: kafkaURI,
// Send stdout and stderr to our stderr so that we see error messages from
// the kafka process.
OutputWriter: os.Stderr,
}
var timeout time.Duration
var clientEventTestData []string
func init() {
var err error
timeout, err = time.ParseDuration(timeoutString)
if err != nil {
panic(err)
}
for _, s := range outputRoomEventTestData {
clientEventTestData = append(clientEventTestData, clientEventJSONForOutputRoomEvent(s))
}
}
func createTestUser(database, username, token string) error {
cmd := exec.Command(
filepath.Join(filepath.Dir(os.Args[0]), "create-account"),
"--database", database,
"--username", username,
"--token", token,
)
// Send stdout and stderr to our stderr so that we see error messages from
// the create-account process
cmd.Stdout = os.Stderr
cmd.Stderr = os.Stderr
return cmd.Run()
}
// clientEventJSONForOutputRoomEvent parses the given output room event and extracts the 'Event' JSON. It is
// trimmed to the client format and then canonicalised and returned as a string.
// Panics if there are any problems.
func clientEventJSONForOutputRoomEvent(outputRoomEvent string) string {
var out api.OutputEvent
if err := json.Unmarshal([]byte(outputRoomEvent), &out); err != nil {
panic("failed to unmarshal output room event: " + err.Error())
}
clientEvs := gomatrixserverlib.ToClientEvents([]*gomatrixserverlib.Event{
out.NewRoomEvent.Event.Event,
}, gomatrixserverlib.FormatSync)
b, err := json.Marshal(clientEvs[0])
if err != nil {
panic("failed to marshal client event as json: " + err.Error())
}
jsonBytes, err := gomatrixserverlib.CanonicalJSON(b)
if err != nil {
panic("failed to turn event json into canonical json: " + err.Error())
}
return string(jsonBytes)
}
// startSyncServer creates the database and config file needed for the sync server to run and
// then starts the sync server. The Cmd being executed is returned. A channel is also returned,
// which will have any termination errors sent down it, followed immediately by the channel being closed.
func startSyncServer() (*exec.Cmd, chan error) {
dir, err := ioutil.TempDir("", "syncapi-server-test")
if err != nil {
panic(err)
}
cfg, _, err := test.MakeConfig(dir, kafkaURI, testDatabase, "localhost", 10000)
if err != nil {
panic(err)
}
// TODO use the address assigned by the config generator rather than clobbering.
cfg.Global.ServerName = "localhost"
cfg.SyncAPI.InternalAPI.Listen = config.HTTPAddress("http://" + syncserverAddr)
cfg.SyncAPI.InternalAPI.Connect = cfg.SyncAPI.InternalAPI.Listen
if err := test.WriteConfig(cfg, dir); err != nil {
panic(err)
}
serverArgs := []string{
"--config", filepath.Join(dir, test.ConfigFile),
}
databases := []string{
testDatabaseName,
}
test.InitDatabase(
postgresDatabase,
postgresContainerName,
databases,
)
if err := createTestUser(testDatabase, "alice", "@alice:localhost"); err != nil {
panic(err)
}
if err := createTestUser(testDatabase, "bob", "@bob:localhost"); err != nil {
panic(err)
}
if err := createTestUser(testDatabase, "charlie", "@charlie:localhost"); err != nil {
panic(err)
}
cmd, cmdChan := test.CreateBackgroundCommand(
filepath.Join(filepath.Dir(os.Args[0]), "dendrite-sync-api-server"),
serverArgs,
)
return cmd, cmdChan
}
// prepareKafka creates the topics which will be written to by the tests.
func prepareKafka() {
err := exe.DeleteTopic(inputTopic)
if err != nil {
panic(err)
}
if err = exe.CreateTopic(inputTopic); err != nil {
panic(err)
}
err = exe.DeleteTopic(clientTopic)
if err != nil {
panic(err)
}
if err = exe.CreateTopic(clientTopic); err != nil {
panic(err)
}
}
func testSyncServer(syncServerCmdChan chan error, userID, since, want string) {
fmt.Printf("==TESTING== testSyncServer(%s,%s)\n", userID, since)
sinceQuery := ""
if since != "" {
sinceQuery = "&since=" + since
}
req, err := http.NewRequest(
"GET",
"http://"+syncserverAddr+"/api/_matrix/client/r0/sync?timeout=100&access_token="+userID+sinceQuery,
nil,
)
if err != nil {
panic(err)
}
testReq := &test.Request{
Req: req,
WantedStatusCode: 200,
WantedBody: test.CanonicalJSONInput([]string{want})[0],
}
testReq.Run("sync-api", timeout, syncServerCmdChan)
}
func writeToRoomServerLog(indexes ...int) {
var roomEvents []string
for _, i := range indexes {
roomEvents = append(roomEvents, outputRoomEventTestData[i])
}
if err := exe.WriteToTopic(inputTopic, test.CanonicalJSONInput(roomEvents)); err != nil {
panic(err)
}
}
// Runs a battery of sync server tests against test data in testdata.go
// testdata.go has a list of OutputRoomEvents which will be fed into the kafka log which the sync server will consume.
// The tests will pause at various points in this list to conduct tests on the /sync responses before continuing.
// For ease of understanding, the curl commands used to create the OutputRoomEvents are listed along with each write to kafka.
func main() {
fmt.Println("==TESTING==", os.Args[0])
prepareKafka()
cmd, syncServerCmdChan := startSyncServer()
// ensure server is dead, only cleaning up so don't care about errors this returns.
defer cmd.Process.Kill() // nolint: errcheck
// $ curl -XPOST -d '{}' "http://localhost:8009/_matrix/client/r0/createRoom?access_token=@alice:localhost"
// $ curl -XPUT -d '{"msgtype":"m.text","body":"hello world"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/1?access_token=@alice:localhost"
// $ curl -XPUT -d '{"msgtype":"m.text","body":"hello world 2"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/2?access_token=@alice:localhost"
// $ curl -XPUT -d '{"msgtype":"m.text","body":"hello world 3"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/3?access_token=@alice:localhost"
// $ curl -XPUT -d '{"name":"Custom Room Name"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.name?access_token=@alice:localhost"
writeToRoomServerLog(
i0StateRoomCreate, i1StateAliceJoin, i2StatePowerLevels, i3StateJoinRules, i4StateHistoryVisibility,
i5AliceMsg, i6AliceMsg, i7AliceMsg, i8StateAliceRoomName,
)
// Make sure initial sync works TODO: prev_batch
testSyncServer(syncServerCmdChan, "@alice:localhost", "", `{
"account_data": {
"events": []
},
"next_batch": "9",
"presence": {
"events": []
},
"rooms": {
"invite": {},
"join": {
"!PjrbIMW2cIiaYF4t:localhost": {
"account_data": {
"events": []
},
"ephemeral": {
"events": []
},
"state": {
"events": []
},
"timeline": {
"events": [`+
clientEventTestData[i0StateRoomCreate]+","+
clientEventTestData[i1StateAliceJoin]+","+
clientEventTestData[i2StatePowerLevels]+","+
clientEventTestData[i3StateJoinRules]+","+
clientEventTestData[i4StateHistoryVisibility]+","+
clientEventTestData[i5AliceMsg]+","+
clientEventTestData[i6AliceMsg]+","+
clientEventTestData[i7AliceMsg]+","+
clientEventTestData[i8StateAliceRoomName]+`],
"limited": true,
"prev_batch": ""
}
}
},
"leave": {}
}
}`)
// Make sure alice's rooms don't leak to bob
testSyncServer(syncServerCmdChan, "@bob:localhost", "", `{
"account_data": {
"events": []
},
"next_batch": "9",
"presence": {
"events": []
},
"rooms": {
"invite": {},
"join": {},
"leave": {}
}
}`)
// Make sure polling with an up-to-date token returns nothing new
testSyncServer(syncServerCmdChan, "@alice:localhost", "9", `{
"account_data": {
"events": []
},
"next_batch": "9",
"presence": {
"events": []
},
"rooms": {
"invite": {},
"join": {},
"leave": {}
}
}`)
// $ curl -XPUT -d '{"membership":"join"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@bob:localhost?access_token=@bob:localhost"
writeToRoomServerLog(i9StateBobJoin)
// Make sure alice sees it TODO: prev_batch
testSyncServer(syncServerCmdChan, "@alice:localhost", "9", `{
"account_data": {
"events": []
},
"next_batch": "10",
"presence": {
"events": []
},
"rooms": {
"invite": {},
"join": {
"!PjrbIMW2cIiaYF4t:localhost": {
"account_data": {
"events": []
},
"ephemeral": {
"events": []
},
"state": {
"events": []
},
"timeline": {
"limited": false,
"prev_batch": "",
"events": [`+clientEventTestData[i9StateBobJoin]+`]
}
}
},
"leave": {}
}
}`)
// Make sure bob sees the room AND all the current room state TODO: history visibility
testSyncServer(syncServerCmdChan, "@bob:localhost", "9", `{
"account_data": {
"events": []
},
"next_batch": "10",
"presence": {
"events": []
},
"rooms": {
"invite": {},
"join": {
"!PjrbIMW2cIiaYF4t:localhost": {
"account_data": {
"events": []
},
"ephemeral": {
"events": []
},
"state": {
"events": [`+
clientEventTestData[i0StateRoomCreate]+","+
clientEventTestData[i1StateAliceJoin]+","+
clientEventTestData[i2StatePowerLevels]+","+
clientEventTestData[i3StateJoinRules]+","+
clientEventTestData[i4StateHistoryVisibility]+","+
clientEventTestData[i8StateAliceRoomName]+`]
},
"timeline": {
"limited": false,
"prev_batch": "",
"events": [`+
clientEventTestData[i9StateBobJoin]+`]
}
}
},
"leave": {}
}
}`)
// $ curl -XPUT -d '{"msgtype":"m.text","body":"hello alice"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/1?access_token=@bob:localhost"
writeToRoomServerLog(i10BobMsg)
// Make sure alice can see everything around the join point for bob TODO: prev_batch
testSyncServer(syncServerCmdChan, "@alice:localhost", "7", `{
"account_data": {
"events": []
},
"next_batch": "11",
"presence": {
"events": []
},
"rooms": {
"invite": {},
"join": {
"!PjrbIMW2cIiaYF4t:localhost": {
"account_data": {
"events": []
},
"ephemeral": {
"events": []
},
"state": {
"events": []
},
"timeline": {
"limited": false,
"prev_batch": "",
"events": [`+
clientEventTestData[i7AliceMsg]+","+
clientEventTestData[i8StateAliceRoomName]+","+
clientEventTestData[i9StateBobJoin]+","+
clientEventTestData[i10BobMsg]+`]
}
}
},
"leave": {}
}
}`)
// $ curl -XPUT -d '{"name":"A Different Custom Room Name"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.name?access_token=@alice:localhost"
// $ curl -XPUT -d '{"msgtype":"m.text","body":"hello bob"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/2?access_token=@alice:localhost"
// $ curl -XPUT -d '{"membership":"invite"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@charlie:localhost?access_token=@bob:localhost"
writeToRoomServerLog(i11StateAliceRoomName, i12AliceMsg, i13StateBobInviteCharlie)
// Make sure charlie sees the invite both with and without a ?since= token
// TODO: Invite state should include the invite event and the room name.
charlieInviteData := `{
"account_data": {
"events": []
},
"next_batch": "14",
"presence": {
"events": []
},
"rooms": {
"invite": {
"!PjrbIMW2cIiaYF4t:localhost": {
"invite_state": {
"events": []
}
}
},
"join": {},
"leave": {}
}
}`
testSyncServer(syncServerCmdChan, "@charlie:localhost", "7", charlieInviteData)
testSyncServer(syncServerCmdChan, "@charlie:localhost", "", charlieInviteData)
// $ curl -XPUT -d '{"membership":"join"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@charlie:localhost?access_token=@charlie:localhost"
// $ curl -XPUT -d '{"msgtype":"m.text","body":"not charlie..."}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/3?access_token=@alice:localhost"
// $ curl -XPUT -d '{"membership":"leave"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@charlie:localhost?access_token=@alice:localhost"
// $ curl -XPUT -d '{"msgtype":"m.text","body":"why did you kick charlie"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/3?access_token=@bob:localhost"
writeToRoomServerLog(i14StateCharlieJoin, i15AliceMsg, i16StateAliceKickCharlie, i17BobMsg)
// Check transitions to leave work
testSyncServer(syncServerCmdChan, "@charlie:localhost", "15", `{
"account_data": {
"events": []
},
"next_batch": "18",
"presence": {
"events": []
},
"rooms": {
"invite": {},
"join": {},
"leave": {
"!PjrbIMW2cIiaYF4t:localhost": {
"state": {
"events": []
},
"timeline": {
"limited": false,
"prev_batch": "",
"events": [`+
clientEventTestData[i15AliceMsg]+","+
clientEventTestData[i16StateAliceKickCharlie]+`]
}
}
}
}
}`)
// Test joining and leaving the same room in a single /sync request puts the room in the 'leave' section.
// TODO: Use an earlier since value to assert that the /sync response doesn't leak messages
// from before charlie was joined to the room. Currently it does leak because RecentEvents doesn't
// take membership into account.
testSyncServer(syncServerCmdChan, "@charlie:localhost", "14", `{
"account_data": {
"events": []
},
"next_batch": "18",
"presence": {
"events": []
},
"rooms": {
"invite": {},
"join": {},
"leave": {
"!PjrbIMW2cIiaYF4t:localhost": {
"state": {
"events": []
},
"timeline": {
"limited": false,
"prev_batch": "",
"events": [`+
clientEventTestData[i14StateCharlieJoin]+","+
clientEventTestData[i15AliceMsg]+","+
clientEventTestData[i16StateAliceKickCharlie]+`]
}
}
}
}
}`)
// $ curl -XPUT -d '{"name":"No Charlies"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.name?access_token=@alice:localhost"
writeToRoomServerLog(i18StateAliceRoomName)
// Check that users don't see state changes in rooms after they have left
testSyncServer(syncServerCmdChan, "@charlie:localhost", "17", `{
"account_data": {
"events": []
},
"next_batch": "19",
"presence": {
"events": []
},
"rooms": {
"invite": {},
"join": {},
"leave": {}
}
}`)
// $ curl -XPUT -d '{"msgtype":"m.text","body":"whatever"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/3?access_token=@bob:localhost"
// $ curl -XPUT -d '{"membership":"leave"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@bob:localhost?access_token=@bob:localhost"
// $ curl -XPUT -d '{"msgtype":"m.text","body":"im alone now"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/3?access_token=@alice:localhost"
// $ curl -XPUT -d '{"membership":"invite"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@bob:localhost?access_token=@alice:localhost"
// $ curl -XPUT -d '{"membership":"leave"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@bob:localhost?access_token=@bob:localhost"
// $ curl -XPUT -d '{"msgtype":"m.text","body":"so alone"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/3?access_token=@alice:localhost"
// $ curl -XPUT -d '{"name":"Everyone welcome"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.name?access_token=@alice:localhost"
// $ curl -XPUT -d '{"membership":"join"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@charlie:localhost?access_token=@charlie:localhost"
// $ curl -XPUT -d '{"msgtype":"m.text","body":"hiiiii"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/3?access_token=@charlie:localhost"
}

102
cmd/syncserver-integration-tests/testdata.go
View file

@ -1,102 +0,0 @@
// Copyright 2017 Vector Creations Ltd
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
// nolint: varcheck, deadcode, unused, megacheck
const (
i0StateRoomCreate = iota
i1StateAliceJoin
i2StatePowerLevels
i3StateJoinRules
i4StateHistoryVisibility
i5AliceMsg
i6AliceMsg
i7AliceMsg
i8StateAliceRoomName
i9StateBobJoin
i10BobMsg
i11StateAliceRoomName
i12AliceMsg
i13StateBobInviteCharlie
i14StateCharlieJoin
i15AliceMsg
i16StateAliceKickCharlie
i17BobMsg
i18StateAliceRoomName
i19BobMsg
i20StateBobLeave
i21AliceMsg
i22StateAliceInviteBob
i23StateBobRejectInvite
i24AliceMsg
i25StateAliceRoomName
i26StateCharlieJoin
i27CharlieMsg
)
var outputRoomEventTestData = []string{
// $ curl -XPOST -d '{}' "http://localhost:8009/_matrix/client/r0/createRoom?access_token=@alice:localhost"
`{"type":"new_room_event","new_room_event":{"event":{"auth_events":[],"content":{"creator":"@alice:localhost"},"depth":1,"event_id":"$xz0fUB8zNMTGFh1W:localhost","hashes":{"sha256":"KKkpxS8NoH0igBbL3J+nJ39MRlmA7QgW4BGL7Fv4ASI"},"origin":"localhost","origin_server_ts":1494411218382,"prev_events":[],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"uZG5Q/Hs2Z611gFlZPdwomomRJKf70xV2FQV+gLWM1XgzkLDRlRF3cBZc9y3CnHKnV/upTcXs7Op2/GmgD3UBw"}},"state_key":"","type":"m.room.create"},"latest_event_ids":["$xz0fUB8zNMTGFh1W:localhost"],"adds_state_event_ids":["$xz0fUB8zNMTGFh1W:localhost"],"last_sent_event_id":""}}`,
`{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}]],"content":{"membership":"join"},"depth":2,"event_id":"$QTen1vksfcRTpUCk:localhost","hashes":{"sha256":"tTukc9ab1fJfzgc5EMA/UD3swqfl/ic9Y9Zkt4fJo0Q"},"origin":"localhost","origin_server_ts":1494411218385,"prev_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"OPysDn/wT7yHeALXLTcEgR+iaKjv0p7VPuR/Mzvyg2IMAwPUjSOw8SQZlhSioWRtVPUp9VHbhIhJxQaPUg9yBQ"}},"state_key":"@alice:localhost","type":"m.room.member"},"latest_event_ids":["$QTen1vksfcRTpUCk:localhost"],"adds_state_event_ids":["$QTen1vksfcRTpUCk:localhost"],"last_sent_event_id":"$xz0fUB8zNMTGFh1W:localhost"}}`,
`{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}]],"content":{"ban":50,"events":{"m.room.avatar":50,"m.room.canonical_alias":50,"m.room.history_visibility":100,"m.room.name":50,"m.room.power_levels":100},"events_default":0,"invite":0,"kick":50,"redact":50,"state_default":50,"users":{"@alice:localhost":100},"users_default":0},"depth":3,"event_id":"$RWsxGlfPHAcijTgu:localhost","hashes":{"sha256":"ueZWiL/Q8bagRQGFktpnYJAJV6V6U3QKcUEmWYeyaaM"},"origin":"localhost","origin_server_ts":1494411218385,"prev_events":[["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"hZwWx3lyW61zMYmqLOxLTlfW2CnbjJQsZPLjZFa97TVG4ISz8CixMPsnVAIu5is29UCmiHyP8RvLecJjbLCtAQ"}},"state_key":"","type":"m.room.power_levels"},"latest_event_ids":["$RWsxGlfPHAcijTgu:localhost"],"adds_state_event_ids":["$RWsxGlfPHAcijTgu:localhost"],"last_sent_event_id":"$QTen1vksfcRTpUCk:localhost"}}`,
`{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}]],"content":{"join_rule":"public"},"depth":4,"event_id":"$2O2DpHB37CuwwJOe:localhost","hashes":{"sha256":"3P3HxAXI8gc094i020EoV/gissYiMVWv8+JAbrakM4E"},"origin":"localhost","origin_server_ts":1494411218386,"prev_events":[["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"L2yZoBbG/6TNsRHz+UtHY0SK4FgrdAYPR1l7RBWaNFbm+k/7kVhnoGlJ9yptpdLJjPMR2InqKXH8BBxRC83BCg"}},"state_key":"","type":"m.room.join_rules"},"latest_event_ids":["$2O2DpHB37CuwwJOe:localhost"],"adds_state_event_ids":["$2O2DpHB37CuwwJOe:localhost"],"last_sent_event_id":"$RWsxGlfPHAcijTgu:localhost"}}`,
`{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}]],"content":{"history_visibility":"joined"},"depth":5,"event_id":"$5LRiBskVCROnL5WY:localhost","hashes":{"sha256":"341alVufcKSVKLPr9WsJNTnW33QkBTn9eTfVWbyoa0o"},"origin":"localhost","origin_server_ts":1494411218387,"prev_events":[["$2O2DpHB37CuwwJOe:localhost",{"sha256":"ulaRD63dbCyolLTwvInIQpcrtU2c7ex/BHmhpLXAUoE"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"kRyt68cstwYgK8NtYzf0V5CnAbqUO47ixCCWYzRCi0WNstEwUw4XW1GHc8BllQsXwSj+nNv9g/66zZgG0DtxCA"}},"state_key":"","type":"m.room.history_visibility"},"latest_event_ids":["$5LRiBskVCROnL5WY:localhost"],"adds_state_event_ids":["$5LRiBskVCROnL5WY:localhost"],"last_sent_event_id":"$2O2DpHB37CuwwJOe:localhost"}}`,
// $ curl -XPUT -d '{"msgtype":"m.text","body":"hello world"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/1?access_token=@alice:localhost"
`{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}]],"content":{"body":"hello world","msgtype":"m.text"},"depth":0,"event_id":"$Z8ZJik7ghwzSYTH9:localhost","hashes":{"sha256":"ahN1T5aiSZCzllf0pqNWJkF+x2h2S3kic+40pQ1X6BE"},"origin":"localhost","origin_server_ts":1494411339207,"prev_events":[["$5LRiBskVCROnL5WY:localhost",{"sha256":"3jULNC9b9Q0AhvnDQqpjhbtYwmkioHzPzdTJZvn8vOI"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"ylEpahRwEfGpqk+UCv0IF8YAxmut7w7udgHy3sVDfdJhs/4uJ6EkFEsKLknpXRc1vTIy1etKCBQ63QbCmRC2Bw"}},"type":"m.room.message"},"latest_event_ids":["$Z8ZJik7ghwzSYTH9:localhost"],"last_sent_event_id":"$5LRiBskVCROnL5WY:localhost"}}`,
// $ curl -XPUT -d '{"msgtype":"m.text","body":"hello world 2"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/2?access_token=@alice:localhost"
`{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}]],"content":{"body":"hello world 2","msgtype":"m.text"},"depth":0,"event_id":"$8382Ah682eL4hxjN:localhost","hashes":{"sha256":"hQElDGSYc6KOdylrbMMm3+LlvUiCKo6S9G9n58/qtns"},"origin":"localhost","origin_server_ts":1494411380282,"prev_events":[["$Z8ZJik7ghwzSYTH9:localhost",{"sha256":"FBDwP+2FeqDENe7AEa3iAFAVKl1/IVq43mCH0uPRn90"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"LFXi6jTG7qn9xzi4rhIiHbkLD+4AZ9Yg7UTS2gqm1gt2lXQsgTYH1wE4Fol2fq4lvGlQVpxhtEr2huAYSbT7DA"}},"type":"m.room.message"},"latest_event_ids":["$8382Ah682eL4hxjN:localhost"],"last_sent_event_id":"$Z8ZJik7ghwzSYTH9:localhost"}}`,
// $ curl -XPUT -d '{"msgtype":"m.text","body":"hello world 3"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/3?access_token=@alice:localhost"
`{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}]],"content":{"body":"hello world 3","msgtype":"m.text"},"depth":0,"event_id":"$17SfHsvSeTQthSWF:localhost","hashes":{"sha256":"eS6VFQI0l2U8rA8U17jgSHr9lQ73SNSnlnZu+HD0IjE"},"origin":"localhost","origin_server_ts":1494411396560,"prev_events":[["$8382Ah682eL4hxjN:localhost",{"sha256":"c6I/PUY7WnvxQ+oUEp/w2HEEuD3g8Vq7QwPUOSUjuc8"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"dvu9bSHZmX+yZoEqHioK7YDMtLH9kol0DdFqc5aHsbhZe/fKRZpfJMrlf1iXQdXSCMhikvnboPAXN3guiZCUBQ"}},"type":"m.room.message"},"latest_event_ids":["$17SfHsvSeTQthSWF:localhost"],"last_sent_event_id":"$8382Ah682eL4hxjN:localhost"}}`,
// $ curl -XPUT -d '{"name":"Custom Room Name"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.name?access_token=@alice:localhost"
`{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}]],"content":{"name":"Custom Room Name"},"depth":0,"event_id":"$j7KtuOzM0K15h3Kr:localhost","hashes":{"sha256":"QIKj5Klr50ugll4EjaNUATJmrru4CDp6TvGPv0v15bo"},"origin":"localhost","origin_server_ts":1494411482625,"prev_events":[["$17SfHsvSeTQthSWF:localhost",{"sha256":"iMTefewJ4W5sKQy7osQv4ilJAi7X0NsK791kqEUmYX0"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"WU7lwSWUAk7bsyDnBs128PyXxPZZoD1sN4AiDcvk+W1mDezJbFvWHDWymclxWESlP7TDrFTZEumRWGGCakjyAg"}},"state_key":"","type":"m.room.name"},"latest_event_ids":["$j7KtuOzM0K15h3Kr:localhost"],"adds_state_event_ids":["$j7KtuOzM0K15h3Kr:localhost"],"last_sent_event_id":"$17SfHsvSeTQthSWF:localhost"}}`,
// $ curl -XPUT -d '{"membership":"join"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@bob:localhost?access_token=@bob:localhost"
`{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}],["$2O2DpHB37CuwwJOe:localhost",{"sha256":"ulaRD63dbCyolLTwvInIQpcrtU2c7ex/BHmhpLXAUoE"}]],"content":{"membership":"join"},"depth":0,"event_id":"$wPepDhIla765Odre:localhost","hashes":{"sha256":"KeKqWLvM+LTvyFbwx6y3Y4W5Pj6nBSFUQ6jpkSf1oTE"},"origin":"localhost","origin_server_ts":1494411534290,"prev_events":[["$j7KtuOzM0K15h3Kr:localhost",{"sha256":"oDrWG5/sy1Ea3hYDOSJZRuGKCcjaHQlDYPDn2gB0/L0"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@bob:localhost","signatures":{"localhost":{"ed25519:something":"oVtvjZbWFe+iJhoDvLcQKnFpSYQ94dOodM4gGsx26P6fs2sFJissYwSIqpoxlElCJnmBAgy5iv4JK/5x21R2CQ"}},"state_key":"@bob:localhost","type":"m.room.member"},"latest_event_ids":["$wPepDhIla765Odre:localhost"],"adds_state_event_ids":["$wPepDhIla765Odre:localhost"],"last_sent_event_id":"$j7KtuOzM0K15h3Kr:localhost"}}`,
// $ curl -XPUT -d '{"msgtype":"m.text","body":"hello alice"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/1?access_token=@bob:localhost"
`{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}],["$wPepDhIla765Odre:localhost",{"sha256":"GqUhRiAkRvPrNBDyUxj+emRfK2P8j6iWtvsXDOUltiI"}]],"content":{"body":"hello alice","msgtype":"m.text"},"depth":0,"event_id":"$RHNjeYUvXVZfb93t:localhost","hashes":{"sha256":"Ic1QLxTWFrWt1o31DS93ftrNHkunf4O6ubFvdD4ydNI"},"origin":"localhost","origin_server_ts":1494411593196,"prev_events":[["$wPepDhIla765Odre:localhost",{"sha256":"GqUhRiAkRvPrNBDyUxj+emRfK2P8j6iWtvsXDOUltiI"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@bob:localhost","signatures":{"localhost":{"ed25519:something":"8BHHkiThWwiIZbXCegRjIKNVGIa2kqrZW8VuL7nASfJBORhZ9R9p34UsmhsxVwTs/2/dX7M2ogMB28gIGdLQCg"}},"type":"m.room.message"},"latest_event_ids":["$RHNjeYUvXVZfb93t:localhost"],"last_sent_event_id":"$wPepDhIla765Odre:localhost"}}`,
// $ curl -XPUT -d '{"name":"A Different Custom Room Name"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.name?access_token=@alice:localhost"
`{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}]],"content":{"name":"A Different Custom Room Name"},"depth":0,"event_id":"$1xoUuqOFjFFJgwA5:localhost","hashes":{"sha256":"2pNnLhoHxNeSUpqxrd3c0kZUA4I+cdWZgYcJ8V3e2tk"},"origin":"localhost","origin_server_ts":1494411643348,"prev_events":[["$RHNjeYUvXVZfb93t:localhost",{"sha256":"LqFmTIzULgUDSf5xM3REObvnsRGLQliWBUf1hEDT4+w"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"gsY4B6TIBdVvLyFAaXw0xez9N5/Cn/ZaJ4z+j9gJU/ZR8j1t3OYlcVQN6uln9JwEU1k20AsGnIqvOaayd+bfCg"}},"state_key":"","type":"m.room.name"},"latest_event_ids":["$1xoUuqOFjFFJgwA5:localhost"],"adds_state_event_ids":["$1xoUuqOFjFFJgwA5:localhost"],"removes_state_event_ids":["$j7KtuOzM0K15h3Kr:localhost"],"last_sent_event_id":"$RHNjeYUvXVZfb93t:localhost"}}`,
// $ curl -XPUT -d '{"msgtype":"m.text","body":"hello bob"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/2?access_token=@alice:localhost"
`{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}]],"content":{"body":"hello bob","msgtype":"m.text"},"depth":0,"event_id":"$4NBTdIwDxq5fDGpv:localhost","hashes":{"sha256":"msCIESAya8kD7nLCopxkEqrgVuGfrlr9YBIADH5czTA"},"origin":"localhost","origin_server_ts":1494411674630,"prev_events":[["$1xoUuqOFjFFJgwA5:localhost",{"sha256":"ZXj+kY6sqQpf5vsNqvCMSvNoXXKDKxRE4R7+gZD9Tkk"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"bZRT3NxVlfBWw1PxSlKlgfnJixG+NI5H9QmUK2AjECg+l887BZJNCvAK0eD27N8e9V+c2glyXWYje2wexP2CBw"}},"type":"m.room.message"},"latest_event_ids":["$4NBTdIwDxq5fDGpv:localhost"],"last_sent_event_id":"$1xoUuqOFjFFJgwA5:localhost"}}`,
// $ curl -XPUT -d '{"membership":"invite"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@charlie:localhost?access_token=@bob:localhost"
`{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}],["$wPepDhIla765Odre:localhost",{"sha256":"GqUhRiAkRvPrNBDyUxj+emRfK2P8j6iWtvsXDOUltiI"}]],"content":{"membership":"invite"},"depth":0,"event_id":"$zzLHVlHIWPrnE7DI:localhost","hashes":{"sha256":"LKk7tnYJAHsyffbi9CzfdP+TU4KQ5g6YTgYGKjJ7NxU"},"origin":"localhost","origin_server_ts":1494411709192,"prev_events":[["$4NBTdIwDxq5fDGpv:localhost",{"sha256":"EpqmxEoJP93Zb2Nt2fS95SJWTqqIutHm/Ne8OHqp6Ps"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@bob:localhost","signatures":{"localhost":{"ed25519:something":"GdUzkC+7YKl1XDi7kYuD39yi2L/+nv+YrecIQHS+0BLDQqnEj+iRXfNBuZfTk6lUBCJCHXZlk7MnEIjvWDlZCg"}},"state_key":"@charlie:localhost","type":"m.room.member"},"latest_event_ids":["$zzLHVlHIWPrnE7DI:localhost"],"adds_state_event_ids":["$zzLHVlHIWPrnE7DI:localhost"],"last_sent_event_id":"$4NBTdIwDxq5fDGpv:localhost"}}`,
// $ curl -XPUT -d '{"membership":"join"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@charlie:localhost?access_token=@charlie:localhost"
`{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}],["$2O2DpHB37CuwwJOe:localhost",{"sha256":"ulaRD63dbCyolLTwvInIQpcrtU2c7ex/BHmhpLXAUoE"}],["$zzLHVlHIWPrnE7DI:localhost",{"sha256":"Jw28x9W+GoZYw7sEynsi1fcRzqRQiLddolOa/p26PV0"}]],"content":{"membership":"join"},"unsigned":{"prev_content":{"membership":"invite"},"prev_sender":"@bob:localhost","replaces_state":"$zzLHVlHIWPrnE7DI:localhost"},"depth":0,"event_id":"$uJVKyzZi8ZX0kOd9:localhost","hashes":{"sha256":"9ZZs/Cg0ewpBiCB6iFXXYlmW8koFiesCNGFrOLDTolE"},"origin":"localhost","origin_server_ts":1494411745015,"prev_events":[["$zzLHVlHIWPrnE7DI:localhost",{"sha256":"Jw28x9W+GoZYw7sEynsi1fcRzqRQiLddolOa/p26PV0"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@charlie:localhost","signatures":{"localhost":{"ed25519:something":"+TM0gFPM/M3Ji2BjYuTUTgDyCOWlOq8aTMCxLg7EBvS62yPxJ558f13OWWTczUO5aRAt+PvXsMVM/bp8u6c8DQ"}},"state_key":"@charlie:localhost","type":"m.room.member"},"latest_event_ids":["$uJVKyzZi8ZX0kOd9:localhost"],"adds_state_event_ids":["$uJVKyzZi8ZX0kOd9:localhost"],"removes_state_event_ids":["$zzLHVlHIWPrnE7DI:localhost"],"last_sent_event_id":"$zzLHVlHIWPrnE7DI:localhost"}}`,
// $ curl -XPUT -d '{"msgtype":"m.text","body":"not charlie..."}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/3?access_token=@alice:localhost"
`{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}]],"content":{"body":"not charlie...","msgtype":"m.text"},"depth":0,"event_id":"$Ixfn5WT9ocWTYxfy:localhost","hashes":{"sha256":"hRChdyMQ3AY4jvrPpI8PEX6Taux83Qo5hdSeHlhPxGo"},"origin":"localhost","origin_server_ts":1494411792737,"prev_events":[["$uJVKyzZi8ZX0kOd9:localhost",{"sha256":"BtesLFnHZOREQCeilFM+xvDU/Wdj+nyHMw7IGTh/9gU"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"LC/Zqwu/XdqjmLdTOp/NQaFaE0niSAGgEpa39gCxsnsqEX80P7P5WDn/Kzx6rjWTnhIszrLsnoycqkXQT0Z4DQ"}},"type":"m.room.message"},"latest_event_ids":["$Ixfn5WT9ocWTYxfy:localhost"],"last_sent_event_id":"$uJVKyzZi8ZX0kOd9:localhost"}}`,
// $ curl -XPUT -d '{"membership":"leave"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@charlie:localhost?access_token=@alice:localhost"
`{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}],["$uJVKyzZi8ZX0kOd9:localhost",{"sha256":"BtesLFnHZOREQCeilFM+xvDU/Wdj+nyHMw7IGTh/9gU"}]],"content":{"membership":"leave"},"unsigned":{"prev_content":{"membership":"join"},"prev_sender":"@charlie:localhost","replaces_state":"$uJVKyzZi8ZX0kOd9:localhost"},"depth":0,"event_id":"$om1F4AI8tCYlHUSp:localhost","hashes":{"sha256":"7JVI0uCxSUyEqDJ+o36/zUIlIZkXVK/R6wkrZGvQXDE"},"origin":"localhost","origin_server_ts":1494411855278,"prev_events":[["$Ixfn5WT9ocWTYxfy:localhost",{"sha256":"hOoPIDQFvvNqQJzA5ggjoQi4v1BOELnhnmwU4UArDOY"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"3sxoDLUPnKuDJgFgS3C647BbiXrozxhhxrZOlFP3KgJKzBYv/ht+Jd2V2iSZOvsv94wgRBf0A/lEcJRIqeLgDA"}},"state_key":"@charlie:localhost","type":"m.room.member"},"latest_event_ids":["$om1F4AI8tCYlHUSp:localhost"],"adds_state_event_ids":["$om1F4AI8tCYlHUSp:localhost"],"removes_state_event_ids":["$uJVKyzZi8ZX0kOd9:localhost"],"last_sent_event_id":"$Ixfn5WT9ocWTYxfy:localhost"}}`,
// $ curl -XPUT -d '{"msgtype":"m.text","body":"why did you kick charlie"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/3?access_token=@bob:localhost"
`{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}],["$wPepDhIla765Odre:localhost",{"sha256":"GqUhRiAkRvPrNBDyUxj+emRfK2P8j6iWtvsXDOUltiI"}]],"content":{"body":"why did you kick charlie","msgtype":"m.text"},"depth":0,"event_id":"$hgao5gTmr3r9TtK2:localhost","hashes":{"sha256":"Aa2ZCrvwjX5xhvkVqIOFUeEGqrnrQZjjNFiZRybjsPY"},"origin":"localhost","origin_server_ts":1494411912809,"prev_events":[["$om1F4AI8tCYlHUSp:localhost",{"sha256":"yVs+CW7AiJrJOYouL8xPIBrtIHAhnbxaegna8MxeCto"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@bob:localhost","signatures":{"localhost":{"ed25519:something":"sGkpbEXGsvAuCvE3wb5E9H5fjCVKpRdWNt6csj1bCB9Fmg4Rg4mvj3TAJ+91DjO8IPsgSxDKdqqRYF0OtcynBA"}},"type":"m.room.message"},"latest_event_ids":["$hgao5gTmr3r9TtK2:localhost"],"last_sent_event_id":"$om1F4AI8tCYlHUSp:localhost"}}`,
// $ curl -XPUT -d '{"name":"No Charlies"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.name?access_token=@alice:localhost"
`{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}]],"content":{"name":"No Charlies"},"depth":0,"event_id":"$CY4XDoxjbns3a4Pc:localhost","hashes":{"sha256":"chk72pVkp3AGR2FtdC0mORBWS1b9ePnRN4WK3BP0BiI"},"origin":"localhost","origin_server_ts":1494411959114,"prev_events":[["$hgao5gTmr3r9TtK2:localhost",{"sha256":"/4/OG4Q2YalIeBtN76BEPIieBKA/3UFshR9T+WJip4o"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"mapvA3KJYgw5FmzJMhSFa/+JSuNyv2eKAkiGomAeBB7LQ1e9nK9XhW/Fp7a5Z2Sy2ENwHyd3ij7FEGiLOnSIAw"}},"state_key":"","type":"m.room.name"},"latest_event_ids":["$CY4XDoxjbns3a4Pc:localhost"],"adds_state_event_ids":["$CY4XDoxjbns3a4Pc:localhost"],"removes_state_event_ids":["$1xoUuqOFjFFJgwA5:localhost"],"last_sent_event_id":"$hgao5gTmr3r9TtK2:localhost"}}`,
// $ curl -XPUT -d '{"msgtype":"m.text","body":"whatever"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/3?access_token=@bob:localhost"
`{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}],["$wPepDhIla765Odre:localhost",{"sha256":"GqUhRiAkRvPrNBDyUxj+emRfK2P8j6iWtvsXDOUltiI"}]],"content":{"body":"whatever","msgtype":"m.text"},"depth":0,"event_id":"$pl8VBHRPYDmsnDh4:localhost","hashes":{"sha256":"FYqY9+/cepwIxxjfFV3AjOFBXkTlyEI2jep87dUc+SU"},"origin":"localhost","origin_server_ts":1494411988548,"prev_events":[["$CY4XDoxjbns3a4Pc:localhost",{"sha256":"hCoV63fp8eiquVdEefsOqJtLmJhw4wTlRv+wNTS20Ac"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@bob:localhost","signatures":{"localhost":{"ed25519:something":"sQKwRzE59eZyb8rDySo/pVwZXBh0nA5zx+kjEyXglxIQrTre+8Gj3R7Prni+RE3Dq7oWfKYV7QklTLURAaSICQ"}},"type":"m.room.message"},"latest_event_ids":["$pl8VBHRPYDmsnDh4:localhost"],"last_sent_event_id":"$CY4XDoxjbns3a4Pc:localhost"}}`,
// $ curl -XPUT -d '{"membership":"leave"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@bob:localhost?access_token=@bob:localhost"
`{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}],["$wPepDhIla765Odre:localhost",{"sha256":"GqUhRiAkRvPrNBDyUxj+emRfK2P8j6iWtvsXDOUltiI"}]],"content":{"membership":"leave"},"depth":0,"event_id":"$acCW4IgnBo8YD3jw:localhost","hashes":{"sha256":"porP+E2yftBGjfS381+WpZeDM9gZHsM3UydlBcRKBLw"},"origin":"localhost","origin_server_ts":1494412037042,"prev_events":[["$pl8VBHRPYDmsnDh4:localhost",{"sha256":"b+qQ380JDFq7quVU9EbIJ2sbpUKM1LAUNX0ZZUoVMZw"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@bob:localhost","signatures":{"localhost":{"ed25519:something":"kxbjTIC0/UR4cOYUAOTNiUc0SSVIF4BY6Rq6IEgYJemq4jcU2fYqum4mFxIQTDKKXMSRHEoNPDmYMFIJwkrsCg"}},"state_key":"@bob:localhost","type":"m.room.member"},"latest_event_ids":["$acCW4IgnBo8YD3jw:localhost"],"adds_state_event_ids":["$acCW4IgnBo8YD3jw:localhost"],"removes_state_event_ids":["$wPepDhIla765Odre:localhost"],"last_sent_event_id":"$pl8VBHRPYDmsnDh4:localhost"}}`,
// $ curl -XPUT -d '{"msgtype":"m.text","body":"im alone now"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/3?access_token=@alice:localhost"
`{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}]],"content":{"body":"im alone now","msgtype":"m.text"},"depth":0,"event_id":"$nYdEXrvTDeb7DfkC:localhost","hashes":{"sha256":"qibC5NmlJpSRMBWSWxy1pv73FXymhPDXQFMmGosfsV0"},"origin":"localhost","origin_server_ts":1494412084668,"prev_events":[["$acCW4IgnBo8YD3jw:localhost",{"sha256":"8h3uXoE6pnI9iLnXI6493qJ0HeuRQfenRIu9PcgH72g"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"EHRoZznhXywhYeIn83o4FSFm3No/aOdLQPHQ68YGtNgESWwpuWLkkGVjoISjz3QgXQ06Fl3cHt7nlTaAHpCNAg"}},"type":"m.room.message"},"latest_event_ids":["$nYdEXrvTDeb7DfkC:localhost"],"last_sent_event_id":"$acCW4IgnBo8YD3jw:localhost"}}`,
// $ curl -XPUT -d '{"membership":"invite"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@bob:localhost?access_token=@alice:localhost"
`{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}],["$acCW4IgnBo8YD3jw:localhost",{"sha256":"8h3uXoE6pnI9iLnXI6493qJ0HeuRQfenRIu9PcgH72g"}]],"content":{"membership":"invite"},"depth":0,"event_id":"$gKNfcXLlWvs2cFad:localhost","hashes":{"sha256":"iYDOUjYkaGSFbVp7TRVFvGJyGMEuBHMQrJ9XqwhzmPI"},"origin":"localhost","origin_server_ts":1494412135845,"prev_events":[["$nYdEXrvTDeb7DfkC:localhost",{"sha256":"83T5Q3+nDvtS0oJTEhHxIw02twBDa1A7QR2bHtnxv1Y"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"ofw009aMJMqVjww9eDXgeTjOQqSlJl/GN/AAb+6mZAPcUI8aVgRlXOSESfhu1ONEuV/yNUycxNXWfMwuvoWsDg"}},"state_key":"@bob:localhost","type":"m.room.member"},"latest_event_ids":["$gKNfcXLlWvs2cFad:localhost"],"adds_state_event_ids":["$gKNfcXLlWvs2cFad:localhost"],"removes_state_event_ids":["$acCW4IgnBo8YD3jw:localhost"],"last_sent_event_id":"$nYdEXrvTDeb7DfkC:localhost"}}`,
// $ curl -XPUT -d '{"membership":"leave"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@bob:localhost?access_token=@bob:localhost"
`{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}],["$gKNfcXLlWvs2cFad:localhost",{"sha256":"/TYIY+L9qjg516Bzl8sadu+Np21KkxE4KdPXALeJ9eE"}]],"content":{"membership":"leave"},"depth":0,"event_id":"$B2q9Tepb6Xc1Rku0:localhost","hashes":{"sha256":"RbHTVdceAEfTALQDZdGrOmakKeTYnChaKjlVuoNUdSY"},"origin":"localhost","origin_server_ts":1494412187614,"prev_events":[["$gKNfcXLlWvs2cFad:localhost",{"sha256":"/TYIY+L9qjg516Bzl8sadu+Np21KkxE4KdPXALeJ9eE"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@bob:localhost","signatures":{"localhost":{"ed25519:something":"dNtUL86j2zUe5+DkfOkil5VujvFZg4FeTjbtcpeF+3E4SUChCAG3lyR6YOAIYBnjtD0/kqT7OcP3pM6vMEp1Aw"}},"state_key":"@bob:localhost","type":"m.room.member"},"latest_event_ids":["$B2q9Tepb6Xc1Rku0:localhost"],"adds_state_event_ids":["$B2q9Tepb6Xc1Rku0:localhost"],"removes_state_event_ids":["$gKNfcXLlWvs2cFad:localhost"],"last_sent_event_id":"$gKNfcXLlWvs2cFad:localhost"}}`,
// $ curl -XPUT -d '{"msgtype":"m.text","body":"so alone"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/3?access_token=@alice:localhost"
`{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}]],"content":{"body":"so alone","msgtype":"m.text"},"depth":0,"event_id":"$W1nrYHQIbCTTSJOV:localhost","hashes":{"sha256":"uUKSa4U1coDoT3LUcNF25dt+UpUa2pLXzRJ3ljgxXZs"},"origin":"localhost","origin_server_ts":1494412229742,"prev_events":[["$B2q9Tepb6Xc1Rku0:localhost",{"sha256":"0CLru7nGPgyF9AWlZnarCElscSVrXl2MMY2atrz80Uc"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"YlBJyDnE34UhaCB9hirQN5OySfTDoqiBDnNvxomXjU94z4a8g2CLWKjApwd/q/j4HamCUtjgkjJ2um6hNjsVBA"}},"type":"m.room.message"},"latest_event_ids":["$W1nrYHQIbCTTSJOV:localhost"],"last_sent_event_id":"$B2q9Tepb6Xc1Rku0:localhost"}}`,
// $ curl -XPUT -d '{"name":"Everyone welcome"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.name?access_token=@alice:localhost"
`{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}]],"content":{"name":"Everyone welcome"},"depth":0,"event_id":"$nLzxoBC4A0QRvJ1k:localhost","hashes":{"sha256":"PExCybjaMW1TfgFr57MdIRYJ642FY2jnrdW/tpPOf1Y"},"origin":"localhost","origin_server_ts":1494412294551,"prev_events":[["$W1nrYHQIbCTTSJOV:localhost",{"sha256":"HXk/ACcsiaZ/z1f2aZSIhJF8Ih3BWeh1vp+cV/fwoE0"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"RK09L8sQv78y69PNbOLaX8asq5kp51mbqUuct5gd7ZNmaHKnVds6ew06QEn+gHSDAxqQo2tpcfoajp+yMj1HBw"}},"state_key":"","type":"m.room.name"},"latest_event_ids":["$nLzxoBC4A0QRvJ1k:localhost"],"adds_state_event_ids":["$nLzxoBC4A0QRvJ1k:localhost"],"removes_state_event_ids":["$CY4XDoxjbns3a4Pc:localhost"],"last_sent_event_id":"$W1nrYHQIbCTTSJOV:localhost"}}`,
// $ curl -XPUT -d '{"membership":"join"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@charlie:localhost?access_token=@charlie:localhost"
`{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}],["$2O2DpHB37CuwwJOe:localhost",{"sha256":"ulaRD63dbCyolLTwvInIQpcrtU2c7ex/BHmhpLXAUoE"}],["$om1F4AI8tCYlHUSp:localhost",{"sha256":"yVs+CW7AiJrJOYouL8xPIBrtIHAhnbxaegna8MxeCto"}]],"content":{"membership":"join"},"depth":0,"event_id":"$Zo6P8r9bczF6kctV:localhost","hashes":{"sha256":"R3J2iUWnGxVdmly8ah+Dgb5VbJ2i/e8BLaWM0z9eZKU"},"origin":"localhost","origin_server_ts":1494412338689,"prev_events":[["$nLzxoBC4A0QRvJ1k:localhost",{"sha256":"TDcFaArAXpxIJ1noSubcFqkLXiQTrc1Dw1+kgCtx3XY"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@charlie:localhost","signatures":{"localhost":{"ed25519:something":"tVnjLVoJ9SLlMQIJSK/6zANWaEu8tVVkx3AEJiC3y5JmhPORb3PyG8eE+e/9hC4aJSQL8LGLaJNWXukMpb2SBg"}},"state_key":"@charlie:localhost","type":"m.room.member"},"latest_event_ids":["$Zo6P8r9bczF6kctV:localhost"],"adds_state_event_ids":["$Zo6P8r9bczF6kctV:localhost"],"removes_state_event_ids":["$om1F4AI8tCYlHUSp:localhost"],"last_sent_event_id":"$nLzxoBC4A0QRvJ1k:localhost"}}`,
// $ curl -XPUT -d '{"msgtype":"m.text","body":"hiiiii"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/3?access_token=@charlie:localhost"
`{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}],["$Zo6P8r9bczF6kctV:localhost",{"sha256":"mnjt3WTYqwtuyl2Fca+0cgm6moHaNL+W9BqRJTQzdEY"}]],"content":{"body":"hiiiii","msgtype":"m.text"},"depth":0,"event_id":"$YAEvK8u2zkTsjf5P:localhost","hashes":{"sha256":"6hKy61h1tuHjYdfpq2MnaPtGEBAZOUz8FLTtxLwjK5A"},"origin":"localhost","origin_server_ts":1494412375465,"prev_events":[["$Zo6P8r9bczF6kctV:localhost",{"sha256":"mnjt3WTYqwtuyl2Fca+0cgm6moHaNL+W9BqRJTQzdEY"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@charlie:localhost","signatures":{"localhost":{"ed25519:something":"BsSLaMM5U/YkyvBZ00J/+si9My+wAJZOcBhBeato0oHayiag7FW77ZpSTfADazPdNH62kjB0sdP9CN6vQA7yDg"}},"type":"m.room.message"},"latest_event_ids":["$YAEvK8u2zkTsjf5P:localhost"],"last_sent_event_id":"$Zo6P8r9bczF6kctV:localhost"}}`,
}

2
dendrite-config.yaml
View file

@ -112,7 +112,7 @@ global:
# Maximum number of entries to hold in the DNS cache, and
# for how long those items should be considered valid in seconds.
cache_size: 256
cache_lifetime: 300
cache_lifetime: "5m" # 5minutes; see https://pkg.go.dev/time@master#ParseDuration for more
# Configuration for the Appservice API.
app_service_api:

54
federationapi/routing/join.go
View file

@ -26,6 +26,7 @@ import (
"github.com/matrix-org/dendrite/setup/config"
"github.com/matrix-org/gomatrixserverlib"
"github.com/matrix-org/util"
"github.com/sirupsen/logrus"
)
// MakeJoin implements the /make_join API
@ -228,6 +229,21 @@ func SendJoin(
}
}
// Check that this is in fact a join event
membership, err := event.Membership()
if err != nil {
return util.JSONResponse{
Code: http.StatusBadRequest,
JSON: jsonerror.BadJSON("missing content.membership key"),
}
}
if membership != gomatrixserverlib.Join {
return util.JSONResponse{
Code: http.StatusBadRequest,
JSON: jsonerror.BadJSON("membership must be 'join'"),
}
}
// Check that the event is signed by the server sending the request.
redacted := event.Redact()
verifyRequests := []gomatrixserverlib.VerifyJSONRequest{{
@ -271,31 +287,51 @@ func SendJoin(
// Check if the user is already in the room. If they're already in then
// there isn't much point in sending another join event into the room.
// Also check to see if they are banned: if they are then we reject them.
alreadyJoined := false
isBanned := false
for _, se := range stateAndAuthChainResponse.StateEvents {
if !se.StateKeyEquals(*event.StateKey()) {
continue
}
if membership, merr := se.Membership(); merr == nil {
alreadyJoined = (membership == gomatrixserverlib.Join)
isBanned = (membership == gomatrixserverlib.Ban)
break
}
}
if isBanned {
return util.JSONResponse{
Code: http.StatusForbidden,
JSON: jsonerror.Forbidden("user is banned"),
}
}
// Send the events to the room server.
// We are responsible for notifying other servers that the user has joined
// the room, so set SendAsServer to cfg.Matrix.ServerName
if !alreadyJoined {
if err = api.SendEvents(
httpReq.Context(), rsAPI,
api.KindNew,
[]*gomatrixserverlib.HeaderedEvent{
event.Headered(stateAndAuthChainResponse.RoomVersion),
var response api.InputRoomEventsResponse
rsAPI.InputRoomEvents(httpReq.Context(), &api.InputRoomEventsRequest{
InputRoomEvents: []api.InputRoomEvent{
{
Kind: api.KindNew,
Event: event.Headered(stateAndAuthChainResponse.RoomVersion),
AuthEventIDs: event.AuthEventIDs(),
SendAsServer: string(cfg.Matrix.ServerName),
TransactionID: nil,
},
},
cfg.Matrix.ServerName,
nil,
); err != nil {
util.GetLogger(httpReq.Context()).WithError(err).Error("SendEvents failed")
}, &response)
if response.ErrMsg != "" {
util.GetLogger(httpReq.Context()).WithField(logrus.ErrorKey, response.ErrMsg).Error("SendEvents failed")
if response.NotAllowed {
return util.JSONResponse{
Code: http.StatusBadRequest,
JSON: jsonerror.Forbidden(response.ErrMsg),
}
}
return jsonerror.InternalServerError()
}
}

46
federationapi/routing/keys.go
View file

@ -188,40 +188,46 @@ func NotaryKeys(
}
response.ServerKeys = []json.RawMessage{}
for serverName := range req.ServerKeys {
var keys *gomatrixserverlib.ServerKeys
for serverName, kidToCriteria := range req.ServerKeys {
var keyList []gomatrixserverlib.ServerKeys
if serverName == cfg.Matrix.ServerName {
if k, err := localKeys(cfg, time.Now().Add(cfg.Matrix.KeyValidityPeriod)); err == nil {
keys = k
keyList = append(keyList, *k)
} else {
return util.ErrorResponse(err)
}
} else {
if k, err := fsAPI.GetServerKeys(httpReq.Context(), serverName); err == nil {
keys = &k
} else {
var resp federationSenderAPI.QueryServerKeysResponse
err := fsAPI.QueryServerKeys(httpReq.Context(), &federationSenderAPI.QueryServerKeysRequest{
ServerName: serverName,
KeyIDToCriteria: kidToCriteria,
}, &resp)
if err != nil {
return util.ErrorResponse(err)
}
keyList = append(keyList, resp.ServerKeys...)
}
if keys == nil {
if len(keyList) == 0 {
continue
}
j, err := json.Marshal(keys)
if err != nil {
logrus.WithError(err).Errorf("Failed to marshal %q response", serverName)
return jsonerror.InternalServerError()
}
for _, keys := range keyList {
j, err := json.Marshal(keys)
if err != nil {
logrus.WithError(err).Errorf("Failed to marshal %q response", serverName)
return jsonerror.InternalServerError()
}
js, err := gomatrixserverlib.SignJSON(
string(cfg.Matrix.ServerName), cfg.Matrix.KeyID, cfg.Matrix.PrivateKey, j,
)
if err != nil {
logrus.WithError(err).Errorf("Failed to sign %q response", serverName)
return jsonerror.InternalServerError()
}
js, err := gomatrixserverlib.SignJSON(
string(cfg.Matrix.ServerName), cfg.Matrix.KeyID, cfg.Matrix.PrivateKey, j,
)
if err != nil {
logrus.WithError(err).Errorf("Failed to sign %q response", serverName)
return jsonerror.InternalServerError()
}
response.ServerKeys = append(response.ServerKeys, js)
response.ServerKeys = append(response.ServerKeys, js)
}
}
return util.JSONResponse{

42
federationapi/routing/leave.go
View file

@ -22,6 +22,7 @@ import (
"github.com/matrix-org/dendrite/setup/config"
"github.com/matrix-org/gomatrixserverlib"
"github.com/matrix-org/util"
"github.com/sirupsen/logrus"
)
// MakeLeave implements the /make_leave API
@ -174,6 +175,13 @@ func SendLeave(
}
}
if event.StateKey() == nil {
return util.JSONResponse{
Code: http.StatusBadRequest,
JSON: jsonerror.InvalidArgumentValue("missing state_key"),
}
}
// Check if the user has already left. If so, no-op!
queryReq := &api.QueryLatestEventsAndStateRequest{
RoomID: roomID,
@ -240,7 +248,10 @@ func SendLeave(
mem, err := event.Membership()
if err != nil {
util.GetLogger(httpReq.Context()).WithError(err).Error("event.Membership failed")
return jsonerror.InternalServerError()
return util.JSONResponse{
Code: http.StatusBadRequest,
JSON: jsonerror.BadJSON("missing content.membership key"),
}
}
if mem != gomatrixserverlib.Leave {
return util.JSONResponse{
@ -252,16 +263,27 @@ func SendLeave(
// Send the events to the room server.
// We are responsible for notifying other servers that the user has left
// the room, so set SendAsServer to cfg.Matrix.ServerName
if err = api.SendEvents(
httpReq.Context(), rsAPI,
api.KindNew,
[]*gomatrixserverlib.HeaderedEvent{
event.Headered(verRes.RoomVersion),
var response api.InputRoomEventsResponse
rsAPI.InputRoomEvents(httpReq.Context(), &api.InputRoomEventsRequest{
InputRoomEvents: []api.InputRoomEvent{
{
Kind: api.KindNew,
Event: event.Headered(verRes.RoomVersion),
AuthEventIDs: event.AuthEventIDs(),
SendAsServer: string(cfg.Matrix.ServerName),
TransactionID: nil,
},
},
cfg.Matrix.ServerName,
nil,
); err != nil {
util.GetLogger(httpReq.Context()).WithError(err).Error("producer.SendEvents failed")
}, &response)
if response.ErrMsg != "" {
util.GetLogger(httpReq.Context()).WithField(logrus.ErrorKey, response.ErrMsg).WithField("not_allowed", response.NotAllowed).Error("producer.SendEvents failed")
if response.NotAllowed {
return util.JSONResponse{
Code: http.StatusBadRequest,
JSON: jsonerror.Forbidden(response.ErrMsg),
}
}
return jsonerror.InternalServerError()
}

291
federationapi/routing/send.go
View file

@ -16,7 +16,6 @@ package routing
import (
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
@ -24,7 +23,6 @@ import (
"sync"
"time"
"github.com/getsentry/sentry-go"
"github.com/matrix-org/dendrite/clientapi/jsonerror"
eduserverAPI "github.com/matrix-org/dendrite/eduserver/api"
federationAPI "github.com/matrix-org/dendrite/federationapi/api"
@ -36,6 +34,7 @@ import (
"github.com/matrix-org/util"
"github.com/prometheus/client_golang/prometheus"
"github.com/sirupsen/logrus"
"go.uber.org/atomic"
)
const (
@ -90,6 +89,67 @@ func init() {
)
}
type sendFIFOQueue struct {
tasks []*inputTask
count int
mutex sync.Mutex
notifs chan struct{}
}
func newSendFIFOQueue() *sendFIFOQueue {
q := &sendFIFOQueue{
notifs: make(chan struct{}, 1),
}
return q
}
func (q *sendFIFOQueue) push(frame *inputTask) {
q.mutex.Lock()
defer q.mutex.Unlock()
q.tasks = append(q.tasks, frame)
q.count++
select {
case q.notifs <- struct{}{}:
default:
}
}
// pop returns the first item of the queue, if there is one.
// The second return value will indicate if a task was returned.
func (q *sendFIFOQueue) pop() (*inputTask, bool) {
q.mutex.Lock()
defer q.mutex.Unlock()
if q.count == 0 {
return nil, false
}
frame := q.tasks[0]
q.tasks[0] = nil
q.tasks = q.tasks[1:]
q.count--
if q.count == 0 {
// Force a GC of the underlying array, since it might have
// grown significantly if the queue was hammered for some reason
q.tasks = nil
}
return frame, true
}
type inputTask struct {
ctx context.Context
t *txnReq
event *gomatrixserverlib.Event
wg *sync.WaitGroup
err error // written back by worker, only safe to read when all tasks are done
duration time.Duration // written back by worker, only safe to read when all tasks are done
}
type inputWorker struct {
running atomic.Bool
input *sendFIFOQueue
}
var inputWorkers sync.Map // room ID -> *inputWorker
// Send implements /_matrix/federation/v1/send/{txnID}
func Send(
httpReq *http.Request,
@ -145,7 +205,7 @@ func Send(
util.GetLogger(httpReq.Context()).Infof("Received transaction %q from %q containing %d PDUs, %d EDUs", txnID, request.Origin(), len(t.PDUs), len(t.EDUs))
resp, jsonErr := t.processTransaction(context.Background())
resp, jsonErr := t.processTransaction(httpReq.Context())
if jsonErr != nil {
util.GetLogger(httpReq.Context()).WithField("jsonErr", jsonErr).Error("t.processTransaction failed")
return *jsonErr
@ -172,7 +232,8 @@ type txnReq struct {
// something that can tell us about which servers are in a room right now
servers federationAPI.ServersInRoomProvider
// a list of events from the auth and prev events which we already had
hadEvents map[string]bool
hadEvents map[string]bool
hadEventsMutex sync.Mutex
// local cache of events for auth checks, etc - this may include events
// which the roomserver is unaware of.
haveEvents map[string]*gomatrixserverlib.HeaderedEvent
@ -180,6 +241,12 @@ type txnReq struct {
work string // metrics
}
func (t *txnReq) hadEvent(eventID string, had bool) {
t.hadEventsMutex.Lock()
defer t.hadEventsMutex.Unlock()
t.hadEvents[eventID] = had
}
// A subset of FederationClient functionality that txn requires. Useful for testing.
type txnFederationClient interface {
LookupState(ctx context.Context, s gomatrixserverlib.ServerName, roomID string, eventID string, roomVersion gomatrixserverlib.RoomVersion) (
@ -193,8 +260,9 @@ type txnFederationClient interface {
func (t *txnReq) processTransaction(ctx context.Context) (*gomatrixserverlib.RespSend, *util.JSONResponse) {
results := make(map[string]gomatrixserverlib.PDUResult)
var wg sync.WaitGroup
var tasks []*inputTask
pdus := []*gomatrixserverlib.HeaderedEvent{}
for _, pdu := range t.PDUs {
pduCountTotal.WithLabelValues("total").Inc()
var header struct {
@ -245,83 +313,94 @@ func (t *txnReq) processTransaction(ctx context.Context) (*gomatrixserverlib.Res
}
continue
}
pdus = append(pdus, event.Headered(verRes.RoomVersion))
}
// Process the events.
for _, e := range pdus {
evStart := time.Now()
if err := t.processEvent(ctx, e.Unwrap()); err != nil {
// If the error is due to the event itself being bad then we skip
// it and move onto the next event. We report an error so that the
// sender knows that we have skipped processing it.
//
// However if the event is due to a temporary failure in our server
// such as a database being unavailable then we should bail, and
// hope that the sender will retry when we are feeling better.
//
// It is uncertain what we should do if an event fails because
// we failed to fetch more information from the sending server.
// For example if a request to /state fails.
// If we skip the event then we risk missing the event until we
// receive another event referencing it.
// If we bail and stop processing then we risk wedging incoming
// transactions from that server forever.
if isProcessingErrorFatal(err) {
sentry.CaptureException(err)
// Any other error should be the result of a temporary error in
// our server so we should bail processing the transaction entirely.
util.GetLogger(ctx).Warnf("Processing %s failed fatally: %s", e.EventID(), err)
jsonErr := util.ErrorResponse(err)
processEventSummary.WithLabelValues(t.work, MetricsOutcomeFatal).Observe(
float64(time.Since(evStart).Nanoseconds()) / 1000.,
)
return nil, &jsonErr
} else {
// Auth errors mean the event is 'rejected' which have to be silent to appease sytest
errMsg := ""
outcome := MetricsOutcomeRejected
_, rejected := err.(*gomatrixserverlib.NotAllowed)
if !rejected {
errMsg = err.Error()
outcome = MetricsOutcomeFail
}
util.GetLogger(ctx).WithError(err).WithField("event_id", e.EventID()).WithField("rejected", rejected).Warn(
"Failed to process incoming federation event, skipping",
)
processEventSummary.WithLabelValues(t.work, outcome).Observe(
float64(time.Since(evStart).Nanoseconds()) / 1000.,
)
results[e.EventID()] = gomatrixserverlib.PDUResult{
Error: errMsg,
}
}
} else {
results[e.EventID()] = gomatrixserverlib.PDUResult{}
pduCountTotal.WithLabelValues("success").Inc()
processEventSummary.WithLabelValues(t.work, MetricsOutcomeOK).Observe(
float64(time.Since(evStart).Nanoseconds()) / 1000.,
)
v, _ := inputWorkers.LoadOrStore(event.RoomID(), &inputWorker{
input: newSendFIFOQueue(),
})
worker := v.(*inputWorker)
wg.Add(1)
task := &inputTask{
ctx: ctx,
t: t,
event: event,
wg: &wg,
}
tasks = append(tasks, task)
worker.input.push(task)
if worker.running.CAS(false, true) {
go worker.run()
}
}
t.processEDUs(ctx)
wg.Wait()
for _, task := range tasks {
if task.err != nil {
results[task.event.EventID()] = gomatrixserverlib.PDUResult{
Error: task.err.Error(),
}
} else {
results[task.event.EventID()] = gomatrixserverlib.PDUResult{}
}
}
if c := len(results); c > 0 {
util.GetLogger(ctx).Infof("Processed %d PDUs from transaction %q", c, t.TransactionID)
}
return &gomatrixserverlib.RespSend{PDUs: results}, nil
}
// isProcessingErrorFatal returns true if the error is really bad and
// we should stop processing the transaction, and returns false if it
// is just some less serious error about a specific event.
func isProcessingErrorFatal(err error) bool {
switch err {
case sql.ErrConnDone:
case sql.ErrTxDone:
return true
func (t *inputWorker) run() {
defer t.running.Store(false)
for {
task, ok := t.input.pop()
if !ok {
return
}
if task == nil {
continue
}
func() {
defer task.wg.Done()
select {
case <-task.ctx.Done():
task.err = context.DeadlineExceeded
pduCountTotal.WithLabelValues("expired").Inc()
return
default:
evStart := time.Now()
// TODO: Is 5 minutes too long?
ctx, cancel := context.WithTimeout(context.Background(), time.Minute*5)
task.err = task.t.processEvent(ctx, task.event)
cancel()
task.duration = time.Since(evStart)
if err := task.err; err != nil {
switch err.(type) {
case *gomatrixserverlib.NotAllowed:
processEventSummary.WithLabelValues(task.t.work, MetricsOutcomeRejected).Observe(
float64(time.Since(evStart).Nanoseconds()) / 1000.,
)
util.GetLogger(task.ctx).WithError(err).WithField("event_id", task.event.EventID()).WithField("rejected", true).Warn(
"Failed to process incoming federation event, skipping",
)
task.err = nil // make "rejected" failures silent
default:
processEventSummary.WithLabelValues(task.t.work, MetricsOutcomeFail).Observe(
float64(time.Since(evStart).Nanoseconds()) / 1000.,
)
util.GetLogger(task.ctx).WithError(err).WithField("event_id", task.event.EventID()).WithField("rejected", false).Warn(
"Failed to process incoming federation event, skipping",
)
}
} else {
pduCountTotal.WithLabelValues("success").Inc()
processEventSummary.WithLabelValues(task.t.work, MetricsOutcomeOK).Observe(
float64(time.Since(evStart).Nanoseconds()) / 1000.,
)
}
}
}()
}
return false
}
type roomNotFoundError struct {
@ -494,6 +573,23 @@ func (t *txnReq) processEvent(ctx context.Context, e *gomatrixserverlib.Event) e
logger := util.GetLogger(ctx).WithField("event_id", e.EventID()).WithField("room_id", e.RoomID())
t.work = "" // reset from previous event
// Ask the roomserver if we know about the room and/or if we're joined
// to it. If we aren't then we won't bother processing the event.
joinedReq := api.QueryServerJoinedToRoomRequest{
RoomID: e.RoomID(),
}
var joinedRes api.QueryServerJoinedToRoomResponse
if err := t.rsAPI.QueryServerJoinedToRoom(ctx, &joinedReq, &joinedRes); err != nil {
return fmt.Errorf("t.rsAPI.QueryServerJoinedToRoom: %w", err)
}
if !joinedRes.RoomExists || !joinedRes.IsInRoom {
// We don't believe we're a member of this room, therefore there's
// no point in wasting work trying to figure out what to do with
// missing auth or prev events. Drop the event.
return roomNotFoundError{e.RoomID()}
}
// Work out if the roomserver knows everything it needs to know to auth
// the event. This includes the prev_events and auth_events.
// NOTE! This is going to include prev_events that have an empty state
@ -510,23 +606,13 @@ func (t *txnReq) processEvent(ctx context.Context, e *gomatrixserverlib.Event) e
return fmt.Errorf("t.rsAPI.QueryMissingAuthPrevEvents: %w", err)
}
if !stateResp.RoomExists {
// TODO: When synapse receives a message for a room it is not in it
// asks the remote server for the state of the room so that it can
// check if the remote server knows of a join "m.room.member" event
// that this server is unaware of.
// However generally speaking we should reject events for rooms we
// aren't a member of.
return roomNotFoundError{e.RoomID()}
}
// Prepare a map of all the events we already had before this point, so
// that we don't send them to the roomserver again.
for _, eventID := range append(e.AuthEventIDs(), e.PrevEventIDs()...) {
t.hadEvents[eventID] = true
t.hadEvent(eventID, true)
}
for _, eventID := range append(stateResp.MissingAuthEventIDs, stateResp.MissingPrevEventIDs...) {
t.hadEvents[eventID] = false
t.hadEvent(eventID, false)
}
if len(stateResp.MissingAuthEventIDs) > 0 {
@ -601,7 +687,7 @@ withNextEvent:
); err != nil {
return fmt.Errorf("api.SendEvents: %w", err)
}
t.hadEvents[ev.EventID()] = true // if the roomserver didn't know about the event before, it does now
t.hadEvent(ev.EventID(), true) // if the roomserver didn't know about the event before, it does now
t.cacheAndReturn(ev.Headered(stateResp.RoomVersion))
delete(missingAuthEvents, missingAuthEventID)
continue withNextEvent
@ -625,19 +711,9 @@ func checkAllowedByState(e *gomatrixserverlib.Event, stateEvents []*gomatrixserv
return gomatrixserverlib.Allowed(e, &authUsingState)
}
var processEventWithMissingStateMutexes = internal.NewMutexByRoom()
func (t *txnReq) processEventWithMissingState(
ctx context.Context, e *gomatrixserverlib.Event, roomVersion gomatrixserverlib.RoomVersion,
) error {
processEventWithMissingStateMutexes.Lock(e.RoomID())
defer processEventWithMissingStateMutexes.Unlock(e.RoomID())
// Do this with a fresh context, so that we keep working even if the
// original request times out. With any luck, by the time the remote
// side retries, we'll have fetched the missing state.
gmectx, cancel := context.WithTimeout(context.Background(), time.Minute*5)
defer cancel()
// We are missing the previous events for this events.
// This means that there is a gap in our view of the history of the
// room. There two ways that we can handle such a gap:
@ -658,7 +734,7 @@ func (t *txnReq) processEventWithMissingState(
// - fill in the gap completely then process event `e` returning no backwards extremity
// - fail to fill in the gap and tell us to terminate the transaction err=not nil
// - fail to fill in the gap and tell us to fetch state at the new backwards extremity, and to not terminate the transaction
newEvents, err := t.getMissingEvents(gmectx, e, roomVersion)
newEvents, err := t.getMissingEvents(ctx, e, roomVersion)
if err != nil {
return err
}
@ -685,7 +761,7 @@ func (t *txnReq) processEventWithMissingState(
// Look up what the state is after the backward extremity. This will either
// come from the roomserver, if we know all the required events, or it will
// come from a remote server via /state_ids if not.
prevState, trustworthy, lerr := t.lookupStateAfterEvent(gmectx, roomVersion, backwardsExtremity.RoomID(), prevEventID)
prevState, trustworthy, lerr := t.lookupStateAfterEvent(ctx, roomVersion, backwardsExtremity.RoomID(), prevEventID)
if lerr != nil {
util.GetLogger(ctx).WithError(lerr).Errorf("Failed to lookup state after prev_event: %s", prevEventID)
return lerr
@ -729,7 +805,7 @@ func (t *txnReq) processEventWithMissingState(
}
// There's more than one previous state - run them all through state res
t.roomsMu.Lock(e.RoomID())
resolvedState, err = t.resolveStatesAndCheck(gmectx, roomVersion, respStates, backwardsExtremity)
resolvedState, err = t.resolveStatesAndCheck(ctx, roomVersion, respStates, backwardsExtremity)
t.roomsMu.Unlock(e.RoomID())
if err != nil {
util.GetLogger(ctx).WithError(err).Errorf("Failed to resolve state conflicts for event %s", backwardsExtremity.EventID())
@ -739,14 +815,23 @@ func (t *txnReq) processEventWithMissingState(
// First of all, send the backward extremity into the roomserver with the
// newly resolved state. This marks the "oldest" point in the backfill and
// sets the baseline state for any new events after this.
// sets the baseline state for any new events after this. We'll make a
// copy of the hadEvents map so that it can be taken downstream without
// worrying about concurrent map reads/writes, since t.hadEvents is meant
// to be protected by a mutex.
hadEvents := map[string]bool{}
t.hadEventsMutex.Lock()
for k, v := range t.hadEvents {
hadEvents[k] = v
}
t.hadEventsMutex.Unlock()
err = api.SendEventWithState(
context.Background(),
t.rsAPI,
api.KindOld,
resolvedState,
backwardsExtremity.Headered(roomVersion),
t.hadEvents,
hadEvents,
)
if err != nil {
return fmt.Errorf("api.SendEventWithState: %w", err)
@ -842,7 +927,7 @@ func (t *txnReq) lookupStateAfterEventLocally(ctx context.Context, roomID, event
// set the event from the haveEvents cache - this means we will share pointers with other prev_event branches for this
// processEvent request, which is better for memory.
stateEvents[i] = t.cacheAndReturn(ev)
t.hadEvents[ev.EventID()] = true
t.hadEvent(ev.EventID(), true)
}
// we should never access res.StateEvents again so we delete it here to make GC faster
res.StateEvents = nil
@ -877,7 +962,7 @@ func (t *txnReq) lookupStateAfterEventLocally(ctx context.Context, roomID, event
}
for i, ev := range queryRes.Events {
authEvents = append(authEvents, t.cacheAndReturn(queryRes.Events[i]).Unwrap())
t.hadEvents[ev.EventID()] = true
t.hadEvent(ev.EventID(), true)
}
queryRes.Events = nil
}
@ -954,7 +1039,7 @@ func (t *txnReq) getMissingEvents(ctx context.Context, e *gomatrixserverlib.Even
latestEvents := make([]string, len(res.LatestEvents))
for i, ev := range res.LatestEvents {
latestEvents[i] = res.LatestEvents[i].EventID
t.hadEvents[ev.EventID] = true
t.hadEvent(ev.EventID, true)
}
var missingResp *gomatrixserverlib.RespMissingEvents
@ -1090,7 +1175,7 @@ func (t *txnReq) lookupMissingStateViaStateIDs(ctx context.Context, roomID, even
}
for i, ev := range queryRes.Events {
queryRes.Events[i] = t.cacheAndReturn(queryRes.Events[i])
t.hadEvents[ev.EventID()] = true
t.hadEvent(ev.EventID(), true)
evID := queryRes.Events[i].EventID()
if missing[evID] {
delete(missing, evID)

4
federationapi/routing/send_test.go
View file

@ -190,7 +190,9 @@ func (t *testRoomserverAPI) QueryServerJoinedToRoom(
request *api.QueryServerJoinedToRoomRequest,
response *api.QueryServerJoinedToRoomResponse,
) error {
return fmt.Errorf("not implemented")
response.RoomExists = true
response.IsInRoom = true
return nil
}
// Query whether a server is allowed to see an event

22
federationsender/api/api.go
View file

@ -20,7 +20,6 @@ type FederationClient interface {
ClaimKeys(ctx context.Context, s gomatrixserverlib.ServerName, oneTimeKeys map[string]map[string]string) (res gomatrixserverlib.RespClaimKeys, err error)
QueryKeys(ctx context.Context, s gomatrixserverlib.ServerName, keys map[string][]string) (res gomatrixserverlib.RespQueryKeys, err error)
GetEvent(ctx context.Context, s gomatrixserverlib.ServerName, eventID string) (res gomatrixserverlib.Transaction, err error)
GetServerKeys(ctx context.Context, matrixServer gomatrixserverlib.ServerName) (gomatrixserverlib.ServerKeys, error)
MSC2836EventRelationships(ctx context.Context, dst gomatrixserverlib.ServerName, r gomatrixserverlib.MSC2836EventRelationshipsRequest, roomVersion gomatrixserverlib.RoomVersion) (res gomatrixserverlib.MSC2836EventRelationshipsResponse, err error)
MSC2946Spaces(ctx context.Context, dst gomatrixserverlib.ServerName, roomID string, r gomatrixserverlib.MSC2946SpacesRequest) (res gomatrixserverlib.MSC2946SpacesResponse, err error)
LookupServerKeys(ctx context.Context, s gomatrixserverlib.ServerName, keyRequests map[gomatrixserverlib.PublicKeyLookupRequest]gomatrixserverlib.Timestamp) ([]gomatrixserverlib.ServerKeys, error)
@ -41,6 +40,8 @@ func (e *FederationClientError) Error() string {
type FederationSenderInternalAPI interface {
FederationClient
QueryServerKeys(ctx context.Context, request *QueryServerKeysRequest, response *QueryServerKeysResponse) error
// PerformDirectoryLookup looks up a remote room ID from a room alias.
PerformDirectoryLookup(
ctx context.Context,
@ -94,6 +95,25 @@ type FederationSenderInternalAPI interface {
) error
}
type QueryServerKeysRequest struct {
ServerName gomatrixserverlib.ServerName
KeyIDToCriteria map[gomatrixserverlib.KeyID]gomatrixserverlib.PublicKeyNotaryQueryCriteria
}
func (q *QueryServerKeysRequest) KeyIDs() []gomatrixserverlib.KeyID {
kids := make([]gomatrixserverlib.KeyID, len(q.KeyIDToCriteria))
i := 0
for keyID := range q.KeyIDToCriteria {
kids[i] = keyID
i++
}
return kids
}
type QueryServerKeysResponse struct {
ServerKeys []gomatrixserverlib.ServerKeys
}
type PerformDirectoryLookupRequest struct {
RoomAlias string `json:"room_alias"`
ServerName gomatrixserverlib.ServerName `json:"server_name"`

14
federationsender/internal/api.go
View file

@ -202,20 +202,6 @@ func (a *FederationSenderInternalAPI) GetEvent(
return ires.(gomatrixserverlib.Transaction), nil
}
func (a *FederationSenderInternalAPI) GetServerKeys(
ctx context.Context, s gomatrixserverlib.ServerName,
) (gomatrixserverlib.ServerKeys, error) {
ctx, cancel := context.WithTimeout(ctx, time.Second*30)
defer cancel()
ires, err := a.doRequest(s, func() (interface{}, error) {
return a.federation.GetServerKeys(ctx, s)
})
if err != nil {
return gomatrixserverlib.ServerKeys{}, err
}
return ires.(gomatrixserverlib.ServerKeys), nil
}
func (a *FederationSenderInternalAPI) LookupServerKeys(
ctx context.Context, s gomatrixserverlib.ServerName, keyRequests map[gomatrixserverlib.PublicKeyLookupRequest]gomatrixserverlib.Timestamp,
) ([]gomatrixserverlib.ServerKeys, error) {

75
federationsender/internal/query.go
View file

@ -2,8 +2,12 @@ package internal
import (
"context"
"fmt"
"time"
"github.com/matrix-org/dendrite/federationsender/api"
"github.com/matrix-org/gomatrixserverlib"
"github.com/matrix-org/util"
)
// QueryJoinedHostServerNamesInRoom implements api.FederationSenderInternalAPI
@ -20,3 +24,74 @@ func (f *FederationSenderInternalAPI) QueryJoinedHostServerNamesInRoom(
return
}
func (a *FederationSenderInternalAPI) fetchServerKeysDirectly(ctx context.Context, serverName gomatrixserverlib.ServerName) (*gomatrixserverlib.ServerKeys, error) {
ctx, cancel := context.WithTimeout(ctx, time.Second*30)
defer cancel()
ires, err := a.doRequest(serverName, func() (interface{}, error) {
return a.federation.GetServerKeys(ctx, serverName)
})
if err != nil {
return nil, err
}
sks := ires.(gomatrixserverlib.ServerKeys)
return &sks, nil
}
func (a *FederationSenderInternalAPI) fetchServerKeysFromCache(
ctx context.Context, req *api.QueryServerKeysRequest,
) ([]gomatrixserverlib.ServerKeys, error) {
var results []gomatrixserverlib.ServerKeys
for keyID, criteria := range req.KeyIDToCriteria {
serverKeysResponses, _ := a.db.GetNotaryKeys(ctx, req.ServerName, []gomatrixserverlib.KeyID{keyID})
if len(serverKeysResponses) == 0 {
return nil, fmt.Errorf("failed to find server key response for key ID %s", keyID)
}
// we should only get 1 result as we only gave 1 key ID
sk := serverKeysResponses[0]
util.GetLogger(ctx).Infof("fetchServerKeysFromCache: minvalid:%v keys: %+v", criteria.MinimumValidUntilTS, sk)
if criteria.MinimumValidUntilTS != 0 {
// check if it's still valid. if they have the same value that's also valid
if sk.ValidUntilTS < criteria.MinimumValidUntilTS {
return nil, fmt.Errorf(
"found server response for key ID %s but it is no longer valid, min: %v valid_until: %v",
keyID, criteria.MinimumValidUntilTS, sk.ValidUntilTS,
)
}
}
results = append(results, sk)
}
return results, nil
}
func (a *FederationSenderInternalAPI) QueryServerKeys(
ctx context.Context, req *api.QueryServerKeysRequest, res *api.QueryServerKeysResponse,
) error {
// attempt to satisfy the entire request from the cache first
results, err := a.fetchServerKeysFromCache(ctx, req)
if err == nil {
// satisfied entirely from cache, return it
res.ServerKeys = results
return nil
}
util.GetLogger(ctx).WithField("server", req.ServerName).WithError(err).Warn("notary: failed to satisfy keys request entirely from cache, hitting direct")
serverKeys, err := a.fetchServerKeysDirectly(ctx, req.ServerName)
if err != nil {
// try to load as much as we can from the cache in a best effort basis
util.GetLogger(ctx).WithField("server", req.ServerName).WithError(err).Warn("notary: failed to ask server for keys, returning best effort keys")
serverKeysResponses, dbErr := a.db.GetNotaryKeys(ctx, req.ServerName, req.KeyIDs())
if dbErr != nil {
return fmt.Errorf("notary: server returned %s, and db returned %s", err, dbErr)
}
res.ServerKeys = serverKeysResponses
return nil
}
// cache it!
if err = a.db.UpdateNotaryKeys(context.Background(), req.ServerName, *serverKeys); err != nil {
// non-fatal, still return the response
util.GetLogger(ctx).WithError(err).Warn("failed to UpdateNotaryKeys")
}
res.ServerKeys = []gomatrixserverlib.ServerKeys{*serverKeys}
return nil
}

31
federationsender/inthttp/client.go
View file

@ -15,6 +15,7 @@ import (
// HTTP paths for the internal HTTP API
const (
FederationSenderQueryJoinedHostServerNamesInRoomPath = "/federationsender/queryJoinedHostServerNamesInRoom"
FederationSenderQueryServerKeysPath = "/federationsender/queryServerKeys"
FederationSenderPerformDirectoryLookupRequestPath = "/federationsender/performDirectoryLookup"
FederationSenderPerformJoinRequestPath = "/federationsender/performJoinRequest"
@ -31,7 +32,6 @@ const (
FederationSenderLookupStatePath = "/federationsender/client/lookupState"
FederationSenderLookupStateIDsPath = "/federationsender/client/lookupStateIDs"
FederationSenderGetEventPath = "/federationsender/client/getEvent"
FederationSenderGetServerKeysPath = "/federationsender/client/getServerKeys"
FederationSenderLookupServerKeysPath = "/federationsender/client/lookupServerKeys"
FederationSenderEventRelationshipsPath = "/federationsender/client/msc2836eventRelationships"
FederationSenderSpacesSummaryPath = "/federationsender/client/msc2946spacesSummary"
@ -377,31 +377,14 @@ func (h *httpFederationSenderInternalAPI) GetEvent(
return *response.Res, nil
}
type getServerKeys struct {
S gomatrixserverlib.ServerName
ServerKeys gomatrixserverlib.ServerKeys
Err *api.FederationClientError
}
func (h *httpFederationSenderInternalAPI) GetServerKeys(
ctx context.Context, s gomatrixserverlib.ServerName,
) (gomatrixserverlib.ServerKeys, error) {
span, ctx := opentracing.StartSpanFromContext(ctx, "GetServerKeys")
func (h *httpFederationSenderInternalAPI) QueryServerKeys(
ctx context.Context, req *api.QueryServerKeysRequest, res *api.QueryServerKeysResponse,
) error {
span, ctx := opentracing.StartSpanFromContext(ctx, "QueryServerKeys")
defer span.Finish()
request := getServerKeys{
S: s,
}
var response getServerKeys
apiURL := h.federationSenderURL + FederationSenderGetServerKeysPath
err := httputil.PostJSON(ctx, span, h.httpClient, apiURL, &request, &response)
if err != nil {
return gomatrixserverlib.ServerKeys{}, err
}
if response.Err != nil {
return gomatrixserverlib.ServerKeys{}, response.Err
}
return response.ServerKeys, nil
apiURL := h.federationSenderURL + FederationSenderQueryServerKeysPath
return httputil.PostJSON(ctx, span, h.httpClient, apiURL, req, res)
}
type lookupServerKeys struct {

22
federationsender/inthttp/server.go
View file

@ -264,25 +264,17 @@ func AddRoutes(intAPI api.FederationSenderInternalAPI, internalAPIMux *mux.Route
}),
)
internalAPIMux.Handle(
FederationSenderGetServerKeysPath,
httputil.MakeInternalAPI("GetServerKeys", func(req *http.Request) util.JSONResponse {
var request getServerKeys
FederationSenderQueryServerKeysPath,
httputil.MakeInternalAPI("QueryServerKeys", func(req *http.Request) util.JSONResponse {
var request api.QueryServerKeysRequest
var response api.QueryServerKeysResponse
if err := json.NewDecoder(req.Body).Decode(&request); err != nil {
return util.MessageResponse(http.StatusBadRequest, err.Error())
}
res, err := intAPI.GetServerKeys(req.Context(), request.S)
if err != nil {
ferr, ok := err.(*api.FederationClientError)
if ok {
request.Err = ferr
} else {
request.Err = &api.FederationClientError{
Err: err.Error(),
}
}
if err := intAPI.QueryServerKeys(req.Context(), &request, &response); err != nil {
return util.ErrorResponse(err)
}
request.ServerKeys = res
return util.JSONResponse{Code: http.StatusOK, JSON: request}
return util.JSONResponse{Code: http.StatusOK, JSON: &response}
}),
)
internalAPIMux.Handle(

6
federationsender/storage/interface.go
View file

@ -66,4 +66,10 @@ type Database interface {
RenewInboundPeek(ctx context.Context, serverName gomatrixserverlib.ServerName, roomID, peekID string, renewalInterval int64) error
GetInboundPeek(ctx context.Context, serverName gomatrixserverlib.ServerName, roomID, peekID string) (*types.InboundPeek, error)
GetInboundPeeks(ctx context.Context, roomID string) ([]types.InboundPeek, error)
// Update the notary with the given server keys from the given server name.
UpdateNotaryKeys(ctx context.Context, serverName gomatrixserverlib.ServerName, serverKeys gomatrixserverlib.ServerKeys) error
// Query the notary for the server keys for the given server. If `optKeyIDs` is not empty, multiple server keys may be returned (between 1 - len(optKeyIDs))
// such that the combination of all server keys will include all the `optKeyIDs`.
GetNotaryKeys(ctx context.Context, serverName gomatrixserverlib.ServerName, optKeyIDs []gomatrixserverlib.KeyID) ([]gomatrixserverlib.ServerKeys, error)
}

64
federationsender/storage/postgres/notary_server_keys_json_table.go Normal file
View file

@ -0,0 +1,64 @@
// Copyright 2021 The Matrix.org Foundation C.I.C.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package postgres
import (
"context"
"database/sql"
"github.com/matrix-org/dendrite/federationsender/storage/tables"
"github.com/matrix-org/gomatrixserverlib"
)
const notaryServerKeysJSONSchema = `
CREATE SEQUENCE IF NOT EXISTS federationsender_notary_server_keys_json_pkey;
CREATE TABLE IF NOT EXISTS federationsender_notary_server_keys_json (
notary_id BIGINT PRIMARY KEY NOT NULL DEFAULT nextval('federationsender_notary_server_keys_json_pkey'),
response_json TEXT NOT NULL,
server_name TEXT NOT NULL,
valid_until BIGINT NOT NULL
);
`
const insertServerKeysJSONSQL = "" +
"INSERT INTO federationsender_notary_server_keys_json (response_json, server_name, valid_until) VALUES ($1, $2, $3)" +
" RETURNING notary_id"
type notaryServerKeysStatements struct {
db *sql.DB
insertServerKeysJSONStmt *sql.Stmt
}
func NewPostgresNotaryServerKeysTable(db *sql.DB) (s *notaryServerKeysStatements, err error) {
s = &notaryServerKeysStatements{
db: db,
}
_, err = db.Exec(notaryServerKeysJSONSchema)
if err != nil {
return
}
if s.insertServerKeysJSONStmt, err = db.Prepare(insertServerKeysJSONSQL); err != nil {
return
}
return
}
func (s *notaryServerKeysStatements) InsertJSONResponse(
ctx context.Context, txn *sql.Tx, keyQueryResponseJSON gomatrixserverlib.ServerKeys, serverName gomatrixserverlib.ServerName, validUntil gomatrixserverlib.Timestamp,
) (tables.NotaryID, error) {
var notaryID tables.NotaryID
return notaryID, txn.Stmt(s.insertServerKeysJSONStmt).QueryRowContext(ctx, string(keyQueryResponseJSON.Raw), serverName, validUntil).Scan(&notaryID)
}

167
federationsender/storage/postgres/notary_server_keys_metadata_table.go Normal file
View file

@ -0,0 +1,167 @@
// Copyright 2021 The Matrix.org Foundation C.I.C.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package postgres
import (
"context"
"database/sql"
"encoding/json"
"github.com/lib/pq"
"github.com/matrix-org/dendrite/federationsender/storage/tables"
"github.com/matrix-org/dendrite/internal"
"github.com/matrix-org/gomatrixserverlib"
)
const notaryServerKeysMetadataSchema = `
CREATE TABLE IF NOT EXISTS federationsender_notary_server_keys_metadata (
notary_id BIGINT NOT NULL,
server_name TEXT NOT NULL,
key_id TEXT NOT NULL,
UNIQUE (server_name, key_id)
);
`
const upsertServerKeysSQL = "" +
"INSERT INTO federationsender_notary_server_keys_metadata (notary_id, server_name, key_id) VALUES ($1, $2, $3)" +
" ON CONFLICT (server_name, key_id) DO UPDATE SET notary_id = $1"
// for a given (server_name, key_id), find the existing notary ID and valid until. Used to check if we will replace it
// JOINs with the json table
const selectNotaryKeyMetadataSQL = `
SELECT federationsender_notary_server_keys_metadata.notary_id, valid_until FROM federationsender_notary_server_keys_json
JOIN federationsender_notary_server_keys_metadata ON
federationsender_notary_server_keys_metadata.notary_id = federationsender_notary_server_keys_json.notary_id
WHERE federationsender_notary_server_keys_metadata.server_name = $1 AND federationsender_notary_server_keys_metadata.key_id = $2
`
// select the response which has the highest valid_until value
// JOINs with the json table
const selectNotaryKeyResponsesSQL = `
SELECT response_json FROM federationsender_notary_server_keys_json
WHERE server_name = $1 AND valid_until = (
SELECT MAX(valid_until) FROM federationsender_notary_server_keys_json WHERE server_name = $1
)
`
// select the responses which have the given key IDs
// JOINs with the json table
const selectNotaryKeyResponsesWithKeyIDsSQL = `
SELECT response_json FROM federationsender_notary_server_keys_json
JOIN federationsender_notary_server_keys_metadata ON
federationsender_notary_server_keys_metadata.notary_id = federationsender_notary_server_keys_json.notary_id
WHERE federationsender_notary_server_keys_json.server_name = $1 AND federationsender_notary_server_keys_metadata.key_id = ANY ($2)
GROUP BY federationsender_notary_server_keys_json.notary_id
`
// JOINs with the metadata table
const deleteUnusedServerKeysJSONSQL = `
DELETE FROM federationsender_notary_server_keys_json WHERE federationsender_notary_server_keys_json.notary_id NOT IN (
SELECT DISTINCT notary_id FROM federationsender_notary_server_keys_metadata
)
`
type notaryServerKeysMetadataStatements struct {
db *sql.DB
upsertServerKeysStmt *sql.Stmt
selectNotaryKeyResponsesStmt *sql.Stmt
selectNotaryKeyResponsesWithKeyIDsStmt *sql.Stmt
selectNotaryKeyMetadataStmt *sql.Stmt
deleteUnusedServerKeysJSONStmt *sql.Stmt
}
func NewPostgresNotaryServerKeysMetadataTable(db *sql.DB) (s *notaryServerKeysMetadataStatements, err error) {
s = &notaryServerKeysMetadataStatements{
db: db,
}
_, err = db.Exec(notaryServerKeysMetadataSchema)
if err != nil {
return
}
if s.upsertServerKeysStmt, err = db.Prepare(upsertServerKeysSQL); err != nil {
return
}
if s.selectNotaryKeyResponsesStmt, err = db.Prepare(selectNotaryKeyResponsesSQL); err != nil {
return
}
if s.selectNotaryKeyResponsesWithKeyIDsStmt, err = db.Prepare(selectNotaryKeyResponsesWithKeyIDsSQL); err != nil {
return
}
if s.selectNotaryKeyMetadataStmt, err = db.Prepare(selectNotaryKeyMetadataSQL); err != nil {
return
}
if s.deleteUnusedServerKeysJSONStmt, err = db.Prepare(deleteUnusedServerKeysJSONSQL); err != nil {
return
}
return
}
func (s *notaryServerKeysMetadataStatements) UpsertKey(
ctx context.Context, txn *sql.Tx, serverName gomatrixserverlib.ServerName, keyID gomatrixserverlib.KeyID, newNotaryID tables.NotaryID, newValidUntil gomatrixserverlib.Timestamp,
) (tables.NotaryID, error) {
notaryID := newNotaryID
// see if the existing notary ID a) exists, b) has a longer valid_until
var existingNotaryID tables.NotaryID
var existingValidUntil gomatrixserverlib.Timestamp
if err := txn.Stmt(s.selectNotaryKeyMetadataStmt).QueryRowContext(ctx, serverName, keyID).Scan(&existingNotaryID, &existingValidUntil); err != nil {
if err != sql.ErrNoRows {
return 0, err
}
}
if existingValidUntil.Time().After(newValidUntil.Time()) {
// the existing valid_until is valid longer, so use that.
return existingNotaryID, nil
}
// overwrite the notary_id for this (server_name, key_id) tuple
_, err := txn.Stmt(s.upsertServerKeysStmt).ExecContext(ctx, notaryID, serverName, keyID)
return notaryID, err
}
func (s *notaryServerKeysMetadataStatements) SelectKeys(ctx context.Context, txn *sql.Tx, serverName gomatrixserverlib.ServerName, keyIDs []gomatrixserverlib.KeyID) ([]gomatrixserverlib.ServerKeys, error) {
var rows *sql.Rows
var err error
if len(keyIDs) == 0 {
rows, err = txn.Stmt(s.selectNotaryKeyResponsesStmt).QueryContext(ctx, string(serverName))
} else {
keyIDstr := make([]string, len(keyIDs))
for i := range keyIDs {
keyIDstr[i] = string(keyIDs[i])
}
rows, err = txn.Stmt(s.selectNotaryKeyResponsesWithKeyIDsStmt).QueryContext(ctx, string(serverName), pq.StringArray(keyIDstr))
}
if err != nil {
return nil, err
}
defer internal.CloseAndLogIfError(ctx, rows, "selectNotaryKeyResponsesStmt close failed")
var results []gomatrixserverlib.ServerKeys
for rows.Next() {
var sk gomatrixserverlib.ServerKeys
var raw string
if err = rows.Scan(&raw); err != nil {
return nil, err
}
if err = json.Unmarshal([]byte(raw), &sk); err != nil {
return nil, err
}
results = append(results, sk)
}
return results, nil
}
func (s *notaryServerKeysMetadataStatements) DeleteOldJSONResponses(ctx context.Context, txn *sql.Tx) error {
_, err := txn.Stmt(s.deleteUnusedServerKeysJSONStmt).ExecContext(ctx)
return err
}

11
federationsender/storage/postgres/storage.go
View file

@ -17,6 +17,7 @@ package postgres
import (
"database/sql"
"fmt"
"github.com/matrix-org/dendrite/federationsender/storage/postgres/deltas"
"github.com/matrix-org/dendrite/federationsender/storage/shared"
@ -69,6 +70,14 @@ func NewDatabase(dbProperties *config.DatabaseOptions, cache caching.FederationS
if err != nil {
return nil, err
}
notaryJSON, err := NewPostgresNotaryServerKeysTable(d.db)
if err != nil {
return nil, fmt.Errorf("NewPostgresNotaryServerKeysTable: %s", err)
}
notaryMetadata, err := NewPostgresNotaryServerKeysMetadataTable(d.db)
if err != nil {
return nil, fmt.Errorf("NewPostgresNotaryServerKeysMetadataTable: %s", err)
}
m := sqlutil.NewMigrations()
deltas.LoadRemoveRoomsTable(m)
if err = m.RunDeltas(d.db, dbProperties); err != nil {
@ -85,6 +94,8 @@ func NewDatabase(dbProperties *config.DatabaseOptions, cache caching.FederationS
FederationSenderBlacklist: blacklist,
FederationSenderInboundPeeks: inboundPeeks,
FederationSenderOutboundPeeks: outboundPeeks,
NotaryServerKeysJSON: notaryJSON,
NotaryServerKeysMetadata: notaryMetadata,
}
if err = d.PartitionOffsetStatements.Prepare(d.db, d.writer, "federationsender"); err != nil {
return nil, err

47
federationsender/storage/shared/storage.go
View file

@ -18,6 +18,7 @@ import (
"context"
"database/sql"
"fmt"
"time"
"github.com/matrix-org/dendrite/federationsender/storage/tables"
"github.com/matrix-org/dendrite/federationsender/types"
@ -37,6 +38,8 @@ type Database struct {
FederationSenderBlacklist tables.FederationSenderBlacklist
FederationSenderOutboundPeeks tables.FederationSenderOutboundPeeks
FederationSenderInboundPeeks tables.FederationSenderInboundPeeks
NotaryServerKeysJSON tables.FederationSenderNotaryServerKeysJSON
NotaryServerKeysMetadata tables.FederationSenderNotaryServerKeysMetadata
}
// An Receipt contains the NIDs of a call to GetNextTransactionPDUs/EDUs.
@ -197,3 +200,47 @@ func (d *Database) GetInboundPeek(ctx context.Context, serverName gomatrixserver
func (d *Database) GetInboundPeeks(ctx context.Context, roomID string) ([]types.InboundPeek, error) {
return d.FederationSenderInboundPeeks.SelectInboundPeeks(ctx, nil, roomID)
}
func (d *Database) UpdateNotaryKeys(ctx context.Context, serverName gomatrixserverlib.ServerName, serverKeys gomatrixserverlib.ServerKeys) error {
return d.Writer.Do(d.DB, nil, func(txn *sql.Tx) error {
validUntil := serverKeys.ValidUntilTS
// Servers MUST use the lesser of this field and 7 days into the future when determining if a key is valid.
// This is to avoid a situation where an attacker publishes a key which is valid for a significant amount of
// time without a way for the homeserver owner to revoke it.
// https://spec.matrix.org/unstable/server-server-api/#querying-keys-through-another-server
weekIntoFuture := time.Now().Add(7 * 24 * time.Hour)
if weekIntoFuture.Before(validUntil.Time()) {
validUntil = gomatrixserverlib.AsTimestamp(weekIntoFuture)
}
notaryID, err := d.NotaryServerKeysJSON.InsertJSONResponse(ctx, txn, serverKeys, serverName, validUntil)
if err != nil {
return err
}
// update the metadata for the keys
for keyID := range serverKeys.OldVerifyKeys {
_, err = d.NotaryServerKeysMetadata.UpsertKey(ctx, txn, serverName, keyID, notaryID, validUntil)
if err != nil {
return err
}
}
for keyID := range serverKeys.VerifyKeys {
_, err = d.NotaryServerKeysMetadata.UpsertKey(ctx, txn, serverName, keyID, notaryID, validUntil)
if err != nil {
return err
}
}
// clean up old responses
return d.NotaryServerKeysMetadata.DeleteOldJSONResponses(ctx, txn)
})
}
func (d *Database) GetNotaryKeys(
ctx context.Context, serverName gomatrixserverlib.ServerName, optKeyIDs []gomatrixserverlib.KeyID,
) (sks []gomatrixserverlib.ServerKeys, err error) {
err = d.Writer.Do(d.DB, nil, func(txn *sql.Tx) error {
sks, err = d.NotaryServerKeysMetadata.SelectKeys(ctx, txn, serverName, optKeyIDs)
return err
})
return sks, err
}

63
federationsender/storage/sqlite3/notary_server_keys_json_table.go Normal file
View file

@ -0,0 +1,63 @@
// Copyright 2021 The Matrix.org Foundation C.I.C.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package sqlite3
import (
"context"
"database/sql"
"github.com/matrix-org/dendrite/federationsender/storage/tables"
"github.com/matrix-org/gomatrixserverlib"
)
const notaryServerKeysJSONSchema = `
CREATE TABLE IF NOT EXISTS federationsender_notary_server_keys_json (
notary_id INTEGER PRIMARY KEY AUTOINCREMENT,
response_json TEXT NOT NULL,
server_name TEXT NOT NULL,
valid_until BIGINT NOT NULL
);
`
const insertServerKeysJSONSQL = "" +
"INSERT INTO federationsender_notary_server_keys_json (response_json, server_name, valid_until) VALUES ($1, $2, $3)" +
" RETURNING notary_id"
type notaryServerKeysStatements struct {
db *sql.DB
insertServerKeysJSONStmt *sql.Stmt
}
func NewSQLiteNotaryServerKeysTable(db *sql.DB) (s *notaryServerKeysStatements, err error) {
s = &notaryServerKeysStatements{
db: db,
}
_, err = db.Exec(notaryServerKeysJSONSchema)
if err != nil {
return
}
if s.insertServerKeysJSONStmt, err = db.Prepare(insertServerKeysJSONSQL); err != nil {
return
}
return
}
func (s *notaryServerKeysStatements) InsertJSONResponse(
ctx context.Context, txn *sql.Tx, keyQueryResponseJSON gomatrixserverlib.ServerKeys, serverName gomatrixserverlib.ServerName, validUntil gomatrixserverlib.Timestamp,
) (tables.NotaryID, error) {
var notaryID tables.NotaryID
return notaryID, txn.Stmt(s.insertServerKeysJSONStmt).QueryRowContext(ctx, string(keyQueryResponseJSON.Raw), serverName, validUntil).Scan(&notaryID)
}

169
federationsender/storage/sqlite3/notary_server_keys_metadata_table.go Normal file
View file

@ -0,0 +1,169 @@
// Copyright 2021 The Matrix.org Foundation C.I.C.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package sqlite3
import (
"context"
"database/sql"
"encoding/json"
"fmt"
"strings"
"github.com/matrix-org/dendrite/federationsender/storage/tables"
"github.com/matrix-org/dendrite/internal"
"github.com/matrix-org/dendrite/internal/sqlutil"
"github.com/matrix-org/gomatrixserverlib"
)
const notaryServerKeysMetadataSchema = `
CREATE TABLE IF NOT EXISTS federationsender_notary_server_keys_metadata (
notary_id BIGINT NOT NULL,
server_name TEXT NOT NULL,
key_id TEXT NOT NULL,
UNIQUE (server_name, key_id)
);
`
const upsertServerKeysSQL = "" +
"INSERT INTO federationsender_notary_server_keys_metadata (notary_id, server_name, key_id) VALUES ($1, $2, $3)" +
" ON CONFLICT (server_name, key_id) DO UPDATE SET notary_id = $1"
// for a given (server_name, key_id), find the existing notary ID and valid until. Used to check if we will replace it
// JOINs with the json table
const selectNotaryKeyMetadataSQL = `
SELECT federationsender_notary_server_keys_metadata.notary_id, valid_until FROM federationsender_notary_server_keys_json
JOIN federationsender_notary_server_keys_metadata ON
federationsender_notary_server_keys_metadata.notary_id = federationsender_notary_server_keys_json.notary_id
WHERE federationsender_notary_server_keys_metadata.server_name = $1 AND federationsender_notary_server_keys_metadata.key_id = $2
`
// select the response which has the highest valid_until value
// JOINs with the json table
const selectNotaryKeyResponsesSQL = `
SELECT response_json FROM federationsender_notary_server_keys_json
WHERE server_name = $1 AND valid_until = (
SELECT MAX(valid_until) FROM federationsender_notary_server_keys_json WHERE server_name = $1
)
`
// select the responses which have the given key IDs
// JOINs with the json table
const selectNotaryKeyResponsesWithKeyIDsSQL = `
SELECT response_json FROM federationsender_notary_server_keys_json
JOIN federationsender_notary_server_keys_metadata ON
federationsender_notary_server_keys_metadata.notary_id = federationsender_notary_server_keys_json.notary_id
WHERE federationsender_notary_server_keys_json.server_name = $1 AND federationsender_notary_server_keys_metadata.key_id IN ($2)
GROUP BY federationsender_notary_server_keys_json.notary_id
`
// JOINs with the metadata table
const deleteUnusedServerKeysJSONSQL = `
DELETE FROM federationsender_notary_server_keys_json WHERE federationsender_notary_server_keys_json.notary_id NOT IN (
SELECT DISTINCT notary_id FROM federationsender_notary_server_keys_metadata
)
`
type notaryServerKeysMetadataStatements struct {
db *sql.DB
upsertServerKeysStmt *sql.Stmt
selectNotaryKeyResponsesStmt *sql.Stmt
selectNotaryKeyMetadataStmt *sql.Stmt
deleteUnusedServerKeysJSONStmt *sql.Stmt
}
func NewSQLiteNotaryServerKeysMetadataTable(db *sql.DB) (s *notaryServerKeysMetadataStatements, err error) {
s = &notaryServerKeysMetadataStatements{
db: db,
}
_, err = db.Exec(notaryServerKeysMetadataSchema)
if err != nil {
return
}
if s.upsertServerKeysStmt, err = db.Prepare(upsertServerKeysSQL); err != nil {
return
}
if s.selectNotaryKeyResponsesStmt, err = db.Prepare(selectNotaryKeyResponsesSQL); err != nil {
return
}
if s.selectNotaryKeyMetadataStmt, err = db.Prepare(selectNotaryKeyMetadataSQL); err != nil {
return
}
if s.deleteUnusedServerKeysJSONStmt, err = db.Prepare(deleteUnusedServerKeysJSONSQL); err != nil {
return
}
return
}
func (s *notaryServerKeysMetadataStatements) UpsertKey(
ctx context.Context, txn *sql.Tx, serverName gomatrixserverlib.ServerName, keyID gomatrixserverlib.KeyID, newNotaryID tables.NotaryID, newValidUntil gomatrixserverlib.Timestamp,
) (tables.NotaryID, error) {
notaryID := newNotaryID
// see if the existing notary ID a) exists, b) has a longer valid_until
var existingNotaryID tables.NotaryID
var existingValidUntil gomatrixserverlib.Timestamp
if err := txn.Stmt(s.selectNotaryKeyMetadataStmt).QueryRowContext(ctx, serverName, keyID).Scan(&existingNotaryID, &existingValidUntil); err != nil {
if err != sql.ErrNoRows {
return 0, err
}
}
if existingValidUntil.Time().After(newValidUntil.Time()) {
// the existing valid_until is valid longer, so use that.
return existingNotaryID, nil
}
// overwrite the notary_id for this (server_name, key_id) tuple
_, err := txn.Stmt(s.upsertServerKeysStmt).ExecContext(ctx, notaryID, serverName, keyID)
return notaryID, err
}
func (s *notaryServerKeysMetadataStatements) SelectKeys(ctx context.Context, txn *sql.Tx, serverName gomatrixserverlib.ServerName, keyIDs []gomatrixserverlib.KeyID) ([]gomatrixserverlib.ServerKeys, error) {
var rows *sql.Rows
var err error
if len(keyIDs) == 0 {
rows, err = txn.Stmt(s.selectNotaryKeyResponsesStmt).QueryContext(ctx, string(serverName))
} else {
iKeyIDs := make([]interface{}, len(keyIDs)+1)
iKeyIDs[0] = serverName
for i := range keyIDs {
iKeyIDs[i+1] = string(keyIDs[i])
}
sql := strings.Replace(selectNotaryKeyResponsesWithKeyIDsSQL, "($2)", sqlutil.QueryVariadicOffset(len(keyIDs), 1), 1)
fmt.Println(sql)
fmt.Println(iKeyIDs...)
rows, err = s.db.QueryContext(ctx, sql, iKeyIDs...)
}
if err != nil {
return nil, err
}
defer internal.CloseAndLogIfError(ctx, rows, "selectNotaryKeyResponsesStmt close failed")
var results []gomatrixserverlib.ServerKeys
for rows.Next() {
var sk gomatrixserverlib.ServerKeys
var raw string
if err = rows.Scan(&raw); err != nil {
return nil, err
}
if err = json.Unmarshal([]byte(raw), &sk); err != nil {
return nil, err
}
results = append(results, sk)
}
return results, nil
}
func (s *notaryServerKeysMetadataStatements) DeleteOldJSONResponses(ctx context.Context, txn *sql.Tx) error {
_, err := txn.Stmt(s.deleteUnusedServerKeysJSONStmt).ExecContext(ctx)
return err
}

10
federationsender/storage/sqlite3/storage.go
View file

@ -71,6 +71,14 @@ func NewDatabase(dbProperties *config.DatabaseOptions, cache caching.FederationS
if err != nil {
return nil, err
}
notaryKeys, err := NewSQLiteNotaryServerKeysTable(d.db)
if err != nil {
return nil, err
}
notaryKeysMetadata, err := NewSQLiteNotaryServerKeysMetadataTable(d.db)
if err != nil {
return nil, err
}
m := sqlutil.NewMigrations()
deltas.LoadRemoveRoomsTable(m)
if err = m.RunDeltas(d.db, dbProperties); err != nil {
@ -87,6 +95,8 @@ func NewDatabase(dbProperties *config.DatabaseOptions, cache caching.FederationS
FederationSenderBlacklist: blacklist,
FederationSenderOutboundPeeks: outboundPeeks,
FederationSenderInboundPeeks: inboundPeeks,
NotaryServerKeysJSON: notaryKeys,
NotaryServerKeysMetadata: notaryKeysMetadata,
}
if err = d.PartitionOffsetStatements.Prepare(d.db, d.writer, "federationsender"); err != nil {
return nil, err

24
federationsender/storage/tables/interface.go
View file

@ -22,6 +22,8 @@ import (
"github.com/matrix-org/gomatrixserverlib"
)
type NotaryID int64
type FederationSenderQueuePDUs interface {
InsertQueuePDU(ctx context.Context, txn *sql.Tx, transactionID gomatrixserverlib.TransactionID, serverName gomatrixserverlib.ServerName, nid int64) error
DeleteQueuePDUs(ctx context.Context, txn *sql.Tx, serverName gomatrixserverlib.ServerName, jsonNIDs []int64) error
@ -80,3 +82,25 @@ type FederationSenderInboundPeeks interface {
DeleteInboundPeek(ctx context.Context, txn *sql.Tx, serverName gomatrixserverlib.ServerName, roomID, peekID string) (err error)
DeleteInboundPeeks(ctx context.Context, txn *sql.Tx, roomID string) (err error)
}
// FederationSenderNotaryServerKeysJSON contains the byte-for-byte responses from servers which contain their keys and is signed by them.
type FederationSenderNotaryServerKeysJSON interface {
// InsertJSONResponse inserts a new response JSON. Useless on its own, needs querying via FederationSenderNotaryServerKeysMetadata
// `validUntil` should be the value of `valid_until_ts` with the 7-day check applied from:
// "Servers MUST use the lesser of this field and 7 days into the future when determining if a key is valid.
// This is to avoid a situation where an attacker publishes a key which is valid for a significant amount of time
// without a way for the homeserver owner to revoke it.""
InsertJSONResponse(ctx context.Context, txn *sql.Tx, keyQueryResponseJSON gomatrixserverlib.ServerKeys, serverName gomatrixserverlib.ServerName, validUntil gomatrixserverlib.Timestamp) (NotaryID, error)
}
// FederationSenderNotaryServerKeysMetadata persists the metadata for FederationSenderNotaryServerKeysJSON
type FederationSenderNotaryServerKeysMetadata interface {
// UpsertKey updates or inserts a (server_name, key_id) tuple, pointing it via NotaryID at the the response which has the longest valid_until_ts
// `newNotaryID` and `newValidUntil` should be the notary ID / valid_until which has this (server_name, key_id) tuple already, e.g one you just inserted.
UpsertKey(ctx context.Context, txn *sql.Tx, serverName gomatrixserverlib.ServerName, keyID gomatrixserverlib.KeyID, newNotaryID NotaryID, newValidUntil gomatrixserverlib.Timestamp) (NotaryID, error)
// SelectKeys returns the signed JSON objects which contain the given key IDs. This will be at most the length of `keyIDs` and at least 1 (assuming
// the keys exist in the first place). If `keyIDs` is empty, the signed JSON object with the longest valid_until_ts will be returned.
SelectKeys(ctx context.Context, txn *sql.Tx, serverName gomatrixserverlib.ServerName, keyIDs []gomatrixserverlib.KeyID) ([]gomatrixserverlib.ServerKeys, error)
// DeleteOldJSONResponses removes all responses which are not referenced in FederationSenderNotaryServerKeysMetadata
DeleteOldJSONResponses(ctx context.Context, txn *sql.Tx) error
}

29
go.mod
View file

@ -1,14 +1,22 @@
module github.com/matrix-org/dendrite
require (
github.com/Arceliar/ironwood v0.0.0-20210619124114-6ad55cae5031
github.com/DATA-DOG/go-sqlmock v1.5.0
github.com/HdrHistogram/hdrhistogram-go v1.0.1 // indirect
github.com/Masterminds/semver/v3 v3.1.1
github.com/Shopify/sarama v1.28.0
github.com/codeclysm/extract v2.2.0+incompatible
github.com/containerd/containerd v1.5.2 // indirect
github.com/docker/docker v20.10.7+incompatible
github.com/docker/go-connections v0.4.0
github.com/getsentry/sentry-go v0.10.0
github.com/gologme/log v1.2.0
github.com/gorilla/mux v1.8.0
github.com/gorilla/websocket v1.4.2
github.com/h2non/filetype v1.1.1 // indirect
github.com/hashicorp/golang-lru v0.5.4
github.com/juju/testing v0.0.0-20210324180055-18c50b0c2098 // indirect
github.com/lib/pq v1.9.0
github.com/libp2p/go-libp2p v0.13.0
github.com/libp2p/go-libp2p-circuit v0.4.0
@ -22,28 +30,33 @@ require (
github.com/matrix-org/dugong v0.0.0-20180820122854-51a565b5666b
github.com/matrix-org/go-http-js-libp2p v0.0.0-20200518170932-783164aeeda4
github.com/matrix-org/go-sqlite3-js v0.0.0-20210625141222-bd2b7124cee8
github.com/matrix-org/gomatrix v0.0.0-20200827122206-7dd5e2a05bcd
github.com/matrix-org/gomatrixserverlib v0.0.0-20210525110027-8cb7699aa64a
github.com/matrix-org/gomatrix v0.0.0-20210324163249-be2af5ef2e16
github.com/matrix-org/gomatrixserverlib v0.0.0-20210719140634-f44a103bb12e
github.com/matrix-org/naffka v0.0.0-20210623111924-14ff508b58e0
github.com/matrix-org/pinecone v0.0.0-20210623102758-74f885644c1b
github.com/matrix-org/util v0.0.0-20200807132607-55161520e1d4
github.com/mattn/go-sqlite3 v1.14.7-0.20210414154423-1157a4212dcb
github.com/morikuni/aec v1.0.0 // indirect
github.com/neilalexander/utp v0.1.1-0.20210705212447-691f29ad692b
github.com/nfnt/resize v0.0.0-20180221191011-83c6a9932646
github.com/ngrok/sqlmw v0.0.0-20200129213757-d5c93a81bec6
github.com/opentracing/opentracing-go v1.2.0
github.com/patrickmn/go-cache v2.1.0+incompatible
github.com/pkg/errors v0.9.1
github.com/pressly/goose v2.7.0+incompatible
github.com/prometheus/client_golang v1.9.0
github.com/sirupsen/logrus v1.8.0
github.com/tidwall/gjson v1.6.8
github.com/tidwall/sjson v1.1.5
github.com/sirupsen/logrus v1.8.1
github.com/tidwall/gjson v1.8.1
github.com/tidwall/pretty v1.2.0 // indirect
github.com/tidwall/sjson v1.1.7
github.com/uber/jaeger-client-go v2.25.0+incompatible
github.com/uber/jaeger-lib v2.4.0+incompatible
github.com/yggdrasil-network/yggdrasil-go v0.3.15-0.20210218094457-e77ca8019daa
github.com/yggdrasil-network/yggdrasil-go v0.4.1-0.20210715083903-52309d094c00
go.uber.org/atomic v1.7.0
golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83
golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97
golang.org/x/mobile v0.0.0-20210220033013-bdb1ca9a1e08
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110
golang.org/x/net v0.0.0-20210610132358-84b48f89b13b
golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c // indirect
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1
gopkg.in/h2non/bimg.v1 v1.1.5
gopkg.in/yaml.v2 v2.4.0

757
go.sum
View file

File diff suppressed because it is too large Load diff

6
internal/log.go
View file

@ -73,9 +73,9 @@ func callerPrettyfier(f *runtime.Frame) (string, string) {
// Append a newline + tab to it to move the actual log content to its own line
funcname += "\n\t"
// Surround the filepath in brackets and append line number so IDEs can quickly
// navigate
filename := fmt.Sprintf(" [%s:%d]", f.File, f.Line)
// Use a shortened file path which just has the filename to avoid having lots of redundant
// directories which contribute significantly to overall log sizes!
filename := fmt.Sprintf(" [%s:%d]", path.Base(f.File), f.Line)
return funcname, filename
}

4
internal/version.go
View file

@ -16,8 +16,8 @@ var build string
const (
VersionMajor = 0
VersionMinor = 3
VersionPatch = 11
VersionMinor = 4
VersionPatch = 0
VersionTag = "" // example: "rc1"
)

19
mediaapi/routing/upload.go
View file

@ -147,20 +147,17 @@ func (r *uploadRequest) doUpload(
// r.storeFileAndMetadata(ctx, tmpDir, ...)
// before you return from doUpload else we will leak a temp file. We could make this nicer with a `WithTransaction` style of
// nested function to guarantee either storage or cleanup.
// should not happen, but prevents any int overflows
if cfg.MaxFileSizeBytes != nil && *cfg.MaxFileSizeBytes+1 <= 0 {
r.Logger.WithFields(log.Fields{
"MaxFileSizeBytes": *cfg.MaxFileSizeBytes + 1,
}).Error("Error while transferring file, configured max_file_size_bytes overflows int64")
return &util.JSONResponse{
Code: http.StatusBadRequest,
JSON: jsonerror.Unknown("Failed to upload"),
if *cfg.MaxFileSizeBytes > 0 {
if *cfg.MaxFileSizeBytes+1 <= 0 {
r.Logger.WithFields(log.Fields{
"MaxFileSizeBytes": *cfg.MaxFileSizeBytes,
}).Warnf("Configured MaxFileSizeBytes overflows int64, defaulting to %d bytes", config.DefaultMaxFileSizeBytes)
cfg.MaxFileSizeBytes = &config.DefaultMaxFileSizeBytes
}
reqReader = io.LimitReader(reqReader, int64(*cfg.MaxFileSizeBytes)+1)
}
lr := io.LimitReader(reqReader, int64(*cfg.MaxFileSizeBytes)+1)
hash, bytesWritten, tmpDir, err := fileutils.WriteTempFile(ctx, lr, cfg.AbsBasePath)
hash, bytesWritten, tmpDir, err := fileutils.WriteTempFile(ctx, reqReader, cfg.AbsBasePath)
if err != nil {
r.Logger.WithError(err).WithFields(log.Fields{
"MaxFileSizeBytes": *cfg.MaxFileSizeBytes,

6
roomserver/api/query.go
View file

@ -170,7 +170,8 @@ type QueryMembershipsForRoomResponse struct {
// QueryServerJoinedToRoomRequest is a request to QueryServerJoinedToRoom
type QueryServerJoinedToRoomRequest struct {
// Server name of the server to find
// Server name of the server to find. If not specified, we will
// default to checking if the local server is joined.
ServerName gomatrixserverlib.ServerName `json:"server_name"`
// ID of the room to see if we are still joined to
RoomID string `json:"room_id"`
@ -182,7 +183,8 @@ type QueryServerJoinedToRoomResponse struct {
RoomExists bool `json:"room_exists"`
// True if we still believe that we are participating in the room
IsInRoom bool `json:"is_in_room"`
// List of servers that are also in the room
// List of servers that are also in the room. This will not be populated
// if the queried ServerName is the local server name.
ServerNames []gomatrixserverlib.ServerName `json:"server_names"`
}

2
roomserver/internal/api.go
View file

@ -59,6 +59,7 @@ func NewRoomserverAPI(
Queryer: &query.Queryer{
DB: roomserverDB,
Cache: caches,
ServerName: cfg.Matrix.ServerName,
ServerACLs: serverACLs,
},
Inputer: &input.Inputer{
@ -92,6 +93,7 @@ func (r *RoomserverInternalAPI) SetFederationSenderAPI(fsAPI fsAPI.FederationSen
FSAPI: r.fsAPI,
RSAPI: r,
Inputer: r.Inputer,
Queryer: r.Queryer,
}
r.Peeker = &perform.Peeker{
ServerName: r.Cfg.Matrix.ServerName,

8
roomserver/internal/helpers/helpers.go
View file

@ -50,6 +50,10 @@ func UpdateToInviteMembership(
return updates, nil
}
// IsServerCurrentlyInRoom checks if a server is in a given room, based on the room
// memberships. If the servername is not supplied then the local server will be
// checked instead using a faster code path.
// TODO: This should probably be replaced by an API call.
func IsServerCurrentlyInRoom(ctx context.Context, db storage.Database, serverName gomatrixserverlib.ServerName, roomID string) (bool, error) {
info, err := db.RoomInfo(ctx, roomID)
if err != nil {
@ -59,6 +63,10 @@ func IsServerCurrentlyInRoom(ctx context.Context, db storage.Database, serverNam
return false, fmt.Errorf("unknown room %s", roomID)
}
if serverName == "" {
return db.GetLocalServerInRoom(ctx, info.RoomNID)
}
eventNIDs, err := db.GetMembershipEventNIDsForRoom(ctx, info.RoomNID, true, false)
if err != nil {
return false, err

44
roomserver/internal/input/input.go
View file

@ -28,10 +28,18 @@ import (
"github.com/matrix-org/dendrite/roomserver/api"
"github.com/matrix-org/dendrite/roomserver/storage"
"github.com/matrix-org/gomatrixserverlib"
"github.com/prometheus/client_golang/prometheus"
log "github.com/sirupsen/logrus"
"github.com/tidwall/gjson"
"go.uber.org/atomic"
)
var keyContentFields = map[string]string{
"m.room.join_rules": "join_rule",
"m.room.history_visibility": "history_visibility",
"m.room.member": "membership",
}
type Inputer struct {
DB storage.Database
Producer sarama.SyncProducer
@ -64,6 +72,9 @@ func (w *inputWorker) start() {
if !ok {
continue
}
roomserverInputBackpressure.With(prometheus.Labels{
"room_id": task.event.Event.RoomID(),
}).Dec()
hooks.Run(hooks.KindNewEventReceived, task.event.Event)
_, task.err = w.r.processRoomEvent(task.ctx, task.event)
if task.err == nil {
@ -91,15 +102,27 @@ func (r *Inputer) WriteOutputEvents(roomID string, updates []api.OutputEvent) er
"type": updates[i].Type,
})
if updates[i].NewRoomEvent != nil {
eventType := updates[i].NewRoomEvent.Event.Type()
logger = logger.WithFields(log.Fields{
"event_type": updates[i].NewRoomEvent.Event.Type(),
"event_type": eventType,
"event_id": updates[i].NewRoomEvent.Event.EventID(),
"adds_state": len(updates[i].NewRoomEvent.AddsStateEventIDs),
"removes_state": len(updates[i].NewRoomEvent.RemovesStateEventIDs),
"send_as_server": updates[i].NewRoomEvent.SendAsServer,
"sender": updates[i].NewRoomEvent.Event.Sender(),
})
if updates[i].NewRoomEvent.Event.Type() == "m.room.server_acl" && updates[i].NewRoomEvent.Event.StateKeyEquals("") {
if updates[i].NewRoomEvent.Event.StateKey() != nil {
logger = logger.WithField("state_key", *updates[i].NewRoomEvent.Event.StateKey())
}
contentKey := keyContentFields[eventType]
if contentKey != "" {
value := gjson.GetBytes(updates[i].NewRoomEvent.Event.Content(), contentKey)
if value.Exists() {
logger = logger.WithField("content_value", value.String())
}
}
if eventType == "m.room.server_acl" && updates[i].NewRoomEvent.Event.StateKeyEquals("") {
ev := updates[i].NewRoomEvent.Event.Unwrap()
defer r.ACLs.OnServerACLUpdate(ev)
}
@ -120,6 +143,20 @@ func (r *Inputer) WriteOutputEvents(roomID string, updates []api.OutputEvent) er
return errs
}
func init() {
prometheus.MustRegister(roomserverInputBackpressure)
}
var roomserverInputBackpressure = prometheus.NewGaugeVec(
prometheus.GaugeOpts{
Namespace: "dendrite",
Subsystem: "roomserver",
Name: "input_backpressure",
Help: "How many events are queued for input for a given room",
},
[]string{"room_id"},
)
// InputRoomEvents implements api.RoomserverInternalAPI
func (r *Inputer) InputRoomEvents(
_ context.Context,
@ -164,6 +201,9 @@ func (r *Inputer) InputRoomEvents(
go worker.start()
}
worker.input.push(tasks[i])
roomserverInputBackpressure.With(prometheus.Labels{
"room_id": roomID,
}).Inc()
}
// Wait for all of the workers to return results about our tasks.

11
roomserver/internal/perform/perform_join.go
View file

@ -28,6 +28,7 @@ import (
rsAPI "github.com/matrix-org/dendrite/roomserver/api"
"github.com/matrix-org/dendrite/roomserver/internal/helpers"
"github.com/matrix-org/dendrite/roomserver/internal/input"
"github.com/matrix-org/dendrite/roomserver/internal/query"
"github.com/matrix-org/dendrite/roomserver/storage"
"github.com/matrix-org/dendrite/setup/config"
"github.com/matrix-org/gomatrixserverlib"
@ -42,6 +43,7 @@ type Joiner struct {
DB storage.Database
Inputer *input.Inputer
Queryer *query.Queryer
}
// PerformJoin handles joining matrix rooms, including over federation by talking to the federationsender.
@ -205,7 +207,14 @@ func (r *Joiner) performJoinRoomByID(
// Force a federated join if we aren't in the room and we've been
// given some server names to try joining by.
serverInRoom, _ := helpers.IsServerCurrentlyInRoom(ctx, r.DB, r.ServerName, req.RoomIDOrAlias)
inRoomReq := &api.QueryServerJoinedToRoomRequest{
RoomID: req.RoomIDOrAlias,
}
inRoomRes := &api.QueryServerJoinedToRoomResponse{}
if err = r.Queryer.QueryServerJoinedToRoom(ctx, inRoomReq, inRoomRes); err != nil {
return "", "", fmt.Errorf("r.Queryer.QueryServerJoinedToRoom: %w", err)
}
serverInRoom := inRoomRes.IsInRoom
forceFederatedJoin := len(req.ServerNames) > 0 && !serverInRoom
// Force a federated join if we're dealing with a pending invite

20
roomserver/internal/perform/perform_leave.go
View file

@ -26,6 +26,7 @@ import (
"github.com/matrix-org/dendrite/roomserver/storage"
"github.com/matrix-org/dendrite/setup/config"
"github.com/matrix-org/gomatrixserverlib"
"github.com/matrix-org/util"
)
type Leaver struct {
@ -64,7 +65,14 @@ func (r *Leaver) performLeaveRoomByID(
// that.
isInvitePending, senderUser, eventID, err := helpers.IsInvitePending(ctx, r.DB, req.RoomID, req.UserID)
if err == nil && isInvitePending {
return r.performRejectInvite(ctx, req, res, senderUser, eventID)
var host gomatrixserverlib.ServerName
_, host, err = gomatrixserverlib.SplitID('@', senderUser)
if err != nil {
return nil, fmt.Errorf("Sender %q is invalid", senderUser)
}
if host != r.Cfg.Matrix.ServerName {
return r.performFederatedRejectInvite(ctx, req, res, senderUser, eventID)
}
}
// There's no invite pending, so first of all we want to find out
@ -94,9 +102,7 @@ func (r *Leaver) performLeaveRoomByID(
if err != nil {
return nil, fmt.Errorf("Error getting membership: %w", err)
}
if membership != gomatrixserverlib.Join {
// TODO: should be able to handle "invite" in this case too, if
// it's a case of kicking or banning or such
if membership != gomatrixserverlib.Join && membership != gomatrixserverlib.Invite {
return nil, fmt.Errorf("User %q is not joined to the room (membership is %q)", req.UserID, membership)
}
@ -147,7 +153,7 @@ func (r *Leaver) performLeaveRoomByID(
return nil, nil
}
func (r *Leaver) performRejectInvite(
func (r *Leaver) performFederatedRejectInvite(
ctx context.Context,
req *api.PerformLeaveRequest,
res *api.PerformLeaveResponse, // nolint:unparam
@ -166,7 +172,9 @@ func (r *Leaver) performRejectInvite(
}
leaveRes := fsAPI.PerformLeaveResponse{}
if err := r.FSAPI.PerformLeave(ctx, &leaveReq, &leaveRes); err != nil {
return nil, err
// failures in PerformLeave should NEVER stop us from telling other components like the
// sync API that the invite was withdrawn. Otherwise we can end up with stuck invites.
util.GetLogger(ctx).WithError(err).Errorf("failed to PerformLeave, still retiring invite event")
}
// Withdraw the invite, so that the sync API etc are

25
roomserver/internal/query/query.go
View file

@ -36,6 +36,7 @@ import (
type Queryer struct {
DB storage.Database
Cache caching.RoomServerCaches
ServerName gomatrixserverlib.ServerName
ServerACLs *acls.ServerACLs
}
@ -328,6 +329,16 @@ func (r *Queryer) QueryServerJoinedToRoom(
}
response.RoomExists = true
if request.ServerName == r.ServerName || request.ServerName == "" {
var joined bool
joined, err = r.DB.GetLocalServerInRoom(ctx, info.RoomNID)
if err != nil {
return fmt.Errorf("r.DB.GetLocalServerInRoom: %w", err)
}
response.IsInRoom = joined
return nil
}
eventNIDs, err := r.DB.GetMembershipEventNIDsForRoom(ctx, info.RoomNID, true, false)
if err != nil {
return fmt.Errorf("r.DB.GetMembershipEventNIDsForRoom: %w", err)
@ -377,10 +388,16 @@ func (r *Queryer) QueryServerAllowedToSeeEvent(
return
}
roomID := events[0].RoomID()
isServerInRoom, err := helpers.IsServerCurrentlyInRoom(ctx, r.DB, request.ServerName, roomID)
if err != nil {
return
inRoomReq := &api.QueryServerJoinedToRoomRequest{
RoomID: roomID,
ServerName: request.ServerName,
}
inRoomRes := &api.QueryServerJoinedToRoomResponse{}
if err = r.QueryServerJoinedToRoom(ctx, inRoomReq, inRoomRes); err != nil {
return fmt.Errorf("r.Queryer.QueryServerJoinedToRoom: %w", err)
}
info, err := r.DB.RoomInfo(ctx, roomID)
if err != nil {
return err
@ -389,7 +406,7 @@ func (r *Queryer) QueryServerAllowedToSeeEvent(
return fmt.Errorf("QueryServerAllowedToSeeEvent: no room info for room %s", roomID)
}
response.AllowedToSeeEvent, err = helpers.CheckServerAllowedToSeeEvent(
ctx, r.DB, *info, request.EventID, request.ServerName, isServerInRoom,
ctx, r.DB, *info, request.EventID, request.ServerName, inRoomRes.IsInRoom,
)
return
}

14
roomserver/state/state.go
View file

@ -424,12 +424,17 @@ func (v *StateResolution) loadStateAfterEventsForNumericTuples(
return result, nil
}
var calculateStateDurations = prometheus.NewSummaryVec(
prometheus.SummaryOpts{
var calculateStateDurations = prometheus.NewHistogramVec(
prometheus.HistogramOpts{
Namespace: "dendrite",
Subsystem: "roomserver",
Name: "calculate_state_duration_microseconds",
Name: "calculate_state_duration_milliseconds",
Help: "How long it takes to calculate the state after a list of events",
Buckets: []float64{ // milliseconds
5, 10, 25, 50, 75, 100, 200, 300, 400, 500,
1000, 2000, 3000, 4000, 5000, 6000,
7000, 8000, 9000, 10000, 15000, 20000, 30000,
},
},
// Takes two labels:
// algorithm:
@ -496,9 +501,8 @@ func (c *calculateStateMetrics) stop(stateNID types.StateSnapshotNID, err error)
} else {
outcome = "failure"
}
endTime := time.Now()
calculateStateDurations.WithLabelValues(c.algorithm, outcome).Observe(
float64(endTime.Sub(c.startTime).Nanoseconds()) / 1000.,
float64(time.Since(c.startTime).Milliseconds()),
)
calculateStatePrevEventLength.WithLabelValues(c.algorithm, outcome).Observe(
float64(c.prevEventLength),

2
roomserver/storage/interface.go
View file

@ -154,6 +154,8 @@ type Database interface {
GetBulkStateContent(ctx context.Context, roomIDs []string, tuples []gomatrixserverlib.StateKeyTuple, allowWildcards bool) ([]tables.StrippedEvent, error)
// JoinedUsersSetInRooms returns all joined users in the rooms given, along with the count of how many times they appear.
JoinedUsersSetInRooms(ctx context.Context, roomIDs []string) (map[string]int, error)
// GetLocalServerInRoom returns true if we think we're in a given room or false otherwise.
GetLocalServerInRoom(ctx context.Context, roomNID types.RoomNID) (bool, error)
// GetKnownUsers searches all users that userID knows about.
GetKnownUsers(ctx context.Context, userID, searchString string, limit int) ([]string, error)
// GetKnownRooms returns a list of all rooms we know about.

37
roomserver/storage/postgres/deltas/2021041615092700_state_blocks_refactor.go
View file

@ -117,7 +117,7 @@ func UpStateBlocksRefactor(tx *sql.Tx) error {
_roomserver_state_block.event_nid
FROM
_roomserver_state_snapshots
JOIN _roomserver_state_block ON _roomserver_state_block.state_block_nid = ANY (_roomserver_state_snapshots.state_block_nids)
LEFT JOIN _roomserver_state_block ON _roomserver_state_block.state_block_nid = ANY (_roomserver_state_snapshots.state_block_nids)
WHERE
_roomserver_state_snapshots.state_snapshot_nid = ANY (
SELECT
@ -140,22 +140,49 @@ func UpStateBlocksRefactor(tx *sql.Tx) error {
logrus.Warnf("Rewriting snapshots %d-%d of %d...", batchoffset, batchoffset+batchsize, snapshotcount)
var snapshots []stateBlockData
var badCreateSnapshots []stateBlockData
for snapshotrows.Next() {
var snapshot stateBlockData
var eventsarray pq.Int64Array
if err = snapshotrows.Scan(&snapshot.StateSnapshotNID, &snapshot.RoomNID, &snapshot.StateBlockNID, &eventsarray); err != nil {
var eventsarray []sql.NullInt64
var nulStateBlockNID sql.NullInt64
if err = snapshotrows.Scan(&snapshot.StateSnapshotNID, &snapshot.RoomNID, &nulStateBlockNID, pq.Array(&eventsarray)); err != nil {
return fmt.Errorf("rows.Scan: %w", err)
}
if nulStateBlockNID.Valid {
snapshot.StateBlockNID = types.StateBlockNID(nulStateBlockNID.Int64)
}
// Dendrite v0.1.0 would not make a state block for the create event, resulting in [NULL] from the query above.
// Remember the snapshot and we'll fill it in after we close this cursor as we can't have 2 queries running at the same time
if len(eventsarray) == 1 && !eventsarray[0].Valid {
badCreateSnapshots = append(badCreateSnapshots, snapshot)
continue
}
for _, e := range eventsarray {
snapshot.EventNIDs = append(snapshot.EventNIDs, types.EventNID(e))
if e.Valid {
snapshot.EventNIDs = append(snapshot.EventNIDs, types.EventNID(e.Int64))
}
}
snapshot.EventNIDs = snapshot.EventNIDs[:util.SortAndUnique(snapshot.EventNIDs)]
snapshots = append(snapshots, snapshot)
}
if err = snapshotrows.Close(); err != nil {
return fmt.Errorf("snapshots.Close: %w", err)
}
// fill in bad create snapshots
for _, s := range badCreateSnapshots {
var createEventNID types.EventNID
err = tx.QueryRow(
`SELECT event_nid FROM roomserver_events WHERE state_snapshot_nid = $1 AND event_type_nid = 1`, s.StateSnapshotNID,
).Scan(&createEventNID)
if err != nil {
return fmt.Errorf("cannot xref null state block with snapshot %d: %s", s.StateSnapshotNID, err)
}
if createEventNID == 0 {
return fmt.Errorf("cannot xref null state block with snapshot %d, no create event", s.StateSnapshotNID)
}
s.EventNIDs = append(s.EventNIDs, createEventNID)
snapshots = append(snapshots, s)
}
newsnapshots := map[stateSnapshotData]types.StateBlockNIDs{}

23
roomserver/storage/postgres/membership_table.go
View file

@ -124,6 +124,14 @@ var selectKnownUsersSQL = "" +
" SELECT DISTINCT room_nid FROM roomserver_membership WHERE target_nid=$1 AND membership_nid = " + fmt.Sprintf("%d", tables.MembershipStateJoin) +
") AND membership_nid = " + fmt.Sprintf("%d", tables.MembershipStateJoin) + " AND event_state_key LIKE $2 LIMIT $3"
// selectLocalServerInRoomSQL is an optimised case for checking if we, the local server,
// are in the room by using the target_local column of the membership table. Normally when
// we want to know if a server is in a room, we have to unmarshal the entire room state which
// is expensive. The presence of a single row from this query suggests we're still in the
// room, no rows returned suggests we aren't.
const selectLocalServerInRoomSQL = "" +
"SELECT room_nid FROM roomserver_membership WHERE target_local = true AND membership_nid = $1 AND room_nid = $2 LIMIT 1"
type membershipStatements struct {
insertMembershipStmt *sql.Stmt
selectMembershipForUpdateStmt *sql.Stmt
@ -137,6 +145,7 @@ type membershipStatements struct {
selectJoinedUsersSetForRoomsStmt *sql.Stmt
selectKnownUsersStmt *sql.Stmt
updateMembershipForgetRoomStmt *sql.Stmt
selectLocalServerInRoomStmt *sql.Stmt
}
func createMembershipTable(db *sql.DB) error {
@ -160,6 +169,7 @@ func prepareMembershipTable(db *sql.DB) (tables.Membership, error) {
{&s.selectJoinedUsersSetForRoomsStmt, selectJoinedUsersSetForRoomsSQL},
{&s.selectKnownUsersStmt, selectKnownUsersSQL},
{&s.updateMembershipForgetRoomStmt, updateMembershipForgetRoom},
{&s.selectLocalServerInRoomStmt, selectLocalServerInRoomSQL},
}.Prepare(db)
}
@ -324,3 +334,16 @@ func (s *membershipStatements) UpdateForgetMembership(
)
return err
}
func (s *membershipStatements) SelectLocalServerInRoom(ctx context.Context, roomNID types.RoomNID) (bool, error) {
var nid types.RoomNID
err := s.selectLocalServerInRoomStmt.QueryRowContext(ctx, tables.MembershipStateJoin, roomNID).Scan(&nid)
if err != nil {
if err == sql.ErrNoRows {
return false, nil
}
return false, err
}
found := nid > 0
return found, nil
}

2
roomserver/storage/postgres/state_block_table.go
View file

@ -64,7 +64,7 @@ const insertStateDataSQL = "" +
const bulkSelectStateBlockEntriesSQL = "" +
"SELECT state_block_nid, event_nids" +
" FROM roomserver_state_block WHERE state_block_nid = ANY($1)"
" FROM roomserver_state_block WHERE state_block_nid = ANY($1) ORDER BY state_block_nid ASC"
type stateBlockStatements struct {
insertStateDataStmt *sql.Stmt

5
roomserver/storage/shared/storage.go
View file

@ -1063,6 +1063,11 @@ func (d *Database) JoinedUsersSetInRooms(ctx context.Context, roomIDs []string)
return result, nil
}
// GetLocalServerInRoom returns true if we think we're in a given room or false otherwise.
func (d *Database) GetLocalServerInRoom(ctx context.Context, roomNID types.RoomNID) (bool, error) {
return d.MembershipTable.SelectLocalServerInRoom(ctx, roomNID)
}
// GetKnownUsers searches all users that userID knows about.
func (d *Database) GetKnownUsers(ctx context.Context, userID, searchString string, limit int) ([]string, error) {
stateKeyNID, err := d.EventStateKeysTable.SelectEventStateKeyNID(ctx, nil, userID)

23
roomserver/storage/sqlite3/membership_table.go
View file

@ -100,6 +100,14 @@ var selectKnownUsersSQL = "" +
" SELECT DISTINCT room_nid FROM roomserver_membership WHERE target_nid=$1 AND membership_nid = " + fmt.Sprintf("%d", tables.MembershipStateJoin) +
") AND membership_nid = " + fmt.Sprintf("%d", tables.MembershipStateJoin) + " AND event_state_key LIKE $2 LIMIT $3"
// selectLocalServerInRoomSQL is an optimised case for checking if we, the local server,
// are in the room by using the target_local column of the membership table. Normally when
// we want to know if a server is in a room, we have to unmarshal the entire room state which
// is expensive. The presence of a single row from this query suggests we're still in the
// room, no rows returned suggests we aren't.
const selectLocalServerInRoomSQL = "" +
"SELECT room_nid FROM roomserver_membership WHERE target_local = 1 AND membership_nid = $1 AND room_nid = $2 LIMIT 1"
type membershipStatements struct {
db *sql.DB
insertMembershipStmt *sql.Stmt
@ -113,6 +121,7 @@ type membershipStatements struct {
updateMembershipStmt *sql.Stmt
selectKnownUsersStmt *sql.Stmt
updateMembershipForgetRoomStmt *sql.Stmt
selectLocalServerInRoomStmt *sql.Stmt
}
func createMembershipTable(db *sql.DB) error {
@ -137,6 +146,7 @@ func prepareMembershipTable(db *sql.DB) (tables.Membership, error) {
{&s.selectRoomsWithMembershipStmt, selectRoomsWithMembershipSQL},
{&s.selectKnownUsersStmt, selectKnownUsersSQL},
{&s.updateMembershipForgetRoomStmt, updateMembershipForgetRoom},
{&s.selectLocalServerInRoomStmt, selectLocalServerInRoomSQL},
}.Prepare(db)
}
@ -304,3 +314,16 @@ func (s *membershipStatements) UpdateForgetMembership(
)
return err
}
func (s *membershipStatements) SelectLocalServerInRoom(ctx context.Context, roomNID types.RoomNID) (bool, error) {
var nid types.RoomNID
err := s.selectLocalServerInRoomStmt.QueryRowContext(ctx, tables.MembershipStateJoin, roomNID).Scan(&nid)
if err != nil {
if err == sql.ErrNoRows {
return false, nil
}
return false, err
}
found := nid > 0
return found, nil
}

2
roomserver/storage/sqlite3/state_block_table.go
View file

@ -57,7 +57,7 @@ const insertStateDataSQL = `
const bulkSelectStateBlockEntriesSQL = "" +
"SELECT state_block_nid, event_nids" +
" FROM roomserver_state_block WHERE state_block_nid IN ($1)"
" FROM roomserver_state_block WHERE state_block_nid IN ($1) ORDER BY state_block_nid ASC"
type stateBlockStatements struct {
db *sql.DB

1
roomserver/storage/tables/interface.go
View file

@ -135,6 +135,7 @@ type Membership interface {
SelectJoinedUsersSetForRooms(ctx context.Context, roomNIDs []types.RoomNID) (map[types.EventStateKeyNID]int, error)
SelectKnownUsers(ctx context.Context, userID types.EventStateKeyNID, searchString string, limit int) ([]string, error)
UpdateForgetMembership(ctx context.Context, txn *sql.Tx, roomNID types.RoomNID, targetUserNID types.EventStateKeyNID, forget bool) error
SelectLocalServerInRoom(ctx context.Context, roomNID types.RoomNID) (bool, error)
}
type Published interface {

8
setup/base.go
View file

@ -77,6 +77,7 @@ type BaseDendrite struct {
PublicKeyAPIMux *mux.Router
PublicMediaAPIMux *mux.Router
InternalAPIMux *mux.Router
SynapseAdminMux *mux.Router
UseHTTPAPIs bool
apiHttpClient *http.Client
httpClient *http.Client
@ -138,15 +139,14 @@ func NewBaseDendrite(cfg *config.Dendrite, componentName string, useHTTPAPIs boo
var dnsCache *gomatrixserverlib.DNSCache
if cfg.Global.DNSCache.Enabled {
lifetime := time.Second * cfg.Global.DNSCache.CacheLifetime
dnsCache = gomatrixserverlib.NewDNSCache(
cfg.Global.DNSCache.CacheSize,
lifetime,
cfg.Global.DNSCache.CacheLifetime,
)
logrus.Infof(
"DNS cache enabled (size %d, lifetime %s)",
cfg.Global.DNSCache.CacheSize,
lifetime,
cfg.Global.DNSCache.CacheLifetime,
)
}
@ -199,6 +199,7 @@ func NewBaseDendrite(cfg *config.Dendrite, componentName string, useHTTPAPIs boo
PublicKeyAPIMux: mux.NewRouter().SkipClean(true).PathPrefix(httputil.PublicKeyPathPrefix).Subrouter().UseEncodedPath(),
PublicMediaAPIMux: mux.NewRouter().SkipClean(true).PathPrefix(httputil.PublicMediaPathPrefix).Subrouter().UseEncodedPath(),
InternalAPIMux: mux.NewRouter().SkipClean(true).PathPrefix(httputil.InternalPathPrefix).Subrouter().UseEncodedPath(),
SynapseAdminMux: mux.NewRouter().SkipClean(true).PathPrefix("/_synapse/").Subrouter().UseEncodedPath(),
apiHttpClient: &apiClient,
httpClient: &client,
}
@ -391,6 +392,7 @@ func (b *BaseDendrite) SetupAndServeHTTP(
externalRouter.PathPrefix(httputil.PublicKeyPathPrefix).Handler(b.PublicKeyAPIMux)
externalRouter.PathPrefix(httputil.PublicFederationPathPrefix).Handler(federationHandler)
}
externalRouter.PathPrefix("/_synapse/").Handler(b.SynapseAdminMux)
externalRouter.PathPrefix(httputil.PublicMediaPathPrefix).Handler(b.PublicMediaAPIMux)
if internalAddr != NoListener && internalAddr != externalAddr {

12
setup/config/config_mediaapi.go
View file

@ -2,7 +2,6 @@ package config
import (
"fmt"
"math"
)
type MediaAPI struct {
@ -36,6 +35,9 @@ type MediaAPI struct {
ThumbnailSizes []ThumbnailSize `yaml:"thumbnail_sizes"`
}
// DefaultMaxFileSizeBytes defines the default file size allowed in transfers
var DefaultMaxFileSizeBytes = FileSizeBytes(10485760)
func (c *MediaAPI) Defaults() {
c.InternalAPI.Listen = "http://localhost:7774"
c.InternalAPI.Connect = "http://localhost:7774"
@ -43,8 +45,7 @@ func (c *MediaAPI) Defaults() {
c.Database.Defaults(5)
c.Database.ConnectionString = "file:mediaapi.db"
defaultMaxFileSizeBytes := FileSizeBytes(10485760)
c.MaxFileSizeBytes = &defaultMaxFileSizeBytes
c.MaxFileSizeBytes = &DefaultMaxFileSizeBytes
c.MaxThumbnailGenerators = 10
c.BasePath = "./media_store"
}
@ -58,11 +59,6 @@ func (c *MediaAPI) Verify(configErrs *ConfigErrors, isMonolith bool) {
checkNotEmpty(configErrs, "media_api.database.connection_string", string(c.Database.ConnectionString))
checkNotEmpty(configErrs, "media_api.base_path", string(c.BasePath))
// allow "unlimited" file size
if c.MaxFileSizeBytes != nil && *c.MaxFileSizeBytes <= 0 {
unlimitedSize := FileSizeBytes(math.MaxInt64 - 1)
c.MaxFileSizeBytes = &unlimitedSize
}
checkPositive(configErrs, "media_api.max_file_size_bytes", int64(*c.MaxFileSizeBytes))
checkPositive(configErrs, "media_api.max_thumbnail_generators", int64(c.MaxThumbnailGenerators))

4
setup/monolith.go
View file

@ -57,9 +57,9 @@ type Monolith struct {
}
// AddAllPublicRoutes attaches all public paths to the given router
func (m *Monolith) AddAllPublicRoutes(process *process.ProcessContext, csMux, ssMux, keyMux, mediaMux *mux.Router) {
func (m *Monolith) AddAllPublicRoutes(process *process.ProcessContext, csMux, ssMux, keyMux, mediaMux, synapseMux *mux.Router) {
clientapi.AddPublicRoutes(
csMux, &m.Config.ClientAPI, m.AccountDB,
csMux, synapseMux, &m.Config.ClientAPI, m.AccountDB,
m.FedClient, m.RoomserverAPI,
m.EDUInternalAPI, m.AppserviceAPI, transactions.New(),
m.FederationSenderAPI, m.UserAPI, m.KeyAPI, m.ExtPublicRoomsProvider,

16
syncapi/streams/stream_invite.go
View file

@ -2,8 +2,13 @@ package streams
import (
"context"
"crypto/sha256"
"encoding/base64"
"strconv"
"time"
"github.com/matrix-org/dendrite/syncapi/types"
"github.com/matrix-org/gomatrixserverlib"
)
type InviteStreamProvider struct {
@ -56,6 +61,17 @@ func (p *InviteStreamProvider) IncrementalSync(
for roomID := range retiredInvites {
if _, ok := req.Response.Rooms.Join[roomID]; !ok {
lr := types.NewLeaveResponse()
h := sha256.Sum256(append([]byte(roomID), []byte(strconv.FormatInt(int64(to), 10))...))
lr.Timeline.Events = append(lr.Timeline.Events, gomatrixserverlib.ClientEvent{
// fake event ID which muxes in the to position
EventID: "$" + base64.RawURLEncoding.EncodeToString(h[:]),
OriginServerTS: gomatrixserverlib.AsTimestamp(time.Now()),
RoomID: roomID,
Sender: req.Device.UserID,
StateKey: &req.Device.UserID,
Type: "m.room.member",
Content: gomatrixserverlib.RawJSON(`{"membership":"leave"}`),
})
req.Response.Rooms.Leave[roomID] = *lr
}
}

6
sytest-blacklist
View file

@ -72,4 +72,8 @@ Can re-join room if re-invited
Local device key changes get to remote servers with correct prev_id
# Flakey
Local device key changes appear in /keys/changes
Local device key changes appear in /keys/changes
# we don't support groups
Remove group category
Remove group role

15
sytest-whitelist
View file

@ -522,3 +522,18 @@ Invalid openid access tokens are rejected
Requests to userinfo without access tokens are rejected
'ban' event respects room powerlevel
Non-present room members cannot ban others
POST /_synapse/admin/v1/register with shared secret
POST /_synapse/admin/v1/register admin with shared secret
POST /_synapse/admin/v1/register with shared secret downcases capitals
POST /_synapse/admin/v1/register with shared secret disallows symbols
Membership event with an invalid displayname in the send_join response should not cause room join to fail
Inbound federation rejects incorrectly-signed invite rejections
Inbound federation can receive invite rejections
Inbound federation can receive invite and reject when remote replies with a 403
Inbound federation can receive invite and reject when remote replies with a 500
Inbound federation can receive invite and reject when remote is unreachable
Remote servers cannot set power levels in rooms without existing powerlevels
Remote servers should reject attempts by non-creators to set the power levels
Federation handles empty auth_events in state_ids sanely
Key notary server should return an expired key if it can't find any others
Key notary server must not overwrite a valid key with a spurious result from the origin server