From 30dbc5ac448e30f69305c81b7049d6bd30ae76d0 Mon Sep 17 00:00:00 2001 From: timaeos <104638316+timaeos@users.noreply.github.com> Date: Sat, 13 Aug 2022 13:24:52 -0500 Subject: [PATCH 01/42] Adding FAQ Questions with common questions (#2641) * Adding FAQ Questions with common questions * changing order of the FAQ to reflect suggestions --- docs/FAQ.md | 26 +++++++++++++++++++++++- docs/administration/5_troubleshooting.md | 9 +++++++- 2 files changed, 33 insertions(+), 2 deletions(-) diff --git a/docs/FAQ.md b/docs/FAQ.md index f8255684e..85ad83db5 100644 --- a/docs/FAQ.md +++ b/docs/FAQ.md @@ -12,7 +12,13 @@ Mostly, although there are still bugs and missing features. If you are a confide ## Is Dendrite feature-complete? -No, although a good portion of the Matrix specification has been implemented. Mostly missing are client features - see the readme at the root of the repository for more information. +No, although a good portion of the Matrix specification has been implemented. Mostly missing are client features - see the [readme](../README.md) at the root of the repository for more information. + +## Why doesn't Dendrite have yet? + +Dendrite development is currently supported by a small team of developers and due to those limited resources, the majority of the effort is focused on getting Dendrite to be +specification complete. If there are major features you're requesting (e.g. new administration endpoints), we'd like to strongly encourage you to join the community in supporting +the development efforts through [contributing](https://matrix-org.github.io/dendrite/development/contributing). ## Is there a migration path from Synapse to Dendrite? @@ -43,6 +49,20 @@ It should do, although we are aware of some minor issues: * **Element Android**: registration does not work, but logging in with an existing account does * **Hydrogen**: occasionally sync can fail due to gaps in the `since` parameter, but clearing the cache fixes this +## Does Dendrite support Space Summaries? + +Yes, [Space Summaries](https://github.com/matrix-org/matrix-spec-proposals/pull/2946) were merged into the Matrix Spec as of 2022-01-17 however, they are still treated as an MSC (Matrix Specification Change) in Dendrite. In order to enable Space Summaries in Dendrite, you must add the MSC to the MSC configuration section in the configuration YAML. If the MSC is not enabled, a user will typically see a perpetual loading icon on the summary page. See below for a demonstration of how to add to the Dendrite configuration: + +``` +mscs: + mscs: + - msc2946 +``` + +Similarly, [msc2836](https://github.com/matrix-org/matrix-spec-proposals/pull/2836) would need to be added to mscs configuration in order to support Threading. Other MSCs are not currently supported. + +Please note that MSCs should be considered experimental and can result in significant usability issues when enabled. If you'd like more details on how MSCs are ratified or the current status of MSCs, please see the [Matrix specification documentation](https://spec.matrix.org/proposals/) on the subject. + ## Does Dendrite support push notifications? Yes, we have experimental support for push notifications. Configure them in the usual way in your Matrix client. @@ -86,6 +106,10 @@ would be a huge help too, as that will help us to understand where the memory us You may need to revisit the connection limit of your PostgreSQL server and/or make changes to the `max_connections` lines in your Dendrite configuration. Be aware that each Dendrite component opens its own database connections and has its own connection limit, even in monolith mode! +## VOIP and Video Calls don't appear to work on Dendrite + +There is likely an issue with your STUN/TURN configuration on the server. If you believe your configuration to be correct, please see the [troubleshooting](administration/5_troubleshooting.md) for troubleshooting recommendations. + ## What is being reported when enabling phone-home statistics? Phone-home statistics contain your server's domain name, some configuration information about diff --git a/docs/administration/5_troubleshooting.md b/docs/administration/5_troubleshooting.md index 14df2e3fb..8ba510ef6 100644 --- a/docs/administration/5_troubleshooting.md +++ b/docs/administration/5_troubleshooting.md @@ -77,5 +77,12 @@ If there aren't, you will see a log lines like this: level=warning msg="IMPORTANT: Process file descriptor limit is currently 65535, it is recommended to raise the limit for Dendrite to at least 65535 to avoid issues" ``` -Follow the [Optimisation](../installation/10_optimisation.md) instructions to correct the +Follow the [Optimisation](../installation/11_optimisation.md) instructions to correct the available number of file descriptors. + +## 6. STUN/TURN Server tester + +If you are experiencing problems with VoIP or video calls, you should check that Dendrite +is able to successfully connect your TURN server using +[Matrix VoIP Tester](https://test.voip.librepush.net/). This can highlight any issues +that the server may encounter so that you can begin the troubleshooting process. From da7d209b10c01ea3762bf5e524dcc17549de934d Mon Sep 17 00:00:00 2001 From: timaeos <104638316+timaeos@users.noreply.github.com> Date: Sat, 13 Aug 2022 13:25:39 -0500 Subject: [PATCH 02/42] Admin endpoint documentation (#2640) * Adding documentation for administration end points that aren't currently documented * Additional information on using the administration API * Fix inaccurate information on server notice documentation * Fix typo in the curl command * Add details about using the whois endpoint * fix formatting in documentation for admin whois endpoint * Fix more documentation formatting * Additional formatting updates to match other formats * Add a link to the server notice information in the Matrix Spec * Fix broken link on server notices * Adding access token information and admin info * adding a warning regarding access_token Co-authored-by: Timothy Arnold --- docs/administration/4_adminapi.md | 46 +++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/docs/administration/4_adminapi.md b/docs/administration/4_adminapi.md index 783fee95a..a34bfde1f 100644 --- a/docs/administration/4_adminapi.md +++ b/docs/administration/4_adminapi.md @@ -13,6 +13,25 @@ without warning. More endpoints will be added in the future. +Endpoints may be used directly through curl: + +``` +curl --header "Authorization: Bearer " -X -d '' +``` + +An `access_token` can be obtained through most Element-based matrix clients by going to `Settings` -> `Help & About` -> `Advanced` -> `Access Token`. +Be aware that an `access_token` allows a client to perform actions as an user and should be kept **secret**. + +The user must be an administrator in the `account_accounts` table in order to use these endpoints. + +Existing user accounts can be set to administrative accounts by changing `account_type` to `3` in `account_accounts` + +``` +UPDATE account_accounts SET account_type = 3 WHERE localpart = '$localpart'; +``` + +Where `$localpart` is the username only (e.g. `alice`). + ## GET `/_dendrite/admin/evacuateRoom/{roomID}` This endpoint will instruct Dendrite to part all local users from the given `roomID` @@ -38,7 +57,34 @@ Request body format: Reset the password of a local user. The `localpart` is the username only, i.e. if the full user ID is `@alice:domain.com` then the local part is `alice`. +## POST `/_synapse/admin/v1/send_server_notice` + +Request body format: +``` +{ + "user_id": "@target_user:server_name", + "content": { + "msgtype": "m.text", + "body": "This is my message" + } +} +``` + +Send a server notice to a specific user. See the [Matrix Spec](https://spec.matrix.org/v1.3/client-server-api/#server-notices) for additional details on server notice behaviour. +If successfully sent, the API will return the following response: + +``` +{ + "event_id": "" +} +``` + ## GET `/_synapse/admin/v1/register` Shared secret registration — please see the [user creation page](createusers) for guidance on configuring and using this endpoint. + +## GET `/_matrix/client/v3/admin/whois/{userId}` + +From the [Matrix Spec](https://spec.matrix.org/v1.3/client-server-api/#get_matrixclientv3adminwhoisuserid). +Gets information about a particular user. `userId` is the full user ID (e.g. `@alice:domain.com`) From 65b7db633c07311fea109ca045fcdb20c87d22a3 Mon Sep 17 00:00:00 2001 From: Till Faelligen <2353100+S7evinK@users.noreply.github.com> Date: Sat, 13 Aug 2022 20:31:43 +0200 Subject: [PATCH 03/42] Fix FAQ question --- docs/FAQ.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/FAQ.md b/docs/FAQ.md index 85ad83db5..ca72b151d 100644 --- a/docs/FAQ.md +++ b/docs/FAQ.md @@ -14,7 +14,7 @@ Mostly, although there are still bugs and missing features. If you are a confide No, although a good portion of the Matrix specification has been implemented. Mostly missing are client features - see the [readme](../README.md) at the root of the repository for more information. -## Why doesn't Dendrite have yet? +## Why doesn't Dendrite have "x" yet? Dendrite development is currently supported by a small team of developers and due to those limited resources, the majority of the effort is focused on getting Dendrite to be specification complete. If there are major features you're requesting (e.g. new administration endpoints), we'd like to strongly encourage you to join the community in supporting From bcdbd5c00a757971f753622b5a9f78347d5160d4 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Mon, 15 Aug 2022 10:56:54 +0100 Subject: [PATCH 04/42] Version 0.9.3 --- CHANGES.md | 10 ++++++++++ go.mod | 2 +- go.sum | 4 ++-- internal/version.go | 2 +- 4 files changed, 14 insertions(+), 4 deletions(-) diff --git a/CHANGES.md b/CHANGES.md index 041816ae3..36a1a6311 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -1,5 +1,15 @@ # Changelog +## Dendrite 0.9.3 (2022-08-15) + +### Important + +* This is a **security release** to fix a vulnerability within event auth, affecting all versions of Dendrite before 0.9.3. Upgrading to this version is highly recommended. For more information, [see here](https://github.com/matrix-org/gomatrixserverlib/security/advisories/GHSA-grvv-h2f9-7v9c). + +### Fixes + +* Dendrite will now correctly parse the `"events_default"` power level value for event auth. + ## Dendrite 0.9.2 (2022-08-12) ### Features diff --git a/go.mod b/go.mod index 108165ba2..fe4604a03 100644 --- a/go.mod +++ b/go.mod @@ -21,7 +21,7 @@ require ( github.com/matrix-org/dugong v0.0.0-20210921133753-66e6b1c67e2e github.com/matrix-org/go-sqlite3-js v0.0.0-20220419092513-28aa791a1c91 github.com/matrix-org/gomatrix v0.0.0-20210324163249-be2af5ef2e16 - github.com/matrix-org/gomatrixserverlib v0.0.0-20220801083850-5ff38e2c2839 + github.com/matrix-org/gomatrixserverlib v0.0.0-20220815094957-74b7ff4ae09c github.com/matrix-org/pinecone v0.0.0-20220803093810-b7a830c08fb9 github.com/matrix-org/util v0.0.0-20200807132607-55161520e1d4 github.com/mattn/go-sqlite3 v1.14.13 diff --git a/go.sum b/go.sum index e925d6415..7ac9fc6e2 100644 --- a/go.sum +++ b/go.sum @@ -343,8 +343,8 @@ github.com/matrix-org/go-sqlite3-js v0.0.0-20220419092513-28aa791a1c91/go.mod h1 github.com/matrix-org/gomatrix v0.0.0-20190528120928-7df988a63f26/go.mod h1:3fxX6gUjWyI/2Bt7J1OLhpCzOfO/bB3AiX0cJtEKud0= github.com/matrix-org/gomatrix v0.0.0-20210324163249-be2af5ef2e16 h1:ZtO5uywdd5dLDCud4r0r55eP4j9FuUNpl60Gmntcop4= github.com/matrix-org/gomatrix v0.0.0-20210324163249-be2af5ef2e16/go.mod h1:/gBX06Kw0exX1HrwmoBibFA98yBk/jxKpGVeyQbff+s= -github.com/matrix-org/gomatrixserverlib v0.0.0-20220801083850-5ff38e2c2839 h1:QEFxKWH8PlEt3ZQKl31yJNAm8lvpNUwT51IMNTl9v1k= -github.com/matrix-org/gomatrixserverlib v0.0.0-20220801083850-5ff38e2c2839/go.mod h1:jX38yp3SSLJNftBg3PXU1ayd0PCLIiDHQ4xAc9DIixk= +github.com/matrix-org/gomatrixserverlib v0.0.0-20220815094957-74b7ff4ae09c h1:GhKmb8s9iXA9qsFD1SbiRo6Ee7cnbfcgJQ/iy43wczM= +github.com/matrix-org/gomatrixserverlib v0.0.0-20220815094957-74b7ff4ae09c/go.mod h1:jX38yp3SSLJNftBg3PXU1ayd0PCLIiDHQ4xAc9DIixk= github.com/matrix-org/pinecone v0.0.0-20220803093810-b7a830c08fb9 h1:ed8yvWhTLk7+sNeK/eOZRTvESFTOHDRevoRoyeqPtvY= github.com/matrix-org/pinecone v0.0.0-20220803093810-b7a830c08fb9/go.mod h1:P4MqPf+u83OPulPJ+XTbSDbbWrdFYNY4LZ/B1PIduFE= github.com/matrix-org/util v0.0.0-20190711121626-527ce5ddefc7/go.mod h1:vVQlW/emklohkZnOPwD3LrZUBqdfsbiyO3p1lNV8F6U= diff --git a/internal/version.go b/internal/version.go index 18db2dcc8..561e6c06f 100644 --- a/internal/version.go +++ b/internal/version.go @@ -17,7 +17,7 @@ var build string const ( VersionMajor = 0 VersionMinor = 9 - VersionPatch = 2 + VersionPatch = 3 VersionTag = "" // example: "rc1" ) From 0642ffc0f6c6d21af0e4a17acd84200e35bf279a Mon Sep 17 00:00:00 2001 From: Till <2353100+S7evinK@users.noreply.github.com> Date: Tue, 16 Aug 2022 10:29:36 +0200 Subject: [PATCH 05/42] Only return non-retired invites (#2643) * Only return non-retired invites * Revert "Only return non-retired invites" This reverts commit 1150aa7f385b7d7cf5378297f3e17566d5aabcc6. * Check if we're doing an initial sync in the stream --- syncapi/streams/stream_invite.go | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/syncapi/streams/stream_invite.go b/syncapi/streams/stream_invite.go index ddac9be2c..925da32f2 100644 --- a/syncapi/streams/stream_invite.go +++ b/syncapi/streams/stream_invite.go @@ -7,8 +7,9 @@ import ( "strconv" "time" - "github.com/matrix-org/dendrite/syncapi/types" "github.com/matrix-org/gomatrixserverlib" + + "github.com/matrix-org/dendrite/syncapi/types" ) type InviteStreamProvider struct { @@ -62,6 +63,11 @@ func (p *InviteStreamProvider) IncrementalSync( req.Response.Rooms.Invite[roomID] = *ir } + // When doing an initial sync, we don't want to add retired invites, as this + // can add rooms we were invited to, but already left. + if from == 0 { + return to + } for roomID := range retiredInvites { if _, ok := req.Response.Rooms.Join[roomID]; !ok { lr := types.NewLeaveResponse() From 5424b88f3061dddbeae53df8b67bd064e8cbc400 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Tue, 16 Aug 2022 11:55:06 +0100 Subject: [PATCH 06/42] Use `is_direct` flag from `/createRoom`, update stripped state (#2644) * Use `is_direct` flag from `/createRoom`, update stripped state * Add comment --- clientapi/routing/createroom.go | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/clientapi/routing/createroom.go b/clientapi/routing/createroom.go index 874908639..3e837c864 100644 --- a/clientapi/routing/createroom.go +++ b/clientapi/routing/createroom.go @@ -49,6 +49,7 @@ type createRoomRequest struct { GuestCanJoin bool `json:"guest_can_join"` RoomVersion gomatrixserverlib.RoomVersion `json:"room_version"` PowerLevelContentOverride json.RawMessage `json:"power_level_content_override"` + IsDirect bool `json:"is_direct"` } const ( @@ -499,9 +500,17 @@ func createRoom( // Build some stripped state for the invite. var globalStrippedState []gomatrixserverlib.InviteV2StrippedState for _, event := range builtEvents { + // Chosen events from the spec: + // https://spec.matrix.org/v1.3/client-server-api/#stripped-state switch event.Type() { + case gomatrixserverlib.MRoomCreate: + fallthrough case gomatrixserverlib.MRoomName: fallthrough + case gomatrixserverlib.MRoomAvatar: + fallthrough + case gomatrixserverlib.MRoomTopic: + fallthrough case gomatrixserverlib.MRoomCanonicalAlias: fallthrough case gomatrixserverlib.MRoomEncryption: @@ -522,7 +531,7 @@ func createRoom( // Build the invite event. inviteEvent, err := buildMembershipEvent( ctx, invitee, "", profileAPI, device, gomatrixserverlib.Invite, - roomID, true, cfg, evTime, rsAPI, asAPI, + roomID, r.IsDirect, cfg, evTime, rsAPI, asAPI, ) if err != nil { util.GetLogger(ctx).WithError(err).Error("buildMembershipEvent failed") From 804653e55118a8155dfbfe5187e941a2f8ce8337 Mon Sep 17 00:00:00 2001 From: Till <2353100+S7evinK@users.noreply.github.com> Date: Tue, 16 Aug 2022 13:21:22 +0200 Subject: [PATCH 07/42] Verify a shared secret is set in `create-account` (#2645) --- cmd/create-account/main.go | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/cmd/create-account/main.go b/cmd/create-account/main.go index fbd14b8d7..bd053f2f7 100644 --- a/cmd/create-account/main.go +++ b/cmd/create-account/main.go @@ -85,6 +85,10 @@ func main() { logrus.Fatalf("The reset-password flag has been replaced by the POST /_dendrite/admin/resetPassword/{localpart} admin API.") } + if cfg.ClientAPI.RegistrationSharedSecret == "" { + logrus.Fatalln("Shared secret registration is not enabled, enable it by setting a shared secret in the config: 'client_api.registration_shared_secret'") + } + if *username == "" { flag.Usage() os.Exit(1) From ec16c944eb646c6ef61f0b7783f2f3869b9bc10c Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Tue, 16 Aug 2022 14:42:00 +0100 Subject: [PATCH 08/42] Lazy-loading fixes (#2646) * Use existing current room state if we have it * Don't dedupe before applying the history vis filter * Revert "Don't dedupe before applying the history vis filter" This reverts commit d27c4a0874dabb77c2eda6b23eb7c00478bc9e90. * Revert "Use existing current room state if we have it" This reverts commit 5819b4a7ce511204c4fb48d3c4741612b136e2ea. * Tweaks --- syncapi/streams/stream_pdu.go | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/syncapi/streams/stream_pdu.go b/syncapi/streams/stream_pdu.go index 136cbea5a..0e9dda577 100644 --- a/syncapi/streams/stream_pdu.go +++ b/syncapi/streams/stream_pdu.go @@ -560,14 +560,13 @@ func (p *PDUStreamProvider) lazyLoadMembers( // If this is a gapped incremental sync, we still want this membership isGappedIncremental := limited && incremental // We want this users membership event, keep it in the list - _, ok := timelineUsers[event.Sender()] - wantMembership := ok || isGappedIncremental - if wantMembership { + stateKey := *event.StateKey() + if _, ok := timelineUsers[stateKey]; ok || isGappedIncremental { newStateEvents = append(newStateEvents, event) if !includeRedundant { - p.lazyLoadCache.StoreLazyLoadedUser(device, roomID, event.Sender(), event.EventID()) + p.lazyLoadCache.StoreLazyLoadedUser(device, roomID, stateKey, event.EventID()) } - delete(timelineUsers, event.Sender()) + delete(timelineUsers, stateKey) } } else { newStateEvents = append(newStateEvents, event) @@ -578,17 +577,16 @@ func (p *PDUStreamProvider) lazyLoadMembers( wantUsers = append(wantUsers, userID) } // Query missing membership events - memberships, err := p.DB.GetStateEventsForRoom(ctx, roomID, &gomatrixserverlib.StateFilter{ - Limit: 100, - Senders: &wantUsers, - Types: &[]string{gomatrixserverlib.MRoomMember}, - }) + filter := gomatrixserverlib.DefaultStateFilter() + filter.Senders = &wantUsers + filter.Types = &[]string{gomatrixserverlib.MRoomMember} + memberships, err := p.DB.GetStateEventsForRoom(ctx, roomID, &filter) if err != nil { return stateEvents, err } // cache the membership events for _, membership := range memberships { - p.lazyLoadCache.StoreLazyLoadedUser(device, roomID, membership.Sender(), membership.EventID()) + p.lazyLoadCache.StoreLazyLoadedUser(device, roomID, *membership.StateKey(), membership.EventID()) } stateEvents = append(newStateEvents, memberships...) return stateEvents, nil From ad4ac2c016d2916c51695fd11d3a7b52e38d0ca2 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Tue, 16 Aug 2022 14:42:35 +0100 Subject: [PATCH 09/42] Stop spamming the logs with `StateBetween: ignoring deleted state` event IDs --- syncapi/storage/postgres/output_room_events_table.go | 4 ++-- syncapi/storage/sqlite3/output_room_events_table.go | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/syncapi/storage/postgres/output_room_events_table.go b/syncapi/storage/postgres/output_room_events_table.go index 8f633640e..9c758cac1 100644 --- a/syncapi/storage/postgres/output_room_events_table.go +++ b/syncapi/storage/postgres/output_room_events_table.go @@ -279,8 +279,8 @@ func (s *outputRoomEventsStatements) SelectStateInRange( log.WithFields(log.Fields{ "since": r.From, "current": r.To, - "adds": addIDs, - "dels": delIDs, + "adds": len(addIDs), + "dels": len(delIDs), }).Warn("StateBetween: ignoring deleted state") } diff --git a/syncapi/storage/sqlite3/output_room_events_table.go b/syncapi/storage/sqlite3/output_room_events_table.go index 91fd35b5b..cd9a46d4c 100644 --- a/syncapi/storage/sqlite3/output_room_events_table.go +++ b/syncapi/storage/sqlite3/output_room_events_table.go @@ -234,8 +234,8 @@ func (s *outputRoomEventsStatements) SelectStateInRange( log.WithFields(log.Fields{ "since": r.From, "current": r.To, - "adds": addIDsJSON, - "dels": delIDsJSON, + "adds": len(addIDsJSON), + "dels": len(delIDsJSON), }).Warn("StateBetween: ignoring deleted state") } From 8d9c8f11c5c5673d48b8e5308e1f565927bdd341 Mon Sep 17 00:00:00 2001 From: Till Faelligen <2353100+S7evinK@users.noreply.github.com> Date: Thu, 18 Aug 2022 08:56:57 +0200 Subject: [PATCH 10/42] Add a delay after sending events to the roomserver --- syncapi/syncapi_test.go | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/syncapi/syncapi_test.go b/syncapi/syncapi_test.go index dc073a16e..089bdafaf 100644 --- a/syncapi/syncapi_test.go +++ b/syncapi/syncapi_test.go @@ -10,6 +10,10 @@ import ( "testing" "time" + "github.com/matrix-org/gomatrixserverlib" + "github.com/nats-io/nats.go" + "github.com/tidwall/gjson" + "github.com/matrix-org/dendrite/clientapi/producers" keyapi "github.com/matrix-org/dendrite/keyserver/api" "github.com/matrix-org/dendrite/roomserver" @@ -21,9 +25,6 @@ import ( "github.com/matrix-org/dendrite/test" "github.com/matrix-org/dendrite/test/testrig" userapi "github.com/matrix-org/dendrite/userapi/api" - "github.com/matrix-org/gomatrixserverlib" - "github.com/nats-io/nats.go" - "github.com/tidwall/gjson" ) type syncRoomserverAPI struct { @@ -422,6 +423,7 @@ func testHistoryVisibility(t *testing.T, dbType test.DBType) { if err := api.SendEvents(ctx, rsAPI, api.KindNew, eventsToSend, "test", "test", nil, false); err != nil { t.Fatalf("failed to send events: %v", err) } + time.Sleep(100 * time.Millisecond) // TODO: find a better way // There is only one event, we expect only to be able to see this, if the room is world_readable w := httptest.NewRecorder() @@ -454,6 +456,7 @@ func testHistoryVisibility(t *testing.T, dbType test.DBType) { if err := api.SendEvents(ctx, rsAPI, api.KindNew, eventsToSend, "test", "test", nil, false); err != nil { t.Fatalf("failed to send events: %v", err) } + time.Sleep(100 * time.Millisecond) // TODO: find a better way // Verify the messages after/before invite are visible or not w = httptest.NewRecorder() From 59bc0a6f4ed0324da0387118e1761b4551aaf103 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Thu, 18 Aug 2022 10:37:47 +0100 Subject: [PATCH 11/42] Reprocess rejected input events (#2647) * Reprocess outliers that were previously rejected * Might as well do all events this way * More useful errors * Fix queries * Tweak condition * Don't wrap errors * Report more useful error * Flatten error on `r.Queryer.QueryStateAfterEvents` * Some more debug logging * Flatten error in `QueryRestrictedJoinAllowed` * Revert "Flatten error in `QueryRestrictedJoinAllowed`" This reverts commit 1238b4184c30e0c31ffb0f364806fa1275aba483. * Tweak `QueryStateAfterEvents` * Handle MissingStateError too * Scope to room * Clean up * Fix the error * Only apply rejection check to outliers --- roomserver/internal/input/input_events.go | 47 +++++++++++---------- roomserver/internal/query/query.go | 13 +++--- roomserver/storage/interface.go | 2 + roomserver/storage/postgres/events_table.go | 13 ++++++ roomserver/storage/shared/storage.go | 4 ++ roomserver/storage/sqlite3/events_table.go | 13 ++++++ roomserver/storage/tables/interface.go | 1 + 7 files changed, 65 insertions(+), 28 deletions(-) diff --git a/roomserver/internal/input/input_events.go b/roomserver/internal/input/input_events.go index 81541260c..53ccd5973 100644 --- a/roomserver/internal/input/input_events.go +++ b/roomserver/internal/input/input_events.go @@ -17,8 +17,8 @@ package input import ( - "bytes" "context" + "database/sql" "fmt" "time" @@ -107,28 +107,6 @@ func (r *Inputer) processRoomEvent( }) } - // if we have already got this event then do not process it again, if the input kind is an outlier. - // Outliers contain no extra information which may warrant a re-processing. - if input.Kind == api.KindOutlier { - evs, err2 := r.DB.EventsFromIDs(ctx, []string{event.EventID()}) - if err2 == nil && len(evs) == 1 { - // check hash matches if we're on early room versions where the event ID was a random string - idFormat, err2 := headered.RoomVersion.EventIDFormat() - if err2 == nil { - switch idFormat { - case gomatrixserverlib.EventIDFormatV1: - if bytes.Equal(event.EventReference().EventSHA256, evs[0].EventReference().EventSHA256) { - logger.Debugf("Already processed event; ignoring") - return nil - } - default: - logger.Debugf("Already processed event; ignoring") - return nil - } - } - } - } - // Don't waste time processing the event if the room doesn't exist. // A room entry locally will only be created in response to a create // event. @@ -141,6 +119,29 @@ func (r *Inputer) processRoomEvent( return fmt.Errorf("room %s does not exist for event %s", event.RoomID(), event.EventID()) } + // If we already know about this outlier and it hasn't been rejected + // then we won't attempt to reprocess it. If it was rejected or has now + // arrived as a different kind of event, then we can attempt to reprocess, + // in case we have learned something new or need to weave the event into + // the DAG now. + if input.Kind == api.KindOutlier && roomInfo != nil { + wasRejected, werr := r.DB.IsEventRejected(ctx, roomInfo.RoomNID, event.EventID()) + switch { + case werr == sql.ErrNoRows: + // We haven't seen this event before so continue. + case werr != nil: + // Something has gone wrong trying to find out if we rejected + // this event already. + logger.WithError(werr).Errorf("Failed to check if event %q is already seen", event.EventID()) + return werr + case !wasRejected: + // We've seen this event before and it wasn't rejected so we + // should ignore it. + logger.Debugf("Already processed event %q, ignoring", event.EventID()) + return nil + } + } + var missingAuth, missingPrev bool serverRes := &fedapi.QueryJoinedHostServerNamesInRoomResponse{} if !isCreateEvent { diff --git a/roomserver/internal/query/query.go b/roomserver/internal/query/query.go index f3061c222..f5d8c2d49 100644 --- a/roomserver/internal/query/query.go +++ b/roomserver/internal/query/query.go @@ -72,13 +72,10 @@ func (r *Queryer) QueryStateAfterEvents( prevStates, err := r.DB.StateAtEventIDs(ctx, request.PrevEventIDs) if err != nil { - switch err.(type) { - case types.MissingEventError: - util.GetLogger(ctx).Errorf("QueryStateAfterEvents: MissingEventError: %s", err) + if _, ok := err.(types.MissingEventError); ok { return nil - default: - return err } + return err } response.PrevEventsExist = true @@ -95,6 +92,12 @@ func (r *Queryer) QueryStateAfterEvents( ) } if err != nil { + if _, ok := err.(types.MissingEventError); ok { + return nil + } + if _, ok := err.(types.MissingStateError); ok { + return nil + } return err } diff --git a/roomserver/storage/interface.go b/roomserver/storage/interface.go index b12025c41..5c068873a 100644 --- a/roomserver/storage/interface.go +++ b/roomserver/storage/interface.go @@ -94,6 +94,8 @@ type Database interface { // Opens and returns a room updater, which locks the room and opens a transaction. // The GetRoomUpdater must have Commit or Rollback called on it if this doesn't return an error. // If this returns an error then no further action is required. + // IsEventRejected returns true if the event is known and rejected. + IsEventRejected(ctx context.Context, roomNID types.RoomNID, eventID string) (rejected bool, err error) GetRoomUpdater(ctx context.Context, roomInfo *types.RoomInfo) (*shared.RoomUpdater, error) // Look up event references for the latest events in the room and the current state snapshot. // Returns the latest events, the current state and the maximum depth of the latest events plus 1. diff --git a/roomserver/storage/postgres/events_table.go b/roomserver/storage/postgres/events_table.go index a4d05756d..c7748d2be 100644 --- a/roomserver/storage/postgres/events_table.go +++ b/roomserver/storage/postgres/events_table.go @@ -136,6 +136,9 @@ const selectMaxEventDepthSQL = "" + const selectRoomNIDsForEventNIDsSQL = "" + "SELECT event_nid, room_nid FROM roomserver_events WHERE event_nid = ANY($1)" +const selectEventRejectedSQL = "" + + "SELECT is_rejected FROM roomserver_events WHERE room_nid = $1 AND event_id = $2" + type eventStatements struct { insertEventStmt *sql.Stmt selectEventStmt *sql.Stmt @@ -153,6 +156,7 @@ type eventStatements struct { bulkSelectUnsentEventNIDStmt *sql.Stmt selectMaxEventDepthStmt *sql.Stmt selectRoomNIDsForEventNIDsStmt *sql.Stmt + selectEventRejectedStmt *sql.Stmt } func CreateEventsTable(db *sql.DB) error { @@ -180,6 +184,7 @@ func PrepareEventsTable(db *sql.DB) (tables.Events, error) { {&s.bulkSelectUnsentEventNIDStmt, bulkSelectUnsentEventNIDSQL}, {&s.selectMaxEventDepthStmt, selectMaxEventDepthSQL}, {&s.selectRoomNIDsForEventNIDsStmt, selectRoomNIDsForEventNIDsSQL}, + {&s.selectEventRejectedStmt, selectEventRejectedSQL}, }.Prepare(db) } @@ -540,3 +545,11 @@ func eventNIDsAsArray(eventNIDs []types.EventNID) pq.Int64Array { } return nids } + +func (s *eventStatements) SelectEventRejected( + ctx context.Context, txn *sql.Tx, roomNID types.RoomNID, eventID string, +) (rejected bool, err error) { + stmt := sqlutil.TxStmt(txn, s.selectEventRejectedStmt) + err = stmt.QueryRowContext(ctx, roomNID, eventID).Scan(&rejected) + return +} diff --git a/roomserver/storage/shared/storage.go b/roomserver/storage/shared/storage.go index cbf9c8b20..4f92adf1f 100644 --- a/roomserver/storage/shared/storage.go +++ b/roomserver/storage/shared/storage.go @@ -567,6 +567,10 @@ func (d *Database) GetRoomUpdater( return updater, err } +func (d *Database) IsEventRejected(ctx context.Context, roomNID types.RoomNID, eventID string) (bool, error) { + return d.EventsTable.SelectEventRejected(ctx, nil, roomNID, eventID) +} + func (d *Database) StoreEvent( ctx context.Context, event *gomatrixserverlib.Event, authEventNIDs []types.EventNID, isRejected bool, diff --git a/roomserver/storage/sqlite3/events_table.go b/roomserver/storage/sqlite3/events_table.go index 1dda34c36..174e3a9a7 100644 --- a/roomserver/storage/sqlite3/events_table.go +++ b/roomserver/storage/sqlite3/events_table.go @@ -109,6 +109,9 @@ const selectMaxEventDepthSQL = "" + const selectRoomNIDsForEventNIDsSQL = "" + "SELECT event_nid, room_nid FROM roomserver_events WHERE event_nid IN ($1)" +const selectEventRejectedSQL = "" + + "SELECT is_rejected FROM roomserver_events WHERE room_nid = $1 AND event_id = $2" + type eventStatements struct { db *sql.DB insertEventStmt *sql.Stmt @@ -122,6 +125,7 @@ type eventStatements struct { bulkSelectStateAtEventAndReferenceStmt *sql.Stmt bulkSelectEventReferenceStmt *sql.Stmt bulkSelectEventIDStmt *sql.Stmt + selectEventRejectedStmt *sql.Stmt //bulkSelectEventNIDStmt *sql.Stmt //bulkSelectUnsentEventNIDStmt *sql.Stmt //selectRoomNIDsForEventNIDsStmt *sql.Stmt @@ -152,6 +156,7 @@ func PrepareEventsTable(db *sql.DB) (tables.Events, error) { //{&s.bulkSelectEventNIDStmt, bulkSelectEventNIDSQL}, //{&s.bulkSelectUnsentEventNIDStmt, bulkSelectUnsentEventNIDSQL}, //{&s.selectRoomNIDForEventNIDStmt, selectRoomNIDForEventNIDSQL}, + {&s.selectEventRejectedStmt, selectEventRejectedSQL}, }.Prepare(db) } @@ -614,3 +619,11 @@ func eventNIDsAsArray(eventNIDs []types.EventNID) string { b, _ := json.Marshal(eventNIDs) return string(b) } + +func (s *eventStatements) SelectEventRejected( + ctx context.Context, txn *sql.Tx, roomNID types.RoomNID, eventID string, +) (rejected bool, err error) { + stmt := sqlutil.TxStmt(txn, s.selectEventRejectedStmt) + err = stmt.QueryRowContext(ctx, roomNID, eventID).Scan(&rejected) + return +} diff --git a/roomserver/storage/tables/interface.go b/roomserver/storage/tables/interface.go index 0bc389b80..ed67c43d8 100644 --- a/roomserver/storage/tables/interface.go +++ b/roomserver/storage/tables/interface.go @@ -66,6 +66,7 @@ type Events interface { BulkSelectUnsentEventNID(ctx context.Context, txn *sql.Tx, eventIDs []string) (map[string]types.EventNID, error) SelectMaxEventDepth(ctx context.Context, txn *sql.Tx, eventNIDs []types.EventNID) (int64, error) SelectRoomNIDsForEventNIDs(ctx context.Context, txn *sql.Tx, eventNIDs []types.EventNID) (roomNIDs map[types.EventNID]types.RoomNID, err error) + SelectEventRejected(ctx context.Context, txn *sql.Tx, roomNID types.RoomNID, eventID string) (rejected bool, err error) } type Rooms interface { From 7484689ad124c5759c8a10bebaabe9265602fd7d Mon Sep 17 00:00:00 2001 From: Till <2353100+S7evinK@users.noreply.github.com> Date: Thu, 18 Aug 2022 12:14:42 +0200 Subject: [PATCH 12/42] Actually store EDUs once we retrieved from the database (#2651) We now actually cache the EDUs once we got them from the database and ensures we only evict them if we successfully deleted them. --- federationapi/storage/shared/storage_edus.go | 21 ++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/federationapi/storage/shared/storage_edus.go b/federationapi/storage/shared/storage_edus.go index b62e5d9c5..ce9632ed3 100644 --- a/federationapi/storage/shared/storage_edus.go +++ b/federationapi/storage/shared/storage_edus.go @@ -110,6 +110,7 @@ func (d *Database) GetPendingEDUs( return fmt.Errorf("json.Unmarshal: %w", err) } edus[&Receipt{nid}] = &event + d.Cache.StoreFederationQueuedEDU(nid, &event) } return nil @@ -177,20 +178,18 @@ func (d *Database) GetPendingEDUServerNames( return d.FederationQueueEDUs.SelectQueueEDUServerNames(ctx, nil) } -// DeleteExpiredEDUs deletes expired EDUs +// DeleteExpiredEDUs deletes expired EDUs and evicts them from the cache. func (d *Database) DeleteExpiredEDUs(ctx context.Context) error { - return d.Writer.Do(d.DB, nil, func(txn *sql.Tx) error { + var jsonNIDs []int64 + err := d.Writer.Do(d.DB, nil, func(txn *sql.Tx) (err error) { expiredBefore := gomatrixserverlib.AsTimestamp(time.Now()) - jsonNIDs, err := d.FederationQueueEDUs.SelectExpiredEDUs(ctx, txn, expiredBefore) + jsonNIDs, err = d.FederationQueueEDUs.SelectExpiredEDUs(ctx, txn, expiredBefore) if err != nil { return err } if len(jsonNIDs) == 0 { return nil } - for i := range jsonNIDs { - d.Cache.EvictFederationQueuedEDU(jsonNIDs[i]) - } if err = d.FederationQueueJSON.DeleteQueueJSON(ctx, txn, jsonNIDs); err != nil { return err @@ -198,4 +197,14 @@ func (d *Database) DeleteExpiredEDUs(ctx context.Context) error { return d.FederationQueueEDUs.DeleteExpiredEDUs(ctx, txn, expiredBefore) }) + + if err != nil { + return err + } + + for i := range jsonNIDs { + d.Cache.EvictFederationQueuedEDU(jsonNIDs[i]) + } + + return nil } From 606cb6750639b1e680cbe1dd1b8026ac1536d642 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Thu, 18 Aug 2022 13:50:58 +0100 Subject: [PATCH 13/42] Enable `workflow_dispatch` in GHA --- .github/workflows/dendrite.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/dendrite.yml b/.github/workflows/dendrite.yml index 3044b0b9d..6cd7a60e2 100644 --- a/.github/workflows/dendrite.yml +++ b/.github/workflows/dendrite.yml @@ -7,6 +7,7 @@ on: pull_request: release: types: [published] + workflow_dispatch: concurrency: group: ${{ github.workflow }}-${{ github.ref }} From 6b48ce0d757279965a6deb77a1ca8d72c15d4bff Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Thu, 18 Aug 2022 17:06:13 +0100 Subject: [PATCH 14/42] State handling tweaks (#2652) This tweaks how rejected events are handled in room state and also to not apply checks we can't complete to outliers. --- roomserver/internal/helpers/auth.go | 4 +- roomserver/internal/input/input_events.go | 6 +- .../internal/perform/perform_backfill.go | 4 +- roomserver/storage/interface.go | 2 +- roomserver/storage/postgres/events_table.go | 64 ++++++++++++------- roomserver/storage/shared/storage.go | 4 +- roomserver/storage/sqlite3/events_table.go | 57 +++++++++++------ .../storage/tables/events_table_test.go | 2 +- roomserver/storage/tables/interface.go | 2 +- 9 files changed, 92 insertions(+), 53 deletions(-) diff --git a/roomserver/internal/helpers/auth.go b/roomserver/internal/helpers/auth.go index 648c50cf6..935a045df 100644 --- a/roomserver/internal/helpers/auth.go +++ b/roomserver/internal/helpers/auth.go @@ -39,7 +39,7 @@ func CheckForSoftFail( var authStateEntries []types.StateEntry var err error if rewritesState { - authStateEntries, err = db.StateEntriesForEventIDs(ctx, stateEventIDs) + authStateEntries, err = db.StateEntriesForEventIDs(ctx, stateEventIDs, true) if err != nil { return true, fmt.Errorf("StateEntriesForEventIDs failed: %w", err) } @@ -97,7 +97,7 @@ func CheckAuthEvents( authEventIDs []string, ) ([]types.EventNID, error) { // Grab the numeric IDs for the supplied auth state events from the database. - authStateEntries, err := db.StateEntriesForEventIDs(ctx, authEventIDs) + authStateEntries, err := db.StateEntriesForEventIDs(ctx, authEventIDs, true) if err != nil { return nil, fmt.Errorf("db.StateEntriesForEventIDs: %w", err) } diff --git a/roomserver/internal/input/input_events.go b/roomserver/internal/input/input_events.go index 53ccd5973..0ece9d145 100644 --- a/roomserver/internal/input/input_events.go +++ b/roomserver/internal/input/input_events.go @@ -301,7 +301,7 @@ func (r *Inputer) processRoomEvent( // bother doing this if the event was already rejected as it just ends up // burning CPU time. historyVisibility := gomatrixserverlib.HistoryVisibilityShared // Default to shared. - if rejectionErr == nil && !isRejected && !softfail { + if input.Kind != api.KindOutlier && rejectionErr == nil && !isRejected && !softfail { var err error historyVisibility, rejectionErr, err = r.processStateBefore(ctx, input, missingPrev) if err != nil { @@ -356,6 +356,8 @@ func (r *Inputer) processRoomEvent( // We stop here if the event is rejected: We've stored it but won't update forward extremities or notify anyone about it. if isRejected || softfail { logger.WithError(rejectionErr).WithFields(logrus.Fields{ + "room_id": event.RoomID(), + "event_id": event.EventID(), "soft_fail": softfail, "missing_prev": missingPrev, }).Warn("Stored rejected event") @@ -661,7 +663,7 @@ func (r *Inputer) calculateAndSetState( // We've been told what the state at the event is so we don't need to calculate it. // Check that those state events are in the database and store the state. var entries []types.StateEntry - if entries, err = r.DB.StateEntriesForEventIDs(ctx, input.StateEventIDs); err != nil { + if entries, err = r.DB.StateEntriesForEventIDs(ctx, input.StateEventIDs, true); err != nil { return fmt.Errorf("updater.StateEntriesForEventIDs: %w", err) } entries = types.DeduplicateStateEntries(entries) diff --git a/roomserver/internal/perform/perform_backfill.go b/roomserver/internal/perform/perform_backfill.go index 298ba04f6..aecff8b88 100644 --- a/roomserver/internal/perform/perform_backfill.go +++ b/roomserver/internal/perform/perform_backfill.go @@ -140,11 +140,11 @@ func (r *Backfiller) backfillViaFederation(ctx context.Context, req *api.Perform continue } var entries []types.StateEntry - if entries, err = r.DB.StateEntriesForEventIDs(ctx, stateIDs); err != nil { + if entries, err = r.DB.StateEntriesForEventIDs(ctx, stateIDs, true); err != nil { // attempt to fetch the missing events r.fetchAndStoreMissingEvents(ctx, info.RoomVersion, requester, stateIDs) // try again - entries, err = r.DB.StateEntriesForEventIDs(ctx, stateIDs) + entries, err = r.DB.StateEntriesForEventIDs(ctx, stateIDs, true) if err != nil { logrus.WithError(err).WithField("event_id", ev.EventID()).Error("backfillViaFederation: failed to get state entries for event") return err diff --git a/roomserver/storage/interface.go b/roomserver/storage/interface.go index 5c068873a..43e8da7bb 100644 --- a/roomserver/storage/interface.go +++ b/roomserver/storage/interface.go @@ -79,7 +79,7 @@ type Database interface { // Look up the state entries for a list of string event IDs // Returns an error if the there is an error talking to the database // Returns a types.MissingEventError if the event IDs aren't in the database. - StateEntriesForEventIDs(ctx context.Context, eventIDs []string) ([]types.StateEntry, error) + StateEntriesForEventIDs(ctx context.Context, eventIDs []string, excludeRejected bool) ([]types.StateEntry, error) // Look up the string event state keys for a list of numeric event state keys // Returns an error if there was a problem talking to the database. EventStateKeys(ctx context.Context, eventStateKeyNIDs []types.EventStateKeyNID) (map[types.EventStateKeyNID]string, error) diff --git a/roomserver/storage/postgres/events_table.go b/roomserver/storage/postgres/events_table.go index c7748d2be..a310c3963 100644 --- a/roomserver/storage/postgres/events_table.go +++ b/roomserver/storage/postgres/events_table.go @@ -88,6 +88,14 @@ const bulkSelectStateEventByIDSQL = "" + " WHERE event_id = ANY($1)" + " ORDER BY event_type_nid, event_state_key_nid ASC" +// Bulk lookup of events by string ID that aren't excluded. +// Sort by the numeric IDs for event type and state key. +// This means we can use binary search to lookup entries by type and state key. +const bulkSelectStateEventByIDExcludingRejectedSQL = "" + + "SELECT event_type_nid, event_state_key_nid, event_nid FROM roomserver_events" + + " WHERE event_id = ANY($1) AND is_rejected = FALSE" + + " ORDER BY event_type_nid, event_state_key_nid ASC" + // Bulk look up of events by event NID, optionally filtering by the event type // or event state key NIDs if provided. (The CARDINALITY check will return true // if the provided arrays are empty, ergo no filtering). @@ -140,23 +148,24 @@ const selectEventRejectedSQL = "" + "SELECT is_rejected FROM roomserver_events WHERE room_nid = $1 AND event_id = $2" type eventStatements struct { - insertEventStmt *sql.Stmt - selectEventStmt *sql.Stmt - bulkSelectStateEventByIDStmt *sql.Stmt - bulkSelectStateEventByNIDStmt *sql.Stmt - bulkSelectStateAtEventByIDStmt *sql.Stmt - updateEventStateStmt *sql.Stmt - selectEventSentToOutputStmt *sql.Stmt - updateEventSentToOutputStmt *sql.Stmt - selectEventIDStmt *sql.Stmt - bulkSelectStateAtEventAndReferenceStmt *sql.Stmt - bulkSelectEventReferenceStmt *sql.Stmt - bulkSelectEventIDStmt *sql.Stmt - bulkSelectEventNIDStmt *sql.Stmt - bulkSelectUnsentEventNIDStmt *sql.Stmt - selectMaxEventDepthStmt *sql.Stmt - selectRoomNIDsForEventNIDsStmt *sql.Stmt - selectEventRejectedStmt *sql.Stmt + insertEventStmt *sql.Stmt + selectEventStmt *sql.Stmt + bulkSelectStateEventByIDStmt *sql.Stmt + bulkSelectStateEventByIDExcludingRejectedStmt *sql.Stmt + bulkSelectStateEventByNIDStmt *sql.Stmt + bulkSelectStateAtEventByIDStmt *sql.Stmt + updateEventStateStmt *sql.Stmt + selectEventSentToOutputStmt *sql.Stmt + updateEventSentToOutputStmt *sql.Stmt + selectEventIDStmt *sql.Stmt + bulkSelectStateAtEventAndReferenceStmt *sql.Stmt + bulkSelectEventReferenceStmt *sql.Stmt + bulkSelectEventIDStmt *sql.Stmt + bulkSelectEventNIDStmt *sql.Stmt + bulkSelectUnsentEventNIDStmt *sql.Stmt + selectMaxEventDepthStmt *sql.Stmt + selectRoomNIDsForEventNIDsStmt *sql.Stmt + selectEventRejectedStmt *sql.Stmt } func CreateEventsTable(db *sql.DB) error { @@ -171,6 +180,7 @@ func PrepareEventsTable(db *sql.DB) (tables.Events, error) { {&s.insertEventStmt, insertEventSQL}, {&s.selectEventStmt, selectEventSQL}, {&s.bulkSelectStateEventByIDStmt, bulkSelectStateEventByIDSQL}, + {&s.bulkSelectStateEventByIDExcludingRejectedStmt, bulkSelectStateEventByIDExcludingRejectedSQL}, {&s.bulkSelectStateEventByNIDStmt, bulkSelectStateEventByNIDSQL}, {&s.bulkSelectStateAtEventByIDStmt, bulkSelectStateAtEventByIDSQL}, {&s.updateEventStateStmt, updateEventStateSQL}, @@ -221,11 +231,18 @@ func (s *eventStatements) SelectEvent( } // bulkSelectStateEventByID lookups a list of state events by event ID. -// If any of the requested events are missing from the database it returns a types.MissingEventError +// If not excluding rejected events, and any of the requested events are missing from +// the database it returns a types.MissingEventError. If excluding rejected events, +// the events will be silently omitted without error. func (s *eventStatements) BulkSelectStateEventByID( - ctx context.Context, txn *sql.Tx, eventIDs []string, + ctx context.Context, txn *sql.Tx, eventIDs []string, excludeRejected bool, ) ([]types.StateEntry, error) { - stmt := sqlutil.TxStmt(txn, s.bulkSelectStateEventByIDStmt) + var stmt *sql.Stmt + if excludeRejected { + stmt = sqlutil.TxStmt(txn, s.bulkSelectStateEventByIDExcludingRejectedStmt) + } else { + stmt = sqlutil.TxStmt(txn, s.bulkSelectStateEventByIDStmt) + } rows, err := stmt.QueryContext(ctx, pq.StringArray(eventIDs)) if err != nil { return nil, err @@ -235,10 +252,10 @@ func (s *eventStatements) BulkSelectStateEventByID( // because of the unique constraint on event IDs. // So we can allocate an array of the correct size now. // We might get fewer results than IDs so we adjust the length of the slice before returning it. - results := make([]types.StateEntry, len(eventIDs)) + results := make([]types.StateEntry, 0, len(eventIDs)) i := 0 for ; rows.Next(); i++ { - result := &results[i] + var result types.StateEntry if err = rows.Scan( &result.EventTypeNID, &result.EventStateKeyNID, @@ -246,11 +263,12 @@ func (s *eventStatements) BulkSelectStateEventByID( ); err != nil { return nil, err } + results = append(results, result) } if err = rows.Err(); err != nil { return nil, err } - if i != len(eventIDs) { + if !excludeRejected && i != len(eventIDs) { // If there are fewer rows returned than IDs then we were asked to lookup event IDs we don't have. // We don't know which ones were missing because we don't return the string IDs in the query. // However it should be possible debug this by replaying queries or entries from the input kafka logs. diff --git a/roomserver/storage/shared/storage.go b/roomserver/storage/shared/storage.go index 4f92adf1f..f35592a76 100644 --- a/roomserver/storage/shared/storage.go +++ b/roomserver/storage/shared/storage.go @@ -113,9 +113,9 @@ func (d *Database) eventStateKeyNIDs( } func (d *Database) StateEntriesForEventIDs( - ctx context.Context, eventIDs []string, + ctx context.Context, eventIDs []string, excludeRejected bool, ) ([]types.StateEntry, error) { - return d.EventsTable.BulkSelectStateEventByID(ctx, nil, eventIDs) + return d.EventsTable.BulkSelectStateEventByID(ctx, nil, eventIDs, excludeRejected) } func (d *Database) StateEntriesForTuples( diff --git a/roomserver/storage/sqlite3/events_table.go b/roomserver/storage/sqlite3/events_table.go index 174e3a9a7..943f256eb 100644 --- a/roomserver/storage/sqlite3/events_table.go +++ b/roomserver/storage/sqlite3/events_table.go @@ -65,6 +65,14 @@ const bulkSelectStateEventByIDSQL = "" + " WHERE event_id IN ($1)" + " ORDER BY event_type_nid, event_state_key_nid ASC" +// Bulk lookup of events by string ID that aren't rejected. +// Sort by the numeric IDs for event type and state key. +// This means we can use binary search to lookup entries by type and state key. +const bulkSelectStateEventByIDExcludingRejectedSQL = "" + + "SELECT event_type_nid, event_state_key_nid, event_nid FROM roomserver_events" + + " WHERE event_id IN ($1) AND is_rejected = 0" + + " ORDER BY event_type_nid, event_state_key_nid ASC" + const bulkSelectStateEventByNIDSQL = "" + "SELECT event_type_nid, event_state_key_nid, event_nid FROM roomserver_events" + " WHERE event_nid IN ($1)" @@ -113,19 +121,20 @@ const selectEventRejectedSQL = "" + "SELECT is_rejected FROM roomserver_events WHERE room_nid = $1 AND event_id = $2" type eventStatements struct { - db *sql.DB - insertEventStmt *sql.Stmt - selectEventStmt *sql.Stmt - bulkSelectStateEventByIDStmt *sql.Stmt - bulkSelectStateAtEventByIDStmt *sql.Stmt - updateEventStateStmt *sql.Stmt - selectEventSentToOutputStmt *sql.Stmt - updateEventSentToOutputStmt *sql.Stmt - selectEventIDStmt *sql.Stmt - bulkSelectStateAtEventAndReferenceStmt *sql.Stmt - bulkSelectEventReferenceStmt *sql.Stmt - bulkSelectEventIDStmt *sql.Stmt - selectEventRejectedStmt *sql.Stmt + db *sql.DB + insertEventStmt *sql.Stmt + selectEventStmt *sql.Stmt + bulkSelectStateEventByIDStmt *sql.Stmt + bulkSelectStateEventByIDExcludingRejectedStmt *sql.Stmt + bulkSelectStateAtEventByIDStmt *sql.Stmt + updateEventStateStmt *sql.Stmt + selectEventSentToOutputStmt *sql.Stmt + updateEventSentToOutputStmt *sql.Stmt + selectEventIDStmt *sql.Stmt + bulkSelectStateAtEventAndReferenceStmt *sql.Stmt + bulkSelectEventReferenceStmt *sql.Stmt + bulkSelectEventIDStmt *sql.Stmt + selectEventRejectedStmt *sql.Stmt //bulkSelectEventNIDStmt *sql.Stmt //bulkSelectUnsentEventNIDStmt *sql.Stmt //selectRoomNIDsForEventNIDsStmt *sql.Stmt @@ -145,6 +154,7 @@ func PrepareEventsTable(db *sql.DB) (tables.Events, error) { {&s.insertEventStmt, insertEventSQL}, {&s.selectEventStmt, selectEventSQL}, {&s.bulkSelectStateEventByIDStmt, bulkSelectStateEventByIDSQL}, + {&s.bulkSelectStateEventByIDExcludingRejectedStmt, bulkSelectStateEventByIDExcludingRejectedSQL}, {&s.bulkSelectStateAtEventByIDStmt, bulkSelectStateAtEventByIDSQL}, {&s.updateEventStateStmt, updateEventStateSQL}, {&s.updateEventSentToOutputStmt, updateEventSentToOutputSQL}, @@ -194,16 +204,24 @@ func (s *eventStatements) SelectEvent( } // bulkSelectStateEventByID lookups a list of state events by event ID. -// If any of the requested events are missing from the database it returns a types.MissingEventError +// If not excluding rejected events, and any of the requested events are missing from +// the database it returns a types.MissingEventError. If excluding rejected events, +// the events will be silently omitted without error. func (s *eventStatements) BulkSelectStateEventByID( - ctx context.Context, txn *sql.Tx, eventIDs []string, + ctx context.Context, txn *sql.Tx, eventIDs []string, excludeRejected bool, ) ([]types.StateEntry, error) { /////////////// + var sql string + if excludeRejected { + sql = bulkSelectStateEventByIDExcludingRejectedSQL + } else { + sql = bulkSelectStateEventByIDSQL + } iEventIDs := make([]interface{}, len(eventIDs)) for k, v := range eventIDs { iEventIDs[k] = v } - selectOrig := strings.Replace(bulkSelectStateEventByIDSQL, "($1)", sqlutil.QueryVariadic(len(iEventIDs)), 1) + selectOrig := strings.Replace(sql, "($1)", sqlutil.QueryVariadic(len(iEventIDs)), 1) selectPrep, err := s.db.Prepare(selectOrig) if err != nil { return nil, err @@ -221,10 +239,10 @@ func (s *eventStatements) BulkSelectStateEventByID( // because of the unique constraint on event IDs. // So we can allocate an array of the correct size now. // We might get fewer results than IDs so we adjust the length of the slice before returning it. - results := make([]types.StateEntry, len(eventIDs)) + results := make([]types.StateEntry, 0, len(eventIDs)) i := 0 for ; rows.Next(); i++ { - result := &results[i] + var result types.StateEntry if err = rows.Scan( &result.EventTypeNID, &result.EventStateKeyNID, @@ -232,8 +250,9 @@ func (s *eventStatements) BulkSelectStateEventByID( ); err != nil { return nil, err } + results = append(results, result) } - if i != len(eventIDs) { + if !excludeRejected && i != len(eventIDs) { // If there are fewer rows returned than IDs then we were asked to lookup event IDs we don't have. // We don't know which ones were missing because we don't return the string IDs in the query. // However it should be possible debug this by replaying queries or entries from the input kafka logs. diff --git a/roomserver/storage/tables/events_table_test.go b/roomserver/storage/tables/events_table_test.go index 74502a31f..107af4784 100644 --- a/roomserver/storage/tables/events_table_test.go +++ b/roomserver/storage/tables/events_table_test.go @@ -102,7 +102,7 @@ func Test_EventsTable(t *testing.T) { }) } - stateEvents, err := tab.BulkSelectStateEventByID(ctx, nil, eventIDs) + stateEvents, err := tab.BulkSelectStateEventByID(ctx, nil, eventIDs, false) assert.NoError(t, err) assert.Equal(t, len(stateEvents), len(eventIDs)) nids := make([]types.EventNID, 0, len(stateEvents)) diff --git a/roomserver/storage/tables/interface.go b/roomserver/storage/tables/interface.go index ed67c43d8..68d30f994 100644 --- a/roomserver/storage/tables/interface.go +++ b/roomserver/storage/tables/interface.go @@ -46,7 +46,7 @@ type Events interface { SelectEvent(ctx context.Context, txn *sql.Tx, eventID string) (types.EventNID, types.StateSnapshotNID, error) // bulkSelectStateEventByID lookups a list of state events by event ID. // If any of the requested events are missing from the database it returns a types.MissingEventError - BulkSelectStateEventByID(ctx context.Context, txn *sql.Tx, eventIDs []string) ([]types.StateEntry, error) + BulkSelectStateEventByID(ctx context.Context, txn *sql.Tx, eventIDs []string, excludeRejected bool) ([]types.StateEntry, error) BulkSelectStateEventByNID(ctx context.Context, txn *sql.Tx, eventNIDs []types.EventNID, stateKeyTuples []types.StateKeyTuple) ([]types.StateEntry, error) // BulkSelectStateAtEventByID lookups the state at a list of events by event ID. // If any of the requested events are missing from the database it returns a types.MissingEventError. From a379d3e8141cfceb0ca49435ffb7a596d57619bb Mon Sep 17 00:00:00 2001 From: Winter Date: Fri, 19 Aug 2022 01:28:33 -0400 Subject: [PATCH 15/42] De-race `TestExpireEDUs` (#2654) In some conditions (fast CPUs), this test would race the clock for EDU expiration when all we want to make sure of is that the expired EDUs are properly deleted. Given this, we set the expiry time to 0 so the specified EDUs are always deleted when DeleteExpiredEDUs is called. Fixes #2650. Signed-off-by: Winter --- federationapi/storage/storage_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/federationapi/storage/storage_test.go b/federationapi/storage/storage_test.go index 7eba2cbee..3b0268e55 100644 --- a/federationapi/storage/storage_test.go +++ b/federationapi/storage/storage_test.go @@ -31,7 +31,7 @@ func mustCreateFederationDatabase(t *testing.T, dbType test.DBType) (storage.Dat func TestExpireEDUs(t *testing.T) { var expireEDUTypes = map[string]time.Duration{ - gomatrixserverlib.MReceipt: time.Millisecond, + gomatrixserverlib.MReceipt: 0, } ctx := context.Background() From 5cacca92d2b888d022f9fa346b8068ce13087b00 Mon Sep 17 00:00:00 2001 From: Till <2353100+S7evinK@users.noreply.github.com> Date: Fri, 19 Aug 2022 11:03:55 +0200 Subject: [PATCH 16/42] Make SyncAPI unit tests more reliable (#2655) This should hopefully make some SyncAPI tests more reliable --- syncapi/syncapi_test.go | 86 ++++++++++++++++++++++++++++++++++++----- 1 file changed, 76 insertions(+), 10 deletions(-) diff --git a/syncapi/syncapi_test.go b/syncapi/syncapi_test.go index 089bdafaf..8b33c5e43 100644 --- a/syncapi/syncapi_test.go +++ b/syncapi/syncapi_test.go @@ -154,8 +154,12 @@ func testSyncAccessTokens(t *testing.T, dbType test.DBType) { wantJoinedRooms: []string{room.ID}, }, } - // TODO: find a better way - time.Sleep(500 * time.Millisecond) + + syncUntil(t, base, alice.AccessToken, false, func(syncBody string) bool { + // wait for the last sent eventID to come down sync + path := fmt.Sprintf(`rooms.join.%s.timeline.events.#(event_id=="%s")`, room.ID, room.Events()[len(room.Events())-1].EventID()) + return gjson.Get(syncBody, path).Exists() + }) for _, tc := range testCases { w := httptest.NewRecorder() @@ -343,6 +347,13 @@ func testHistoryVisibility(t *testing.T, dbType test.DBType) { // create the users alice := test.NewUser(t) + aliceDev := userapi.Device{ + ID: "ALICEID", + UserID: alice.ID, + AccessToken: "ALICE_BEARER_TOKEN", + DisplayName: "ALICE", + } + bob := test.NewUser(t) bobDev := userapi.Device{ @@ -409,7 +420,7 @@ func testHistoryVisibility(t *testing.T, dbType test.DBType) { rsAPI := roomserver.NewInternalAPI(base) rsAPI.SetFederationAPI(nil, nil) - AddPublicRoutes(base, &syncUserAPI{accounts: []userapi.Device{bobDev}}, rsAPI, &syncKeyAPI{}) + AddPublicRoutes(base, &syncUserAPI{accounts: []userapi.Device{aliceDev, bobDev}}, rsAPI, &syncKeyAPI{}) for _, tc := range testCases { testname := fmt.Sprintf("%s - %s", tc.historyVisibility, userType) @@ -418,12 +429,18 @@ func testHistoryVisibility(t *testing.T, dbType test.DBType) { room := test.NewRoom(t, alice, test.RoomHistoryVisibility(tc.historyVisibility)) // send the events/messages to NATS to create the rooms - beforeJoinEv := room.CreateAndInsert(t, alice, "m.room.message", map[string]interface{}{"body": fmt.Sprintf("Before invite in a %s room", tc.historyVisibility)}) + beforeJoinBody := fmt.Sprintf("Before invite in a %s room", tc.historyVisibility) + beforeJoinEv := room.CreateAndInsert(t, alice, "m.room.message", map[string]interface{}{"body": beforeJoinBody}) eventsToSend := append(room.Events(), beforeJoinEv) if err := api.SendEvents(ctx, rsAPI, api.KindNew, eventsToSend, "test", "test", nil, false); err != nil { t.Fatalf("failed to send events: %v", err) } - time.Sleep(100 * time.Millisecond) // TODO: find a better way + syncUntil(t, base, aliceDev.AccessToken, false, + func(syncBody string) bool { + path := fmt.Sprintf(`rooms.join.%s.timeline.events.#(content.body=="%s")`, room.ID, beforeJoinBody) + return gjson.Get(syncBody, path).Exists() + }, + ) // There is only one event, we expect only to be able to see this, if the room is world_readable w := httptest.NewRecorder() @@ -449,14 +466,20 @@ func testHistoryVisibility(t *testing.T, dbType test.DBType) { inviteEv := room.CreateAndInsert(t, alice, "m.room.member", map[string]interface{}{"membership": "invite"}, test.WithStateKey(bob.ID)) afterInviteEv := room.CreateAndInsert(t, alice, "m.room.message", map[string]interface{}{"body": fmt.Sprintf("After invite in a %s room", tc.historyVisibility)}) joinEv := room.CreateAndInsert(t, bob, "m.room.member", map[string]interface{}{"membership": "join"}, test.WithStateKey(bob.ID)) - msgEv := room.CreateAndInsert(t, alice, "m.room.message", map[string]interface{}{"body": fmt.Sprintf("After join in a %s room", tc.historyVisibility)}) + afterJoinBody := fmt.Sprintf("After join in a %s room", tc.historyVisibility) + msgEv := room.CreateAndInsert(t, alice, "m.room.message", map[string]interface{}{"body": afterJoinBody}) eventsToSend = append([]*gomatrixserverlib.HeaderedEvent{}, inviteEv, afterInviteEv, joinEv, msgEv) if err := api.SendEvents(ctx, rsAPI, api.KindNew, eventsToSend, "test", "test", nil, false); err != nil { t.Fatalf("failed to send events: %v", err) } - time.Sleep(100 * time.Millisecond) // TODO: find a better way + syncUntil(t, base, aliceDev.AccessToken, false, + func(syncBody string) bool { + path := fmt.Sprintf(`rooms.join.%s.timeline.events.#(content.body=="%s")`, room.ID, afterJoinBody) + return gjson.Get(syncBody, path).Exists() + }, + ) // Verify the messages after/before invite are visible or not w = httptest.NewRecorder() @@ -511,8 +534,8 @@ func testSendToDevice(t *testing.T, dbType test.DBType) { AccountType: userapi.AccountTypeUser, } - base, close := testrig.CreateBaseDendrite(t, dbType) - defer close() + base, baseClose := testrig.CreateBaseDendrite(t, dbType) + defer baseClose() jsctx, _ := base.NATS.Prepare(base.ProcessContext, &base.Cfg.Global.JetStream) defer jetstream.DeleteAllStreams(jsctx, &base.Cfg.Global.JetStream) @@ -607,7 +630,14 @@ func testSendToDevice(t *testing.T, dbType test.DBType) { t.Fatalf("unable to send to device message: %v", err) } } - time.Sleep((time.Millisecond * 15) * time.Duration(tc.sendMessagesCount)) // wait a bit, so the messages can be processed + + syncUntil(t, base, alice.AccessToken, + len(tc.want) == 0, + func(body string) bool { + return gjson.Get(body, fmt.Sprintf(`to_device.events.#(content.dummy=="message %d")`, msgCounter)).Exists() + }, + ) + // Execute a /sync request, recording the response w := httptest.NewRecorder() base.PublicClientAPIMux.ServeHTTP(w, test.NewRequest(t, "GET", "/_matrix/client/v3/sync", test.WithQueryParams(map[string]string{ @@ -630,6 +660,42 @@ func testSendToDevice(t *testing.T, dbType test.DBType) { } } +func syncUntil(t *testing.T, + base *base.BaseDendrite, accessToken string, + skip bool, + checkFunc func(syncBody string) bool, +) { + if checkFunc == nil { + t.Fatalf("No checkFunc defined") + } + if skip { + return + } + // loop on /sync until we receive the last send message or timeout after 5 seconds, since we don't know if the message made it + // to the syncAPI when hitting /sync + done := make(chan bool) + defer close(done) + go func() { + for { + w := httptest.NewRecorder() + base.PublicClientAPIMux.ServeHTTP(w, test.NewRequest(t, "GET", "/_matrix/client/v3/sync", test.WithQueryParams(map[string]string{ + "access_token": accessToken, + "timeout": "1000", + }))) + if checkFunc(w.Body.String()) { + done <- true + return + } + } + }() + + select { + case <-done: + case <-time.After(time.Second * 5): + t.Fatalf("Timed out waiting for messages") + } +} + func toNATSMsgs(t *testing.T, base *base.BaseDendrite, input ...*gomatrixserverlib.HeaderedEvent) []*nats.Msg { result := make([]*nats.Msg, len(input)) for i, ev := range input { From 365da70a23cec9595a2854ed47f970f03cfde0a9 Mon Sep 17 00:00:00 2001 From: Till <2353100+S7evinK@users.noreply.github.com> Date: Fri, 19 Aug 2022 11:04:26 +0200 Subject: [PATCH 17/42] Set historyVisibility for backfilled events over federation (#2656) This should hopefully deflake Backfill works correctly with history visibility set to joined as we were using the default shared visibility, even if the events are set to joined (or something else) --- roomserver/api/perform.go | 6 +++-- .../internal/perform/perform_backfill.go | 24 ++++++++++++------- syncapi/routing/messages.go | 14 ++++++++--- 3 files changed, 31 insertions(+), 13 deletions(-) diff --git a/roomserver/api/perform.go b/roomserver/api/perform.go index d9ea9dd1c..20931f807 100644 --- a/roomserver/api/perform.go +++ b/roomserver/api/perform.go @@ -5,9 +5,10 @@ import ( "fmt" "net/http" - "github.com/matrix-org/dendrite/clientapi/jsonerror" "github.com/matrix-org/gomatrixserverlib" "github.com/matrix-org/util" + + "github.com/matrix-org/dendrite/clientapi/jsonerror" ) type PerformErrorCode int @@ -161,7 +162,8 @@ func (r *PerformBackfillRequest) PrevEventIDs() []string { // PerformBackfillResponse is a response to PerformBackfill. type PerformBackfillResponse struct { // Missing events, arbritrary order. - Events []*gomatrixserverlib.HeaderedEvent `json:"events"` + Events []*gomatrixserverlib.HeaderedEvent `json:"events"` + HistoryVisibility gomatrixserverlib.HistoryVisibility `json:"history_visibility"` } type PerformPublishRequest struct { diff --git a/roomserver/internal/perform/perform_backfill.go b/roomserver/internal/perform/perform_backfill.go index aecff8b88..de76b6412 100644 --- a/roomserver/internal/perform/perform_backfill.go +++ b/roomserver/internal/perform/perform_backfill.go @@ -164,6 +164,7 @@ func (r *Backfiller) backfillViaFederation(ctx context.Context, req *api.Perform // TODO: update backwards extremities, as that should be moved from syncapi to roomserver at some point. res.Events = events + res.HistoryVisibility = requester.historyVisiblity return nil } @@ -248,6 +249,7 @@ type backfillRequester struct { servers []gomatrixserverlib.ServerName eventIDToBeforeStateIDs map[string][]string eventIDMap map[string]*gomatrixserverlib.Event + historyVisiblity gomatrixserverlib.HistoryVisibility } func newBackfillRequester( @@ -266,6 +268,7 @@ func newBackfillRequester( eventIDMap: make(map[string]*gomatrixserverlib.Event), bwExtrems: bwExtrems, preferServer: preferServer, + historyVisiblity: gomatrixserverlib.HistoryVisibilityShared, } } @@ -447,7 +450,8 @@ FindSuccessor: } // possibly return all joined servers depending on history visiblity - memberEventsFromVis, err := joinEventsFromHistoryVisibility(ctx, b.db, roomID, stateEntries, b.thisServer) + memberEventsFromVis, visibility, err := joinEventsFromHistoryVisibility(ctx, b.db, roomID, stateEntries, b.thisServer) + b.historyVisiblity = visibility if err != nil { logrus.WithError(err).Error("ServersAtEvent: failed calculate servers from history visibility rules") return nil @@ -528,7 +532,7 @@ func (b *backfillRequester) ProvideEvents(roomVer gomatrixserverlib.RoomVersion, // pull all events and then filter by that table. func joinEventsFromHistoryVisibility( ctx context.Context, db storage.Database, roomID string, stateEntries []types.StateEntry, - thisServer gomatrixserverlib.ServerName) ([]types.Event, error) { + thisServer gomatrixserverlib.ServerName) ([]types.Event, gomatrixserverlib.HistoryVisibility, error) { var eventNIDs []types.EventNID for _, entry := range stateEntries { @@ -542,7 +546,9 @@ func joinEventsFromHistoryVisibility( // Get all of the events in this state stateEvents, err := db.Events(ctx, eventNIDs) if err != nil { - return nil, err + // even though the default should be shared, restricting the visibility to joined + // feels more secure here. + return nil, gomatrixserverlib.HistoryVisibilityJoined, err } events := make([]*gomatrixserverlib.Event, len(stateEvents)) for i := range stateEvents { @@ -551,20 +557,22 @@ func joinEventsFromHistoryVisibility( // Can we see events in the room? canSeeEvents := auth.IsServerAllowed(thisServer, true, events) + visibility := gomatrixserverlib.HistoryVisibility(auth.HistoryVisibilityForRoom(events)) if !canSeeEvents { - logrus.Infof("ServersAtEvent history not visible to us: %s", auth.HistoryVisibilityForRoom(events)) - return nil, nil + logrus.Infof("ServersAtEvent history not visible to us: %s", visibility) + return nil, visibility, nil } // get joined members info, err := db.RoomInfo(ctx, roomID) if err != nil { - return nil, err + return nil, visibility, nil } joinEventNIDs, err := db.GetMembershipEventNIDsForRoom(ctx, info.RoomNID, true, false) if err != nil { - return nil, err + return nil, visibility, err } - return db.Events(ctx, joinEventNIDs) + evs, err := db.Events(ctx, joinEventNIDs) + return evs, visibility, err } func persistEvents(ctx context.Context, db storage.Database, events []*gomatrixserverlib.HeaderedEvent) (types.RoomNID, map[string]types.Event) { diff --git a/syncapi/routing/messages.go b/syncapi/routing/messages.go index 9db3d8e17..03614302c 100644 --- a/syncapi/routing/messages.go +++ b/syncapi/routing/messages.go @@ -350,8 +350,10 @@ func (r *messagesReq) retrieveEvents() ( startTime := time.Now() filteredEvents, err := internal.ApplyHistoryVisibilityFilter(r.ctx, r.db, r.rsAPI, events, nil, r.device.UserID, "messages") logrus.WithFields(logrus.Fields{ - "duration": time.Since(startTime), - "room_id": r.roomID, + "duration": time.Since(startTime), + "room_id": r.roomID, + "events_before": len(events), + "events_after": len(filteredEvents), }).Debug("applied history visibility (messages)") return gomatrixserverlib.HeaderedToClientEvents(filteredEvents, gomatrixserverlib.FormatAll), start, end, err } @@ -513,6 +515,9 @@ func (r *messagesReq) backfill(roomID string, backwardsExtremities map[string][] // Store the events in the database, while marking them as unfit to show // up in responses to sync requests. + if res.HistoryVisibility == "" { + res.HistoryVisibility = gomatrixserverlib.HistoryVisibilityShared + } for i := range res.Events { _, err = r.db.WriteEvent( context.Background(), @@ -521,7 +526,7 @@ func (r *messagesReq) backfill(roomID string, backwardsExtremities map[string][] []string{}, []string{}, nil, true, - gomatrixserverlib.HistoryVisibilityShared, + res.HistoryVisibility, ) if err != nil { return nil, err @@ -534,6 +539,9 @@ func (r *messagesReq) backfill(roomID string, backwardsExtremities map[string][] // last `limit` events events = events[len(events)-limit:] } + for _, ev := range events { + ev.Visibility = res.HistoryVisibility + } return events, nil } From 5513f182ccd978866354eaa45effd293b0745207 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Fri, 19 Aug 2022 10:23:09 +0100 Subject: [PATCH 18/42] Enforce device list backoffs (#2653) This ensures that if the device list updater is already backing off a node, we don't try to call processServer again anyway for server just because the server name arrived in the channel. Otherwise we can keep trying to hit a remote server that is offline or not behaving every second and that spams the logs too. --- keyserver/internal/device_list_update.go | 13 ++++++++++--- sytest-whitelist | 2 -- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/keyserver/internal/device_list_update.go b/keyserver/internal/device_list_update.go index 80efbec51..304b67b23 100644 --- a/keyserver/internal/device_list_update.go +++ b/keyserver/internal/device_list_update.go @@ -335,8 +335,9 @@ func (u *DeviceListUpdater) worker(ch chan gomatrixserverlib.ServerName) { retriesMu := &sync.Mutex{} // restarter goroutine which will inject failed servers into ch when it is time go func() { + var serversToRetry []gomatrixserverlib.ServerName for { - var serversToRetry []gomatrixserverlib.ServerName + serversToRetry = serversToRetry[:0] // reuse memory time.Sleep(time.Second) retriesMu.Lock() now := time.Now() @@ -355,11 +356,17 @@ func (u *DeviceListUpdater) worker(ch chan gomatrixserverlib.ServerName) { } }() for serverName := range ch { + retriesMu.Lock() + _, exists := retries[serverName] + retriesMu.Unlock() + if exists { + // Don't retry a server that we're already waiting for. + continue + } waitTime, shouldRetry := u.processServer(serverName) if shouldRetry { retriesMu.Lock() - _, exists := retries[serverName] - if !exists { + if _, exists = retries[serverName]; !exists { retries[serverName] = time.Now().Add(waitTime) } retriesMu.Unlock() diff --git a/sytest-whitelist b/sytest-whitelist index dcffeaffb..5c8896b99 100644 --- a/sytest-whitelist +++ b/sytest-whitelist @@ -144,7 +144,6 @@ Server correctly handles incoming m.device_list_update If remote user leaves room, changes device and rejoins we see update in sync If remote user leaves room, changes device and rejoins we see update in /keys/changes If remote user leaves room we no longer receive device updates -If a device list update goes missing, the server resyncs on the next one Server correctly resyncs when client query keys and there is no remote cache Server correctly resyncs when server leaves and rejoins a room Device list doesn't change if remote server is down @@ -633,7 +632,6 @@ Test that rejected pushers are removed. Trying to add push rule with no scope fails with 400 Trying to add push rule with invalid scope fails with 400 Forward extremities remain so even after the next events are populated as outliers -If a device list update goes missing, the server resyncs on the next one uploading self-signing key notifies over federation uploading signed devices gets propagated over federation Device list doesn't change if remote server is down From 56b55a28f591cd2e920c5f16bee6f7a97b0898c0 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Fri, 19 Aug 2022 12:46:14 +0100 Subject: [PATCH 19/42] Version 0.9.4 (#2658) ### Fixes * A bug in the roomserver around handling rejected outliers has been fixed * Backfilled events will now use the correct history visibility where possible * The device list updater backoff has been fixed, which should reduce the number of outbound HTTP requests and `Failed to query device keys for some users` log entries for dead servers * The `/sync` endpoint will no longer incorrectly return room entries for retired invites which could cause some rooms to show up in the client "Historical" section * The `/createRoom` endpoint will now correctly populate `is_direct` in invite membership events, which may help clients to classify direct messages correctly * The `create-account` tool will now log an error if the shared secret is not set in the Dendrite config * A couple of minor bugs have been fixed in the membership lazy-loading * Queued EDUs in the federation API are now cached properly --- CHANGES.md | 13 +++++++++++++ internal/version.go | 2 +- 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/CHANGES.md b/CHANGES.md index 36a1a6311..aaf5836ba 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -1,5 +1,18 @@ # Changelog +## Dendrite 0.9.4 (2022-08-19) + +### Fixes + +* A bug in the roomserver around handling rejected outliers has been fixed +* Backfilled events will now use the correct history visibility where possible +* The device list updater backoff has been fixed, which should reduce the number of outbound HTTP requests and `Failed to query device keys for some users` log entries for dead servers +* The `/sync` endpoint will no longer incorrectly return room entries for retired invites which could cause some rooms to show up in the client "Historical" section +* The `/createRoom` endpoint will now correctly populate `is_direct` in invite membership events, which may help clients to classify direct messages correctly +* The `create-account` tool will now log an error if the shared secret is not set in the Dendrite config +* A couple of minor bugs have been fixed in the membership lazy-loading +* Queued EDUs in the federation API are now cached properly + ## Dendrite 0.9.3 (2022-08-15) ### Important diff --git a/internal/version.go b/internal/version.go index 561e6c06f..384f091a0 100644 --- a/internal/version.go +++ b/internal/version.go @@ -17,7 +17,7 @@ var build string const ( VersionMajor = 0 VersionMinor = 9 - VersionPatch = 3 + VersionPatch = 4 VersionTag = "" // example: "rc1" ) From 9dc57122d991d54ea6750448ba88c8763a569830 Mon Sep 17 00:00:00 2001 From: Till <2353100+S7evinK@users.noreply.github.com> Date: Fri, 19 Aug 2022 15:32:24 +0200 Subject: [PATCH 20/42] Fetch more data for newly joined rooms in an incremental sync (#2657) If we've joined a new room in an incremental sync, try fetching more data. This deflakes the complement server notices test (at least in my tests). --- syncapi/internal/keychange.go | 12 ++++++--- syncapi/streams/stream_pdu.go | 39 +++++++++++++++++++++++++++--- syncapi/streams/stream_presence.go | 8 +++++- syncapi/syncapi_test.go | 1 + 4 files changed, 52 insertions(+), 8 deletions(-) diff --git a/syncapi/internal/keychange.go b/syncapi/internal/keychange.go index 23824e366..c5180e338 100644 --- a/syncapi/internal/keychange.go +++ b/syncapi/internal/keychange.go @@ -18,14 +18,16 @@ import ( "context" "strings" + "github.com/matrix-org/gomatrixserverlib" + "github.com/matrix-org/util" + "github.com/sirupsen/logrus" + "github.com/tidwall/gjson" + keyapi "github.com/matrix-org/dendrite/keyserver/api" keytypes "github.com/matrix-org/dendrite/keyserver/types" roomserverAPI "github.com/matrix-org/dendrite/roomserver/api" "github.com/matrix-org/dendrite/syncapi/storage" "github.com/matrix-org/dendrite/syncapi/types" - "github.com/matrix-org/gomatrixserverlib" - "github.com/matrix-org/util" - "github.com/sirupsen/logrus" ) // DeviceOTKCounts adds one-time key counts to the /sync response @@ -277,6 +279,10 @@ func membershipEventPresent(events []gomatrixserverlib.ClientEvent, userID strin // it's enough to know that we have our member event here, don't need to check membership content // as it's implied by being in the respective section of the sync response. if ev.Type == gomatrixserverlib.MRoomMember && ev.StateKey != nil && *ev.StateKey == userID { + // ignore e.g. join -> join changes + if gjson.GetBytes(ev.Unsigned, "prev_content.membership").Str == gjson.GetBytes(ev.Content, "membership").Str { + continue + } return true } } diff --git a/syncapi/streams/stream_pdu.go b/syncapi/streams/stream_pdu.go index 0e9dda577..2818aad87 100644 --- a/syncapi/streams/stream_pdu.go +++ b/syncapi/streams/stream_pdu.go @@ -178,24 +178,24 @@ func (p *PDUStreamProvider) IncrementalSync( var err error var stateDeltas []types.StateDelta - var joinedRooms []string + var syncJoinedRooms []string stateFilter := req.Filter.Room.State eventFilter := req.Filter.Room.Timeline if req.WantFullState { - if stateDeltas, joinedRooms, err = p.DB.GetStateDeltasForFullStateSync(ctx, req.Device, r, req.Device.UserID, &stateFilter); err != nil { + if stateDeltas, syncJoinedRooms, err = p.DB.GetStateDeltasForFullStateSync(ctx, req.Device, r, req.Device.UserID, &stateFilter); err != nil { req.Log.WithError(err).Error("p.DB.GetStateDeltasForFullStateSync failed") return } } else { - if stateDeltas, joinedRooms, err = p.DB.GetStateDeltas(ctx, req.Device, r, req.Device.UserID, &stateFilter); err != nil { + if stateDeltas, syncJoinedRooms, err = p.DB.GetStateDeltas(ctx, req.Device, r, req.Device.UserID, &stateFilter); err != nil { req.Log.WithError(err).Error("p.DB.GetStateDeltas failed") return } } - for _, roomID := range joinedRooms { + for _, roomID := range syncJoinedRooms { req.Rooms[roomID] = gomatrixserverlib.Join } @@ -222,6 +222,37 @@ func (p *PDUStreamProvider) IncrementalSync( } } + // If we joined a new room in this sync, make sure we add enough information about it. + // This does an "initial sync" for the newly joined rooms + newlyJoinedRooms := joinedRooms(req.Response, req.Device.UserID) + if len(newlyJoinedRooms) > 0 { + // remove already added rooms, as we're doing an "initial sync" + for _, x := range newlyJoinedRooms { + delete(req.Response.Rooms.Join, x) + } + r = types.Range{ + From: to, + To: 0, + Backwards: true, + } + // We only care about the newly joined rooms, so update the stateFilter to reflect that + stateFilter.Rooms = &newlyJoinedRooms + if stateDeltas, _, err = p.DB.GetStateDeltas(ctx, req.Device, r, req.Device.UserID, &stateFilter); err != nil { + req.Log.WithError(err).Error("p.DB.GetStateDeltas failed") + return newPos + } + for _, delta := range stateDeltas { + // Ignore deltas for rooms we didn't newly join + if _, ok := req.Response.Rooms.Join[delta.RoomID]; ok { + continue + } + if _, err = p.addRoomDeltaToResponse(ctx, req.Device, r, delta, &eventFilter, &stateFilter, req.Response); err != nil { + req.Log.WithError(err).Error("d.addRoomDeltaToResponse failed") + return newPos + } + } + } + return newPos } diff --git a/syncapi/streams/stream_presence.go b/syncapi/streams/stream_presence.go index 877bcf141..15db4d30e 100644 --- a/syncapi/streams/stream_presence.go +++ b/syncapi/streams/stream_presence.go @@ -19,9 +19,11 @@ import ( "encoding/json" "sync" + "github.com/matrix-org/gomatrixserverlib" + "github.com/tidwall/gjson" + "github.com/matrix-org/dendrite/syncapi/notifier" "github.com/matrix-org/dendrite/syncapi/types" - "github.com/matrix-org/gomatrixserverlib" ) type PresenceStreamProvider struct { @@ -175,6 +177,10 @@ func membershipEventPresent(events []gomatrixserverlib.ClientEvent, userID strin // it's enough to know that we have our member event here, don't need to check membership content // as it's implied by being in the respective section of the sync response. if ev.Type == gomatrixserverlib.MRoomMember && ev.StateKey != nil && *ev.StateKey == userID { + // ignore e.g. join -> join changes + if gjson.GetBytes(ev.Unsigned, "prev_content.membership").Str == gjson.GetBytes(ev.Content, "membership").Str { + continue + } return true } } diff --git a/syncapi/syncapi_test.go b/syncapi/syncapi_test.go index 8b33c5e43..76d51c86b 100644 --- a/syncapi/syncapi_test.go +++ b/syncapi/syncapi_test.go @@ -195,6 +195,7 @@ func TestSyncAPICreateRoomSyncEarly(t *testing.T) { } func testSyncAPICreateRoomSyncEarly(t *testing.T, dbType test.DBType) { + t.SkipNow() // Temporary? user := test.NewUser(t) room := test.NewRoom(t, user) alice := userapi.Device{ From 2668050e535be7527d5dea2e97309027cbc12560 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Mon, 22 Aug 2022 10:30:35 +0100 Subject: [PATCH 21/42] Tweak soft-failure handling in roomserver commit 1929b688e31987c46e0c8a546f0f9cb0a46bf9a3 Author: Neil Alexander Date: Mon Aug 22 10:09:44 2022 +0100 Still process state-before for soft-failed events commit e83c0b701d40d78b92072c4643f6bc6f71b72800 Author: Neil Alexander Date: Mon Aug 22 10:06:50 2022 +0100 Improve logging commit 29e26124bc27cb83d449de2a4214b253c594aa93 Author: Neil Alexander Date: Mon Aug 22 09:58:13 2022 +0100 Don't store soft-failed events as rejected --- roomserver/internal/input/input_events.go | 24 +++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/roomserver/internal/input/input_events.go b/roomserver/internal/input/input_events.go index 0ece9d145..29af649ad 100644 --- a/roomserver/internal/input/input_events.go +++ b/roomserver/internal/input/input_events.go @@ -301,7 +301,7 @@ func (r *Inputer) processRoomEvent( // bother doing this if the event was already rejected as it just ends up // burning CPU time. historyVisibility := gomatrixserverlib.HistoryVisibilityShared // Default to shared. - if input.Kind != api.KindOutlier && rejectionErr == nil && !isRejected && !softfail { + if input.Kind != api.KindOutlier && rejectionErr == nil && !isRejected { var err error historyVisibility, rejectionErr, err = r.processStateBefore(ctx, input, missingPrev) if err != nil { @@ -313,7 +313,7 @@ func (r *Inputer) processRoomEvent( } // Store the event. - _, _, stateAtEvent, redactionEvent, redactedEventID, err := r.DB.StoreEvent(ctx, event, authEventNIDs, isRejected || softfail) + _, _, stateAtEvent, redactionEvent, redactedEventID, err := r.DB.StoreEvent(ctx, event, authEventNIDs, isRejected) if err != nil { return fmt.Errorf("updater.StoreEvent: %w", err) } @@ -353,14 +353,18 @@ func (r *Inputer) processRoomEvent( } } - // We stop here if the event is rejected: We've stored it but won't update forward extremities or notify anyone about it. - if isRejected || softfail { - logger.WithError(rejectionErr).WithFields(logrus.Fields{ - "room_id": event.RoomID(), - "event_id": event.EventID(), - "soft_fail": softfail, - "missing_prev": missingPrev, - }).Warn("Stored rejected event") + // We stop here if the event is rejected: We've stored it but won't update + // forward extremities or notify downstream components about it. + switch { + case isRejected: + logger.WithError(rejectionErr).Warn("Stored rejected event") + if rejectionErr != nil { + return types.RejectedError(rejectionErr.Error()) + } + return nil + + case softfail: + logger.WithError(rejectionErr).Warn("Stored soft-failed event") if rejectionErr != nil { return types.RejectedError(rejectionErr.Error()) } From 33129c02f79c951189bc4ab7018e855b1f563bf0 Mon Sep 17 00:00:00 2001 From: Till <2353100+S7evinK@users.noreply.github.com> Date: Tue, 23 Aug 2022 11:10:41 +0200 Subject: [PATCH 22/42] Add timeout parameter & trim URL (#2666) A timeout of 10 seconds could cause issues with servers having a high `bcrypt_cost` configured in the config. This adds a parameter to manually configure the timeout, defaults to 30 seconds. --- cmd/create-account/main.go | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/cmd/create-account/main.go b/cmd/create-account/main.go index bd053f2f7..a9357f6db 100644 --- a/cmd/create-account/main.go +++ b/cmd/create-account/main.go @@ -66,10 +66,11 @@ var ( resetPassword = flag.Bool("reset-password", false, "Deprecated") serverURL = flag.String("url", "https://localhost:8448", "The URL to connect to.") validUsernameRegex = regexp.MustCompile(`^[0-9a-z_\-=./]+$`) + timeout = flag.Duration("timeout", time.Second*30, "Timeout for the http client when connecting to the server") ) var cl = http.Client{ - Timeout: time.Second * 10, + Timeout: time.Second * 30, Transport: http.DefaultTransport, } @@ -108,6 +109,8 @@ func main() { logrus.Fatalln(err) } + cl.Timeout = *timeout + accessToken, err := sharedSecretRegister(cfg.ClientAPI.RegistrationSharedSecret, *serverURL, *username, pass, *isAdmin) if err != nil { logrus.Fatalln("Failed to create the account:", err.Error()) @@ -124,8 +127,8 @@ type sharedSecretRegistrationRequest struct { Admin bool `json:"admin"` } -func sharedSecretRegister(sharedSecret, serverURL, localpart, password string, admin bool) (accesToken string, err error) { - registerURL := fmt.Sprintf("%s/_synapse/admin/v1/register", serverURL) +func sharedSecretRegister(sharedSecret, serverURL, localpart, password string, admin bool) (accessToken string, err error) { + registerURL := fmt.Sprintf("%s/_synapse/admin/v1/register", strings.Trim(serverURL, "/")) nonceReq, err := http.NewRequest(http.MethodGet, registerURL, nil) if err != nil { return "", fmt.Errorf("unable to create http request: %w", err) From 95a509757a6ad160e8d480cf21acb999c3309099 Mon Sep 17 00:00:00 2001 From: Till <2353100+S7evinK@users.noreply.github.com> Date: Tue, 23 Aug 2022 13:10:29 +0200 Subject: [PATCH 23/42] Complement QoL changes (#2663) This PR does the following: - adds a `keysize` parameter to `generate-keys`, so we can use lower sized keys when running in CI - updates the Complement docker files to use BuildKit (requires Docker >18.09) - uses `exec` when executing `dendrite-monotlith-server`, making it PID 1 inside docker, which results in Dendrite actually receiving the `SIGTERM` signal send by Docker. (Making it faster when running tests with Complement, as we don't take 10 seconds to timeout) --- .github/workflows/dendrite.yml | 2 ++ build/scripts/Complement.Dockerfile | 20 +++++++++--------- build/scripts/ComplementLocal.Dockerfile | 23 ++++++++++----------- build/scripts/ComplementPostgres.Dockerfile | 20 +++++++++--------- cmd/generate-keys/main.go | 5 +++-- test/http.go | 2 +- test/keys.go | 12 +++++------ 7 files changed, 43 insertions(+), 41 deletions(-) diff --git a/.github/workflows/dendrite.yml b/.github/workflows/dendrite.yml index 6cd7a60e2..be3c7c173 100644 --- a/.github/workflows/dendrite.yml +++ b/.github/workflows/dendrite.yml @@ -376,6 +376,8 @@ jobs: # Build initial Dendrite image - run: docker build -t complement-dendrite -f build/scripts/Complement${{ matrix.postgres }}.Dockerfile . working-directory: dendrite + env: + DOCKER_BUILDKIT: 1 # Run Complement - run: | diff --git a/build/scripts/Complement.Dockerfile b/build/scripts/Complement.Dockerfile index 56877051b..14b28498b 100644 --- a/build/scripts/Complement.Dockerfile +++ b/build/scripts/Complement.Dockerfile @@ -1,3 +1,5 @@ +#syntax=docker/dockerfile:1.2 + FROM golang:1.18-stretch as build RUN apt-get update && apt-get install -y sqlite3 WORKDIR /build @@ -8,14 +10,12 @@ RUN mkdir /dendrite # Utilise Docker caching when downloading dependencies, this stops us needlessly # downloading dependencies every time. -COPY go.mod . -COPY go.sum . -RUN go mod download - -COPY . . -RUN go build -o /dendrite ./cmd/dendrite-monolith-server -RUN go build -o /dendrite ./cmd/generate-keys -RUN go build -o /dendrite ./cmd/generate-config +RUN --mount=target=. \ + --mount=type=cache,target=/go/pkg/mod \ + --mount=type=cache,target=/root/.cache/go-build \ + go build -o /dendrite ./cmd/generate-config && \ + go build -o /dendrite ./cmd/generate-keys && \ + go build -o /dendrite ./cmd/dendrite-monolith-server WORKDIR /dendrite RUN ./generate-keys --private-key matrix_key.pem @@ -26,7 +26,7 @@ EXPOSE 8008 8448 # At runtime, generate TLS cert based on the CA now mounted at /ca # At runtime, replace the SERVER_NAME with what we are told -CMD ./generate-keys --server $SERVER_NAME --tls-cert server.crt --tls-key server.key --tls-authority-cert /complement/ca/ca.crt --tls-authority-key /complement/ca/ca.key && \ +CMD ./generate-keys -keysize 1024 --server $SERVER_NAME --tls-cert server.crt --tls-key server.key --tls-authority-cert /complement/ca/ca.crt --tls-authority-key /complement/ca/ca.key && \ ./generate-config -server $SERVER_NAME --ci > dendrite.yaml && \ cp /complement/ca/ca.crt /usr/local/share/ca-certificates/ && update-ca-certificates && \ - ./dendrite-monolith-server --really-enable-open-registration --tls-cert server.crt --tls-key server.key --config dendrite.yaml -api=${API:-0} + exec ./dendrite-monolith-server --really-enable-open-registration --tls-cert server.crt --tls-key server.key --config dendrite.yaml -api=${API:-0} diff --git a/build/scripts/ComplementLocal.Dockerfile b/build/scripts/ComplementLocal.Dockerfile index 704359a28..3a019fc20 100644 --- a/build/scripts/ComplementLocal.Dockerfile +++ b/build/scripts/ComplementLocal.Dockerfile @@ -1,3 +1,5 @@ +#syntax=docker/dockerfile:1.2 + # A local development Complement dockerfile, to be used with host mounts # /cache -> Contains the entire dendrite code at Dockerfile build time. Builds binaries but only keeps the generate-* ones. Pre-compilation saves time. # /dendrite -> Host-mounted sources @@ -9,11 +11,10 @@ FROM golang:1.18-stretch RUN apt-get update && apt-get install -y sqlite3 -WORKDIR /runtime - ENV SERVER_NAME=localhost EXPOSE 8008 8448 +WORKDIR /runtime # This script compiles Dendrite for us. RUN echo '\ #!/bin/bash -eux \n\ @@ -29,25 +30,23 @@ RUN echo '\ RUN echo '\ #!/bin/bash -eu \n\ ./generate-keys --private-key matrix_key.pem \n\ - ./generate-keys --server $SERVER_NAME --tls-cert server.crt --tls-key server.key --tls-authority-cert /complement/ca/ca.crt --tls-authority-key /complement/ca/ca.key \n\ + ./generate-keys -keysize 1024 --server $SERVER_NAME --tls-cert server.crt --tls-key server.key --tls-authority-cert /complement/ca/ca.crt --tls-authority-key /complement/ca/ca.key \n\ ./generate-config -server $SERVER_NAME --ci > dendrite.yaml \n\ cp /complement/ca/ca.crt /usr/local/share/ca-certificates/ && update-ca-certificates \n\ - ./dendrite-monolith-server --really-enable-open-registration --tls-cert server.crt --tls-key server.key --config dendrite.yaml \n\ + exec ./dendrite-monolith-server --really-enable-open-registration --tls-cert server.crt --tls-key server.key --config dendrite.yaml \n\ ' > run.sh && chmod +x run.sh WORKDIR /cache -# Pre-download deps; we don't need to do this if the GOPATH is mounted. -COPY go.mod . -COPY go.sum . -RUN go mod download - # Build the monolith in /cache - we won't actually use this but will rely on build artifacts to speed # up the real compilation. Build the generate-* binaries in the true /runtime locations. # If the generate-* source is changed, this dockerfile needs re-running. -COPY . . -RUN go build ./cmd/dendrite-monolith-server && go build -o /runtime ./cmd/generate-keys && go build -o /runtime ./cmd/generate-config +RUN --mount=target=. \ + --mount=type=cache,target=/go/pkg/mod \ + --mount=type=cache,target=/root/.cache/go-build \ + go build -o /runtime ./cmd/generate-config && \ + go build -o /runtime ./cmd/generate-keys WORKDIR /runtime -CMD /runtime/compile.sh && /runtime/run.sh +CMD /runtime/compile.sh && exec /runtime/run.sh diff --git a/build/scripts/ComplementPostgres.Dockerfile b/build/scripts/ComplementPostgres.Dockerfile index a8b4fbb1d..699540120 100644 --- a/build/scripts/ComplementPostgres.Dockerfile +++ b/build/scripts/ComplementPostgres.Dockerfile @@ -1,3 +1,5 @@ +#syntax=docker/dockerfile:1.2 + FROM golang:1.18-stretch as build RUN apt-get update && apt-get install -y postgresql WORKDIR /build @@ -26,14 +28,12 @@ RUN mkdir /dendrite # Utilise Docker caching when downloading dependencies, this stops us needlessly # downloading dependencies every time. -COPY go.mod . -COPY go.sum . -RUN go mod download - -COPY . . -RUN go build -o /dendrite ./cmd/dendrite-monolith-server -RUN go build -o /dendrite ./cmd/generate-keys -RUN go build -o /dendrite ./cmd/generate-config +RUN --mount=target=. \ + --mount=type=cache,target=/go/pkg/mod \ + --mount=type=cache,target=/root/.cache/go-build \ + go build -o /dendrite ./cmd/generate-config && \ + go build -o /dendrite ./cmd/generate-keys && \ + go build -o /dendrite ./cmd/dendrite-monolith-server WORKDIR /dendrite RUN ./generate-keys --private-key matrix_key.pem @@ -45,10 +45,10 @@ EXPOSE 8008 8448 # At runtime, generate TLS cert based on the CA now mounted at /ca # At runtime, replace the SERVER_NAME with what we are told -CMD /build/run_postgres.sh && ./generate-keys --server $SERVER_NAME --tls-cert server.crt --tls-key server.key --tls-authority-cert /complement/ca/ca.crt --tls-authority-key /complement/ca/ca.key && \ +CMD /build/run_postgres.sh && ./generate-keys --keysize 1024 --server $SERVER_NAME --tls-cert server.crt --tls-key server.key --tls-authority-cert /complement/ca/ca.crt --tls-authority-key /complement/ca/ca.key && \ ./generate-config -server $SERVER_NAME --ci > dendrite.yaml && \ # Replace the connection string with a single postgres DB, using user/db = 'postgres' and no password, bump max_conns sed -i "s%connection_string:.*$%connection_string: postgresql://postgres@localhost/postgres?sslmode=disable%g" dendrite.yaml && \ sed -i 's/max_open_conns:.*$/max_open_conns: 100/g' dendrite.yaml && \ cp /complement/ca/ca.crt /usr/local/share/ca-certificates/ && update-ca-certificates && \ - ./dendrite-monolith-server --really-enable-open-registration --tls-cert server.crt --tls-key server.key --config dendrite.yaml -api=${API:-0} \ No newline at end of file + exec ./dendrite-monolith-server --really-enable-open-registration --tls-cert server.crt --tls-key server.key --config dendrite.yaml -api=${API:-0} \ No newline at end of file diff --git a/cmd/generate-keys/main.go b/cmd/generate-keys/main.go index 8acd28be0..d4c8cf78a 100644 --- a/cmd/generate-keys/main.go +++ b/cmd/generate-keys/main.go @@ -38,6 +38,7 @@ var ( authorityCertFile = flag.String("tls-authority-cert", "", "Optional: Create TLS certificate/keys based on this CA authority. Useful for integration testing.") authorityKeyFile = flag.String("tls-authority-key", "", "Optional: Create TLS certificate/keys based on this CA authority. Useful for integration testing.") serverName = flag.String("server", "", "Optional: Create TLS certificate/keys with this domain name set. Useful for integration testing.") + keySize = flag.Int("keysize", 4096, "Optional: Create TLS RSA private key with the given key size") ) func main() { @@ -58,12 +59,12 @@ func main() { log.Fatal("Zero or both of --tls-key and --tls-cert must be supplied") } if *authorityCertFile == "" && *authorityKeyFile == "" { - if err := test.NewTLSKey(*tlsKeyFile, *tlsCertFile); err != nil { + if err := test.NewTLSKey(*tlsKeyFile, *tlsCertFile, *keySize); err != nil { panic(err) } } else { // generate the TLS cert/key based on the authority given. - if err := test.NewTLSKeyWithAuthority(*serverName, *tlsKeyFile, *tlsCertFile, *authorityKeyFile, *authorityCertFile); err != nil { + if err := test.NewTLSKeyWithAuthority(*serverName, *tlsKeyFile, *tlsCertFile, *authorityKeyFile, *authorityCertFile, *keySize); err != nil { panic(err) } } diff --git a/test/http.go b/test/http.go index 37b3648f8..8cd83d0a6 100644 --- a/test/http.go +++ b/test/http.go @@ -68,7 +68,7 @@ func ListenAndServe(t *testing.T, router http.Handler, withTLS bool) (apiURL str if withTLS { certFile := filepath.Join(t.TempDir(), "dendrite.cert") keyFile := filepath.Join(t.TempDir(), "dendrite.key") - err = NewTLSKey(keyFile, certFile) + err = NewTLSKey(keyFile, certFile, 1024) if err != nil { t.Errorf("failed to make TLS key: %s", err) return diff --git a/test/keys.go b/test/keys.go index 327c6ed7b..fb156ef27 100644 --- a/test/keys.go +++ b/test/keys.go @@ -69,8 +69,8 @@ func NewMatrixKey(matrixKeyPath string) (err error) { const certificateDuration = time.Hour * 24 * 365 * 10 -func generateTLSTemplate(dnsNames []string) (*rsa.PrivateKey, *x509.Certificate, error) { - priv, err := rsa.GenerateKey(rand.Reader, 4096) +func generateTLSTemplate(dnsNames []string, bitSize int) (*rsa.PrivateKey, *x509.Certificate, error) { + priv, err := rsa.GenerateKey(rand.Reader, bitSize) if err != nil { return nil, nil, err } @@ -118,8 +118,8 @@ func writePrivateKey(tlsKeyPath string, priv *rsa.PrivateKey) error { } // NewTLSKey generates a new RSA TLS key and certificate and writes it to a file. -func NewTLSKey(tlsKeyPath, tlsCertPath string) error { - priv, template, err := generateTLSTemplate(nil) +func NewTLSKey(tlsKeyPath, tlsCertPath string, keySize int) error { + priv, template, err := generateTLSTemplate(nil, keySize) if err != nil { return err } @@ -136,8 +136,8 @@ func NewTLSKey(tlsKeyPath, tlsCertPath string) error { return writePrivateKey(tlsKeyPath, priv) } -func NewTLSKeyWithAuthority(serverName, tlsKeyPath, tlsCertPath, authorityKeyPath, authorityCertPath string) error { - priv, template, err := generateTLSTemplate([]string{serverName}) +func NewTLSKeyWithAuthority(serverName, tlsKeyPath, tlsCertPath, authorityKeyPath, authorityCertPath string, keySize int) error { + priv, template, err := generateTLSTemplate([]string{serverName}, keySize) if err != nil { return err } From 14fea600bbeeb745c1ff1fc1519e4db968c17f86 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Tue, 23 Aug 2022 13:57:11 +0100 Subject: [PATCH 24/42] Detect `types.MissingStateError` in `CheckServerAllowedToSeeEvent` (#2667) This will hopefully stop some 500 errors on `/event` where there is no state-before known. --- roomserver/internal/helpers/helpers.go | 11 +++++++++-- roomserver/storage/postgres/events_table.go | 2 +- roomserver/storage/sqlite3/events_table.go | 2 +- 3 files changed, 11 insertions(+), 4 deletions(-) diff --git a/roomserver/internal/helpers/helpers.go b/roomserver/internal/helpers/helpers.go index 6091f8ec2..cbd1561f7 100644 --- a/roomserver/internal/helpers/helpers.go +++ b/roomserver/internal/helpers/helpers.go @@ -254,8 +254,15 @@ func CheckServerAllowedToSeeEvent( return false, err } default: - // Something else went wrong - return false, err + switch err.(type) { + case types.MissingStateError: + // If there's no state then we assume it's open visibility, as Synapse does: + // https://github.com/matrix-org/synapse/blob/aec87a0f9369a3015b2a53469f88d1de274e8b71/synapse/visibility.py#L654-L655 + return true, nil + default: + // Something else went wrong + return false, err + } } return auth.IsServerAllowed(serverName, isServerInRoom, stateAtEvent), nil } diff --git a/roomserver/storage/postgres/events_table.go b/roomserver/storage/postgres/events_table.go index a310c3963..e758837a0 100644 --- a/roomserver/storage/postgres/events_table.go +++ b/roomserver/storage/postgres/events_table.go @@ -346,7 +346,7 @@ func (s *eventStatements) BulkSelectStateAtEventByID( // Genuine create events are the only case where it's OK to have no previous state. isCreate := result.EventTypeNID == types.MRoomCreateNID && result.EventStateKeyNID == 1 if result.BeforeStateSnapshotNID == 0 && !isCreate { - return nil, types.MissingEventError( + return nil, types.MissingStateError( fmt.Sprintf("storage: missing state for event NID %d", result.EventNID), ) } diff --git a/roomserver/storage/sqlite3/events_table.go b/roomserver/storage/sqlite3/events_table.go index 943f256eb..b5cc84bc5 100644 --- a/roomserver/storage/sqlite3/events_table.go +++ b/roomserver/storage/sqlite3/events_table.go @@ -362,7 +362,7 @@ func (s *eventStatements) BulkSelectStateAtEventByID( // Genuine create events are the only case where it's OK to have no previous state. isCreate := result.EventTypeNID == types.MRoomCreateNID && result.EventStateKeyNID == 1 if result.BeforeStateSnapshotNID == 0 && !isCreate { - return nil, types.MissingEventError( + return nil, types.MissingStateError( fmt.Sprintf("storage: missing state for event NID %d", result.EventNID), ) } From 78e5d05efc95c959b380d6ee4230f75885a597ad Mon Sep 17 00:00:00 2001 From: Till <2353100+S7evinK@users.noreply.github.com> Date: Tue, 23 Aug 2022 16:54:42 +0200 Subject: [PATCH 25/42] Only set backOffStarted to false if until is not zero (#2669) --- federationapi/statistics/statistics.go | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/federationapi/statistics/statistics.go b/federationapi/statistics/statistics.go index 8bac99cbc..b8e16a259 100644 --- a/federationapi/statistics/statistics.go +++ b/federationapi/statistics/statistics.go @@ -5,10 +5,11 @@ import ( "sync" "time" - "github.com/matrix-org/dendrite/federationapi/storage" "github.com/matrix-org/gomatrixserverlib" "github.com/sirupsen/logrus" "go.uber.org/atomic" + + "github.com/matrix-org/dendrite/federationapi/storage" ) // Statistics contains information about all of the remote federated @@ -126,13 +127,13 @@ func (s *ServerStatistics) Failure() (time.Time, bool) { go func() { until, ok := s.backoffUntil.Load().(time.Time) - if ok { + if ok && !until.IsZero() { select { case <-time.After(time.Until(until)): case <-s.interrupt: } + s.backoffStarted.Store(false) } - s.backoffStarted.Store(false) }() } From 522bd2999f605258e95565c6d648d2f7ea001ea4 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Wed, 24 Aug 2022 14:03:06 +0100 Subject: [PATCH 26/42] Allow un-rejecting events on reprocessing --- roomserver/api/wrapper.go | 8 ++++++++ roomserver/storage/postgres/events_table.go | 2 +- roomserver/storage/sqlite3/events_table.go | 2 +- syncapi/internal/keychange.go | 2 +- 4 files changed, 11 insertions(+), 3 deletions(-) diff --git a/roomserver/api/wrapper.go b/roomserver/api/wrapper.go index bc2f28176..8b031982c 100644 --- a/roomserver/api/wrapper.go +++ b/roomserver/api/wrapper.go @@ -19,6 +19,7 @@ import ( "github.com/matrix-org/gomatrixserverlib" "github.com/matrix-org/util" + "github.com/sirupsen/logrus" ) // SendEvents to the roomserver The events are written with KindNew. @@ -69,6 +70,13 @@ func SendEventWithState( stateEventIDs[i] = stateEvents[i].EventID() } + logrus.WithContext(ctx).WithFields(logrus.Fields{ + "room_id": event.RoomID(), + "event_id": event.EventID(), + "outliers": len(ires), + "state_ids": len(stateEventIDs), + }).Infof("Submitting %q event to roomserver with state snapshot", event.Type()) + ires = append(ires, InputRoomEvent{ Kind: kind, Event: event, diff --git a/roomserver/storage/postgres/events_table.go b/roomserver/storage/postgres/events_table.go index e758837a0..1e7ca7669 100644 --- a/roomserver/storage/postgres/events_table.go +++ b/roomserver/storage/postgres/events_table.go @@ -74,7 +74,7 @@ const insertEventSQL = "" + "INSERT INTO roomserver_events AS e (room_nid, event_type_nid, event_state_key_nid, event_id, reference_sha256, auth_event_nids, depth, is_rejected)" + " VALUES ($1, $2, $3, $4, $5, $6, $7, $8)" + " ON CONFLICT ON CONSTRAINT roomserver_event_id_unique DO UPDATE" + - " SET is_rejected = $8 WHERE e.event_id = $4 AND e.is_rejected = FALSE" + + " SET is_rejected = $8 WHERE e.event_id = $4 AND e.is_rejected = TRUE" + " RETURNING event_nid, state_snapshot_nid" const selectEventSQL = "" + diff --git a/roomserver/storage/sqlite3/events_table.go b/roomserver/storage/sqlite3/events_table.go index b5cc84bc5..950d03b03 100644 --- a/roomserver/storage/sqlite3/events_table.go +++ b/roomserver/storage/sqlite3/events_table.go @@ -50,7 +50,7 @@ const insertEventSQL = ` INSERT INTO roomserver_events (room_nid, event_type_nid, event_state_key_nid, event_id, reference_sha256, auth_event_nids, depth, is_rejected) VALUES ($1, $2, $3, $4, $5, $6, $7, $8) ON CONFLICT DO UPDATE - SET is_rejected = $8 WHERE is_rejected = 0 + SET is_rejected = $8 WHERE is_rejected = 1 RETURNING event_nid, state_snapshot_nid; ` diff --git a/syncapi/internal/keychange.go b/syncapi/internal/keychange.go index c5180e338..3d6b2a7f3 100644 --- a/syncapi/internal/keychange.go +++ b/syncapi/internal/keychange.go @@ -127,7 +127,7 @@ func DeviceListCatchup( "from": offset, "to": toOffset, "response_offset": queryRes.Offset, - }).Debugf("QueryKeyChanges request result: %+v", res.DeviceLists) + }).Tracef("QueryKeyChanges request result: %+v", res.DeviceLists) return types.StreamPosition(queryRes.Offset), hasNew, nil } From 16156b0b0988e7b1746b2834e6357c3c90bc8465 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Thu, 25 Aug 2022 09:51:36 +0100 Subject: [PATCH 27/42] Fix 500s on `/state`, `/state_ids` when state not known (#2672) This was due to bad error bubbling. --- federationapi/routing/join.go | 6 ++++++ federationapi/routing/state.go | 6 ++++++ roomserver/api/query.go | 1 + roomserver/internal/query/query.go | 17 ++++++++++------- 4 files changed, 23 insertions(+), 7 deletions(-) diff --git a/federationapi/routing/join.go b/federationapi/routing/join.go index b48eaf78e..1a1219873 100644 --- a/federationapi/routing/join.go +++ b/federationapi/routing/join.go @@ -329,6 +329,12 @@ func SendJoin( JSON: jsonerror.NotFound("Room does not exist"), } } + if !stateAndAuthChainResponse.StateKnown { + return util.JSONResponse{ + Code: http.StatusForbidden, + JSON: jsonerror.Forbidden("State not known"), + } + } // Check if the user is already in the room. If they're already in then // there isn't much point in sending another join event into the room. diff --git a/federationapi/routing/state.go b/federationapi/routing/state.go index 6fdce20ce..5377eb88f 100644 --- a/federationapi/routing/state.go +++ b/federationapi/routing/state.go @@ -135,6 +135,12 @@ func getState( return nil, nil, &resErr } + if !response.StateKnown { + return nil, nil, &util.JSONResponse{ + Code: http.StatusNotFound, + JSON: jsonerror.NotFound("State not known"), + } + } if response.IsRejected { return nil, nil, &util.JSONResponse{ Code: http.StatusNotFound, diff --git a/roomserver/api/query.go b/roomserver/api/query.go index c8e6f9dc6..32d63bb51 100644 --- a/roomserver/api/query.go +++ b/roomserver/api/query.go @@ -227,6 +227,7 @@ type QueryStateAndAuthChainResponse struct { // Do all the previous events exist on this roomserver? // If some of previous events do not exist this will be false and StateEvents will be empty. PrevEventsExist bool `json:"prev_events_exist"` + StateKnown bool `json:"state_known"` // The state and auth chain events that were requested. // The lists will be in an arbitrary order. StateEvents []*gomatrixserverlib.HeaderedEvent `json:"state_events"` diff --git a/roomserver/internal/query/query.go b/roomserver/internal/query/query.go index f5d8c2d49..6dce2bc3e 100644 --- a/roomserver/internal/query/query.go +++ b/roomserver/internal/query/query.go @@ -503,10 +503,11 @@ func (r *Queryer) QueryStateAndAuthChain( } var stateEvents []*gomatrixserverlib.Event - stateEvents, rejected, err := r.loadStateAtEventIDs(ctx, info, request.PrevEventIDs) + stateEvents, rejected, stateMissing, err := r.loadStateAtEventIDs(ctx, info, request.PrevEventIDs) if err != nil { return err } + response.StateKnown = !stateMissing response.IsRejected = rejected response.PrevEventsExist = true @@ -542,15 +543,18 @@ func (r *Queryer) QueryStateAndAuthChain( return err } -func (r *Queryer) loadStateAtEventIDs(ctx context.Context, roomInfo *types.RoomInfo, eventIDs []string) ([]*gomatrixserverlib.Event, bool, error) { +// first bool: is rejected, second bool: state missing +func (r *Queryer) loadStateAtEventIDs(ctx context.Context, roomInfo *types.RoomInfo, eventIDs []string) ([]*gomatrixserverlib.Event, bool, bool, error) { roomState := state.NewStateResolution(r.DB, roomInfo) prevStates, err := r.DB.StateAtEventIDs(ctx, eventIDs) if err != nil { switch err.(type) { case types.MissingEventError: - return nil, false, nil + return nil, false, true, nil + case types.MissingStateError: + return nil, false, true, nil default: - return nil, false, err + return nil, false, false, err } } // Currently only used on /state and /state_ids @@ -567,12 +571,11 @@ func (r *Queryer) loadStateAtEventIDs(ctx context.Context, roomInfo *types.RoomI ctx, prevStates, ) if err != nil { - return nil, rejected, err + return nil, rejected, false, err } events, err := helpers.LoadStateEvents(ctx, r.DB, stateEntries) - - return events, rejected, err + return events, rejected, false, err } type eventsFromIDs func(context.Context, []string) ([]types.Event, error) From cd7fa34595c0b4107677e32439b0d02599db8e51 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Thu, 25 Aug 2022 10:57:27 +0100 Subject: [PATCH 28/42] Tweak logging and Sentry reporting for roomserver input --- roomserver/internal/input/input.go | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/roomserver/internal/input/input.go b/roomserver/internal/input/input.go index 8d24f3c59..429cc4bd2 100644 --- a/roomserver/internal/input/input.go +++ b/roomserver/internal/input/input.go @@ -36,6 +36,7 @@ import ( "github.com/matrix-org/dendrite/roomserver/internal/query" "github.com/matrix-org/dendrite/roomserver/producers" "github.com/matrix-org/dendrite/roomserver/storage" + "github.com/matrix-org/dendrite/roomserver/types" "github.com/matrix-org/dendrite/setup/base" "github.com/matrix-org/dendrite/setup/config" "github.com/matrix-org/dendrite/setup/jetstream" @@ -247,14 +248,24 @@ func (w *worker) _next() { // it was a synchronous request. var errString string if err = w.r.processRoomEvent(w.r.ProcessContext.Context(), &inputRoomEvent); err != nil { - if !errors.Is(err, context.DeadlineExceeded) && !errors.Is(err, context.Canceled) { - sentry.CaptureException(err) + switch err.(type) { + case types.RejectedError: + // Don't send events that were rejected to Sentry + logrus.WithError(err).WithFields(logrus.Fields{ + "room_id": w.roomID, + "event_id": inputRoomEvent.Event.EventID(), + "type": inputRoomEvent.Event.Type(), + }).Warn("Roomserver rejected event") + default: + if !errors.Is(err, context.DeadlineExceeded) && !errors.Is(err, context.Canceled) { + sentry.CaptureException(err) + } + logrus.WithError(err).WithFields(logrus.Fields{ + "room_id": w.roomID, + "event_id": inputRoomEvent.Event.EventID(), + "type": inputRoomEvent.Event.Type(), + }).Warn("Roomserver failed to process event") } - logrus.WithError(err).WithFields(logrus.Fields{ - "room_id": w.roomID, - "event_id": inputRoomEvent.Event.EventID(), - "type": inputRoomEvent.Event.Type(), - }).Warn("Roomserver failed to process async event") _ = msg.Term() errString = err.Error() } else { From 8ff3f1a7c9578a3a4f95755c4698da3219777097 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Thu, 25 Aug 2022 11:01:07 +0100 Subject: [PATCH 29/42] Remove a couple unnecessary Sentry captures from backfill --- roomserver/internal/perform/perform_backfill.go | 3 --- 1 file changed, 3 deletions(-) diff --git a/roomserver/internal/perform/perform_backfill.go b/roomserver/internal/perform/perform_backfill.go index de76b6412..51c66415a 100644 --- a/roomserver/internal/perform/perform_backfill.go +++ b/roomserver/internal/perform/perform_backfill.go @@ -18,7 +18,6 @@ import ( "context" "fmt" - "github.com/getsentry/sentry-go" "github.com/matrix-org/gomatrixserverlib" "github.com/matrix-org/util" "github.com/sirupsen/logrus" @@ -320,7 +319,6 @@ FederationHit: b.eventIDToBeforeStateIDs[targetEvent.EventID()] = res return res, nil } - sentry.CaptureException(lastErr) // temporary to see if we might need to raise the server limit return nil, lastErr } @@ -398,7 +396,6 @@ func (b *backfillRequester) StateBeforeEvent(ctx context.Context, roomVer gomatr } return result, nil } - sentry.CaptureException(lastErr) // temporary to see if we might need to raise the server limit return nil, lastErr } From 07dd9bd9954d0740664651bf9755a81c3ba2d011 Mon Sep 17 00:00:00 2001 From: Till <2353100+S7evinK@users.noreply.github.com> Date: Thu, 25 Aug 2022 14:42:47 +0200 Subject: [PATCH 30/42] SyncAPI tweaks/fixes (#2671) - Reverts 9dc57122d991d54ea6750448ba88c8763a569830 as it was causing issues https://github.com/matrix-org/dendrite/issues/2660 - Updates the GMSL `DefaultStateFilter` to use a limit of 20 events - Uses the timeline events to determine the new position instead of the state events --- go.mod | 2 +- go.sum | 4 +- syncapi/storage/interface.go | 3 +- syncapi/storage/shared/syncserver.go | 34 ++++++++--------- syncapi/streams/stream_pdu.go | 57 +++++++++++----------------- syncapi/sync/request.go | 28 ++++++++------ syncapi/sync/requestpool.go | 2 +- syncapi/syncapi_test.go | 2 +- syncapi/types/types.go | 1 + 9 files changed, 64 insertions(+), 69 deletions(-) diff --git a/go.mod b/go.mod index fe4604a03..0833f2908 100644 --- a/go.mod +++ b/go.mod @@ -21,7 +21,7 @@ require ( github.com/matrix-org/dugong v0.0.0-20210921133753-66e6b1c67e2e github.com/matrix-org/go-sqlite3-js v0.0.0-20220419092513-28aa791a1c91 github.com/matrix-org/gomatrix v0.0.0-20210324163249-be2af5ef2e16 - github.com/matrix-org/gomatrixserverlib v0.0.0-20220815094957-74b7ff4ae09c + github.com/matrix-org/gomatrixserverlib v0.0.0-20220824082345-662dca17bf94 github.com/matrix-org/pinecone v0.0.0-20220803093810-b7a830c08fb9 github.com/matrix-org/util v0.0.0-20200807132607-55161520e1d4 github.com/mattn/go-sqlite3 v1.14.13 diff --git a/go.sum b/go.sum index 7ac9fc6e2..fcbdb7e41 100644 --- a/go.sum +++ b/go.sum @@ -343,8 +343,8 @@ github.com/matrix-org/go-sqlite3-js v0.0.0-20220419092513-28aa791a1c91/go.mod h1 github.com/matrix-org/gomatrix v0.0.0-20190528120928-7df988a63f26/go.mod h1:3fxX6gUjWyI/2Bt7J1OLhpCzOfO/bB3AiX0cJtEKud0= github.com/matrix-org/gomatrix v0.0.0-20210324163249-be2af5ef2e16 h1:ZtO5uywdd5dLDCud4r0r55eP4j9FuUNpl60Gmntcop4= github.com/matrix-org/gomatrix v0.0.0-20210324163249-be2af5ef2e16/go.mod h1:/gBX06Kw0exX1HrwmoBibFA98yBk/jxKpGVeyQbff+s= -github.com/matrix-org/gomatrixserverlib v0.0.0-20220815094957-74b7ff4ae09c h1:GhKmb8s9iXA9qsFD1SbiRo6Ee7cnbfcgJQ/iy43wczM= -github.com/matrix-org/gomatrixserverlib v0.0.0-20220815094957-74b7ff4ae09c/go.mod h1:jX38yp3SSLJNftBg3PXU1ayd0PCLIiDHQ4xAc9DIixk= +github.com/matrix-org/gomatrixserverlib v0.0.0-20220824082345-662dca17bf94 h1:zoTv/qxg7C/O995JBPvp+Z8KMR69HhB+M+P22A8Hmm0= +github.com/matrix-org/gomatrixserverlib v0.0.0-20220824082345-662dca17bf94/go.mod h1:jX38yp3SSLJNftBg3PXU1ayd0PCLIiDHQ4xAc9DIixk= github.com/matrix-org/pinecone v0.0.0-20220803093810-b7a830c08fb9 h1:ed8yvWhTLk7+sNeK/eOZRTvESFTOHDRevoRoyeqPtvY= github.com/matrix-org/pinecone v0.0.0-20220803093810-b7a830c08fb9/go.mod h1:P4MqPf+u83OPulPJ+XTbSDbbWrdFYNY4LZ/B1PIduFE= github.com/matrix-org/util v0.0.0-20190711121626-527ce5ddefc7/go.mod h1:vVQlW/emklohkZnOPwD3LrZUBqdfsbiyO3p1lNV8F6U= diff --git a/syncapi/storage/interface.go b/syncapi/storage/interface.go index 43a75da95..0c8ba4e3d 100644 --- a/syncapi/storage/interface.go +++ b/syncapi/storage/interface.go @@ -19,10 +19,11 @@ import ( "github.com/matrix-org/dendrite/internal/eventutil" + "github.com/matrix-org/gomatrixserverlib" + "github.com/matrix-org/dendrite/roomserver/api" "github.com/matrix-org/dendrite/syncapi/types" userapi "github.com/matrix-org/dendrite/userapi/api" - "github.com/matrix-org/gomatrixserverlib" ) type Database interface { diff --git a/syncapi/storage/shared/syncserver.go b/syncapi/storage/shared/syncserver.go index a46e55256..b06d2c6a9 100644 --- a/syncapi/storage/shared/syncserver.go +++ b/syncapi/storage/shared/syncserver.go @@ -20,15 +20,18 @@ import ( "encoding/json" "fmt" + "github.com/tidwall/gjson" + userapi "github.com/matrix-org/dendrite/userapi/api" + "github.com/matrix-org/gomatrixserverlib" + "github.com/sirupsen/logrus" + "github.com/matrix-org/dendrite/internal/eventutil" "github.com/matrix-org/dendrite/internal/sqlutil" "github.com/matrix-org/dendrite/roomserver/api" "github.com/matrix-org/dendrite/syncapi/storage/tables" "github.com/matrix-org/dendrite/syncapi/types" - "github.com/matrix-org/gomatrixserverlib" - "github.com/sirupsen/logrus" ) // Database is a temporary struct until we have made syncserver.go the same for both pq/sqlite @@ -683,7 +686,7 @@ func (d *Database) GetStateDeltas( ctx context.Context, device *userapi.Device, r types.Range, userID string, stateFilter *gomatrixserverlib.StateFilter, -) ([]types.StateDelta, []string, error) { +) (deltas []types.StateDelta, joinedRoomsIDs []string, err error) { // Implement membership change algorithm: https://github.com/matrix-org/synapse/blob/v0.19.3/synapse/handlers/sync.py#L821 // - Get membership list changes for this user in this sync response // - For each room which has membership list changes: @@ -718,8 +721,6 @@ func (d *Database) GetStateDeltas( } } - var deltas []types.StateDelta - // get all the state events ever (i.e. for all available rooms) between these two positions stateNeeded, eventMap, err := d.OutputEvents.SelectStateInRange(ctx, txn, r, stateFilter, allRoomIDs) if err != nil { @@ -767,15 +768,11 @@ func (d *Database) GetStateDeltas( } // handle newly joined rooms and non-joined rooms + newlyJoinedRooms := make(map[string]bool, len(state)) for roomID, stateStreamEvents := range state { for _, ev := range stateStreamEvents { - // TODO: Currently this will incorrectly add rooms which were ALREADY joined but they sent another no-op join event. - // We should be checking if the user was already joined at fromPos and not proceed if so. As a result of this, - // dupe join events will result in the entire room state coming down to the client again. This is added in - // the 'state' part of the response though, so is transparent modulo bandwidth concerns as it is not added to - // the timeline. - if membership := getMembershipFromEvent(ev.Event, userID); membership != "" { - if membership == gomatrixserverlib.Join { + if membership, prevMembership := getMembershipFromEvent(ev.Event, userID); membership != "" { + if membership == gomatrixserverlib.Join && prevMembership != membership { // send full room state down instead of a delta var s []types.StreamEvent s, err = d.currentStateStreamEventsForRoom(ctx, txn, roomID, stateFilter) @@ -786,6 +783,7 @@ func (d *Database) GetStateDeltas( return nil, nil, err } state[roomID] = s + newlyJoinedRooms[roomID] = true continue // we'll add this room in when we do joined rooms } @@ -806,6 +804,7 @@ func (d *Database) GetStateDeltas( Membership: gomatrixserverlib.Join, StateEvents: d.StreamEventsToEvents(device, state[joinedRoomID]), RoomID: joinedRoomID, + NewlyJoined: newlyJoinedRooms[joinedRoomID], }) } @@ -892,7 +891,7 @@ func (d *Database) GetStateDeltasForFullStateSync( for roomID, stateStreamEvents := range state { for _, ev := range stateStreamEvents { - if membership := getMembershipFromEvent(ev.Event, userID); membership != "" { + if membership, _ := getMembershipFromEvent(ev.Event, userID); membership != "" { if membership != gomatrixserverlib.Join { // We've already added full state for all joined rooms above. deltas[roomID] = types.StateDelta{ Membership: membership, @@ -1003,15 +1002,16 @@ func (d *Database) CleanSendToDeviceUpdates( // getMembershipFromEvent returns the value of content.membership iff the event is a state event // with type 'm.room.member' and state_key of userID. Otherwise, an empty string is returned. -func getMembershipFromEvent(ev *gomatrixserverlib.Event, userID string) string { +func getMembershipFromEvent(ev *gomatrixserverlib.Event, userID string) (string, string) { if ev.Type() != "m.room.member" || !ev.StateKeyEquals(userID) { - return "" + return "", "" } membership, err := ev.Membership() if err != nil { - return "" + return "", "" } - return membership + prevMembership := gjson.GetBytes(ev.Unsigned(), "prev_content.membership").Str + return membership, prevMembership } // StoreReceipt stores user receipts diff --git a/syncapi/streams/stream_pdu.go b/syncapi/streams/stream_pdu.go index 2818aad87..fa4c722ce 100644 --- a/syncapi/streams/stream_pdu.go +++ b/syncapi/streams/stream_pdu.go @@ -209,11 +209,27 @@ func (p *PDUStreamProvider) IncrementalSync( newPos = from for _, delta := range stateDeltas { + newRange := r + // If this room was joined in this sync, try to fetch + // as much timeline events as allowed by the filter. + if delta.NewlyJoined { + // Reverse the range, so we get the most recent first. + // This will be limited by the eventFilter. + newRange = types.Range{ + From: r.To, + To: 0, + Backwards: true, + } + } var pos types.StreamPosition - if pos, err = p.addRoomDeltaToResponse(ctx, req.Device, r, delta, &eventFilter, &stateFilter, req.Response); err != nil { + if pos, err = p.addRoomDeltaToResponse(ctx, req.Device, newRange, delta, &eventFilter, &stateFilter, req.Response); err != nil { req.Log.WithError(err).Error("d.addRoomDeltaToResponse failed") return to } + // Reset the position, as it is only for the special case of newly joined rooms + if delta.NewlyJoined { + pos = newRange.From + } switch { case r.Backwards && pos < newPos: fallthrough @@ -222,37 +238,6 @@ func (p *PDUStreamProvider) IncrementalSync( } } - // If we joined a new room in this sync, make sure we add enough information about it. - // This does an "initial sync" for the newly joined rooms - newlyJoinedRooms := joinedRooms(req.Response, req.Device.UserID) - if len(newlyJoinedRooms) > 0 { - // remove already added rooms, as we're doing an "initial sync" - for _, x := range newlyJoinedRooms { - delete(req.Response.Rooms.Join, x) - } - r = types.Range{ - From: to, - To: 0, - Backwards: true, - } - // We only care about the newly joined rooms, so update the stateFilter to reflect that - stateFilter.Rooms = &newlyJoinedRooms - if stateDeltas, _, err = p.DB.GetStateDeltas(ctx, req.Device, r, req.Device.UserID, &stateFilter); err != nil { - req.Log.WithError(err).Error("p.DB.GetStateDeltas failed") - return newPos - } - for _, delta := range stateDeltas { - // Ignore deltas for rooms we didn't newly join - if _, ok := req.Response.Rooms.Join[delta.RoomID]; ok { - continue - } - if _, err = p.addRoomDeltaToResponse(ctx, req.Device, r, delta, &eventFilter, &stateFilter, req.Response); err != nil { - req.Log.WithError(err).Error("d.addRoomDeltaToResponse failed") - return newPos - } - } - } - return newPos } @@ -340,12 +325,12 @@ func (p *PDUStreamProvider) addRoomDeltaToResponse( logrus.WithError(err).Error("unable to apply history visibility filter") } - if len(events) > 0 { - updateLatestPosition(events[len(events)-1].EventID()) - } if len(delta.StateEvents) > 0 { updateLatestPosition(delta.StateEvents[len(delta.StateEvents)-1].EventID()) } + if len(events) > 0 { + updateLatestPosition(events[len(events)-1].EventID()) + } switch delta.Membership { case gomatrixserverlib.Join: @@ -418,6 +403,8 @@ func applyHistoryVisibilityFilter( logrus.WithFields(logrus.Fields{ "duration": time.Since(startTime), "room_id": roomID, + "before": len(recentEvents), + "after": len(events), }).Debug("applied history visibility (sync)") return events, nil } diff --git a/syncapi/sync/request.go b/syncapi/sync/request.go index 9d4740e93..268ed70c6 100644 --- a/syncapi/sync/request.go +++ b/syncapi/sync/request.go @@ -23,12 +23,13 @@ import ( "strconv" "time" - "github.com/matrix-org/dendrite/syncapi/storage" - "github.com/matrix-org/dendrite/syncapi/types" - userapi "github.com/matrix-org/dendrite/userapi/api" "github.com/matrix-org/gomatrixserverlib" "github.com/matrix-org/util" "github.com/sirupsen/logrus" + + "github.com/matrix-org/dendrite/syncapi/storage" + "github.com/matrix-org/dendrite/syncapi/types" + userapi "github.com/matrix-org/dendrite/userapi/api" ) const defaultSyncTimeout = time.Duration(0) @@ -46,15 +47,9 @@ func newSyncRequest(req *http.Request, device userapi.Device, syncDB storage.Dat return nil, err } } - // TODO: read from stored filters too + + // Create a default filter and apply a stored filter on top of it (if specified) filter := gomatrixserverlib.DefaultFilter() - if since.IsEmpty() { - // Send as much account data down for complete syncs as possible - // by default, otherwise clients do weird things while waiting - // for the rest of the data to trickle down. - filter.AccountData.Limit = math.MaxInt32 - filter.Room.AccountData.Limit = math.MaxInt32 - } filterQuery := req.URL.Query().Get("filter") if filterQuery != "" { if filterQuery[0] == '{' { @@ -76,6 +71,17 @@ func newSyncRequest(req *http.Request, device userapi.Device, syncDB storage.Dat } } + // A loaded filter might have overwritten these values, + // so set them after loading the filter. + if since.IsEmpty() { + // Send as much account data down for complete syncs as possible + // by default, otherwise clients do weird things while waiting + // for the rest of the data to trickle down. + filter.AccountData.Limit = math.MaxInt32 + filter.Room.AccountData.Limit = math.MaxInt32 + filter.Room.State.Limit = math.MaxInt32 + } + logger := util.GetLogger(req.Context()).WithFields(logrus.Fields{ "user_id": device.UserID, "device_id": device.ID, diff --git a/syncapi/sync/requestpool.go b/syncapi/sync/requestpool.go index d908a9629..c2c9616e8 100644 --- a/syncapi/sync/requestpool.go +++ b/syncapi/sync/requestpool.go @@ -298,8 +298,8 @@ func (rp *RequestPool) OnIncomingSyncRequest(req *http.Request, device *userapi. return giveup() case <-userStreamListener.GetNotifyChannel(syncReq.Since): - syncReq.Log.Debugln("Responding to sync after wake-up") currentPos.ApplyUpdates(userStreamListener.GetSyncPosition()) + syncReq.Log.WithField("currentPos", currentPos).Debugln("Responding to sync after wake-up") } } else { syncReq.Log.WithField("currentPos", currentPos).Debugln("Responding to sync immediately") diff --git a/syncapi/syncapi_test.go b/syncapi/syncapi_test.go index 76d51c86b..c81256aa7 100644 --- a/syncapi/syncapi_test.go +++ b/syncapi/syncapi_test.go @@ -195,7 +195,7 @@ func TestSyncAPICreateRoomSyncEarly(t *testing.T) { } func testSyncAPICreateRoomSyncEarly(t *testing.T, dbType test.DBType) { - t.SkipNow() // Temporary? + t.Skip("Skipped, possibly fixed") user := test.NewUser(t) room := test.NewRoom(t, user) alice := userapi.Device{ diff --git a/syncapi/types/types.go b/syncapi/types/types.go index 39b085d9c..d75d53ca9 100644 --- a/syncapi/types/types.go +++ b/syncapi/types/types.go @@ -37,6 +37,7 @@ var ( type StateDelta struct { RoomID string StateEvents []*gomatrixserverlib.HeaderedEvent + NewlyJoined bool Membership string // The PDU stream position of the latest membership event for this user, if applicable. // Can be 0 if there is no membership event in this delta. From ed79e8626aa3fc909b9ff2dbe4a1e16620ee0e37 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Thu, 25 Aug 2022 14:14:10 +0100 Subject: [PATCH 31/42] Version 0.9.5 (#2673) Changelog and version bump. --- CHANGES.md | 12 ++++++++++++ internal/version.go | 2 +- 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/CHANGES.md b/CHANGES.md index aaf5836ba..0f57bffcb 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -1,5 +1,17 @@ # Changelog +## Dendrite 0.9.5 (2022-08-25) + +### Fixes + +* The roomserver will now correctly unreject previously rejected events if necessary when reprocessing +* The handling of event soft-failure has been improved on the roomserver input by no longer applying rejection rules and still calculating state before the event if possible +* The federation `/state` and `/state_ids` endpoints should now return the correct error code when the state isn't known instead of returning a HTTP 500 +* The federation `/event` should now return outlier events correctly instead of returning a HTTP 500 +* A bug in the federation backoff allowing zero intervals has been corrected +* The `create-account` utility will no longer error if the homeserver URL ends in a trailing slash +* A regression in `/sync` introduced in 0.9.4 should be fixed + ## Dendrite 0.9.4 (2022-08-19) ### Fixes diff --git a/internal/version.go b/internal/version.go index 384f091a0..108b8ab0f 100644 --- a/internal/version.go +++ b/internal/version.go @@ -17,7 +17,7 @@ var build string const ( VersionMajor = 0 VersionMinor = 9 - VersionPatch = 4 + VersionPatch = 5 VersionTag = "" // example: "rc1" ) From 38bed30b411d8e438d430eae2670482eb2778628 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Fri, 26 Aug 2022 09:56:41 +0100 Subject: [PATCH 32/42] Restrict username length properly (#2676) This fixes #2674 by fixing the username check to comply with the [spec appendices](https://spec.matrix.org/v1.3/appendices/#user-identifiers): > The length of a user ID, including the @ sigil and the domain, MUST NOT exceed 255 characters. --- clientapi/routing/register.go | 30 +++++++++++++++--------------- clientapi/routing/routing.go | 2 +- 2 files changed, 16 insertions(+), 16 deletions(-) diff --git a/clientapi/routing/register.go b/clientapi/routing/register.go index af0329a48..0bda1e488 100644 --- a/clientapi/routing/register.go +++ b/clientapi/routing/register.go @@ -276,19 +276,19 @@ type recaptchaResponse struct { } // validateUsername returns an error response if the username is invalid -func validateUsername(username string) *util.JSONResponse { +func validateUsername(localpart string, domain gomatrixserverlib.ServerName) *util.JSONResponse { // https://github.com/matrix-org/synapse/blob/v0.20.0/synapse/rest/client/v2_alpha/register.py#L161 - if len(username) > maxUsernameLength { + if id := fmt.Sprintf("@%s:%s", localpart, domain); len(id) > maxUsernameLength { return &util.JSONResponse{ Code: http.StatusBadRequest, - JSON: jsonerror.BadJSON(fmt.Sprintf("'username' >%d characters", maxUsernameLength)), + JSON: jsonerror.BadJSON(fmt.Sprintf("%q exceeds the maximum length of %d characters", id, maxUsernameLength)), } - } else if !validUsernameRegex.MatchString(username) { + } else if !validUsernameRegex.MatchString(localpart) { return &util.JSONResponse{ Code: http.StatusBadRequest, JSON: jsonerror.InvalidUsername("Username can only contain characters a-z, 0-9, or '_-./='"), } - } else if username[0] == '_' { // Regex checks its not a zero length string + } else if localpart[0] == '_' { // Regex checks its not a zero length string return &util.JSONResponse{ Code: http.StatusBadRequest, JSON: jsonerror.InvalidUsername("Username cannot start with a '_'"), @@ -298,13 +298,13 @@ func validateUsername(username string) *util.JSONResponse { } // validateApplicationServiceUsername returns an error response if the username is invalid for an application service -func validateApplicationServiceUsername(username string) *util.JSONResponse { - if len(username) > maxUsernameLength { +func validateApplicationServiceUsername(localpart string, domain gomatrixserverlib.ServerName) *util.JSONResponse { + if id := fmt.Sprintf("@%s:%s", localpart, domain); len(id) > maxUsernameLength { return &util.JSONResponse{ Code: http.StatusBadRequest, - JSON: jsonerror.BadJSON(fmt.Sprintf("'username' >%d characters", maxUsernameLength)), + JSON: jsonerror.BadJSON(fmt.Sprintf("%q exceeds the maximum length of %d characters", id, maxUsernameLength)), } - } else if !validUsernameRegex.MatchString(username) { + } else if !validUsernameRegex.MatchString(localpart) { return &util.JSONResponse{ Code: http.StatusBadRequest, JSON: jsonerror.InvalidUsername("Username can only contain characters a-z, 0-9, or '_-./='"), @@ -523,7 +523,7 @@ func validateApplicationService( } // Check username application service is trying to register is valid - if err := validateApplicationServiceUsername(username); err != nil { + if err := validateApplicationServiceUsername(username, cfg.Matrix.ServerName); err != nil { return "", err } @@ -604,7 +604,7 @@ func Register( case r.Type == authtypes.LoginTypeApplicationService && accessTokenErr == nil: // Spec-compliant case (the access_token is specified and the login type // is correctly set, so it's an appservice registration) - if resErr := validateApplicationServiceUsername(r.Username); resErr != nil { + if resErr := validateApplicationServiceUsername(r.Username, cfg.Matrix.ServerName); resErr != nil { return *resErr } case accessTokenErr == nil: @@ -617,7 +617,7 @@ func Register( default: // Spec-compliant case (neither the access_token nor the login type are // specified, so it's a normal user registration) - if resErr := validateUsername(r.Username); resErr != nil { + if resErr := validateUsername(r.Username, cfg.Matrix.ServerName); resErr != nil { return *resErr } } @@ -1018,7 +1018,7 @@ func RegisterAvailable( // Squash username to all lowercase letters username = strings.ToLower(username) - if err := validateUsername(username); err != nil { + if err := validateUsername(username, cfg.Matrix.ServerName); err != nil { return *err } @@ -1059,7 +1059,7 @@ func RegisterAvailable( } } -func handleSharedSecretRegistration(userAPI userapi.ClientUserAPI, sr *SharedSecretRegistration, req *http.Request) util.JSONResponse { +func handleSharedSecretRegistration(cfg *config.ClientAPI, userAPI userapi.ClientUserAPI, sr *SharedSecretRegistration, req *http.Request) util.JSONResponse { ssrr, err := NewSharedSecretRegistrationRequest(req.Body) if err != nil { return util.JSONResponse{ @@ -1080,7 +1080,7 @@ func handleSharedSecretRegistration(userAPI userapi.ClientUserAPI, sr *SharedSec // downcase capitals ssrr.User = strings.ToLower(ssrr.User) - if resErr := validateUsername(ssrr.User); resErr != nil { + if resErr := validateUsername(ssrr.User, cfg.Matrix.ServerName); resErr != nil { return *resErr } if resErr := validatePassword(ssrr.Password); resErr != nil { diff --git a/clientapi/routing/routing.go b/clientapi/routing/routing.go index 6904a2b34..d7a48d228 100644 --- a/clientapi/routing/routing.go +++ b/clientapi/routing/routing.go @@ -133,7 +133,7 @@ func Setup( } } if req.Method == http.MethodPost { - return handleSharedSecretRegistration(userAPI, sr, req) + return handleSharedSecretRegistration(cfg, userAPI, sr, req) } return util.JSONResponse{ Code: http.StatusMethodNotAllowed, From a3eb4e5e9806745f3e6fd36f43bfa07333650cb4 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Sun, 28 Aug 2022 09:52:03 +0100 Subject: [PATCH 33/42] Enforce `type`, `sender`, `state_key` and `room_id` lengths using codepoints rather than bytes (update to matrix-org/gomatrixserverlib@5f66df0, matrix-org/matrix-spec#1001) --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index 0833f2908..1f55ee5b8 100644 --- a/go.mod +++ b/go.mod @@ -21,7 +21,7 @@ require ( github.com/matrix-org/dugong v0.0.0-20210921133753-66e6b1c67e2e github.com/matrix-org/go-sqlite3-js v0.0.0-20220419092513-28aa791a1c91 github.com/matrix-org/gomatrix v0.0.0-20210324163249-be2af5ef2e16 - github.com/matrix-org/gomatrixserverlib v0.0.0-20220824082345-662dca17bf94 + github.com/matrix-org/gomatrixserverlib v0.0.0-20220828085018-5f66df08dc66 github.com/matrix-org/pinecone v0.0.0-20220803093810-b7a830c08fb9 github.com/matrix-org/util v0.0.0-20200807132607-55161520e1d4 github.com/mattn/go-sqlite3 v1.14.13 diff --git a/go.sum b/go.sum index fcbdb7e41..8e98e0bcf 100644 --- a/go.sum +++ b/go.sum @@ -343,8 +343,8 @@ github.com/matrix-org/go-sqlite3-js v0.0.0-20220419092513-28aa791a1c91/go.mod h1 github.com/matrix-org/gomatrix v0.0.0-20190528120928-7df988a63f26/go.mod h1:3fxX6gUjWyI/2Bt7J1OLhpCzOfO/bB3AiX0cJtEKud0= github.com/matrix-org/gomatrix v0.0.0-20210324163249-be2af5ef2e16 h1:ZtO5uywdd5dLDCud4r0r55eP4j9FuUNpl60Gmntcop4= github.com/matrix-org/gomatrix v0.0.0-20210324163249-be2af5ef2e16/go.mod h1:/gBX06Kw0exX1HrwmoBibFA98yBk/jxKpGVeyQbff+s= -github.com/matrix-org/gomatrixserverlib v0.0.0-20220824082345-662dca17bf94 h1:zoTv/qxg7C/O995JBPvp+Z8KMR69HhB+M+P22A8Hmm0= -github.com/matrix-org/gomatrixserverlib v0.0.0-20220824082345-662dca17bf94/go.mod h1:jX38yp3SSLJNftBg3PXU1ayd0PCLIiDHQ4xAc9DIixk= +github.com/matrix-org/gomatrixserverlib v0.0.0-20220828085018-5f66df08dc66 h1:4+ycQUZ4Do5LeEuiesz6gD7LkjRNUtk8oETFqjalZxQ= +github.com/matrix-org/gomatrixserverlib v0.0.0-20220828085018-5f66df08dc66/go.mod h1:jX38yp3SSLJNftBg3PXU1ayd0PCLIiDHQ4xAc9DIixk= github.com/matrix-org/pinecone v0.0.0-20220803093810-b7a830c08fb9 h1:ed8yvWhTLk7+sNeK/eOZRTvESFTOHDRevoRoyeqPtvY= github.com/matrix-org/pinecone v0.0.0-20220803093810-b7a830c08fb9/go.mod h1:P4MqPf+u83OPulPJ+XTbSDbbWrdFYNY4LZ/B1PIduFE= github.com/matrix-org/util v0.0.0-20190711121626-527ce5ddefc7/go.mod h1:vVQlW/emklohkZnOPwD3LrZUBqdfsbiyO3p1lNV8F6U= From aba171d9bcfcfa31e8f1b16e2d91b256d0cfa07c Mon Sep 17 00:00:00 2001 From: Till Faelligen <2353100+S7evinK@users.noreply.github.com> Date: Mon, 29 Aug 2022 07:23:46 +0200 Subject: [PATCH 34/42] Default to limit of 20 state events --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index 1f55ee5b8..ee025c354 100644 --- a/go.mod +++ b/go.mod @@ -21,7 +21,7 @@ require ( github.com/matrix-org/dugong v0.0.0-20210921133753-66e6b1c67e2e github.com/matrix-org/go-sqlite3-js v0.0.0-20220419092513-28aa791a1c91 github.com/matrix-org/gomatrix v0.0.0-20210324163249-be2af5ef2e16 - github.com/matrix-org/gomatrixserverlib v0.0.0-20220828085018-5f66df08dc66 + github.com/matrix-org/gomatrixserverlib v0.0.0-20220829052120-b3746121e530 github.com/matrix-org/pinecone v0.0.0-20220803093810-b7a830c08fb9 github.com/matrix-org/util v0.0.0-20200807132607-55161520e1d4 github.com/mattn/go-sqlite3 v1.14.13 diff --git a/go.sum b/go.sum index 8e98e0bcf..d584a7333 100644 --- a/go.sum +++ b/go.sum @@ -343,8 +343,8 @@ github.com/matrix-org/go-sqlite3-js v0.0.0-20220419092513-28aa791a1c91/go.mod h1 github.com/matrix-org/gomatrix v0.0.0-20190528120928-7df988a63f26/go.mod h1:3fxX6gUjWyI/2Bt7J1OLhpCzOfO/bB3AiX0cJtEKud0= github.com/matrix-org/gomatrix v0.0.0-20210324163249-be2af5ef2e16 h1:ZtO5uywdd5dLDCud4r0r55eP4j9FuUNpl60Gmntcop4= github.com/matrix-org/gomatrix v0.0.0-20210324163249-be2af5ef2e16/go.mod h1:/gBX06Kw0exX1HrwmoBibFA98yBk/jxKpGVeyQbff+s= -github.com/matrix-org/gomatrixserverlib v0.0.0-20220828085018-5f66df08dc66 h1:4+ycQUZ4Do5LeEuiesz6gD7LkjRNUtk8oETFqjalZxQ= -github.com/matrix-org/gomatrixserverlib v0.0.0-20220828085018-5f66df08dc66/go.mod h1:jX38yp3SSLJNftBg3PXU1ayd0PCLIiDHQ4xAc9DIixk= +github.com/matrix-org/gomatrixserverlib v0.0.0-20220829052120-b3746121e530 h1:G0xIy9wpOGpv/xkCneAM22bJ36cXsSpa7AThmtvClyE= +github.com/matrix-org/gomatrixserverlib v0.0.0-20220829052120-b3746121e530/go.mod h1:jX38yp3SSLJNftBg3PXU1ayd0PCLIiDHQ4xAc9DIixk= github.com/matrix-org/pinecone v0.0.0-20220803093810-b7a830c08fb9 h1:ed8yvWhTLk7+sNeK/eOZRTvESFTOHDRevoRoyeqPtvY= github.com/matrix-org/pinecone v0.0.0-20220803093810-b7a830c08fb9/go.mod h1:P4MqPf+u83OPulPJ+XTbSDbbWrdFYNY4LZ/B1PIduFE= github.com/matrix-org/util v0.0.0-20190711121626-527ce5ddefc7/go.mod h1:vVQlW/emklohkZnOPwD3LrZUBqdfsbiyO3p1lNV8F6U= From 704cc5c9f500bdd41a1c99ff2ae2fcef5eb74abb Mon Sep 17 00:00:00 2001 From: Brian Meek Date: Mon, 29 Aug 2022 00:10:42 -0700 Subject: [PATCH 35/42] Race in keyserver intialization (#2619) Signed-off-by: Brian Meek --- keyserver/keyserver.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/keyserver/keyserver.go b/keyserver/keyserver.go index cd506f981..75cdaedb4 100644 --- a/keyserver/keyserver.go +++ b/keyserver/keyserver.go @@ -60,7 +60,7 @@ func NewInternalAPI( updater := internal.NewDeviceListUpdater(db, ap, keyChangeProducer, fedClient, 8) // 8 workers TODO: configurable ap.Updater = updater go func() { - if err = updater.Start(); err != nil { + if err := updater.Start(); err != nil { logrus.WithError(err).Panicf("failed to start device list updater") } }() @@ -68,7 +68,7 @@ func NewInternalAPI( dlConsumer := consumers.NewDeviceListUpdateConsumer( base.ProcessContext, cfg, js, updater, ) - if err = dlConsumer.Start(); err != nil { + if err := dlConsumer.Start(); err != nil { logrus.WithError(err).Panic("failed to start device list consumer") } From 21ef487ff8c36f7302e5fc47d3602ea0fc5ae3d3 Mon Sep 17 00:00:00 2001 From: Till Faelligen <2353100+S7evinK@users.noreply.github.com> Date: Mon, 29 Aug 2022 14:18:33 +0200 Subject: [PATCH 36/42] Log roomserver error --- appservice/consumers/roomserver.go | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/appservice/consumers/roomserver.go b/appservice/consumers/roomserver.go index 37d4ef9c2..d567408be 100644 --- a/appservice/consumers/roomserver.go +++ b/appservice/consumers/roomserver.go @@ -18,14 +18,15 @@ import ( "context" "encoding/json" + "github.com/matrix-org/gomatrixserverlib" + "github.com/nats-io/nats.go" + "github.com/matrix-org/dendrite/appservice/storage" "github.com/matrix-org/dendrite/appservice/types" "github.com/matrix-org/dendrite/roomserver/api" "github.com/matrix-org/dendrite/setup/config" "github.com/matrix-org/dendrite/setup/jetstream" "github.com/matrix-org/dendrite/setup/process" - "github.com/matrix-org/gomatrixserverlib" - "github.com/nats-io/nats.go" log "github.com/sirupsen/logrus" ) @@ -103,6 +104,7 @@ func (s *OutputRoomEventConsumer) onMessage(ctx context.Context, msg *nats.Msg) } if len(eventsReq.EventIDs) > 0 { if err := s.rsAPI.QueryEventsByID(s.ctx, eventsReq, eventsRes); err != nil { + log.WithError(err).Errorf("s.rsAPI.QueryEventsByID failed") return false } events = append(events, eventsRes.Events...) From bbb3ade4a2b49cfdaf7ec86ddf079ff7d48e0cf3 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Tue, 30 Aug 2022 11:44:26 +0100 Subject: [PATCH 37/42] Fix the Cat Overflow bug (update to matrix-org/gomatrixserverlib@2020bdc) --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index ee025c354..8c334f1e8 100644 --- a/go.mod +++ b/go.mod @@ -21,7 +21,7 @@ require ( github.com/matrix-org/dugong v0.0.0-20210921133753-66e6b1c67e2e github.com/matrix-org/go-sqlite3-js v0.0.0-20220419092513-28aa791a1c91 github.com/matrix-org/gomatrix v0.0.0-20210324163249-be2af5ef2e16 - github.com/matrix-org/gomatrixserverlib v0.0.0-20220829052120-b3746121e530 + github.com/matrix-org/gomatrixserverlib v0.0.0-20220830104348-2020bdc55859 github.com/matrix-org/pinecone v0.0.0-20220803093810-b7a830c08fb9 github.com/matrix-org/util v0.0.0-20200807132607-55161520e1d4 github.com/mattn/go-sqlite3 v1.14.13 diff --git a/go.sum b/go.sum index d584a7333..aac86010b 100644 --- a/go.sum +++ b/go.sum @@ -343,8 +343,8 @@ github.com/matrix-org/go-sqlite3-js v0.0.0-20220419092513-28aa791a1c91/go.mod h1 github.com/matrix-org/gomatrix v0.0.0-20190528120928-7df988a63f26/go.mod h1:3fxX6gUjWyI/2Bt7J1OLhpCzOfO/bB3AiX0cJtEKud0= github.com/matrix-org/gomatrix v0.0.0-20210324163249-be2af5ef2e16 h1:ZtO5uywdd5dLDCud4r0r55eP4j9FuUNpl60Gmntcop4= github.com/matrix-org/gomatrix v0.0.0-20210324163249-be2af5ef2e16/go.mod h1:/gBX06Kw0exX1HrwmoBibFA98yBk/jxKpGVeyQbff+s= -github.com/matrix-org/gomatrixserverlib v0.0.0-20220829052120-b3746121e530 h1:G0xIy9wpOGpv/xkCneAM22bJ36cXsSpa7AThmtvClyE= -github.com/matrix-org/gomatrixserverlib v0.0.0-20220829052120-b3746121e530/go.mod h1:jX38yp3SSLJNftBg3PXU1ayd0PCLIiDHQ4xAc9DIixk= +github.com/matrix-org/gomatrixserverlib v0.0.0-20220830104348-2020bdc55859 h1:14mOIE2sGS6YTG1+3GFbVWmAisW5pzyfn3SxJTjkR2I= +github.com/matrix-org/gomatrixserverlib v0.0.0-20220830104348-2020bdc55859/go.mod h1:jX38yp3SSLJNftBg3PXU1ayd0PCLIiDHQ4xAc9DIixk= github.com/matrix-org/pinecone v0.0.0-20220803093810-b7a830c08fb9 h1:ed8yvWhTLk7+sNeK/eOZRTvESFTOHDRevoRoyeqPtvY= github.com/matrix-org/pinecone v0.0.0-20220803093810-b7a830c08fb9/go.mod h1:P4MqPf+u83OPulPJ+XTbSDbbWrdFYNY4LZ/B1PIduFE= github.com/matrix-org/util v0.0.0-20190711121626-527ce5ddefc7/go.mod h1:vVQlW/emklohkZnOPwD3LrZUBqdfsbiyO3p1lNV8F6U= From b0e2ea0f374892a6f9c77623bc6c3fd9ea572b3b Mon Sep 17 00:00:00 2001 From: Till <2353100+S7evinK@users.noreply.github.com> Date: Tue, 30 Aug 2022 13:59:13 +0200 Subject: [PATCH 38/42] Fix race condition on startup (#2679) `SetupAndServeHTTP` would race in `configureHTTPErrors` and while configuring routes. --- setup/base/base.go | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/setup/base/base.go b/setup/base/base.go index b21eeba47..87f415764 100644 --- a/setup/base/base.go +++ b/setup/base/base.go @@ -25,21 +25,23 @@ import ( _ "net/http/pprof" "os" "os/signal" + "sync" "syscall" "time" "github.com/getsentry/sentry-go" sentryhttp "github.com/getsentry/sentry-go/http" - "github.com/matrix-org/dendrite/internal/caching" - "github.com/matrix-org/dendrite/internal/httputil" - "github.com/matrix-org/dendrite/internal/pushgateway" - "github.com/matrix-org/dendrite/internal/sqlutil" "github.com/matrix-org/gomatrixserverlib" "github.com/prometheus/client_golang/prometheus/promhttp" "go.uber.org/atomic" "golang.org/x/net/http2" "golang.org/x/net/http2/h2c" + "github.com/matrix-org/dendrite/internal/caching" + "github.com/matrix-org/dendrite/internal/httputil" + "github.com/matrix-org/dendrite/internal/pushgateway" + "github.com/matrix-org/dendrite/internal/sqlutil" + "github.com/matrix-org/dendrite/internal" "github.com/matrix-org/dendrite/setup/jetstream" "github.com/matrix-org/dendrite/setup/process" @@ -47,6 +49,8 @@ import ( "github.com/gorilla/mux" "github.com/kardianos/minwinsvc" + "github.com/sirupsen/logrus" + appserviceAPI "github.com/matrix-org/dendrite/appservice/api" asinthttp "github.com/matrix-org/dendrite/appservice/inthttp" federationAPI "github.com/matrix-org/dendrite/federationapi/api" @@ -58,7 +62,6 @@ import ( "github.com/matrix-org/dendrite/setup/config" userapi "github.com/matrix-org/dendrite/userapi/api" userapiinthttp "github.com/matrix-org/dendrite/userapi/inthttp" - "github.com/sirupsen/logrus" ) // BaseDendrite is a base for creating new instances of dendrite. It parses @@ -87,6 +90,7 @@ type BaseDendrite struct { Database *sql.DB DatabaseWriter sqlutil.Writer EnableMetrics bool + startupLock sync.Mutex } const NoListener = "" @@ -394,6 +398,9 @@ func (b *BaseDendrite) SetupAndServeHTTP( internalHTTPAddr, externalHTTPAddr config.HTTPAddress, certFile, keyFile *string, ) { + // Manually unlocked right before actually serving requests, + // as we don't return from this method (defer doesn't work). + b.startupLock.Lock() internalAddr, _ := internalHTTPAddr.Address() externalAddr, _ := externalHTTPAddr.Address() @@ -472,6 +479,7 @@ func (b *BaseDendrite) SetupAndServeHTTP( externalRouter.PathPrefix(httputil.PublicMediaPathPrefix).Handler(b.PublicMediaAPIMux) externalRouter.PathPrefix(httputil.PublicWellKnownPrefix).Handler(b.PublicWellKnownAPIMux) + b.startupLock.Unlock() if internalAddr != NoListener && internalAddr != externalAddr { go func() { var internalShutdown atomic.Bool // RegisterOnShutdown can be called more than once From 7313f56f44f6820a5573aad79faaa6037703371f Mon Sep 17 00:00:00 2001 From: Till <2353100+S7evinK@users.noreply.github.com> Date: Tue, 30 Aug 2022 14:18:47 +0200 Subject: [PATCH 39/42] Use existing limit instead of default limit when lazy loading members (#2682) This should fix an issue where we return less than the expected membership events, when doing an initial sync. When doing an initial sync, the state limit is set to `math.MaxInt32`, while the default filter is set to 20. --- syncapi/streams/stream_pdu.go | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/syncapi/streams/stream_pdu.go b/syncapi/streams/stream_pdu.go index fa4c722ce..ffcf64df6 100644 --- a/syncapi/streams/stream_pdu.go +++ b/syncapi/streams/stream_pdu.go @@ -303,7 +303,7 @@ func (p *PDUStreamProvider) addRoomDeltaToResponse( if stateFilter.LazyLoadMembers { delta.StateEvents, err = p.lazyLoadMembers( - ctx, delta.RoomID, true, limited, stateFilter.IncludeRedundantMembers, + ctx, delta.RoomID, true, limited, stateFilter, device, recentEvents, delta.StateEvents, ) if err != nil && err != sql.ErrNoRows { @@ -532,7 +532,7 @@ func (p *PDUStreamProvider) getJoinResponseForCompleteSync( return nil, err } stateEvents, err = p.lazyLoadMembers(ctx, roomID, - false, limited, stateFilter.IncludeRedundantMembers, + false, limited, stateFilter, device, recentEvents, stateEvents, ) if err != nil && err != sql.ErrNoRows { @@ -551,7 +551,7 @@ func (p *PDUStreamProvider) getJoinResponseForCompleteSync( func (p *PDUStreamProvider) lazyLoadMembers( ctx context.Context, roomID string, - incremental, limited, includeRedundant bool, + incremental, limited bool, stateFilter *gomatrixserverlib.StateFilter, device *userapi.Device, timelineEvents, stateEvents []*gomatrixserverlib.HeaderedEvent, ) ([]*gomatrixserverlib.HeaderedEvent, error) { @@ -581,7 +581,7 @@ func (p *PDUStreamProvider) lazyLoadMembers( stateKey := *event.StateKey() if _, ok := timelineUsers[stateKey]; ok || isGappedIncremental { newStateEvents = append(newStateEvents, event) - if !includeRedundant { + if !stateFilter.IncludeRedundantMembers { p.lazyLoadCache.StoreLazyLoadedUser(device, roomID, stateKey, event.EventID()) } delete(timelineUsers, stateKey) @@ -596,6 +596,7 @@ func (p *PDUStreamProvider) lazyLoadMembers( } // Query missing membership events filter := gomatrixserverlib.DefaultStateFilter() + filter.Limit = stateFilter.Limit filter.Senders = &wantUsers filter.Types = &[]string{gomatrixserverlib.MRoomMember} memberships, err := p.DB.GetStateEventsForRoom(ctx, roomID, &filter) From 93a6e2f4d30d72c015c7086b97322f0b106ab061 Mon Sep 17 00:00:00 2001 From: Till <2353100+S7evinK@users.noreply.github.com> Date: Tue, 30 Aug 2022 14:54:57 +0200 Subject: [PATCH 40/42] Set the notification count when sending push notifications (#2683) --- userapi/consumers/syncapi_streamevent.go | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/userapi/consumers/syncapi_streamevent.go b/userapi/consumers/syncapi_streamevent.go index 7807c7637..ec351ef7e 100644 --- a/userapi/consumers/syncapi_streamevent.go +++ b/userapi/consumers/syncapi_streamevent.go @@ -7,6 +7,10 @@ import ( "strings" "time" + "github.com/matrix-org/gomatrixserverlib" + "github.com/nats-io/nats.go" + log "github.com/sirupsen/logrus" + "github.com/matrix-org/dendrite/internal/eventutil" "github.com/matrix-org/dendrite/internal/pushgateway" "github.com/matrix-org/dendrite/internal/pushrules" @@ -20,9 +24,6 @@ import ( "github.com/matrix-org/dendrite/userapi/storage" "github.com/matrix-org/dendrite/userapi/storage/tables" "github.com/matrix-org/dendrite/userapi/util" - "github.com/matrix-org/gomatrixserverlib" - "github.com/nats-io/nats.go" - log "github.com/sirupsen/logrus" ) type OutputStreamEventConsumer struct { @@ -529,7 +530,9 @@ func (s *OutputStreamEventConsumer) notifyHTTP(ctx context.Context, event *gomat case "event_id_only": req = pushgateway.NotifyRequest{ Notification: pushgateway.Notification{ - Counts: &pushgateway.Counts{}, + Counts: &pushgateway.Counts{ + Unread: userNumUnreadNotifs, + }, Devices: devices, EventID: event.EventID(), RoomID: event.RoomID(), From 2be43560ca7d81601e6763c9fcb8b72ecd74009b Mon Sep 17 00:00:00 2001 From: PiotrKozimor <37144818+PiotrKozimor@users.noreply.github.com> Date: Tue, 30 Aug 2022 15:47:54 +0200 Subject: [PATCH 41/42] Index on `syncapi_send_to_device` table (#2684) Introduced index improves select query performance. Example execution time of `selectSendToDeviceMessagesSQL` query dropped from 80 ms to 15 ms. No sytest modifications are required. ### Pull Request Checklist * [x] I have added added tests for PR _or_ I have justified why this PR doesn't need tests. * [x] Pull request includes a [sign off](https://github.com/matrix-org/dendrite/blob/main/docs/CONTRIBUTING.md#sign-off) Signed-off-by: `Piotr Kozimor ` --- syncapi/storage/postgres/send_to_device_table.go | 2 ++ syncapi/storage/sqlite3/send_to_device_table.go | 2 ++ 2 files changed, 4 insertions(+) diff --git a/syncapi/storage/postgres/send_to_device_table.go b/syncapi/storage/postgres/send_to_device_table.go index fd0c1c56b..6ab1f0f48 100644 --- a/syncapi/storage/postgres/send_to_device_table.go +++ b/syncapi/storage/postgres/send_to_device_table.go @@ -41,6 +41,8 @@ CREATE TABLE IF NOT EXISTS syncapi_send_to_device ( -- The event content JSON. content TEXT NOT NULL ); + +CREATE INDEX IF NOT EXISTS syncapi_send_to_device_user_id_device_id_idx ON syncapi_send_to_device(user_id, device_id); ` const insertSendToDeviceMessageSQL = ` diff --git a/syncapi/storage/sqlite3/send_to_device_table.go b/syncapi/storage/sqlite3/send_to_device_table.go index e3aa1b7a1..0da06506c 100644 --- a/syncapi/storage/sqlite3/send_to_device_table.go +++ b/syncapi/storage/sqlite3/send_to_device_table.go @@ -39,6 +39,8 @@ CREATE TABLE IF NOT EXISTS syncapi_send_to_device ( -- The event content JSON. content TEXT NOT NULL ); + +CREATE INDEX IF NOT EXISTS syncapi_send_to_device_user_id_device_id_idx ON syncapi_send_to_device(user_id, device_id); ` const insertSendToDeviceMessageSQL = ` From 02ec00b1bbeb43f68a4f9b1f19ee6282e4a34194 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Tue, 30 Aug 2022 17:40:46 +0100 Subject: [PATCH 42/42] Check for `"creator"` field in `m.room.create` during event auth (update to matrix-org/gomatrixserverlib@c71e518) --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index 8c334f1e8..8bf8f454d 100644 --- a/go.mod +++ b/go.mod @@ -21,7 +21,7 @@ require ( github.com/matrix-org/dugong v0.0.0-20210921133753-66e6b1c67e2e github.com/matrix-org/go-sqlite3-js v0.0.0-20220419092513-28aa791a1c91 github.com/matrix-org/gomatrix v0.0.0-20210324163249-be2af5ef2e16 - github.com/matrix-org/gomatrixserverlib v0.0.0-20220830104348-2020bdc55859 + github.com/matrix-org/gomatrixserverlib v0.0.0-20220830164018-c71e518537a2 github.com/matrix-org/pinecone v0.0.0-20220803093810-b7a830c08fb9 github.com/matrix-org/util v0.0.0-20200807132607-55161520e1d4 github.com/mattn/go-sqlite3 v1.14.13 diff --git a/go.sum b/go.sum index aac86010b..8b8baabcf 100644 --- a/go.sum +++ b/go.sum @@ -343,8 +343,8 @@ github.com/matrix-org/go-sqlite3-js v0.0.0-20220419092513-28aa791a1c91/go.mod h1 github.com/matrix-org/gomatrix v0.0.0-20190528120928-7df988a63f26/go.mod h1:3fxX6gUjWyI/2Bt7J1OLhpCzOfO/bB3AiX0cJtEKud0= github.com/matrix-org/gomatrix v0.0.0-20210324163249-be2af5ef2e16 h1:ZtO5uywdd5dLDCud4r0r55eP4j9FuUNpl60Gmntcop4= github.com/matrix-org/gomatrix v0.0.0-20210324163249-be2af5ef2e16/go.mod h1:/gBX06Kw0exX1HrwmoBibFA98yBk/jxKpGVeyQbff+s= -github.com/matrix-org/gomatrixserverlib v0.0.0-20220830104348-2020bdc55859 h1:14mOIE2sGS6YTG1+3GFbVWmAisW5pzyfn3SxJTjkR2I= -github.com/matrix-org/gomatrixserverlib v0.0.0-20220830104348-2020bdc55859/go.mod h1:jX38yp3SSLJNftBg3PXU1ayd0PCLIiDHQ4xAc9DIixk= +github.com/matrix-org/gomatrixserverlib v0.0.0-20220830164018-c71e518537a2 h1:esbNn9hg//tAStA6TogatAJAursw23A+yfVRQsdiv70= +github.com/matrix-org/gomatrixserverlib v0.0.0-20220830164018-c71e518537a2/go.mod h1:jX38yp3SSLJNftBg3PXU1ayd0PCLIiDHQ4xAc9DIixk= github.com/matrix-org/pinecone v0.0.0-20220803093810-b7a830c08fb9 h1:ed8yvWhTLk7+sNeK/eOZRTvESFTOHDRevoRoyeqPtvY= github.com/matrix-org/pinecone v0.0.0-20220803093810-b7a830c08fb9/go.mod h1:P4MqPf+u83OPulPJ+XTbSDbbWrdFYNY4LZ/B1PIduFE= github.com/matrix-org/util v0.0.0-20190711121626-527ce5ddefc7/go.mod h1:vVQlW/emklohkZnOPwD3LrZUBqdfsbiyO3p1lNV8F6U=