Commit graph

2187 commits

Author SHA1 Message Date
Neil Alexander e9545dc12f
Remove error when state keys are missing for user NIDs (#2213)
* Remove error when state keys are missing for user NIDs

There is still an actual bug here somewhere in the membership updater, but this check does more harm than good, since it means that the key consumers don't actually distribute updates to *anyone*. It's better just to deal with this silently for now.

To find these broken rows:

```
SELECT * FROM roomserver_membership AS m WHERE NOT EXISTS (
	SELECT event_state_key_nid FROM roomserver_event_state_keys AS s
	WHERE m.sender_nid = s.event_state_key_nid
);
```

* Logging
2022-02-22 13:40:08 +00:00
Neil Alexander 600fbae31f
Only emit key change notifications from federation when changes are made (#2217)
* Only emit key changes when poked over federation

* Remove logging

* Fix unit test possibly
2022-02-22 13:35:06 +00:00
Neil Alexander bbe7d37928
Fix logic error on context history visibility (#2211) 2022-02-21 16:38:53 +00:00
Neil Alexander aa6bbf484a
Return ErrRoomNoExists if insufficient state is available for a buildEvent to succeed when joining a room (#2210)
This may help cases like #2206, since it should prompt us to try a federated join again instead.
2022-02-21 16:22:29 +00:00
S7evinK cf525d1f61
Implement /context (#2207)
* Add QueryEventsAfter

* Add /context

* Make all tests pass on sqlite

* Add queries to get the events for /context requests

* Move /context to the syncapi

* Revert "Add QueryEventsAfter"

This reverts commit 440a771d10.

* Simplify getting the required events

* Apply RoomEventFilter when getting events

* Add passing tests

* Remove logging

* Remove unused SQL statements
Update comments & add TODO
2022-02-21 17:12:22 +01:00
Benjamin Nater 280e9b19a1
Don't ignore config flag for yggdrasil demo if set (#2070)
* Don't ignore config flag for yggdrasil demo if set

Signed-off-by: bn4t <me@bn4t.me>

* run goimports

Signed-off-by: bn4t <me@bn4t.me>

* always override ServerName, PrivateKey and KeyID

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-02-21 15:36:03 +00:00
Neil Alexander a02dd7721d
Reset invalid state snapshots for events during state storage refactor migration (#2209)
This should help with #2204. We can't do this for rooms, only events.
2022-02-21 15:25:54 +00:00
Neil Alexander a386fbed2c
Delete one-time keys when deleting a device (#2208) 2022-02-21 12:30:43 +00:00
S7evinK 002429c9e2
Implement server notices (#2180)
* Add server_notices config

* Disallow rejecting "server notice" invites

* Update config

* Slightly refactor sendEvent and CreateRoom so it can be reused

* Implement unspecced server notices

* Validate the request

* Set the user api when starting

* Rename function/variables

* Update comments

* Update config

* Set the avatar on account creation

* Update test

* Only create the account when starting
Only add routes if sever notices are enabled

* Use reserver username
Check that we actually got roomData

* Add check for admin account
Enable server notices for CI
Return same values as Synapse

* Add custom error for rejecting server notice invite

* Move building an invite to it's own function, for reusability

* Don't create new rooms, use the existing one (follow Synapse behavior)

Co-authored-by: kegsay <kegan@matrix.org>
2022-02-18 16:05:03 +01:00
Neil Alexander dbded87525
Expose sync endpoints via /v3 (#2203) 2022-02-18 14:14:16 +00:00
Neil Alexander 9f4a39e8e0
Refactor user API storage (#2202)
* Refactor User API database

* Fix migration bugs
2022-02-18 13:51:59 +00:00
Neil Alexander 9bd5e414c9
Missing commit from #2186 2022-02-18 11:32:45 +00:00
Neil Alexander 153bfbbea5
Merge both user API databases into one (#2186)
* Merge user API databases into one

* Remove DeviceDatabase from config

* Fix tests

* Try that again

* Clean up keyserver device keys when the devices no longer exist in the user API

* Tweak ordering

* Fix UserExists flag, device check

* Allow including empty entries so we can clean them up

* Remove logging
2022-02-18 11:31:05 +00:00
kegsay 0a7dea4450
Update /whoami response to match Spec v1.2 (#2201)
Basically include `is_guest` and `device_id`. The latter is
needed for https://github.com/matrix-org/complement/pull/305
2022-02-18 11:28:02 +00:00
Neil Alexander 131bedc1a1
Remove event type and state key caches (#2200)
* Don't proactively cache event types and state keys when we don't know if the transaction has persisted yet

* Remove event type and state key caches altogether
2022-02-18 10:58:41 +00:00
Neil Alexander e1eb5807b6
Allow preventing guest registration (#2199)
* Allow disabling guest registration separately

* Update sample config

* Set `guests_disabled` to `true` in the sample config
2022-02-18 10:12:26 +00:00
kegsay 5dd203fde3
Listen for /v3 on CSAPI (#2197)
* Listen for /v3 on CSAPI

* Docs

* More docs

* Rename path variable to fix key backup tests

* Update routing.go

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-02-17 17:38:22 +00:00
Neil Alexander 0b123b29f5
Use process context for roomserver input (#2198) 2022-02-17 15:58:54 +00:00
Neil Alexander 140077265e
Make GetUserDevices logging entry more useful 2022-02-17 15:02:06 +00:00
Neil Alexander 7dfc7c3d70
Don't re-send sent events in add_state_events (#2195)
* Only add events to `add_state_events` that haven't already been sent to the roomserver output before

* Filter on event NIDs instead, hopefully bring joy to SQLite

* UnsentFilter, review comments
2022-02-17 13:53:48 +00:00
Neil Alexander 353168a9e9
Fix potential panic in NewStreamTokenFromString caused by off-by-one error (#2196)
Line 291 could panic when trying to set `positions[i]` if `i == len(positions)`.
2022-02-17 13:25:41 +00:00
S7evinK 89b7519089
Raise waitTime for network related issues (#2192) 2022-02-17 13:15:49 +00:00
Neil Alexander 934491eda5
Update NATS Server to v2.7.2 (#2193)
* Update NATS JetStream to v2.7.2

* Remove deprecated option
2022-02-17 13:15:35 +00:00
S7evinK f51e2a99e9
Remove outbound proxy, http.ProxyFromEnvironment is now used (#2191) 2022-02-17 13:54:29 +01:00
kegsay a4681bc7f7
Set 'complement' as the shared secret for CI (#2194) 2022-02-17 10:59:44 +00:00
S7evinK 5a39512f5f
Add account type (#2171)
* Add account_type for sqlite3

* Add account_type for postgres

* Remove CreateGuestAccount from interface

* Add new AccountTypes & update test

* Use newly added AccountType for account creation

* Add migrations

* Reuse type

* Add AccounnType to Device, so it can be verified on requests

* Rename migration, add missing update for appservices

* Rename sqlite3 migration

* Add missing AccountType to return value

* Update sqlite migration
Change allowance check on /admin/whois

* Fix migration, add IS NULL

* Move accountType to completeRegistration

* Fix migrations

* Add passing test
2022-02-16 18:55:38 +01:00
S7evinK e9b672a34e
Make "Device list doesn't change if remote server is down" pass (#2190) 2022-02-16 16:56:45 +00:00
Neil Alexander fa1e12b503
Don't panic on retiring an invite that we haven't seen yet (#2189) 2022-02-16 11:56:08 +00:00
kegsay f92b048fec
Add host mount compatible Complement image (#2187) 2022-02-15 17:40:48 +00:00
Neil Alexander 4c8c53244e
Update prometheus 2022-02-15 16:27:22 +00:00
Neil Alexander e22e87c012
Update to matrix-org/gomatrixserverlib@20632dd 2022-02-15 11:14:43 +00:00
Neil Alexander 5106cc807c
Ensure only one transaction is used for RS input per room (#2178)
* Ensure the input API only uses a single transaction

* Remove more of the dead query API call

* Tidy up

* Fix tests hopefully

* Don't do unnecessary work for rooms that don't exist

* Improve error, fix another case where transaction wasn't used properly

* Add a unit test for checking single transaction on RS input API

* Fix logic oops when deciding whether to use a transaction in storeEvent
2022-02-11 17:40:14 +00:00
S7evinK a4e7d471af
Remove FederationDisabled error type (#2174) 2022-02-11 18:15:44 +01:00
kegsay a566d53b0b
Don't allow parallel complement tests (#2169)
Fixes flakiness seemingly. See https://github.com/matrix-org/synapse/pull/11910
2022-02-11 16:26:23 +00:00
Neil Alexander 88b45d5cd2
Drop m.room.create events in federation /send transaction (#2179) 2022-02-11 15:18:14 +00:00
kegsay 4e75ab9930
Add postgres complement support (#2177) 2022-02-11 12:35:47 +00:00
kegsay f800cae6d2
Point to /complement/ca not /ca (#2172) 2022-02-10 18:12:11 +00:00
Neil Alexander 9ac27cabc5
Version 0.6.3 (#2170) 2022-02-10 13:50:13 +00:00
tommie c36e4546c3
Support for m.login.token (#2014)
* Add GOPATH to PATH in find-lint.sh.

The user doesn't necessarily have it in PATH.

* Refactor LoginTypePassword and Type to support m.login.token and m.login.sso.

For login token:

* m.login.token will require deleting the token after completeAuth has
  generated an access token, so a cleanup function is returned by
  Type.Login.
* Allowing different login types will require parsing the /login body
  twice: first to extract the "type" and then the type-specific parsing.
  Thus, we will have to buffer the request JSON in /login, like
  UserInteractive already does.

For SSO:

* NewUserInteractive will have to also use GetAccountByLocalpart. It
  makes more sense to just pass a (narrowed-down) accountDB interface
  to it than adding more function pointers.

Code quality:

* Passing around (and down-casting) interface{} for login request types
  has drawbacks in terms of type-safety, and no inherent benefits. We
  always decode JSON anyway. Hence renaming to Type.LoginFromJSON. Code
  that directly uses LoginTypePassword with parsed data can still use
  Login.
* Removed a TODO for SSO. This is already tracked in #1297.
* httputil.UnmarshalJSON is useful because it returns a JSONResponse.

This change is intended to have no functional changes.

* Support login tokens in User API.

This adds full lifecycle functions for login tokens: create, query, delete.

* Support m.login.token in /login.

* Fixes for PR review.

* Set @matrix-org/dendrite-core as repository code owner

* Return event NID from `StoreEvent`, match PSQL vs SQLite behaviour, tweak backfill persistence (#2071)

Co-authored-by: kegsay <kegan@matrix.org>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-02-10 10:27:26 +00:00
S7evinK 432c35a307
Allow user to forget a room, even if they never were a member (#2166)
* Allow user to forget a room, even if they never were a member

* Return "M_UNKNOWN" as per the spec

Co-authored-by: kegsay <kegan@matrix.org>
2022-02-10 10:05:37 +00:00
Neil Alexander 2782ae3d56
Fix fetching missing state (#2163)
* Check that we have a populated state snapshot when determining if we closed the gap

* Do the same in the query API

* Use HasState more opportunistically

* Try to avoid falling down the hole of using a trustworthy but empty state snapshot for non-create events

* Refactor missing state and make sure that we really solve the problem for the new event

* Comments

* Review comments

* Tweak that check again

* Tidy up that create check further

* Fix build hopefully

* Update sendOutliers to use OrderAuthAndStateEvents

* Don't go out of bounds on missingEvents
2022-02-10 10:05:14 +00:00
kegsay 9130156b13
Make the Complement Dockerfile use a fresh directory for runtime (#2168) 2022-02-10 09:37:46 +00:00
Neil Alexander 37cbe263ce
Fix transaction issues in events table in PSQL (#2165)
* Revert "Revert "Fix storage bug in PSQL events table""

This reverts commit cf447dd52a.

* Membership updater to use updater

* Fix membership updater to use transactions properly
2022-02-10 09:30:16 +00:00
kegsay aa5c3b88de
Unmarshal events at the Dendrite level not GMSL level (#2164)
* Use new event json types in gmsl

* Fix EventJSON to actually unmarshal events

* Update GMSL

* Bump GMSL and improve error messages

* Send back the correct RespState

* Update GMSL
2022-02-09 20:31:24 +00:00
S7evinK cc688a9a38
Avoid unnecessary logs and marshaling (#2167)
Co-authored-by: kegsay <kegan@matrix.org>
2022-02-09 15:46:52 +01:00
S7evinK ac25065a54
Fix sytest uploading signed devices gets propagated over federation (#2162)
* Remove unneeded logging

* Add MasterKey & SelfSigningKey to update
Avoid panic if signatures are not present

* Add passing test

* Revert "Add MasterKey & SelfSigningKey to update"

This reverts commit 2c81b34884.

* Send MasterKey & SelfSigningKey with update

* Debugging

* Remove delete() so we also query signingkeys
2022-02-09 13:11:43 +01:00
Neil Alexander cf447dd52a
Revert "Fix storage bug in PSQL events table"
This reverts commit b4687f2ed2.
2022-02-09 11:41:21 +00:00
Neil Alexander b4687f2ed2
Fix storage bug in PSQL events table 2022-02-09 11:24:49 +00:00
S7evinK 2771d93748
Remove OutputKeyChangeEvent consumer on keyserver (#2160)
* Remove keyserver consumer

* Remove keyserver from eduserver

* Directly upload device keys without eduserver

* Add passing tests
2022-02-08 18:13:38 +01:00
Neil Alexander 457a07eac5
More relaxed auth event fetching (#2161)
* Tweaks around auth event fetching

* More tweaking
2022-02-08 17:06:13 +00:00