Commit graph

241 commits

Author SHA1 Message Date
Till Faelligen 2b496be2c3 Merge branch 'main' of github.com:matrix-org/dendrite into s7evink/consent-tracking 2022-04-20 17:26:27 +02:00
fcwoknhenuxdfiyv-nextcloud abf71649b0
Make sure resp.Username is defined before hashing. Fixes #2356 (#2357)
Co-authored-by: Jason Quigley <jason@onecha.net>
2022-04-19 09:46:54 +01:00
Till e8dd37d533
Add metrics for internal API requests (#2310)
* Add response size and requests total to internal handler

* Move MustRegister calls to New* funcs

* Move MustRegister back to init

* Init at some place, minimize changes
2022-04-08 12:24:40 +02:00
Neil Alexander b8c97431b9
Prevent setting push rules via account data endpoint (#2333) 2022-04-08 10:46:23 +01:00
Neil Alexander 2dc35403d1
Send avatar updates asynchronously, same as display name updates 2022-04-07 15:50:42 +01:00
Till 60ee7eef4c
Add possibility to ignore users (#2329)
* Add ignore users

* Ignore users in pushrules
Add passing tests

* Update sytest lists

* Store ignore knowledge in the sync API

* Fix copyrights

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-07 15:08:19 +01:00
Neil Alexander 99ef547295
Simplify presence stringification (should help with vector-im/element-android#5712) 2022-04-07 10:10:28 +01:00
Till e5e3350ce1
Add presence module V2 (#2312)
* Syncapi presence

* Clientapi http presence handler

* Why is this here?

* Missing files

* FederationAPI presence implementation

* Add new presence stream

* Pinecone update

* Pinecone update

* Add passing tests

* Make linter happy

* Add presence producer

* Add presence config option

* Set user to unavailable after x minutes

* Only set currently_active if online
Avoid unneeded presence updates when syncing

* Tweaks

* Query devices for last_active_ts
Fixes & tweaks

* Export SharedUsers/SharedUsers

* Presence stream in MemoryStorage

* Remove status_msg_nil

* Fix sytest crashes

* Make presence types const and use stringer for it

* Change options to allow inbound/outbound presence

* Fix option & typo

* Update configs

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-06 13:11:19 +02:00
David Spenler 2defc4249d
Added /upgrade endpoint (#2307)
* Added /upgrade endpoint

* fix

* Fix lints

* More lint lifex

* Move room upgrading to the roomserver

* Remove extraneous arg

* Fix HTTP API for `PerformUpgrade`

* Reduce number of API calls in `generateInitialEvents`, preserve membership fields

* Refactor `generateInitialEvents` to preserve old state events for all but the essential room setup events

* Handle ban events in the state transfer

* Refactor and comment `createTemporaryPowerLevels`

* Only send two power levels if we needed to override the levels, preserve miscellaneous fields in the create event

* Fix copyrights

* Review comments @S7evinK

* Update sytest whitelist

* Specify empty state keys, use `EventLevel`, remove unnecessary check on state copy

* Add comment to `restrictOldRoomPowerLevels`

* Ensure canonical aliases exist before clearing

* Copy invites as well as bans

* Fix return error on `m.room.tombstone` handling in client API

* Relax checks for well-formedness of join rules, membership event etc

Co-authored-by: Alex Kursell <alex@awk.run>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: kegsay <kegan@matrix.org>
2022-04-05 10:04:08 +01:00
S7evinK 49dc49b232
Remove eduserver (#2306)
* Move receipt sending to own JetStream producer

* Move SendToDevice to producer

* Remove most parts of the EDU server

* Fix SendToDevice & copyrights

* Move structs, cleanup EDU Server traces

* Use HeadersOnly subscription

* Missing file

* Fix linter issues

* Move consumers to own files

* Rename durable consumer; Consumer cleanup

* Docs/config cleanup
2022-03-29 14:14:35 +02:00
Neil Alexander 7972915806
User directory for nearby Pinecone peers (P2P demo) (#2311)
* User directory for nearby Pinecone peers

* Fix mux routing

* Use config to determine which server notices user to exclude
2022-03-28 16:25:26 +01:00
Neil Alexander ceb3874469
Allow stored session parameters to be overwritten in the registration request (#2309)
* Allow stored session parameters to be overwritten in the registration request

* Remove logging

* Close request body

* Use `httputil.UnmarshalJSON` as that should enforce UTF-8 correctness

* Return `M_NOT_JSON` on read error

* Whoops, return the value of `httputil.UnmarshalJSON`

* Remove redundant comment
2022-03-28 10:47:35 +01:00
S7evinK f2e550efd8
Refactor appservice & client API to use userapi internal (#2290)
* Refactor user api internal

* Refactor clientapi to use internal userapi

* Use internal userapi instead of user DB directly

* Remove AccountDB dependency

* Fix linter issues

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-03-24 22:45:44 +01:00
Till Faelligen e42ef1706b Merge branch 's7evink/consent-tracking' of github.com:matrix-org/dendrite into s7evink/consent-tracking 2022-03-16 08:43:12 +01:00
Till Faelligen 31ac3ac081 Use DefaultRoomVersion as roomVersion 2022-03-16 08:42:04 +01:00
kegsay 39d9d88b02
Merge branch 'main' into s7evink/consent-tracking 2022-03-09 10:05:50 +00:00
S7evinK 05fa66c9c8
Fix appservice username check (#2223)
* Fix appservice username check

* Flakey test moved to blocklist

* Move tests to blacklist
2022-03-07 18:14:08 +01:00
S7evinK 9fbaa1194b
Add canonical alias support (#2236)
* Add canonical support

* Add test

* Check that the send event is actually an m.room.canonical_alias
Check that we got an event from the database

* Update to get correct required events

* Add flakey test to blacklist
2022-03-07 10:37:04 +01:00
Till Faelligen dcfc0bcd43 URL Encode, use new method to get server notice room 2022-03-07 09:45:59 +01:00
Till Faelligen 519ea13510 Add AuthAPICheck and optional functional checks
Rename several variables
2022-03-04 17:01:18 +01:00
Till Faelligen e80ca307d3 Fix receipts 2022-03-04 09:59:15 +01:00
Till Faelligen df7218e230 Fix parameters 2022-03-04 09:30:46 +01:00
Till Faelligen e6e62497c9 Merge branch 'main' of github.com:matrix-org/dendrite into s7evink/consent-tracking 2022-03-04 09:18:34 +01:00
Neil Alexander f75169c353
Send profile updates asynchronously (#2246) 2022-03-03 18:24:14 +00:00
Neil Alexander 43ab0288f4
Give more time to TestSessionCleanUp tests 2022-03-03 12:37:12 +00:00
Neil Alexander bcc27e9e18
Only store notifications for users with pushers, de-parallelise TestSessionCleanUp for now 2022-03-03 12:01:19 +00:00
Dan f05ce478f0
Implement Push Notifications (#1842)
* Add Pushserver component with Pushers API

Co-authored-by: Tommie Gannert <tommie@gannert.se>
Co-authored-by: Dan Peleg <dan@globekeeper.com>

* Wire Pushserver component

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>

* Add PushGatewayClient.

The full event format is required for Sytest.

* Add a pushrules module.

* Change user API account creation to use the new pushrules module's defaults.

Introduces "scope" as required by client API, and some small field
tweaks to make some 61push Sytests pass.

* Add push rules query/put API in Pushserver.

This manipulates account data over User API, and fires sync messages
for changes. Those sync messages should, according to an existing TODO
in clientapi, be moved to userapi.

Forks clientapi/producers/syncapi.go to pushserver/ for later extension.

* Add clientapi routes for push rules to Pushserver.

A cleanup would be to move more of the name-splitting logic into
pushrules.go, to depollute routing.go.

* Output rooms.join.unread_notifications in /sync.

This is the read-side. Pushserver will be the write-side.

* Implement pushserver/storage for notifications.

* Use PushGatewayClient and the pushrules module in Pushserver's room consumer.

* Use one goroutine per user to avoid locking up the entire server for
  one bad push gateway.
* Split pushing by format.
* Send one device per push. Sytest does not support coalescing
  multiple devices into one push. Matches Synapse. Either we change
  Sytest, or remove the group-by-url-and-format logic.
* Write OutputNotificationData from push server. Sync API is already
  the consumer.

* Implement read receipt consumers in Pushserver.

Supports m.read and m.fully_read receipts.

* Add clientapi route for /unstable/notifications.

* Rename to UpsertPusher for clarity and handle pusher update

* Fix linter errors

* Ignore body.Close() error check

* Fix push server internal http wiring

* Add 40 newly passing 61push tests to whitelist

* Add next 12 newly passing 61push tests to whitelist

* Send notification data before notifying users in EDU server consumer

* NATS JetStream

* Goodbye sarama

* Fix `NewStreamTokenFromString`

* Consume on the correct topic for the roomserver

* Don't panic, NAK instead

* Move push notifications into the User API

* Don't set null values since that apparently causes Element upsetti

* Also set omitempty on conditions

* Fix bug so that we don't override the push rules unnecessarily

* Tweak defaults

* Update defaults

* More tweaks

* Move `/notifications` onto `r0`/`v3` mux

* User API will consume events and read/fully read markers from the sync API with stream positions, instead of consuming directly

Co-authored-by: Piotr Kozimor <p1996k@gmail.com>
Co-authored-by: Tommie Gannert <tommie@gannert.se>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-03-03 11:40:53 +00:00
S7evinK cda2452ba0
Only allow device deletion from session UIA was initiated from (#2235)
* Only allow device deletion if the session matches

* Make the challenge response available to other packages

* Remove userID, as it's not in the spec

* Remove tests

* Add passing test & remove obsolete config

* Rename field, add comment

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-03-01 16:39:57 +00:00
Till Faelligen 2ad15f308f Rename some functions 2022-02-25 15:32:19 +01:00
Till Faelligen ed16a2f107 Merge branch 'main' of github.com:matrix-org/dendrite into s7evink/consent-tracking 2022-02-25 15:21:06 +01:00
S7evinK cf27e26712
Remember parameters on registration (#2225)
* Remember parameters for sessions
Cleanup sessions on successfully registering or after a while

* Add flakey test

* Update to use time.AfterFunc, add more tests

* Try to drain the channel, if possible
2022-02-25 14:33:02 +01:00
Till Faelligen 79e1c9e4bd Merge branch 'main' of github.com:matrix-org/dendrite into s7evink/consent-tracking 2022-02-24 08:47:38 +01:00
S7evinK cfff1b0aaa
Remote banned user is kicked and may not rejoin until unbanned (#2216)
* Remote banned user is kicked and may not rejoin until unbanned

* Use gmsl constant
2022-02-22 15:50:04 +00:00
Till Faelligen 4f2d161401 Remove consentMux 2022-02-21 17:09:25 +01:00
Till Faelligen c2b6019c35 Add possibility to query all membership states 2022-02-21 16:40:59 +01:00
Till Faelligen 61cdb714df Use typed values for Consent 2022-02-21 16:23:28 +01:00
Till Faelligen c0845ea1ad Update logging 2022-02-21 14:44:16 +01:00
Till Faelligen 6622fda08c Add sending server notices on startup 2022-02-21 14:27:59 +01:00
Till Faelligen 9c3a1cfd47 Merge branch 'main' of github.com:matrix-org/dendrite into s7evink/consent-tracking 2022-02-21 12:08:03 +01:00
S7evinK 002429c9e2
Implement server notices (#2180)
* Add server_notices config

* Disallow rejecting "server notice" invites

* Update config

* Slightly refactor sendEvent and CreateRoom so it can be reused

* Implement unspecced server notices

* Validate the request

* Set the user api when starting

* Rename function/variables

* Update comments

* Update config

* Set the avatar on account creation

* Update test

* Only create the account when starting
Only add routes if sever notices are enabled

* Use reserver username
Check that we actually got roomData

* Add check for admin account
Enable server notices for CI
Return same values as Synapse

* Add custom error for rejecting server notice invite

* Move building an invite to it's own function, for reusability

* Don't create new rooms, use the existing one (follow Synapse behavior)

Co-authored-by: kegsay <kegan@matrix.org>
2022-02-18 16:05:03 +01:00
Neil Alexander 153bfbbea5
Merge both user API databases into one (#2186)
* Merge user API databases into one

* Remove DeviceDatabase from config

* Fix tests

* Try that again

* Clean up keyserver device keys when the devices no longer exist in the user API

* Tweak ordering

* Fix UserExists flag, device check

* Allow including empty entries so we can clean them up

* Remove logging
2022-02-18 11:31:05 +00:00
kegsay 0a7dea4450
Update /whoami response to match Spec v1.2 (#2201)
Basically include `is_guest` and `device_id`. The latter is
needed for https://github.com/matrix-org/complement/pull/305
2022-02-18 11:28:02 +00:00
Neil Alexander e1eb5807b6
Allow preventing guest registration (#2199)
* Allow disabling guest registration separately

* Update sample config

* Set `guests_disabled` to `true` in the sample config
2022-02-18 10:12:26 +00:00
kegsay 5dd203fde3
Listen for /v3 on CSAPI (#2197)
* Listen for /v3 on CSAPI

* Docs

* More docs

* Rename path variable to fix key backup tests

* Update routing.go

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-02-17 17:38:22 +00:00
S7evinK 5a39512f5f
Add account type (#2171)
* Add account_type for sqlite3

* Add account_type for postgres

* Remove CreateGuestAccount from interface

* Add new AccountTypes & update test

* Use newly added AccountType for account creation

* Add migrations

* Reuse type

* Add AccounnType to Device, so it can be verified on requests

* Rename migration, add missing update for appservices

* Rename sqlite3 migration

* Add missing AccountType to return value

* Update sqlite migration
Change allowance check on /admin/whois

* Fix migration, add IS NULL

* Move accountType to completeRegistration

* Fix migrations

* Add passing test
2022-02-16 18:55:38 +01:00
Till Faelligen 74da1f0fb3 Remove "magic" Enabled function and use simple bool 2022-02-16 10:11:23 +01:00
Till Faelligen 535d388ec0 Add new login type "m.login.terms" 2022-02-15 14:14:39 +01:00
Till Faelligen d19518fca5 Add ConsentNotGiven error
Verify consent on desired endpoints
Store consent on POST requests
2022-02-15 11:07:24 +01:00
Till Faelligen 89340cfc52 Verify the user has given their consent, otherwise block access 2022-02-14 18:11:56 +01:00
Till Faelligen 11144de92f Implement consent tracking 2022-02-14 16:18:51 +01:00