Commit graph

241 commits

Author SHA1 Message Date
PiotrKozimor 891c8f7db4
Remove io/ioutil package (#24) 2022-08-11 10:56:02 +02:00
Piotr Kozimor d1f906436e Allow for http pusher scheme 2022-08-09 12:39:56 +02:00
Piotr Kozimor 9aceb04b98 Run gofmt over the code 2022-08-05 10:42:14 +02:00
PiotrKozimor da6a8b6f89
Allow to get event by ID if membership is joined or invited (#20)
* Allow to get event by ID if membership is joined or invited

* Fix linter errors

* Use own fork of complement
2022-08-04 16:39:58 +02:00
PiotrKozimor 15cfeb16aa
Upstream release v0.9.0 (#18)
* Correctly redact events over federation (#2526)

* Ensure we check powerlevel/origin before redacting an event

* Add passing test

* Use pl.UserLevel

* Make check more readable, also check for the sender

* Add new next steps page to the documentation

* Highlighting in docs

* Rename the page to "Optimise your installation"

* Attempt to raise the file descriptor limit at startup (#2527)

* Add `--difference` to `resolve-state` tool

* Make the linter happy again

* generic CaddyFile in front of Dendrite (monolith) (#2531)

for Caddy 2.5.x

Co-authored-by: emanuele.aliberti <emanuele.aliberti@mtka.eu>

* Handle state before, send history visibility in output (#2532)

* Check state before event

* Tweaks

* Refactor a bit, include in output events

* Don't waste time if soft failed either

* Tweak control flow, comments, use GMSL history visibility type

* Fix rare panic when returning user devices over federation (#2534)

* Add `InputDeviceListUpdate` to the keyserver, remove old input API (#2536)

* Add `InputDeviceListUpdate` to the keyserver, remove old input API

* Fix copyright

* Log more information when a device list update fails

* Fix nats.go commit (#2540)

Signed-off-by: Jean Lucas <jean@4ray.co>

* Don't return `end` if there are not more messages (#2542)

* Be more spec compliant

* Move lazyLoadMembers to own method

* Return an error if trying to invite a malformed user ID (#2543)

* Add `evacuateUser` endpoint, use it when deactivating accounts (#2545)

* Add `evacuateUser` endpoint, use it when deactivating accounts

* Populate the API

* Clean up user devices when deactivating

* Include invites, delete pushers

* Silence presence logs (#2547)

* Blacklist `Guest users can join guest_access rooms` test until it can be investigated

* Disable WebAssembly builds for now

* Try to fix backfilling (#2548)

* Try to fix backfilling

* Return start/end to not confuse clients

* Update GMSL

* Update GMSL

* Roomserver producers package (#2546)

* Give the roomserver a producers package

* Change init point

* Populate ACLs API

* Fix build issues

* `RoomEventProducer` naming

* Version 0.8.9 (#2549)

* Version 0.8.9

* Update changelog

* feat+fix: Ignore unknown keys and verify required fields are present in appservice registration files (#2550)

* fix: ignore unknown keys in appservice configs

fixes matrix-org/dendrite#1567

* feat: verify required fields in appservice configs

* Use new testrig for key changes tests (#2552)

* Use new testrig for tests

* Log the error message

* Fix QuerySharedUsers for the SyncAPI keychange consumer (#2554)

* Make more use of base.BaseDendrite

* Fix QuerySharedUsers if no UserIDs are supplied

* Return clearer error when no state NID exists for an event (#2555)

* Wrap error from `SnapshotNIDFromEventID`

* Hopefully fix read receipts timestamps (#2557)

This should avoid coercions between signed and unsigned ints which might fix problems like `sql: converting argument $5 type: uint64 values with high bit set are not supported`.

* Fix nil pointer access when redacting events (#2560)

* Fix issue `uint64 values with high bit are not supported` in presence (#2562)

* Fix issue #2528

* Use gomatrixserverlib.Timestamp

* Use ParseUint instead of ParseInt

* Update Pinecone to matrix-org/pinecone@1ce778f

* Ristretto cache (#2563)

* Try Ristretto cache

* Tweak

* It's beautiful

* Update GMSL

* More strict keyable interface

* Fix that some more

* Make less panicky

* Don't enforce mutability checks for now

* Determine mutability using deep equality

* Tweaks

* Namespace keys

* Make federation caches mutable

* Update cost estimation, add metric

* Update GMSL

* Estimate cost for metrics better

* Reduce counters a bit

* Try caching events

* Some guards

* Try again

* Try this

* Use separate caches for hopefully better hash distribution

* Fix bug with admitting events into cache

* Try to fix bugs

* Check nil

* Try that again

* Preserve order jeezo this is messy

* thanks VS Code for doing exactly the wrong thing

* Try this again

* Be more specific

* aaaaargh

* One more time

* That might be better

* Stronger sorting

* Cache expiries, async publishing of EDUs

* Put it back

* Use a shared cache again

* Cost estimation fixes

* Update ristretto

* Reduce counters a bit

* Clean up a bit

* Update GMSL

* 1GB

* Configurable cache sizees

* Tweaks

* Add `config.DataUnit` for specifying friendly cache sizes

* Various tweaks

* Update GMSL

* Add back some lazy loading caching

* Include key in cost

* Include key in cost

* Tweak max age handling, config key name

* Only register prometheus metrics if requested

* Review comments @S7evinK

* Don't return errors when creating caches (it is better just to crash since otherwise we'll `nil`-pointer exception everywhere)

* Review comments

* Update sample configs

* Update GHA Workflow

* Update Complement images to Go 1.18

* Remove the cache test from the federation API as we no longer guarantee immediate cache admission

* Don't check the caches in the renewal test

* Possibly fix the upgrade tests

* Update to matrix-org/gomatrixserverlib#322

* Update documentation to refer to Go 1.18

* Minor SendToDevice fix (#2565)

* Avoid unnecessary marshalling if sending to the local server

* Fix ordering of ToDevice messages

* Revive SendToDevice test

* Use `/v3` to request media from remote servers (update to matrix-org/gomatrixserverlib#324)

* Pointerise `types.RoomInfo` in the cache so we can update it in-place in the latest events updater

* Add a Troubleshooting page

* Update `sytest-whitelist`

* Use sync API database in `filterSharedUsers` (#2572)

* Add function to the sync API storage package for filtering shared users

* Use the database instead of asking the RS API

* Fix unit tests

* Fix map handling in `filterSharedUsers`

* Update 1_createusers.md (#2571)

* Update 1_createusers.md

Added description on how to create user accounts when running in docker.

* Update 1_createusers.md

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>

* Fix connection_string format in dendrite-sample.polylith.yaml (#2574)

* History visibility database changes (#2533)

* Add new history_visibility column

* Update SQL queries to include history_visibility

* Store the history visibilty calculated by the roomserver

* Update GMSL

* Update migrations

* Fix migration

* Update GMSL

* Fix `go.sum`

* Update GMSL to use sql.Scanner & sql.Valuer

* Re-order migration/table creation

* Update gomatrixserverlib

* Add history_visibility column to current_room_state

* Fix migrations

* Return error instead of Fatal log

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>

* Tweak cache counters (#2575)

* Tweak cache counters

This makes the number of counters relative to the
maximum cache size. Since the counters
effectively manage the size of the bloom filter,
larger caches need more counters and smaller
caches need less.

10 counters per 1KB data means that the default
cache size of 1GB should result in a bloom filter
and TinyLRU admission set of about 16MB
estimated.

* Remove line left by accident

* Set historyVisibility in rowsToStreamEvents

* Update FAQ

* Add event state key cache (#2576)

* Explain how SRV works in Matrix and discourage using it (#2577)

* Explain how SRV works in Matrix and discourage using it

* Minor tweaks to formatting

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>

* Fix issue with membership event_nid being 0 (#2580)

* docs: Add build page; correct proxy info; fix Caddy example (#2579)

* Add build page; correct proxy info; fix Caddy example

* Improve Caddyfile example

* Apply review comments; add polylith Caddyfile

* Bump tzinfo from 1.2.9 to 1.2.10 in /docs (#2584)

Bumps [tzinfo](https://github.com/tzinfo/tzinfo) from 1.2.9 to 1.2.10.
- [Release notes](https://github.com/tzinfo/tzinfo/releases)
- [Changelog](https://github.com/tzinfo/tzinfo/blob/master/CHANGES.md)
- [Commits](https://github.com/tzinfo/tzinfo/compare/v1.2.9...v1.2.10)

---
updated-dependencies:
- dependency-name: tzinfo
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Membership updater refactoring (#2541)

* Membership updater refactoring

* Pass in membership state

* Use membership check rather than referring to state directly

* Delete irrelevant membership states

* We don't need the leave event after all

* Tweaks

* Put a log entry in that I might stand a chance of finding

* Be less panicky

* Tweak invite handling

* Don't freak if we can't find the event NID

* Use event NID from `types.Event`

* Clean up

* Better invite handling

* Placate the almighty linter

* Blacklist a Sytest which is otherwise fine under Complement for reasons I don't understand

* Fix the sytest after all (thanks @S7evinK for the spot)

* Try to fix HTTP 500s on `/members` (#2581)

* Update database migrations, remove goose (#2264)

* Add new db migration

* Update migrations
Remove goose

* Add possibility to test direct upgrades

* Try to fix WASM test

* Add checks for specific migrations

* Remove AddMigration
Use WithTransaction
Add Dendrite version to table

* Fix linter issues

* Update tests

* Update comments, outdent if

* Namespace migrations

* Add direct upgrade tests, skipping over one version

* Split migrations

* Update go version in CI

* Fix copy&paste mistake

* Use contexts in migrations

Co-authored-by: kegsay <kegan@matrix.org>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>

* Add .well-known/matrix/client to clientapi (#2551)

Signed-off-by: Jonathan Bartlett <jonathan@jonnobrow.co.uk>

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>

* Remove `room_id` field from MSC2946 stripped events (closes #2588)

* Remove `goose` from Dockerfiles

* Make the User API responsible for sending account data output events (#2592)

* Make the User API responsible for sending account data output events

* Clean up producer

* Review comments

* Update NATS Server and nats.go to use upstream

* Set CORS headers for HTTP 404 and 405 errors (#2599)

* Set CORS headers for the 404s

* Use custom handlers, plus one for HTTP 405 too

* Tweak setup

* Add to muxes too

* Tidy up some more

* Use built-in HTTP 404 handler

* Don't bother setting it for federation-facing

* Optimise checking other servers allowed to see events (#2596)

* Try optimising checking if server is allowed to see event

* Fix error

* Handle case where snapshot NID is 0

* Fix query

* Update SQL

* Clean up `CheckServerAllowedToSeeEvent`

* Not supported on SQLite

* Maybe placate the unit tests

* Review comments

* De-race `types.RoomInfo` (#2600)

* De-race `CompleteSync` (#2601)

The `err` was coming from outside of the goroutine and being written to by concurrent goroutines.

* Version 0.9.0 (#2602)

Co-authored-by: Till <2353100+S7evinK@users.noreply.github.com>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: Till Faelligen <davidf@element.io>
Co-authored-by: Emanuele Aliberti <dev@mtka.eu>
Co-authored-by: emanuele.aliberti <emanuele.aliberti@mtka.eu>
Co-authored-by: Jean Lucas <jean@4ray.co>
Co-authored-by: Kabir Kwatra <kabir@kwatra.me>
Co-authored-by: andreever <52261463+andreever@users.noreply.github.com>
Co-authored-by: Maximilian Gaedig <38767445+MaximilianGaedig@users.noreply.github.com>
Co-authored-by: Tulir Asokan <tulir@maunium.net>
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kegsay <kegan@matrix.org>
Co-authored-by: Jonathan Bartlett <34320158+Jonnobrow@users.noreply.github.com>
2022-08-03 13:35:29 +02:00
PiotrKozimor 507d3dc650
Allow invited users to fetch state events and events by ID (#17)
* Allow invited users to fetch state events and events by ID

* Move failing test case to blacklist

* Fix /sync panicking due to presence udpates

* Use upload-artifact v3

* Yet another attempt to fix sytest artefacts

* Maybe this will work?
2022-08-02 15:39:10 +02:00
Piotr Kozimor 1883126657 Allow to clear avatar url 2022-07-19 10:30:58 +02:00
PiotrKozimor 7823481a0e
Merge v0.8.9 (#13)
Squashed commit of the following:

commit b5c55faf98
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Fri Jul 1 12:00:32 2022 +0100

    Version 0.8.9 (#2549)

    * Version 0.8.9

    * Update changelog

commit b50a24c666
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Fri Jul 1 10:54:07 2022 +0100

    Roomserver producers package (#2546)

    * Give the roomserver a producers package

    * Change init point

    * Populate ACLs API

    * Fix build issues

    * `RoomEventProducer` naming

commit 89cd0e8fc1
Author: Till <2353100+S7evinK@users.noreply.github.com>
Date:   Fri Jul 1 11:49:26 2022 +0200

    Try to fix backfilling (#2548)

    * Try to fix backfilling

    * Return start/end to not confuse clients

    * Update GMSL

    * Update GMSL

commit 086f182e24
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Fri Jul 1 09:50:06 2022 +0100

    Disable WebAssembly builds for now

commit 54bed4c593
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Fri Jul 1 09:37:54 2022 +0100

    Blacklist `Guest users can join guest_access rooms` test until it can be investigated

commit 561c159ad7
Author: Till <2353100+S7evinK@users.noreply.github.com>
Date:   Thu Jun 30 12:34:37 2022 +0200

    Silence presence logs (#2547)

commit 519bc1124b
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Jun 29 15:29:39 2022 +0100

    Add `evacuateUser` endpoint, use it when deactivating accounts (#2545)

    * Add `evacuateUser` endpoint, use it when deactivating accounts

    * Populate the API

    * Clean up user devices when deactivating

    * Include invites, delete pushers

commit 2dea466685
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Jun 29 12:32:24 2022 +0100

    Return an error if trying to invite a malformed user ID (#2543)

commit 2086992caf
Author: Till <2353100+S7evinK@users.noreply.github.com>
Date:   Wed Jun 29 10:49:12 2022 +0200

    Don't return `end` if there are not more messages (#2542)

    * Be more spec compliant

    * Move lazyLoadMembers to own method

commit 920a20821b
Author: Jean Lucas <jean@4ray.co>
Date:   Mon Jun 27 04:15:19 2022 -0400

    Fix nats.go commit (#2540)

    Signed-off-by: Jean Lucas <jean@4ray.co>

commit 7120eb6bc9
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Jun 15 14:27:07 2022 +0100

    Add `InputDeviceListUpdate` to the keyserver, remove old input API (#2536)

    * Add `InputDeviceListUpdate` to the keyserver, remove old input API

    * Fix copyright

    * Log more information when a device list update fails

commit 1b90cc9536
Author: Till <2353100+S7evinK@users.noreply.github.com>
Date:   Wed Jun 15 12:50:02 2022 +0200

    Fix rare panic when returning user devices over federation (#2534)

commit 4c2a10f1a6
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 13 15:11:10 2022 +0100

    Handle state before, send history visibility in output (#2532)

    * Check state before event

    * Tweaks

    * Refactor a bit, include in output events

    * Don't waste time if soft failed either

    * Tweak control flow, comments, use GMSL history visibility type

commit c500958583
Author: Emanuele Aliberti <dev@mtka.eu>
Date:   Mon Jun 13 13:08:46 2022 +0200

    generic CaddyFile in front of Dendrite (monolith) (#2531)

    for Caddy 2.5.x

    Co-authored-by: emanuele.aliberti <emanuele.aliberti@mtka.eu>

commit e1136f4d3e
Author: Till Faelligen <davidf@element.io>
Date:   Mon Jun 13 11:46:59 2022 +0200

    Make the linter happy again

commit 0a7f7dc716
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 13 10:16:30 2022 +0100

    Add `--difference` to `resolve-state` tool

commit 89d2adadbd
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Fri Jun 10 10:58:04 2022 +0100

    Attempt to raise the file descriptor limit at startup (#2527)

commit 1030072285
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Fri Jun 10 10:18:32 2022 +0100

    Rename the page to "Optimise your installation"

commit 16ed1633b6
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Fri Jun 10 10:15:14 2022 +0100

    Highlighting in docs

commit e2a64773ce
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Fri Jun 10 10:14:15 2022 +0100

    Add new next steps page to the documentation

commit 660f7839f5
Author: Till <2353100+S7evinK@users.noreply.github.com>
Date:   Thu Jun 9 18:38:07 2022 +0200

    Correctly redact events over federation (#2526)

    * Ensure we check powerlevel/origin before redacting an event

    * Add passing test

    * Use pl.UserLevel

    * Make check more readable, also check for the sender
2022-07-04 17:39:22 +02:00
PiotrKozimor 374b77a3df
Entry improvements (#11)
* Refactor ApplicationServiceWorkerState to be more robust

* Add launch.json to VS Code

* Implement login with JWT, registering with email, failed login rate limiting and reset password with m.login.email.identity auth type

* Log errors when JWT parsing failed

* Development build script

* Fix linter errors

* Use golangci-lint as a linter in VS Code

* Fix tests with RtFailedLogin

* Pass config load tests - parse JWT public key only if enabled

* Reduce CI steps

Do not support 386 arch and go 1.16, 1.17

* Fix linter errors

* Change RtFailedLogin logic - nil pointer can be provided

* Respect access token in query

* Fix typos

* Use only one mutex in RtFailedLogin

* Remove eventsRemaining across appservice component

* Push dendrite to production registry as well

* Rafactor TestRtFailedLogin
2022-06-30 14:56:45 +02:00
Till 289b3c5608
Allow "registration is idempotent, with username specified" to pass (#2488)
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-06-09 11:26:48 +01:00
Neil Alexander 6d4bd5d890
Rate limiting changes (#2519)
* Rate limiting changes

This makes the following changes:

* For logged in users, the rate limiting now applies to the device session rather than the remote IP address;
* For non-logged in users, the rate limiting continues to apply to remote address as it does today;
* It is now possible to add user IDs to the `exempt_user_ids` option under `rate_limiting` to exclude bots from rate limiting;
* Admin and appservice users are now exempt from rate limiting by default.

* Fix build with media API
2022-06-07 14:24:04 +01:00
Till 3e9c734da5
Make setting state idempotent (#2512)
* Make Setting state twice is idempotent pass

* Add passing tests

* PR comment & comments
2022-06-03 06:43:51 +02:00
Neil Alexander ea16614f71
Update versions 2022-06-01 09:39:17 +01:00
Neil Alexander ead0112aa1
Fix join_authorised_via_users_server key name in SendEvent 2022-05-24 10:22:26 +01:00
Neil Alexander 7379b02b70
Give PL100 to invited users in trusted_private_chat preset (#2485) 2022-05-24 10:10:25 +01:00
Neil Alexander c83837e684
Strip join_authorised_by_users_server when updating membership events using /rooms/{roomID}/state (#2482) 2022-05-24 09:11:23 +01:00
kegsay c15bfefd0d
Add RoomExists flag to QueryMembershipForUser (#2450)
Fixes https://github.com/matrix-org/complement/pull/369
2022-05-11 11:29:23 +01:00
kegsay 85704eff20
Clean up interface definitions (#2427)
* tidy up interfaces

* remove unused GetCreatorIDForAlias

* Add RoomserverUserAPI interface

* Define more interfaces

* Use AppServiceInternalAPI for consistent naming

* clean up federationapi constructor a bit

* Fix monolith in -http mode
2022-05-06 12:39:26 +01:00
kegsay 9957752a9d
Define component interfaces based on consumers (2/2) (#2425)
* convert remaining interfaces

* Tidy up the userapi interfaces
2022-05-05 19:30:38 +01:00
kegsay 506de4bb3d
Define component interfaces based on consumers (1/2) (#2423)
* Specify interfaces used by appservice, do half of clientapi

* convert more deps of clientapi to finer-grained interfaces

* Convert mediaapi and rest of clientapi

* Somehow this got missed
2022-05-05 13:17:38 +01:00
Till 1bfe87aa56
Fix user already joined when using server notices (#2364) 2022-05-05 12:01:28 +02:00
Neil Alexander 2a4517f8e6
Move admin functions into their own file in the client API 2022-04-29 09:10:08 +01:00
Neil Alexander c6ea2c9ff2
Add /_dendrite/admin/evacuateRoom/{roomID} (#2401)
* Add new endpoint to allow admins to evacuate the local server from the room

* Guard endpoint

* Use right prefix

* Auth API

* More useful return error rather than a panic

* More useful return value again

* Update the path

* Try using inputer instead

* oh provide the config

* Try that again

* Return affected user IDs

* Don't create so many forward extremities

* Add missing `Path` to name

Co-authored-by: Till <2353100+S7evinK@users.noreply.github.com>
2022-04-28 16:02:30 +01:00
Till feac9db43f
Add transactionsCache to redact endpoint (#2375) 2022-04-26 10:28:41 +02:00
fcwoknhenuxdfiyv-nextcloud abf71649b0
Make sure resp.Username is defined before hashing. Fixes #2356 (#2357)
Co-authored-by: Jason Quigley <jason@onecha.net>
2022-04-19 09:46:54 +01:00
Till e8dd37d533
Add metrics for internal API requests (#2310)
* Add response size and requests total to internal handler

* Move MustRegister calls to New* funcs

* Move MustRegister back to init

* Init at some place, minimize changes
2022-04-08 12:24:40 +02:00
Neil Alexander b8c97431b9
Prevent setting push rules via account data endpoint (#2333) 2022-04-08 10:46:23 +01:00
Neil Alexander 2dc35403d1
Send avatar updates asynchronously, same as display name updates 2022-04-07 15:50:42 +01:00
Till 60ee7eef4c
Add possibility to ignore users (#2329)
* Add ignore users

* Ignore users in pushrules
Add passing tests

* Update sytest lists

* Store ignore knowledge in the sync API

* Fix copyrights

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-07 15:08:19 +01:00
Neil Alexander 99ef547295
Simplify presence stringification (should help with vector-im/element-android#5712) 2022-04-07 10:10:28 +01:00
Till e5e3350ce1
Add presence module V2 (#2312)
* Syncapi presence

* Clientapi http presence handler

* Why is this here?

* Missing files

* FederationAPI presence implementation

* Add new presence stream

* Pinecone update

* Pinecone update

* Add passing tests

* Make linter happy

* Add presence producer

* Add presence config option

* Set user to unavailable after x minutes

* Only set currently_active if online
Avoid unneeded presence updates when syncing

* Tweaks

* Query devices for last_active_ts
Fixes & tweaks

* Export SharedUsers/SharedUsers

* Presence stream in MemoryStorage

* Remove status_msg_nil

* Fix sytest crashes

* Make presence types const and use stringer for it

* Change options to allow inbound/outbound presence

* Fix option & typo

* Update configs

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-06 13:11:19 +02:00
David Spenler 2defc4249d
Added /upgrade endpoint (#2307)
* Added /upgrade endpoint

* fix

* Fix lints

* More lint lifex

* Move room upgrading to the roomserver

* Remove extraneous arg

* Fix HTTP API for `PerformUpgrade`

* Reduce number of API calls in `generateInitialEvents`, preserve membership fields

* Refactor `generateInitialEvents` to preserve old state events for all but the essential room setup events

* Handle ban events in the state transfer

* Refactor and comment `createTemporaryPowerLevels`

* Only send two power levels if we needed to override the levels, preserve miscellaneous fields in the create event

* Fix copyrights

* Review comments @S7evinK

* Update sytest whitelist

* Specify empty state keys, use `EventLevel`, remove unnecessary check on state copy

* Add comment to `restrictOldRoomPowerLevels`

* Ensure canonical aliases exist before clearing

* Copy invites as well as bans

* Fix return error on `m.room.tombstone` handling in client API

* Relax checks for well-formedness of join rules, membership event etc

Co-authored-by: Alex Kursell <alex@awk.run>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: kegsay <kegan@matrix.org>
2022-04-05 10:04:08 +01:00
S7evinK 49dc49b232
Remove eduserver (#2306)
* Move receipt sending to own JetStream producer

* Move SendToDevice to producer

* Remove most parts of the EDU server

* Fix SendToDevice & copyrights

* Move structs, cleanup EDU Server traces

* Use HeadersOnly subscription

* Missing file

* Fix linter issues

* Move consumers to own files

* Rename durable consumer; Consumer cleanup

* Docs/config cleanup
2022-03-29 14:14:35 +02:00
Neil Alexander 7972915806
User directory for nearby Pinecone peers (P2P demo) (#2311)
* User directory for nearby Pinecone peers

* Fix mux routing

* Use config to determine which server notices user to exclude
2022-03-28 16:25:26 +01:00
Neil Alexander ceb3874469
Allow stored session parameters to be overwritten in the registration request (#2309)
* Allow stored session parameters to be overwritten in the registration request

* Remove logging

* Close request body

* Use `httputil.UnmarshalJSON` as that should enforce UTF-8 correctness

* Return `M_NOT_JSON` on read error

* Whoops, return the value of `httputil.UnmarshalJSON`

* Remove redundant comment
2022-03-28 10:47:35 +01:00
S7evinK f2e550efd8
Refactor appservice & client API to use userapi internal (#2290)
* Refactor user api internal

* Refactor clientapi to use internal userapi

* Use internal userapi instead of user DB directly

* Remove AccountDB dependency

* Fix linter issues

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-03-24 22:45:44 +01:00
S7evinK 05fa66c9c8
Fix appservice username check (#2223)
* Fix appservice username check

* Flakey test moved to blocklist

* Move tests to blacklist
2022-03-07 18:14:08 +01:00
S7evinK 9fbaa1194b
Add canonical alias support (#2236)
* Add canonical support

* Add test

* Check that the send event is actually an m.room.canonical_alias
Check that we got an event from the database

* Update to get correct required events

* Add flakey test to blacklist
2022-03-07 10:37:04 +01:00
Neil Alexander f75169c353
Send profile updates asynchronously (#2246) 2022-03-03 18:24:14 +00:00
Neil Alexander 43ab0288f4
Give more time to TestSessionCleanUp tests 2022-03-03 12:37:12 +00:00
Neil Alexander bcc27e9e18
Only store notifications for users with pushers, de-parallelise TestSessionCleanUp for now 2022-03-03 12:01:19 +00:00
Dan f05ce478f0
Implement Push Notifications (#1842)
* Add Pushserver component with Pushers API

Co-authored-by: Tommie Gannert <tommie@gannert.se>
Co-authored-by: Dan Peleg <dan@globekeeper.com>

* Wire Pushserver component

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>

* Add PushGatewayClient.

The full event format is required for Sytest.

* Add a pushrules module.

* Change user API account creation to use the new pushrules module's defaults.

Introduces "scope" as required by client API, and some small field
tweaks to make some 61push Sytests pass.

* Add push rules query/put API in Pushserver.

This manipulates account data over User API, and fires sync messages
for changes. Those sync messages should, according to an existing TODO
in clientapi, be moved to userapi.

Forks clientapi/producers/syncapi.go to pushserver/ for later extension.

* Add clientapi routes for push rules to Pushserver.

A cleanup would be to move more of the name-splitting logic into
pushrules.go, to depollute routing.go.

* Output rooms.join.unread_notifications in /sync.

This is the read-side. Pushserver will be the write-side.

* Implement pushserver/storage for notifications.

* Use PushGatewayClient and the pushrules module in Pushserver's room consumer.

* Use one goroutine per user to avoid locking up the entire server for
  one bad push gateway.
* Split pushing by format.
* Send one device per push. Sytest does not support coalescing
  multiple devices into one push. Matches Synapse. Either we change
  Sytest, or remove the group-by-url-and-format logic.
* Write OutputNotificationData from push server. Sync API is already
  the consumer.

* Implement read receipt consumers in Pushserver.

Supports m.read and m.fully_read receipts.

* Add clientapi route for /unstable/notifications.

* Rename to UpsertPusher for clarity and handle pusher update

* Fix linter errors

* Ignore body.Close() error check

* Fix push server internal http wiring

* Add 40 newly passing 61push tests to whitelist

* Add next 12 newly passing 61push tests to whitelist

* Send notification data before notifying users in EDU server consumer

* NATS JetStream

* Goodbye sarama

* Fix `NewStreamTokenFromString`

* Consume on the correct topic for the roomserver

* Don't panic, NAK instead

* Move push notifications into the User API

* Don't set null values since that apparently causes Element upsetti

* Also set omitempty on conditions

* Fix bug so that we don't override the push rules unnecessarily

* Tweak defaults

* Update defaults

* More tweaks

* Move `/notifications` onto `r0`/`v3` mux

* User API will consume events and read/fully read markers from the sync API with stream positions, instead of consuming directly

Co-authored-by: Piotr Kozimor <p1996k@gmail.com>
Co-authored-by: Tommie Gannert <tommie@gannert.se>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-03-03 11:40:53 +00:00
S7evinK cda2452ba0
Only allow device deletion from session UIA was initiated from (#2235)
* Only allow device deletion if the session matches

* Make the challenge response available to other packages

* Remove userID, as it's not in the spec

* Remove tests

* Add passing test & remove obsolete config

* Rename field, add comment

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-03-01 16:39:57 +00:00
S7evinK cf27e26712
Remember parameters on registration (#2225)
* Remember parameters for sessions
Cleanup sessions on successfully registering or after a while

* Add flakey test

* Update to use time.AfterFunc, add more tests

* Try to drain the channel, if possible
2022-02-25 14:33:02 +01:00
S7evinK cfff1b0aaa
Remote banned user is kicked and may not rejoin until unbanned (#2216)
* Remote banned user is kicked and may not rejoin until unbanned

* Use gmsl constant
2022-02-22 15:50:04 +00:00
S7evinK 002429c9e2
Implement server notices (#2180)
* Add server_notices config

* Disallow rejecting "server notice" invites

* Update config

* Slightly refactor sendEvent and CreateRoom so it can be reused

* Implement unspecced server notices

* Validate the request

* Set the user api when starting

* Rename function/variables

* Update comments

* Update config

* Set the avatar on account creation

* Update test

* Only create the account when starting
Only add routes if sever notices are enabled

* Use reserver username
Check that we actually got roomData

* Add check for admin account
Enable server notices for CI
Return same values as Synapse

* Add custom error for rejecting server notice invite

* Move building an invite to it's own function, for reusability

* Don't create new rooms, use the existing one (follow Synapse behavior)

Co-authored-by: kegsay <kegan@matrix.org>
2022-02-18 16:05:03 +01:00
Neil Alexander 153bfbbea5
Merge both user API databases into one (#2186)
* Merge user API databases into one

* Remove DeviceDatabase from config

* Fix tests

* Try that again

* Clean up keyserver device keys when the devices no longer exist in the user API

* Tweak ordering

* Fix UserExists flag, device check

* Allow including empty entries so we can clean them up

* Remove logging
2022-02-18 11:31:05 +00:00
kegsay 0a7dea4450
Update /whoami response to match Spec v1.2 (#2201)
Basically include `is_guest` and `device_id`. The latter is
needed for https://github.com/matrix-org/complement/pull/305
2022-02-18 11:28:02 +00:00
Neil Alexander e1eb5807b6
Allow preventing guest registration (#2199)
* Allow disabling guest registration separately

* Update sample config

* Set `guests_disabled` to `true` in the sample config
2022-02-18 10:12:26 +00:00
kegsay 5dd203fde3
Listen for /v3 on CSAPI (#2197)
* Listen for /v3 on CSAPI

* Docs

* More docs

* Rename path variable to fix key backup tests

* Update routing.go

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-02-17 17:38:22 +00:00