1
0
Fork 0
mirror of https://github.com/matrix-org/dendrite.git synced 2025-02-27 13:24:29 -06:00
Commit graph

123 commits

Author SHA1 Message Date
Neil Alexander f6dea712d2
Initial support for multiple server names ()
This PR is the first step towards virtual hosting by laying the
groundwork for multiple server names being configured.
2022-10-26 12:59:19 +01:00
Neboer 2a4c7f45b3
Add support for config "auto_join_rooms" ()
Add support for config "auto_join_rooms". Now new accounts can join the
rooms in config file automatically.

### Pull Request Checklist

<!-- Please read
https://matrix-org.github.io/dendrite/development/contributing before
submitting your pull request -->

* [x] I have justified why this PR doesn't need tests.
* [x] Pull request includes a [sign off below using a legally
identifiable
name](https://matrix-org.github.io/dendrite/development/contributing#sign-off)

Signed-off-by: `Rubin Poster <rubinposter@gmail.com>`
2022-10-26 11:04:53 +02:00
Till e57b301722
Set display_name and/or avatar_url for server notices ()
This should fix  by making sure we actually set the `display_name`
and/or `avatar_url` and create the needed membership event.
To avoid creating a new membership event when starting Dendrite,
`SetAvatarURL` and `SetDisplayName` now return a `Changed` value, which
also makes the regular endpoints idempotent.
2022-10-21 10:48:25 +02:00
Neil Alexander 73e02463cf
Allow m.read.private to clear notifications ()
Otherwise if a user switches to private read receipts, they may not be
able to clear notification counts.
2022-10-21 09:19:52 +01:00
Neil Alexander 3aa92efaa3
Namespace user API tables ()
This migrates all the various user API tables, indices and sequences to
be `userapi_`-namespaced, rather than the mess they are all now.
2022-10-18 15:59:08 +01:00
Till Faelligen fb44e33909
Relax test a bit 2022-10-14 10:37:04 +02:00
Neil Alexander fb6cb2dbcb
Tweak GetAccountByPassword more 2022-10-10 11:14:16 +01:00
Neil Alexander 04bab14290
Add regression test for 980fa55846 2022-10-10 10:45:15 +01:00
Neil Alexander 980fa55846
Stronger passwordless account checks (fixes ) 2022-10-10 10:39:29 +01:00
Till 8c0c3441d8
Add RoomEventType nats.Header to avoid unneeded unmarshalling () 2022-10-05 12:12:42 +02:00
Till Faelligen 3617d5a0ff
Remove SendAccountData, since InputAccountData is already doing that 2022-09-30 17:31:41 +02:00
Till 249b32c4f3
Refactor notifications ()
This PR changes the handling of notifications
- removes the `StreamEvent` and `ReadUpdate` stream
- listens on the `OutputRoomEvent` stream in the UserAPI to inform the
SyncAPI about unread notifications
- listens on the `OutputReceiptEvent` stream in the UserAPI to set
receipts/update notifications
- sets the `read_markers` directly from within the internal UserAPI

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-09-27 15:01:34 +02:00
Neil Alexander 3e87096a21
Use TxStmt in SQLite pusher table 2022-09-26 09:54:54 +01:00
Till Faelligen 3c416517b0
Fix possible "Database is locked" issue 2022-09-26 10:45:35 +02:00
Neil Alexander 34e1dc210b
Use database writer in user API DeleteOldNotifications 2022-09-09 12:59:28 +01:00
Till 64472d9aab
Update getting pushrules, add tests, tweak pushrules ()
This PR
- adds tests for `evaluatePushrules`
- removes the need for the UserAPI on the `OutputStreamEventConsumer`
(for easier testing)
- adds a method to get the pushrules from the database
- adds a new default pushrule for `m.reaction` events (and some other
tweaks)
2022-09-09 13:56:33 +02:00
Neil Alexander ad6b902b84
Refactor appservices component ()
This PR refactors the app services component. It makes the following changes:

* Each appservice now gets its own NATS JetStream consumer
* The appservice database is now removed entirely, since we just use JetStream as a data source instead
* The entire component is now much simpler and we deleted lots of lines of code 💅

The result is that it should be much lighter and hopefully much more performant.
2022-09-01 09:20:40 +01:00
Neil Alexander 175f65407a
Allow batching in JetStreamConsumer ()
This allows us to receive more than one message from NATS at a time if we want.
2022-08-31 12:21:56 +01:00
Till 93a6e2f4d3
Set the notification count when sending push notifications () 2022-08-30 14:54:57 +02:00
Till 48600d5540
Use /admin/v1/register in create-account ()
* Get all account data on CompleteSync

* Revert "Get all account data on CompleteSync"

This reverts commit 44a3e566d8.

* Use /_synapse/admin/v1/register to create account

* Linting

* Linter again :)

* Update docs

* Use HTTP API to reset password, add option to User API `PerformPasswordUpdate` to invalidate sessions

* Fix routing name

* Tell me more about what went wrong

* Deprecate the `-reset-password` flag, document the new API

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-08-12 12:00:07 +01:00
Neil Alexander c45d0936b5
Generic-based internal HTTP API ()
* Generic-based internal HTTP API (tested out on a few endpoints in the federation API)

* Add `PerformInvite`

* More tweaks

* Fix metric name

* Fix LookupStateIDs

* Lots of changes to clients

* Some serverside stuff

* Some error handling

* Use paths as metric names

* Revert "Use paths as metric names"

This reverts commit a9323a6a34.

* Namespace metric names

* Remove duplicate entry

* Remove another duplicate entry

* Tweak error handling

* Some more tweaks

* Update error behaviour

* Some more error tweaking

* Fix API path for `PerformDeleteKeys`

* Fix another path

* Tweak federation client proxying

* Fix another path

* Don't return typed nils

* Some more tweaks, not that it makes any difference

* Tweak federation client proxying

* Maybe fix the key backup test
2022-08-11 15:29:33 +01:00
Till 1b7f84250a
Fix linter issues ()
* Try that again

* All hail the mighty linter?

* And once again

* goimport all the things
2022-08-05 11:12:41 +02:00
Neil Alexander 962b76da44
Make the User API responsible for sending account data output events ()
* Make the User API responsible for sending account data output events

* Clean up producer

* Review comments
2022-07-25 17:30:07 +01:00
Till 081f5e7226
Update database migrations, remove goose ()
* Add new db migration

* Update migrations
Remove goose

* Add possibility to test direct upgrades

* Try to fix WASM test

* Add checks for specific migrations

* Remove AddMigration
Use WithTransaction
Add Dendrite version to table

* Fix linter issues

* Update tests

* Update comments, outdent if

* Namespace migrations

* Add direct upgrade tests, skipping over one version

* Split migrations

* Update go version in CI

* Fix copy&paste mistake

* Use contexts in migrations

Co-authored-by: kegsay <kegan@matrix.org>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-07-25 10:39:22 +01:00
Neil Alexander 583b8ea273
Update FAQ 2022-07-19 11:51:46 +01:00
Neil Alexander 519bc1124b
Add evacuateUser endpoint, use it when deactivating accounts ()
* Add `evacuateUser` endpoint, use it when deactivating accounts

* Populate the API

* Clean up user devices when deactivating

* Include invites, delete pushers
2022-06-29 15:29:39 +01:00
Till ae7b6dd516
Fix () 2022-05-31 14:36:51 +02:00
Till beed39a8f4
Fix pq: invalid input syntax for integer when trying to get a numeric localpart ()
* Match at least once

* Add the test
2022-05-31 11:27:04 +02:00
kegsay 6de29c1cd2
bugfix: E2EE device keys could sometimes not be sent to remote servers ()
* Fix flakey sytest 'Local device key changes get to remote servers'

* Debug logs

* Remove internal/test and use /test only

Remove a lot of ancient code too.

* Use FederationRoomserverAPI in more places

* Use more interfaces in federationapi; begin adding regression test

* Linting

* Add regression test

* Unbreak tests

* ALL THE LOGS

* Fix a race condition which could cause events to not be sent to servers

If a new room event which rewrites state arrives, we remove all joined hosts
then re-calculate them. This wasn't done in a transaction so for a brief period
we would have no joined hosts. During this interim, key change events which arrive
would not be sent to destination servers. This would sporadically fail on sytest.

* Unbreak new tests

* Linting
2022-05-17 13:23:35 +01:00
Neil Alexander 77722c5a4f
Back out by restoring http.Clients
This creates problems with non-HTTPS endpoints and should fix .
2022-05-10 11:08:10 +01:00
Neil Alexander 09d754cfbf
One NATS instance per BaseDendrite ()
* One NATS instance per `BaseDendrite`

* Fix roomserver
2022-05-09 14:15:24 +01:00
kegsay 85704eff20
Clean up interface definitions ()
* tidy up interfaces

* remove unused GetCreatorIDForAlias

* Add RoomserverUserAPI interface

* Define more interfaces

* Use AppServiceInternalAPI for consistent naming

* clean up federationapi constructor a bit

* Fix monolith in -http mode
2022-05-06 12:39:26 +01:00
kegsay 9957752a9d
Define component interfaces based on consumers (2/2) ()
* convert remaining interfaces

* Tidy up the userapi interfaces
2022-05-05 19:30:38 +01:00
Neil Alexander 42f35a57ac
Update table names for user API stats table 2022-05-05 13:42:12 +01:00
kegsay 506de4bb3d
Define component interfaces based on consumers (1/2) ()
* Specify interfaces used by appservice, do half of clientapi

* convert more deps of clientapi to finer-grained interfaces

* Convert mediaapi and rest of clientapi

* Somehow this got missed
2022-05-05 13:17:38 +01:00
Neil Alexander d9e71b93b6
Use gomatrixserverlib.Client instead of http.Client ()
* Update to 

* Use `gomatrixserverlib.Client` for phone-home stats

* Use `gomatrixserverlib.Client` for push notifications

* Use `gomatrixserverlib.Client` for appservices

* Use `gomatrixserverlib.Client` for three-PID invites
2022-05-05 11:33:16 +01:00
Till 658e82f8bc
Don't use in-memory db for userapi tests ()
* Don't use in-memory db

* Use WithAllDatabases where possible
2022-05-05 12:00:18 +02:00
kegsay d86dcbef66
syncapi: define specific interfaces for internal HTTP communications ()
* syncapi: use finer-grained interfaces when making the syncapi

* Use specific interfaces for syncapi-roomserver interactions

* Define query access token api for shared http auth code
2022-05-05 09:56:03 +01:00
Till 3c940c428d
Add opt-in anonymous stats reporting ()
* Initial phone home stats queries

* Add userAgent to UpdateDeviceLastSeen
Add new Table for tracking daily user vists

* Add user_daily_visits table

* Fix queries

* userapi stats tables & queries

* userapi interface and internal api

* sycnapi stats queries

* testing phone home stats

* Add complete config to syncapi

* add missing files

* Fix queries

* Send empty request

* Add version & monolith stats

* Add configuration for phone home stats

* Move WASM to its own file, add config and comments

* Add tracing methods

* Add total rooms

* Add more fields, actually send data somewhere

* Move stats to the userapi

* Move phone home stats to util package

* Cleanup

* Linter & parts of GH comments

* More GH comments changes
- Move comments to SQL statements
- Shrink interface, add struct for stats
- No fatal errors, use defaults

* Be more explicit when querying

* Fix wrong calculation & wrong query params
Add tests

* Add Windows stats

* ADd build constraint

* Use new testing structure
Fix issues with getting values when using SQLite
Fix wrong AddDate value
Export UpdateUserDailyVisits

* Fix query params

* Fix test

* Add comment about countR30UsersSQL and countR30UsersV2SQL; fix test

* Update config

* Also update example config file

* Use OS level proxy, update logging

Co-authored-by: kegsay <kegan@matrix.org>
2022-05-04 19:04:28 +02:00
Neil Alexander 4ad5f9c982
Global database connection pool (for monolith mode) ()
* Allow monolith components to share a single database pool

* Don't yell about missing connection strings

* Rename field

* Setup tweaks

* Fix panic

* Improve configuration checks

* Update config

* Fix lint errors

* Update comments
2022-05-03 16:35:06 +01:00
Brian Meek bfa344e831
Test_Devices, sqlite may return devices in different order, test should still pass () 2022-04-30 00:23:11 +02:00
Till Faelligen 8683ff78b1 Make tests more reliable 2022-04-28 15:06:34 +02:00
Till f023cdf8c4
Add UserAPI storage tests ()
* Add tests for parts of the userapi storage

* Add tests for keybackup

* Add LoginToken tests

* Add OpenID tests

* Add profile tests

* Add pusher tests

* Add ThreePID tests

* Add notification tests

* Add more device tests, fix numeric localpart query

* Fix failing CI

* Fix numeric local part query
2022-04-27 15:05:49 +02:00
Till e95fc5c5e3
Use provided filter for account_data ()
* Reuse IncrementalSync, use provided filter

* Inform SyncAPI about newly created push_rules
2022-04-25 19:04:46 +02:00
Kegan Dougal 073972646f Use unix not second 2022-04-19 13:57:02 +01:00
Neil Alexander 7e745665a4
Change pushkey_ts to be seconds (fix ) () 2022-04-19 09:51:02 +01:00
PiotrKozimor 63b7c3da46
Handle QueryAccountAvailabilityPath in internal user API () 2022-04-08 10:18:27 +01:00
Till 60ee7eef4c
Add possibility to ignore users ()
* Add ignore users

* Ignore users in pushrules
Add passing tests

* Update sytest lists

* Store ignore knowledge in the sync API

* Fix copyrights

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-07 15:08:19 +01:00
Till e5e3350ce1
Add presence module V2 ()
* Syncapi presence

* Clientapi http presence handler

* Why is this here?

* Missing files

* FederationAPI presence implementation

* Add new presence stream

* Pinecone update

* Pinecone update

* Add passing tests

* Make linter happy

* Add presence producer

* Add presence config option

* Set user to unavailable after x minutes

* Only set currently_active if online
Avoid unneeded presence updates when syncing

* Tweaks

* Query devices for last_active_ts
Fixes & tweaks

* Export SharedUsers/SharedUsers

* Presence stream in MemoryStorage

* Remove status_msg_nil

* Fix sytest crashes

* Make presence types const and use stringer for it

* Change options to allow inbound/outbound presence

* Fix option & typo

* Update configs

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-06 13:11:19 +02:00
Neil Alexander 16e2d243fc
Reduce the number of allocations made by localRoomMembers when consuming stream events for push notifications () 2022-04-06 10:43:54 +01:00