Commit graph

127 commits

Author SHA1 Message Date
Till Faelligen cabc5f4bc9 Merge branch 'main' of github.com:matrix-org/dendrite into s7evink/consent-tracking 2022-05-19 14:21:59 +02:00
kegsay 6de29c1cd2
bugfix: E2EE device keys could sometimes not be sent to remote servers (#2466)
* Fix flakey sytest 'Local device key changes get to remote servers'

* Debug logs

* Remove internal/test and use /test only

Remove a lot of ancient code too.

* Use FederationRoomserverAPI in more places

* Use more interfaces in federationapi; begin adding regression test

* Linting

* Add regression test

* Unbreak tests

* ALL THE LOGS

* Fix a race condition which could cause events to not be sent to servers

If a new room event which rewrites state arrives, we remove all joined hosts
then re-calculate them. This wasn't done in a transaction so for a brief period
we would have no joined hosts. During this interim, key change events which arrive
would not be sent to destination servers. This would sporadically fail on sytest.

* Unbreak new tests

* Linting
2022-05-17 13:23:35 +01:00
Neil Alexander 77722c5a4f
Back out matrix-org/dendrite#2421 by restoring http.Clients
This creates problems with non-HTTPS endpoints and should fix #2444.
2022-05-10 11:08:10 +01:00
kegsay c9409078ac
Merge branch 'main' into s7evink/consent-tracking 2022-05-09 16:04:36 +01:00
Neil Alexander 09d754cfbf
One NATS instance per BaseDendrite (#2438)
* One NATS instance per `BaseDendrite`

* Fix roomserver
2022-05-09 14:15:24 +01:00
kegsay 85704eff20
Clean up interface definitions (#2427)
* tidy up interfaces

* remove unused GetCreatorIDForAlias

* Add RoomserverUserAPI interface

* Define more interfaces

* Use AppServiceInternalAPI for consistent naming

* clean up federationapi constructor a bit

* Fix monolith in -http mode
2022-05-06 12:39:26 +01:00
Till Faelligen 3d3773d3d4 Rename migrations so they are executed 2022-05-06 10:15:42 +02:00
Till Faelligen 4bd9a73c13 Fix database locked 2022-05-06 09:50:20 +02:00
Till Faelligen 4d285fff60 Merge branch 'main' of github.com:matrix-org/dendrite into s7evink/consent-tracking 2022-05-06 09:43:50 +02:00
kegsay 9957752a9d
Define component interfaces based on consumers (2/2) (#2425)
* convert remaining interfaces

* Tidy up the userapi interfaces
2022-05-05 19:30:38 +01:00
Neil Alexander 42f35a57ac
Update table names for user API stats table 2022-05-05 13:42:12 +01:00
kegsay 506de4bb3d
Define component interfaces based on consumers (1/2) (#2423)
* Specify interfaces used by appservice, do half of clientapi

* convert more deps of clientapi to finer-grained interfaces

* Convert mediaapi and rest of clientapi

* Somehow this got missed
2022-05-05 13:17:38 +01:00
Neil Alexander d9e71b93b6
Use gomatrixserverlib.Client instead of http.Client (#2421)
* Update to matrix-org/gomatrixserverlib#303

* Use `gomatrixserverlib.Client` for phone-home stats

* Use `gomatrixserverlib.Client` for push notifications

* Use `gomatrixserverlib.Client` for appservices

* Use `gomatrixserverlib.Client` for three-PID invites
2022-05-05 11:33:16 +01:00
Till 658e82f8bc
Don't use in-memory db for userapi tests (#2417)
* Don't use in-memory db

* Use WithAllDatabases where possible
2022-05-05 12:00:18 +02:00
kegsay d86dcbef66
syncapi: define specific interfaces for internal HTTP communications (#2416)
* syncapi: use finer-grained interfaces when making the syncapi

* Use specific interfaces for syncapi-roomserver interactions

* Define query access token api for shared http auth code
2022-05-05 09:56:03 +01:00
Till 3c940c428d
Add opt-in anonymous stats reporting (#2249)
* Initial phone home stats queries

* Add userAgent to UpdateDeviceLastSeen
Add new Table for tracking daily user vists

* Add user_daily_visits table

* Fix queries

* userapi stats tables & queries

* userapi interface and internal api

* sycnapi stats queries

* testing phone home stats

* Add complete config to syncapi

* add missing files

* Fix queries

* Send empty request

* Add version & monolith stats

* Add configuration for phone home stats

* Move WASM to its own file, add config and comments

* Add tracing methods

* Add total rooms

* Add more fields, actually send data somewhere

* Move stats to the userapi

* Move phone home stats to util package

* Cleanup

* Linter & parts of GH comments

* More GH comments changes
- Move comments to SQL statements
- Shrink interface, add struct for stats
- No fatal errors, use defaults

* Be more explicit when querying

* Fix wrong calculation & wrong query params
Add tests

* Add Windows stats

* ADd build constraint

* Use new testing structure
Fix issues with getting values when using SQLite
Fix wrong AddDate value
Export UpdateUserDailyVisits

* Fix query params

* Fix test

* Add comment about countR30UsersSQL and countR30UsersV2SQL; fix test

* Update config

* Also update example config file

* Use OS level proxy, update logging

Co-authored-by: kegsay <kegan@matrix.org>
2022-05-04 19:04:28 +02:00
Till Faelligen 10dc02f1ea Debugging unit tests.. 2022-05-04 18:38:43 +02:00
Till Faelligen 2eb3aab07e Split out UserConsentPolicyAPI for easier testing 2022-05-04 17:56:46 +02:00
Till Faelligen 60ba4b5612 Fix stupid mistake.. and just return the NullString 2022-05-04 17:53:49 +02:00
Till Faelligen 4d5feb2544 Fix userapi issues 2022-05-04 14:33:51 +02:00
kegsay e0cdf64c33
Merge branch 'main' into s7evink/consent-tracking 2022-05-03 17:31:36 +01:00
Neil Alexander 4ad5f9c982
Global database connection pool (for monolith mode) (#2411)
* Allow monolith components to share a single database pool

* Don't yell about missing connection strings

* Rename field

* Setup tweaks

* Fix panic

* Improve configuration checks

* Update config

* Fix lint errors

* Update comments
2022-05-03 16:35:06 +01:00
Till Faelligen ef62255685 Merge branch 'main' of github.com:matrix-org/dendrite into s7evink/consent-tracking 2022-05-03 09:54:25 +02:00
Brian Meek bfa344e831
Test_Devices, sqlite may return devices in different order, test should still pass (#2406) 2022-04-30 00:23:11 +02:00
Till Faelligen 8683ff78b1 Make tests more reliable 2022-04-28 15:06:34 +02:00
Till f023cdf8c4
Add UserAPI storage tests (#2384)
* Add tests for parts of the userapi storage

* Add tests for keybackup

* Add LoginToken tests

* Add OpenID tests

* Add profile tests

* Add pusher tests

* Add ThreePID tests

* Add notification tests

* Add more device tests, fix numeric localpart query

* Fix failing CI

* Fix numeric local part query
2022-04-27 15:05:49 +02:00
Till e95fc5c5e3
Use provided filter for account_data (#2372)
* Reuse IncrementalSync, use provided filter

* Inform SyncAPI about newly created push_rules
2022-04-25 19:04:46 +02:00
Till Faelligen 1f64fc79c8 Merge branch 's7evink/consent-tracking' of github.com:matrix-org/dendrite into s7evink/consent-tracking 2022-04-20 17:35:27 +02:00
Till Faelligen 2b496be2c3 Merge branch 'main' of github.com:matrix-org/dendrite into s7evink/consent-tracking 2022-04-20 17:26:27 +02:00
Kegan Dougal 073972646f Use unix not second 2022-04-19 13:57:02 +01:00
Neil Alexander 7e745665a4
Change pushkey_ts to be seconds (fix #2354) (#2358) 2022-04-19 09:51:02 +01:00
PiotrKozimor 63b7c3da46
Handle QueryAccountAvailabilityPath in internal user API (#2332) 2022-04-08 10:18:27 +01:00
Till 60ee7eef4c
Add possibility to ignore users (#2329)
* Add ignore users

* Ignore users in pushrules
Add passing tests

* Update sytest lists

* Store ignore knowledge in the sync API

* Fix copyrights

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-07 15:08:19 +01:00
Till e5e3350ce1
Add presence module V2 (#2312)
* Syncapi presence

* Clientapi http presence handler

* Why is this here?

* Missing files

* FederationAPI presence implementation

* Add new presence stream

* Pinecone update

* Pinecone update

* Add passing tests

* Make linter happy

* Add presence producer

* Add presence config option

* Set user to unavailable after x minutes

* Only set currently_active if online
Avoid unneeded presence updates when syncing

* Tweaks

* Query devices for last_active_ts
Fixes & tweaks

* Export SharedUsers/SharedUsers

* Presence stream in MemoryStorage

* Remove status_msg_nil

* Fix sytest crashes

* Make presence types const and use stringer for it

* Change options to allow inbound/outbound presence

* Fix option & typo

* Update configs

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-06 13:11:19 +02:00
Neil Alexander 16e2d243fc
Reduce the number of allocations made by localRoomMembers when consuming stream events for push notifications (#2324) 2022-04-06 10:43:54 +01:00
Till Faelligen 733b601aa9 Merge branch 'main' of github.com:matrix-org/dendrite into s7evink/consent-tracking 2022-04-04 15:56:40 +02:00
Neil Alexander 7972915806
User directory for nearby Pinecone peers (P2P demo) (#2311)
* User directory for nearby Pinecone peers

* Fix mux routing

* Use config to determine which server notices user to exclude
2022-03-28 16:25:26 +01:00
Neil Alexander 0692be44d9
Fix account availability on register 2022-03-28 13:31:17 +01:00
Till Faelligen 2a18023a1a Linter again.. 2022-03-28 12:13:47 +02:00
Till Faelligen c99e3aff1b Fix linter issues 2022-03-28 12:01:15 +02:00
Till Faelligen 019f0922ea Merge branch 'main' of github.com:matrix-org/dendrite into s7evink/consent-tracking 2022-03-28 11:30:16 +02:00
S7evinK f2e550efd8
Refactor appservice & client API to use userapi internal (#2290)
* Refactor user api internal

* Refactor clientapi to use internal userapi

* Use internal userapi instead of user DB directly

* Remove AccountDB dependency

* Fix linter issues

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-03-24 22:45:44 +01:00
S7evinK 6e8cca344a
Fix #2287 by trying to fetch account by lowercase localpart (#2292)
* Fix #2287 by trying to fetch account by lowercase localpart and as
passed by request
2022-03-23 13:55:34 +01:00
Neil Alexander 98a5e410d7
Per-room consumers (#2293)
* Roomserver input refactoring — again!

* Ensure the actor runs again

* Preserve consumer after unsubscribe

* Another sprinkling of magic

* Rename `TopicFor` to `Prefixed`

* Recreate the stream if the config is bad

* Check streams too

* Prefix subjects, preserve inboxes

* Recreate if subjects wrong

* Remove stream subject

* Reconstruct properly

* Fix mutex unlock

* Comments

* Fix tests

* Don't drop events

* Review comments

* Separate `queueInputRoomEvents` function

* Re-jig control flow a bit
2022-03-23 10:20:18 +00:00
Neil Alexander 9572f5ed19
Wait for safe shutdown of NATS Server (#2289) 2022-03-21 10:32:34 +00:00
kegsay 6324c1d01f
Merge branch 'main' into s7evink/consent-tracking 2022-03-18 08:37:31 +00:00
Neil Alexander e30aa38fb0
Stream tweaks, use same codepath for sync vs async input room events, wait for error response via NATS messages (#2283) 2022-03-16 14:21:11 +00:00
Till Faelligen 699617ee4d Add server_notice_room_id and methods to update/get it 2022-03-07 09:41:25 +01:00
Till Faelligen e6e62497c9 Merge branch 'main' of github.com:matrix-org/dendrite into s7evink/consent-tracking 2022-03-04 09:18:34 +01:00
Neil Alexander 5592322e13
Clean old notifications regularly (#2244)
* Clean old notifications regularly

We'll keep highlights for a month and non-highlights for a day, to stop the `userapi_notifications` table from growing indefinitely.

We'll also allow storing events even if no pushers are present, because apparently Element Web expects to work that way.

* Fix the milliseconds

* Use process context

* Update sytest lists

* Fix build issue
2022-03-03 16:45:06 +00:00