Commit graph

2568 commits

Author SHA1 Message Date
kegsay 5c37f165ae
Errors from createdb are non-fatal (#2420)
As they are expected if the database already exists.
2022-05-05 10:53:52 +01:00
kegsay d86dcbef66
syncapi: define specific interfaces for internal HTTP communications (#2416)
* syncapi: use finer-grained interfaces when making the syncapi

* Use specific interfaces for syncapi-roomserver interactions

* Define query access token api for shared http auth code
2022-05-05 09:56:03 +01:00
Till 3c940c428d
Add opt-in anonymous stats reporting (#2249)
* Initial phone home stats queries

* Add userAgent to UpdateDeviceLastSeen
Add new Table for tracking daily user vists

* Add user_daily_visits table

* Fix queries

* userapi stats tables & queries

* userapi interface and internal api

* sycnapi stats queries

* testing phone home stats

* Add complete config to syncapi

* add missing files

* Fix queries

* Send empty request

* Add version & monolith stats

* Add configuration for phone home stats

* Move WASM to its own file, add config and comments

* Add tracing methods

* Add total rooms

* Add more fields, actually send data somewhere

* Move stats to the userapi

* Move phone home stats to util package

* Cleanup

* Linter & parts of GH comments

* More GH comments changes
- Move comments to SQL statements
- Shrink interface, add struct for stats
- No fatal errors, use defaults

* Be more explicit when querying

* Fix wrong calculation & wrong query params
Add tests

* Add Windows stats

* ADd build constraint

* Use new testing structure
Fix issues with getting values when using SQLite
Fix wrong AddDate value
Export UpdateUserDailyVisits

* Fix query params

* Fix test

* Add comment about countR30UsersSQL and countR30UsersV2SQL; fix test

* Update config

* Also update example config file

* Use OS level proxy, update logging

Co-authored-by: kegsay <kegan@matrix.org>
2022-05-04 19:04:28 +02:00
Neil Alexander b0a9e85c4a
Fix bug in database global setup 2022-05-03 17:40:56 +01:00
Neil Alexander e01d1e1f5b
Skip tests that require a database if we can't connect to one (#2413)
* Skip tests that require a database if we can't connect to one

* Add `DENDRITE_SKIP_DB_TESTS` environment variable to bring @kegsay joy

* Call it `DENDRITE_TEST_SKIP_NODB` intead
2022-05-03 17:38:54 +01:00
Neil Alexander dd061a172e
Tidy up AddPublicRoutes (#2412)
* Simplify federation API `AddPublicRoutes`

* Simplify client API `AddPublicRoutes`

* Simplify media API `AddPublicRoutes`

* Simplify sync API `AddPublicRoutes`

* Simplify `AddAllPublicRoutes`
2022-05-03 17:17:02 +01:00
Neil Alexander 4ad5f9c982
Global database connection pool (for monolith mode) (#2411)
* Allow monolith components to share a single database pool

* Don't yell about missing connection strings

* Rename field

* Setup tweaks

* Fix panic

* Improve configuration checks

* Update config

* Fix lint errors

* Update comments
2022-05-03 16:35:06 +01:00
Till 979a551f1e
Return null if MaxFileSizeBytes is 0 (#2409)
* Return "null" if MaxFileSizeBytes is 0

* Add comment and nil check (better save than sorry)

* Simplify config
2022-05-02 10:47:16 +02:00
Brian Meek bfa344e831
Test_Devices, sqlite may return devices in different order, test should still pass (#2406) 2022-04-30 00:23:11 +02:00
Till 987d7adc5d
Return "to", if we didn't return any presence events (#2407)
Return correct stream position, if we didn't return any presence events
2022-04-30 00:07:50 +02:00
Neil Alexander 31799a3b2a
Device list display name fixes (#2405)
* Get device names from `unsigned` in `/user/devices`

* Fix display name updates

* Fix bug

* Fix another bug
2022-04-29 16:02:55 +01:00
Brian Meek d28d0ee66e
Fix TestThumbnailsStorage failing when media results come back in non-deterministic order; silence expected
error when tests are run multiple times against the same postgres database (cherry-picked from #2395)

Signed-off-by: Brian Meek <brian@hntlabs.com>
2022-04-29 09:32:58 +01:00
Neil Alexander 2a4517f8e6
Move admin functions into their own file in the client API 2022-04-29 09:10:08 +01:00
Till Faelligen 0d4b8eadaa Add create-account to Getting started 2022-04-29 10:00:28 +02:00
Brian Meek 1e083794ef
Update golangci-lint, how it's installed, and added to the PATH (#2403)
* switch to dendrite server

* minor refactor to merge store code

* Fix issue where m.room.name is being filtered by the dendrite server

* refresh dendrite main

* refresh dendrite main

* missing merges from the last dendrite refresh

* revert unwanted changes in dendrite.yaml

* Update golangci-lint, how it's installed, and added to the PATH

Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
Co-authored-by: tak-slashtalk <64229756+tak-slashtalk@users.noreply.github.com>
2022-04-29 08:55:35 +01:00
Till 26a1512808
Add restrictions for open registration (#2402)
* Add restrications for open registration

* Make enable open registration a parameter

* Enable registration for CI

* Update error message

* Shuffle things around a bit

* Add a warning at every startup just to be extra annoying

* Ignore shared secret when warning about open registration, since it's not strictly required when it is set if registration is otherwise enabled

* Make CI happy?

* Add missing parameter; try new parameter in upgrade-test

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-29 09:31:11 +02:00
Till 2a5b8e0306
Only load members of newly joined rooms (#2389)
* Only load members of newly joined rooms

* Comment that the query is prepared at runtime

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-28 18:53:28 +02:00
Neil Alexander c6ea2c9ff2
Add /_dendrite/admin/evacuateRoom/{roomID} (#2401)
* Add new endpoint to allow admins to evacuate the local server from the room

* Guard endpoint

* Use right prefix

* Auth API

* More useful return error rather than a panic

* More useful return value again

* Update the path

* Try using inputer instead

* oh provide the config

* Try that again

* Return affected user IDs

* Don't create so many forward extremities

* Add missing `Path` to name

Co-authored-by: Till <2353100+S7evinK@users.noreply.github.com>
2022-04-28 16:02:30 +01:00
Till 21ee5b36a4
Limit presence in /sync responses (#2394)
* Use filter and limit presence count

* More limiting

* More limiting

* Fix unit test

* Also limit presence by last_active_ts

* Update query, use "from" as the initial lastPos

* Get 1000 presence events, they are filtered later

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-28 15:12:40 +01:00
Till Faelligen 8683ff78b1 Make tests more reliable 2022-04-28 15:06:34 +02:00
Neil Alexander 65034d1f22
Unlist test since it no longer seems to be flakey (hopefully?) 2022-04-28 11:46:15 +01:00
Neil Alexander 6deb10f3f6
Don't answer expensive federation requests for rooms we no longer belong to (#2398)
This includes `/state`, `/state_ids`, `/get_missing_events` and `/backfill`.

This should fix #2396.
2022-04-28 11:45:56 +01:00
Neil Alexander 2ff75b7c80
Ensure signature map exists (fixes #2393) (#2397) 2022-04-28 11:34:19 +01:00
Till 74259f296f
Fix #2390 (#2392)
Fix duplicate heroes in `/sync` response.
2022-04-27 21:31:30 +02:00
0x1a8510f2 8d69e2f0b8
Use Go 1.18 to build Docker images (#2391)
Go 1.18 has now been released for a while and the CI already tests Dendrite with Go 1.18
so there should be no issues. Go 1.18 brings some performance improvements for ARM via
the register calling convention so it makes sense to switch to it.
2022-04-27 20:19:46 +01:00
Neil Alexander 34221938cc
Version 0.8.2 (#2386)
* Version 0.8.2

* Correct account data position mapping

* Try that again

* Don't duplicate wait-for-shutdowns
2022-04-27 16:04:11 +01:00
Neil Alexander 923f789ca3
Fix graceful shutdown 2022-04-27 15:29:49 +01:00
Neil Alexander 103795d33a
Defer cancel on shutdown context 2022-04-27 15:10:26 +01:00
Neil Alexander cafa2853c5
Use process context as base context for all HTTP 2022-04-27 15:10:20 +01:00
Neil Alexander 655ac3e8fb
Try that again 2022-04-27 15:10:15 +01:00
Neil Alexander 6ee8507955
Correct account data position mapping 2022-04-27 15:10:10 +01:00
Till f023cdf8c4
Add UserAPI storage tests (#2384)
* Add tests for parts of the userapi storage

* Add tests for keybackup

* Add LoginToken tests

* Add OpenID tests

* Add profile tests

* Add pusher tests

* Add ThreePID tests

* Add notification tests

* Add more device tests, fix numeric localpart query

* Fix failing CI

* Fix numeric local part query
2022-04-27 15:05:49 +02:00
Neil Alexander d7cc187ec0
Prevent JetStream from handling OS signals, allow running as a Windows service (#2385)
* Prevent JetStream from handling OS signals, allow running as a Windows service (fixes #2374)

* Remove double import
2022-04-27 13:36:40 +01:00
Neil Alexander 54ff4cf690
Don't try to federated-join via ourselves (#2383) 2022-04-27 12:23:55 +01:00
Neil Alexander dca4afd2f0
Don't send account data or receipts for left/forgotten rooms (#2382)
* Only include account data and receipts for rooms in a complete sync that we care about

* Fix global account data
2022-04-27 12:03:34 +01:00
Neil Alexander 66b397b3c6
Don't create fictitious presence entries (#2381)
* Don't create fictitious presence entries for users that don't have any

* Update whitelist, since that test probably shouldn't be passing

* Fix panics
2022-04-27 11:25:07 +01:00
Neil Alexander 6c5c6d73d7
Use a value that is Go 1.16-friendly 2022-04-26 17:05:31 +01:00
Neil Alexander b527e33c16
Send all account data on complete sync by default
Squashed commit of the following:

commit 0ec8de57261d573a5f88577aa9d7a1174d3999b9
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Apr 26 16:56:30 2022 +0100

    Select filter onto provided target filter

commit da40b6fffbf5737864b223f49900048f557941f9
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Apr 26 16:48:00 2022 +0100

    Specify other field too

commit ffc0b0801f63bb4d3061b6813e3ce5f3b4c8fbcb
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Apr 26 16:45:44 2022 +0100

    Send as much account data as possible during complete sync
2022-04-26 16:58:20 +01:00
Neil Alexander f6d07768a8
Fix account data position 2022-04-26 16:08:01 +01:00
Neil Alexander 6892e0f0e0
Start account data ID from from 2022-04-26 16:02:21 +01:00
Till 4c19f22725
Fix account_data not correctly send in a complete sync (#2379)
* Return the StreamPosition from the database and not the latest

* Fix linter issue
2022-04-26 14:50:56 +01:00
Neil Alexander 5306c73b00
Fix bug when uploading device signatures (#2377)
* Find the complete key ID when uploading signatures

* Try that again

* Try splitting the right thing

* Don't do it for device keys

* Refactor `QuerySignatures`

* Revert "Refactor `QuerySignatures`"

This reverts commit c02832a3e9.

* Both requested key IDs and master/self/user keys

* Fix uniqueness

* Try tweaking GMSL

* Update GMSL again

* Revert "Update GMSL again"

This reverts commit bd6916cc37.

* Revert "Try tweaking GMSL"

This reverts commit 2a054524da.

* Database migrations
2022-04-26 13:08:54 +01:00
Till e8be2b234f
Add heroes to the room summary (#2373)
* Implement room summary heroes

* Add passing tests

* Move MembershipCount to addRoomSummary

* Add comments, close Statement
2022-04-26 10:53:17 +02:00
Till feac9db43f
Add transactionsCache to redact endpoint (#2375) 2022-04-26 10:28:41 +02:00
Till Faelligen 7df5d69a5b Checkout correct branch for Sytest 2022-04-26 08:07:27 +02:00
Till e8ab2154aa
Return M_NOT_FOUND for rejected events (#2371)
* Return M_NOT_FOUND for rejected events

* Add passing tests
2022-04-25 19:05:01 +02:00
Till e95fc5c5e3
Use provided filter for account_data (#2372)
* Reuse IncrementalSync, use provided filter

* Inform SyncAPI about newly created push_rules
2022-04-25 19:04:46 +02:00
Neil Alexander aad81b7b4d
Only call key update process functions if there are updates, don't send things to ourselves over federation 2022-04-25 14:22:46 +01:00
Till 446819e4ac
Store the EDU type in the database (#2370) 2022-04-25 11:56:50 +02:00
Till Faelligen 67fb086c13 Update README 2022-04-24 20:26:20 +02:00