Commit graph

341 commits

Author SHA1 Message Date
Till Faelligen 7198b916b8
Tweaks for incremental syncs 2022-08-09 09:59:05 +02:00
Till Faelligen c6582bdb6a
Merge branch 'main' of github.com:matrix-org/dendrite into s7evink/hisvismessages 2022-08-05 15:41:30 +02:00
Till Faelligen 10a151cb55
Don't panic if we fail to upsert account data 2022-08-05 15:37:13 +02:00
Till Faelligen 33d964b95c
Merge branch 'main' of github.com:matrix-org/dendrite into s7evink/hisvismessages 2022-08-05 14:30:15 +02:00
Till 3a156a434a
Invalidate lazyLoadCache if we're doing an initial sync (#2623)
* Bypass lazyLoadCache if we're doing an initial sync

* Make the linter happy again?

* Revert "Make the linter happy again?"

This reverts commit 52a5691ba3.

* Try that again

* Invalidate LazyLoadCache on initial syncs

* Remove unneeded check

* Add TODO

* Rename Invalite -> InvalidateLazyLoadedUser

* Thanks IDE
2022-08-05 14:27:27 +02:00
Till cecd11be9a
Partly fix notification counts (#2621)
* Fix notification query

* Also for SQLite

* Move tests to whitelist

* Revert "Move tests to whitelist"

This reverts commit a7d0120019.
2022-08-05 13:44:20 +02:00
Neil Alexander c8935fb53f
Do not use ioutil as it is deprecated (#2625) 2022-08-05 10:26:59 +01:00
Till 1b7f84250a
Fix linter issues (#2624)
* Try that again

* All hail the mighty linter?

* And once again

* goimport all the things
2022-08-05 11:12:41 +02:00
Brian Meek de78eab63a
Add race testing to tests, and fix a few small race conditions in the tests (#2587)
* Add race testing to tests, and fix a few small race conditions in the tests

* Enable run-sytest on MacOS

* Remove deadlock detecting mutex, per code review feedback

* Remove autoformatting related changes and a closure that is not needed

* Adjust to importing nats client as 'natsclient'

Signed-off-by: Brian Meek <brian@hntlabs.com>

* Clarify the use of gooseMutex to proect goose internal state

Signed-off-by: Brian Meek <brian@hntlabs.com>

* Remove no longer needed mutex for guarding goose

Signed-off-by: Brian Meek <brian@hntlabs.com>
2022-08-05 09:19:33 +01:00
Till Faelligen 7e2c7e36ad
Merge branch 'main' of github.com:matrix-org/dendrite into s7evink/hisvismessages 2022-08-03 18:48:14 +02:00
Till 9fe509b18d
Fix syncapi shared users query & device lists (#2614)
* Fix query issue, only add "changed" users if we actually share a room

* Avoid log spam if context is done

* Undo changes to filterSharedUsers

* Add logging again..

* Fix SQLite shared users query

* Change query to include invited users
2022-08-03 18:35:17 +02:00
Till df5d4dc7a3
Delete correct Send-to-Device messages (#2608)
* Add send-to-device tests

* Update tests, fix message deletion

* PR comments
2022-08-02 17:00:16 +02:00
Neil Alexander 9295bf0b1f
Merge branch 'main' into s7evink/hisvismessages 2022-08-02 09:53:58 +01:00
sergekh2 6b6b420b9f
Fix issue with sync API not advancing. (#2603)
Issue: During conversation, under some conditions, sync cookie is not advanced, and, as a result, client loops on the same sync API call creating high traffic and CPU load.
Fix: pdu component of cookie was updated incorrectly.
2022-08-02 09:43:48 +01:00
Neil Alexander e94ef84aab
De-race CompleteSync (#2601)
The `err` was coming from outside of the goroutine and being written to by concurrent goroutines.
2022-08-01 15:55:56 +01:00
Till Faelligen 473a402c1c
Fix unit tests 2022-08-01 12:27:25 +02:00
Till Faelligen 3a006495a9
Move setting history visibility to own migration, be more restrictive 2022-08-01 12:07:38 +02:00
Till Faelligen e84fbf5061
PR peview fixes/changes 2022-08-01 11:13:03 +02:00
Neil Alexander 7490a50a4e
Fix imports 2022-07-25 15:19:42 +01:00
Neil Alexander ab3f8a5cd1
Merge branch 'main' into s7evink/hisvismessages 2022-07-25 15:17:54 +01:00
Till 081f5e7226
Update database migrations, remove goose (#2264)
* Add new db migration

* Update migrations
Remove goose

* Add possibility to test direct upgrades

* Try to fix WASM test

* Add checks for specific migrations

* Remove AddMigration
Use WithTransaction
Add Dendrite version to table

* Fix linter issues

* Update tests

* Update comments, outdent if

* Namespace migrations

* Add direct upgrade tests, skipping over one version

* Split migrations

* Update go version in CI

* Fix copy&paste mistake

* Use contexts in migrations

Co-authored-by: kegsay <kegan@matrix.org>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-07-25 10:39:22 +01:00
Neil Alexander f0c8a03649
Membership updater refactoring (#2541)
* Membership updater refactoring

* Pass in membership state

* Use membership check rather than referring to state directly

* Delete irrelevant membership states

* We don't need the leave event after all

* Tweaks

* Put a log entry in that I might stand a chance of finding

* Be less panicky

* Tweak invite handling

* Don't freak if we can't find the event NID

* Use event NID from `types.Event`

* Clean up

* Better invite handling

* Placate the almighty linter

* Blacklist a Sytest which is otherwise fine under Complement for reasons I don't understand

* Fix the sytest after all (thanks @S7evinK for the spot)
2022-07-22 14:44:04 +01:00
Till Faelligen 53885048a8
More cleanup 2022-07-22 09:12:01 +02:00
Till Faelligen 17fee30c0e
Remove noise/unneeded code 2022-07-22 09:06:24 +02:00
Till Faelligen 7cd8f40b48
Revert "Move queries to topology table"
This reverts commit 4f0d41be9c.
2022-07-22 08:59:35 +02:00
Till Faelligen bbbf1036b7
Use actual internal rsAPI, default to shared visibility in tests 2022-07-22 08:32:12 +02:00
Till Faelligen 2502e66592
Tweaks and logging 2022-07-22 07:32:29 +02:00
Till Faelligen 503afceccb Update migrations to set "correct" history visibility 2022-07-19 13:30:43 +02:00
Till Faelligen 56d8a2371f Remove unused statement 2022-07-19 12:41:54 +02:00
Till Faelligen 910bd9b4a8 Update history visibility checks to use gmsl
Update tests
2022-07-19 12:37:16 +02:00
Till Faelligen 1c43dbcc12 Remove unused query 2022-07-19 12:36:23 +02:00
Till Faelligen 9b6b6585c2 Merge branch 'main' of github.com:matrix-org/dendrite into s7evink/hisvismessages 2022-07-18 18:23:02 +02:00
Till Faelligen bcff14adea Set historyVisibility in rowsToStreamEvents 2022-07-18 18:19:44 +02:00
Till Faelligen 993496db9e Merge branch 'main' of github.com:matrix-org/dendrite into s7evink/hisvismessages 2022-07-18 14:47:16 +02:00
Till a7e92f8cb9
History visibility database changes (#2533)
* Add new history_visibility column

* Update SQL queries to include history_visibility

* Store the history visibilty calculated by the roomserver

* Update GMSL

* Update migrations

* Fix migration

* Update GMSL

* Fix `go.sum`

* Update GMSL to use sql.Scanner & sql.Valuer

* Re-order migration/table creation

* Update gomatrixserverlib

* Add history_visibility column to current_room_state

* Fix migrations

* Return error instead of Fatal log

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-07-18 14:46:15 +02:00
Neil Alexander 90bf01d8b1
Use sync API database in filterSharedUsers (#2572)
* Add function to the sync API storage package for filtering shared users

* Use the database instead of asking the RS API

* Fix unit tests

* Fix map handling in `filterSharedUsers`
2022-07-15 16:25:26 +01:00
Till 09f0ff14c8
Minor SendToDevice fix (#2565)
* Avoid unnecessary marshalling if sending to the local server

* Fix ordering of ToDevice messages

* Revive SendToDevice test
2022-07-12 08:23:58 +02:00
Neil Alexander 3ea21273bc
Ristretto cache (#2563)
* Try Ristretto cache

* Tweak

* It's beautiful

* Update GMSL

* More strict keyable interface

* Fix that some more

* Make less panicky

* Don't enforce mutability checks for now

* Determine mutability using deep equality

* Tweaks

* Namespace keys

* Make federation caches mutable

* Update cost estimation, add metric

* Update GMSL

* Estimate cost for metrics better

* Reduce counters a bit

* Try caching events

* Some guards

* Try again

* Try this

* Use separate caches for hopefully better hash distribution

* Fix bug with admitting events into cache

* Try to fix bugs

* Check nil

* Try that again

* Preserve order jeezo this is messy

* thanks VS Code for doing exactly the wrong thing

* Try this again

* Be more specific

* aaaaargh

* One more time

* That might be better

* Stronger sorting

* Cache expiries, async publishing of EDUs

* Put it back

* Use a shared cache again

* Cost estimation fixes

* Update ristretto

* Reduce counters a bit

* Clean up a bit

* Update GMSL

* 1GB

* Configurable cache sizees

* Tweaks

* Add `config.DataUnit` for specifying friendly cache sizes

* Various tweaks

* Update GMSL

* Add back some lazy loading caching

* Include key in cost

* Include key in cost

* Tweak max age handling, config key name

* Only register prometheus metrics if requested

* Review comments @S7evinK

* Don't return errors when creating caches (it is better just to crash since otherwise we'll `nil`-pointer exception everywhere)

* Review comments

* Update sample configs

* Update GHA Workflow

* Update Complement images to Go 1.18

* Remove the cache test from the federation API as we no longer guarantee immediate cache admission

* Don't check the caches in the renewal test

* Possibly fix the upgrade tests

* Update to matrix-org/gomatrixserverlib#322

* Update documentation to refer to Go 1.18
2022-07-11 14:31:31 +01:00
Till f76f28e6db
Fix issue uint64 values with high bit are not supported in presence (#2562)
* Fix issue #2528

* Use gomatrixserverlib.Timestamp

* Use ParseUint instead of ParseInt
2022-07-07 16:29:25 +02:00
Neil Alexander 460dccf93d
Hopefully fix read receipts timestamps (#2557)
This should avoid coercions between signed and unsigned ints which might fix problems like `sql: converting argument $5 type: uint64 values with high bit set are not supported`.
2022-07-05 17:13:26 +01:00
Till 89cd0e8fc1
Try to fix backfilling (#2548)
* Try to fix backfilling

* Return start/end to not confuse clients

* Update GMSL

* Update GMSL
2022-07-01 11:49:26 +02:00
Till 561c159ad7
Silence presence logs (#2547) 2022-06-30 12:34:37 +02:00
Till 2086992caf
Don't return end if there are not more messages (#2542)
* Be more spec compliant

* Move lazyLoadMembers to own method
2022-06-29 10:49:12 +02:00
Neil Alexander 7120eb6bc9
Add InputDeviceListUpdate to the keyserver, remove old input API (#2536)
* Add `InputDeviceListUpdate` to the keyserver, remove old input API

* Fix copyright

* Log more information when a device list update fails
2022-06-15 14:27:07 +01:00
Till Faelligen 075618833f Use newly added GMSL HistoryVisibility 2022-06-14 11:48:38 +02:00
Neil Alexander 4c2a10f1a6
Handle state before, send history visibility in output (#2532)
* Check state before event

* Tweaks

* Refactor a bit, include in output events

* Don't waste time if soft failed either

* Tweak control flow, comments, use GMSL history visibility type
2022-06-13 15:11:10 +01:00
Till Faelligen e8b1fd9ef8 Move filtering to own function to avoid duplication 2022-06-03 14:38:18 +02:00
Till Faelligen 0fed7ed32d Re-add passing tests 2022-06-03 14:25:50 +02:00
Till Faelligen b666b1e014 Add filtering to /sync and /context
Some cleanup
2022-06-03 10:04:47 +02:00
Till Faelligen 4f0d41be9c Move queries to topology table 2022-06-03 09:30:54 +02:00