This prevents us from holding onto durable consumers indefinitely for
rooms that have long since turned inactive, since they do have a bit of
a processing overhead in the NATS Server. If we clear up a consumer and
then a room becomes active again, the consumer gets recreated as needed.
The threshold is set to 24 hours for now, we can tweak it later if needs
be.
Fixes `outliers whose auth_events are in a different room are correctly
rejected`, by validating that auth events are all from the same room and
not using rejected events for event auth.
This should hopefully fix an entire class of problems where components
downstream from the roomserver (i.e. the sync API) could just lose a
whole bunch of state after a rewrite operation like a federated join.
The root of the bug is that we set `RewritesState` in the output event
which instructs downstream components to purge their copy of any room
state, but then didn't send the entire state snapshot in
`adds_state_event_ids` so the downstream state ends up being incomplete
as a result.
commit 1929b688e31987c46e0c8a546f0f9cb0a46bf9a3
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Mon Aug 22 10:09:44 2022 +0100
Still process state-before for soft-failed events
commit e83c0b701d40d78b92072c4643f6bc6f71b72800
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Mon Aug 22 10:06:50 2022 +0100
Improve logging
commit 29e26124bc27cb83d449de2a4214b253c594aa93
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Mon Aug 22 09:58:13 2022 +0100
Don't store soft-failed events as rejected
* Reprocess outliers that were previously rejected
* Might as well do all events this way
* More useful errors
* Fix queries
* Tweak condition
* Don't wrap errors
* Report more useful error
* Flatten error on `r.Queryer.QueryStateAfterEvents`
* Some more debug logging
* Flatten error in `QueryRestrictedJoinAllowed`
* Revert "Flatten error in `QueryRestrictedJoinAllowed`"
This reverts commit 1238b4184c.
* Tweak `QueryStateAfterEvents`
* Handle MissingStateError too
* Scope to room
* Clean up
* Fix the error
* Only apply rejection check to outliers
* Add possibility to set history_visibility and user AccountType
* Add new DB queries
* Add actual history_visibility changes for /messages
* Add passing tests
* Extract check function
* Cleanup
* Cleanup
* Fix build on 386
* Move ApplyHistoryVisibilityFilter to internal
* Move queries to topology table
* Add filtering to /sync and /context
Some cleanup
* Add passing tests; Remove failing tests :(
* Re-add passing tests
* Move filtering to own function to avoid duplication
* Re-add passing test
* Use newly added GMSL HistoryVisibility
* Update gomatrixserverlib
* Set the visibility when creating events
* Default to shared history visibility
* Remove unused query
* Update history visibility checks to use gmsl
Update tests
* Remove unused statement
* Update migrations to set "correct" history visibility
* Add method to fetch the membership at a given event
* Tweaks and logging
* Use actual internal rsAPI, default to shared visibility in tests
* Revert "Move queries to topology table"
This reverts commit 4f0d41be9c.
* Remove noise/unneeded code
* More cleanup
* Try to optimize database requests
* Fix imports
* PR peview fixes/changes
* Move setting history visibility to own migration, be more restrictive
* Fix unit tests
* Lint
* Fix missing entries
* Tweaks for incremental syncs
* Adapt generic changes
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: kegsay <kegan@matrix.org>
* Generic-based internal HTTP API (tested out on a few endpoints in the federation API)
* Add `PerformInvite`
* More tweaks
* Fix metric name
* Fix LookupStateIDs
* Lots of changes to clients
* Some serverside stuff
* Some error handling
* Use paths as metric names
* Revert "Use paths as metric names"
This reverts commit a9323a6a34.
* Namespace metric names
* Remove duplicate entry
* Remove another duplicate entry
* Tweak error handling
* Some more tweaks
* Update error behaviour
* Some more error tweaking
* Fix API path for `PerformDeleteKeys`
* Fix another path
* Tweak federation client proxying
* Fix another path
* Don't return typed nils
* Some more tweaks, not that it makes any difference
* Tweak federation client proxying
* Maybe fix the key backup test
* Try more servers when calling `/state_ids`
* More logging
* Maybe fix concurrent map write
* Revert "Maybe fix concurrent map write"
This reverts commit da0dbb8362.
* Enforce a limit of 20s per server, 5 mins total
* Membership updater refactoring
* Pass in membership state
* Use membership check rather than referring to state directly
* Delete irrelevant membership states
* We don't need the leave event after all
* Tweaks
* Put a log entry in that I might stand a chance of finding
* Be less panicky
* Tweak invite handling
* Don't freak if we can't find the event NID
* Use event NID from `types.Event`
* Clean up
* Better invite handling
* Placate the almighty linter
* Blacklist a Sytest which is otherwise fine under Complement for reasons I don't understand
* Fix the sytest after all (thanks @S7evinK for the spot)
* Try Ristretto cache
* Tweak
* It's beautiful
* Update GMSL
* More strict keyable interface
* Fix that some more
* Make less panicky
* Don't enforce mutability checks for now
* Determine mutability using deep equality
* Tweaks
* Namespace keys
* Make federation caches mutable
* Update cost estimation, add metric
* Update GMSL
* Estimate cost for metrics better
* Reduce counters a bit
* Try caching events
* Some guards
* Try again
* Try this
* Use separate caches for hopefully better hash distribution
* Fix bug with admitting events into cache
* Try to fix bugs
* Check nil
* Try that again
* Preserve order jeezo this is messy
* thanks VS Code for doing exactly the wrong thing
* Try this again
* Be more specific
* aaaaargh
* One more time
* That might be better
* Stronger sorting
* Cache expiries, async publishing of EDUs
* Put it back
* Use a shared cache again
* Cost estimation fixes
* Update ristretto
* Reduce counters a bit
* Clean up a bit
* Update GMSL
* 1GB
* Configurable cache sizees
* Tweaks
* Add `config.DataUnit` for specifying friendly cache sizes
* Various tweaks
* Update GMSL
* Add back some lazy loading caching
* Include key in cost
* Include key in cost
* Tweak max age handling, config key name
* Only register prometheus metrics if requested
* Review comments @S7evinK
* Don't return errors when creating caches (it is better just to crash since otherwise we'll `nil`-pointer exception everywhere)
* Review comments
* Update sample configs
* Update GHA Workflow
* Update Complement images to Go 1.18
* Remove the cache test from the federation API as we no longer guarantee immediate cache admission
* Don't check the caches in the renewal test
* Possibly fix the upgrade tests
* Update to matrix-org/gomatrixserverlib#322
* Update documentation to refer to Go 1.18
* Check state before event
* Tweaks
* Refactor a bit, include in output events
* Don't waste time if soft failed either
* Tweak control flow, comments, use GMSL history visibility type
Squashed commit of the following:
commit 2bd0daf4d61376d2dd56628eaff267b0bc63e116
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Wed Jun 1 09:55:54 2022 +0100
Revert resolving old extremities as well as new
This may no longer be needed with the new state fixes and probably just burns more CPU time than is strictly necessary.
* Fix bugs related to state resolution
* Clean up `resolve-state`
* Don't panic when entries can't be found
* Ensure we have state entries for the auth events
* Revert "Ensure we have state entries for the auth events"
This reverts commit 9b13b7ed37.
* Revert "Revert "Ensure we have state entries for the auth events""
This reverts commit d86db197e3.
* Fix bug
* Try that again
* Update gomatrixserverlib
* Remove recursion from `loadAuthEvents`
* Fix flakey sytest 'Local device key changes get to remote servers'
* Debug logs
* Remove internal/test and use /test only
Remove a lot of ancient code too.
* Use FederationRoomserverAPI in more places
* Use more interfaces in federationapi; begin adding regression test
* Linting
* Add regression test
* Unbreak tests
* ALL THE LOGS
* Fix a race condition which could cause events to not be sent to servers
If a new room event which rewrites state arrives, we remove all joined hosts
then re-calculate them. This wasn't done in a transaction so for a brief period
we would have no joined hosts. During this interim, key change events which arrive
would not be sent to destination servers. This would sporadically fail on sytest.
* Unbreak new tests
* Linting
* Feed existing state into state res when calculating state from new extremities
* Remove duplicates
* Fix bug
* Sort and unique
* Update to matrix-org/gomatrixserverlib#308
* Trim the slice properly
* Update gomatrixserverlib again
* Update to matrix-org/gomatrixserverlib#308
* Add response size and requests total to internal handler
* Move MustRegister calls to New* funcs
* Move MustRegister back to init
* Init at some place, minimize changes
* Add test infrastructure code for dendrite unit/integ tests
Start re-enabling some syncapi storage tests in the process.
* Linting
* Add postgres service to unit tests
* dendrite not syncv3
* Skip test which doesn't work
* Linting
* Add `jetstream.PrepareForTests`
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Roomserver input refactoring — again!
* Ensure the actor runs again
* Preserve consumer after unsubscribe
* Another sprinkling of magic
* Rename `TopicFor` to `Prefixed`
* Recreate the stream if the config is bad
* Check streams too
* Prefix subjects, preserve inboxes
* Recreate if subjects wrong
* Remove stream subject
* Reconstruct properly
* Fix mutex unlock
* Comments
* Fix tests
* Don't drop events
* Review comments
* Separate `queueInputRoomEvents` function
* Re-jig control flow a bit
* Don't send `adds_state_events` in roomserver output events anymore
* Set `omitempty` on some output fields that aren't always set
* Add `AddsState` helper function
* No-op if no added state event IDs
* Revert "No-op if no added state event IDs"
This reverts commit 71a0ef3df1.
* Revert "Add `AddsState` helper function"
This reverts commit c9fbe45475.
* Topologically sort with `SendEventWithState`, so that earlier events should satisfy auth for later ones
* Revert "Topologically sort with `SendEventWithState`, so that earlier events should satisfy auth for later ones"
This reverts commit b0cd706012.
* Update to matrix-org/gomatrixserverlib#293
* `Events` no longer returns an error, other tweaks
* Make sure `Events` is sorted for `parsedRespState` too
