Issue: matrix room id does not always resolve to spaceId or channelId
correctly.
Root cause: The clientApi routing endpoint and the syncapi routing
endpoint uses different stores to query for the current room states.
One is correct, the other has incomplete events. Fix the issue by using
the correct store in both routing code paths.
Closes HNT-244.
The following PR implements Space,Channel soft deletion using on-chain
`disabled` flag scope to space, channel respectively. On message sync,
dendrite will now gate disabled rooms by performing a leave on the user
attempting to sync unless the user is the owner (more on this later). To
re-join, given rooms (spaces,channels) are created by default using
`invite` membership state, the owner will need to undo the on-chain
`disabled` flag, setting it false then re-invite users that left the
room as a side effect of it becoming disabled previously.
The owner does not leave the space, channel because if they did then
there would be no one left to invite users let alone themselves back in
if the action is ever undone.
What is not implemented in this PR:
1. **Transitive leaves on channels in a space** - If a space is
disabled, users will leave the space but not the channels within the
space. To allow for fully disabling a space and all its' channels, the
client can offer a view to the owner that iterates over the channels and
space to disable all on-chain. Furthermore, we could implement a batch
on-chain method that fully disables all channels within a space (plus
the space) in one on-chain call to save the owner gas.
2. **Data deletion** - No data is remove from the DAGs or on-chain.
Therefore deletion is soft and reversible.
3. **New hook to check if a room is disabled** - the client can leverage
existing on-chain public read only methods `getSpaceInfoBySpaceId`,
`getChannelInfoByChannelId` to read the state of each in order to remove
spaces, channels from a member's view that are disabled.
Pulls in upstream latest changes from [dendrite-fork
](https://github.com/HereNotThere/dendrite)to subtree at
servers/dendrite here.
Co-authored-by: Tak Wai Wong <64229756+tak-hntlabs@users.noreply.github.com>
Co-authored-by: Tak Wai Wong <tak@hntlabs.com>
Co-authored-by: John Terzis <john@hntlabs.com>
Based on #2480
This actually indexes events based on their event type. They are removed
from the index if we receive a `m.room.redaction` event on the
`OutputRoomEvent` stream.
An admin endpoint is added to reindex all existing events.
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
This should hopefully deflake Backfill works correctly with history visibility set to joined as we were using the default shared visibility, even if the events are set to joined (or something else)
* Add possibility to set history_visibility and user AccountType
* Add new DB queries
* Add actual history_visibility changes for /messages
* Add passing tests
* Extract check function
* Cleanup
* Cleanup
* Fix build on 386
* Move ApplyHistoryVisibilityFilter to internal
* Move queries to topology table
* Add filtering to /sync and /context
Some cleanup
* Add passing tests; Remove failing tests :(
* Re-add passing tests
* Move filtering to own function to avoid duplication
* Re-add passing test
* Use newly added GMSL HistoryVisibility
* Update gomatrixserverlib
* Set the visibility when creating events
* Default to shared history visibility
* Remove unused query
* Update history visibility checks to use gmsl
Update tests
* Remove unused statement
* Update migrations to set "correct" history visibility
* Add method to fetch the membership at a given event
* Tweaks and logging
* Use actual internal rsAPI, default to shared visibility in tests
* Revert "Move queries to topology table"
This reverts commit 4f0d41be9c.
* Remove noise/unneeded code
* More cleanup
* Try to optimize database requests
* Fix imports
* PR peview fixes/changes
* Move setting history visibility to own migration, be more restrictive
* Fix unit tests
* Lint
* Fix missing entries
* Tweaks for incremental syncs
* Adapt generic changes
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: kegsay <kegan@matrix.org>
* Bypass lazyLoadCache if we're doing an initial sync
* Make the linter happy again?
* Revert "Make the linter happy again?"
This reverts commit 52a5691ba3.
* Try that again
* Invalidate LazyLoadCache on initial syncs
* Remove unneeded check
* Add TODO
* Rename Invalite -> InvalidateLazyLoadedUser
* Thanks IDE
* Check state before event
* Tweaks
* Refactor a bit, include in output events
* Don't waste time if soft failed either
* Tweak control flow, comments, use GMSL history visibility type
* syncapi: use finer-grained interfaces when making the syncapi
* Use specific interfaces for syncapi-roomserver interactions
* Define query access token api for shared http auth code
Squashed commit of the following:
commit 0ec8de57261d573a5f88577aa9d7a1174d3999b9
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Tue Apr 26 16:56:30 2022 +0100
Select filter onto provided target filter
commit da40b6fffbf5737864b223f49900048f557941f9
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Tue Apr 26 16:48:00 2022 +0100
Specify other field too
commit ffc0b0801f63bb4d3061b6813e3ce5f3b4c8fbcb
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Tue Apr 26 16:45:44 2022 +0100
Send as much account data as possible during complete sync
* Convert stream positions into topological positions for both `from` and `to` in `/messages`
* Hopefully it works now
* Remove unnecessary logging
* Return sane values if `StreamToTopologicalPosition` can't work out the right thing to do
* Revert logging change
* tweaks
* Fix `selectEventIDsInRangeASCSQL`
* Test `Getting messages going forward is limited for a departed room (SPEC-216)` was passing incorrectly so un-whitelist it
* Add membership events to the end of the list, to ensure Sytest sees them
* Move tests to allowlist
* Append to correct list, fix logging message
* Add flakey tests to blacklist
* Remove flakey tests from whitelist
It's possible for `GetStateEvent` to return `nil` if there was no error but the state event wasn't found. Therefore we need to be prepared for that case.
This should fix#2247.
* Add some filtering (postgres only for now)
* Fix build error
* Try to use request filter
* Use default filter as a template when retrieving from the database
* Remove unused strut
* Update sytest-whitelist
* Add filtering to SelectEarlyEvents
* Fix Postgres selectEarlyEvents query
* Attempt filtering on SQLite
* Test limit, set field for limit/order in prepareWithFilters
* Remove debug logging, add comments
* Tweaks, debug logging
* Separate SQLite stream IDs
* Fix filtering in current state table
* Fix lock issues
* More tweaks
* Current state requires room ID
* Review comments
* Tweak pagination tokens
* start should be the specified from
* Don't reverse start and end
* Tweak getStartEnd again
* Update sytest-whitelist
* NOTSPEC: Re-add iOS end of topology
* Add basic storage methods
* Add internal api handler
* Add check for forgotten room
* Add /rooms/{roomID}/forget endpoint
* Add missing rsAPI method
* Remove unused parameters
* Add passing tests
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
* Add missing file
* Add postgres migration
* Add sqlite migration
* Use Forgetter to forget room
* Remove empty line
* Update HTTP status codes
It looks like the spec calls for these to be 400, rather than 403: https://matrix.org/docs/spec/client_server/r0.6.1#post-matrix-client-r0-rooms-roomid-forget
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Add history visibility guards
Default to 'joined' visibility to avoid leaking events, until we get
around to implementing history visibility completely. Related #617
* Don't apply his vis checks on shared rooms
* Fix order of checks
* Linting and remove another misleading check
* Update whitelist
* Initial pass at refactoring config (not finished)
* Don't forget current state and EDU servers
* More shifting around
* Update server key API tests
* Fix roomserver test
* Fix more tests
* Further tweaks
* Fix current state server test (sort of)
* Maybe fix appservices
* Fix client API test
* Include database connection string in database options
* Fix sync API build
* Update config test
* Fix unit tests
* Fix federation sender build
* Fix gobind build
* Set Listen address for all services in HTTP monolith mode
* Validate config, reinstate appservice derived in directory, tweaks
* Tweak federation API test
* Set MaxOpenConnections/MaxIdleConnections to previous values
* Update generate-config
* Move filter table to syncapi where it is used
* Implement /sync `limited` and read timeline limit from stored filters
We now fully handle `room.timeline.limit` filters (in-line + stored) and
return the right value for `limited` syncs.
* Update whitelist
* Default to the default timeline limit if it's unset, also strip the extra event correctly
* Update whitelist
* Join room body is optional
* Support deprecated login by user/password
* Implement dummy key upload endpoint
* Make a very determinate end to /messages if we hit the create event in back-pagination
* Linting
