Commit graph

419 commits

Author SHA1 Message Date
Brian Meek c243677795
Merge local harmony changes, Dendrite tests passing
Signed-off-by: Brian Meek <brian@hntlabs.com>
2023-02-08 08:23:06 -08:00
Brian Meek 34504e3cfe
Merge upstream
Signed-off-by: Brian Meek <brian@hntlabs.com>
2023-02-07 17:30:01 -08:00
Giuseppe Rodriguez 7d6ffad2de Audit Fixes 1 - Removes OZ Ownable from Space contract and makes checks for space token ownership to see if caller is space owner (#1424)
Removing the OpenZeppelin ownership logic, and, checking directly for
the ownership of the Space’s SpaceOwner NFT. The SpaceFactory could be
the owner during the bootstrapping phase, and afterwards, could transfer
the NFT to the proper owner. Doing a direct check like
_spaceOwner().ownerOf(tokenId) == _msgSender() would be safe because
that’s what the owner entitlement is going to check eventually.

---------

Co-authored-by: Kerem Kazan <kerem.kazan@gmail.com>
2023-02-07 15:57:39 -08:00
Brian Meek c0d986fdb1
Fix lint warning about ignored err assignment
Signed-off-by: Brian Meek <brian@hntlabs.com>
2023-02-07 09:17:09 -08:00
Till eb29a31550
Optimize /sync and history visibility (#2961)
Should fix the following issues or make a lot less worse when using
Postgres:

The main issue behind #2911: The client gives up after a certain time,
causing a cascade of context errors, because the response couldn't be
built up fast enough. This mostly happens on accounts with many rooms,
due to the inefficient way we're getting recent events and current state

For #2777: The queries for getting the membership events for history
visibility were being executed for each room (I think 185?), resulting
in a whooping 2k queries for membership events. (Getting the
statesnapshot -> block nids -> actual wanted membership event)

Both should now be better by:
- Using a LATERAL join to get all recent events for all joined rooms in
one go (TODO: maybe do the same for room summary and current state etc)
- If we're lazy loading on initial syncs, we're now not getting the
whole current state, just to drop the majority of it because we're lazy
loading members - we add a filter to exclude membership events on the
first call to `CurrentState`.
- Using an optimized query to get the membership events needed to
calculate history visibility

---------

Co-authored-by: kegsay <kegan@matrix.org>
2023-02-07 14:31:23 +01:00
Brian Meek 4f1c8983ad
Merge upstream latest
Signed-off-by: Brian Meek <brian@hntlabs.com>
2023-02-06 20:50:50 -08:00
Till baf118b08c
Add Sytest/Complement coverage to scheduled runs (#2962)
This adds Sytest and Complement coverage reporting to the nightly
scheduled CI runs.

Fixes a few API mode related issues as well, since we seemingly never
really ran them with Complement.

Also fixes a bug related to device list changes: When we pass in an
empty `newlyLeftRooms` slice, we got a list of all currently joined
rooms with the corresponding members. When we then got the
`newlyJoinedRooms`, we wouldn't update the `changed` slice, because we
already got the user from the `newlyLeftRooms` query. This is fixed by
simply ignoring empty `newlyLeftRooms`.
2023-02-03 13:42:35 +01:00
Neil 738686ae68
Add /_dendrite/admin/purgeRoom/{roomID} (#2662)
This adds a new admin endpoint `/_dendrite/admin/purgeRoom/{roomID}`. It
completely erases all database entries for a given room ID.

The roomserver will start by clearing all data for that room and then
will generate an output event to notify downstream components (i.e. the
sync API and federation API) to do the same.

It does not currently clear media and it is currently not implemented
for SQLite since it relies on SQL array operations right now.

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: Till Faelligen <2353100+S7evinK@users.noreply.github.com>
2023-01-19 21:02:32 +01:00
Till 0d0280cf5f
/sync performance optimizations (#2927)
Since #2849 there is no limit for the current state we fetch to
calculate history visibility. In large rooms this can cause us to fetch
thousands of membership events we don't really care about.
This now only gets the state event types and senders in our timeline,
which should significantly reduce the amount of events we fetch from the
database.

Also removes `MaxTopologicalPosition`, as it is an unnecessary DB call,
given we use the result in `topological_position < $1` calls.
2023-01-17 10:08:23 +01:00
Umar Getagazov 8582c7520a
Omit state field from /messages response if empty (#2940)
The field type is `[ClientEvent]` in the
[spec](https://spec.matrix.org/v1.5/client-server-api/#get_matrixclientv3roomsroomidmessages),
but right now `null` can also be returned. Omit the field completely if
it's empty. Some clients (rightfully) assume it's either not present at
all or it's of the right type (see
https://github.com/matrix-org/matrix-react-sdk/pull/9913).

### Pull Request Checklist

<!-- Please read
https://matrix-org.github.io/dendrite/development/contributing before
submitting your pull request -->

* [x] I have added Go unit tests or [Complement integration
tests](https://github.com/matrix-org/complement) for this PR _or_ I have
justified why this PR doesn't need tests
    * The PR is a simple struct tag fix
* [x] Pull request includes a [sign off below using a legally
identifiable
name](https://matrix-org.github.io/dendrite/development/contributing#sign-off)
_or_ I have already signed off privately

Signed-off-by: `Umar Getagazov <umar@handlerug.me>`

Signed-off-by: Umar Getagazov <umar@handlerug.me>
2023-01-17 09:07:42 +01:00
Till 0491a8e343
Fix room summary returning wrong heroes (#2930)
This should fix #2910.
Probably makes Sytest/Complement a bit upset, since this not using
`sort.Strings` anymore.
2023-01-12 10:06:03 +01:00
Till c136a450d5
Fix newly joined users presence (#2854)
Fixes #2803 
Also refactors the presence stream to not hit the database for every
user, instead queries all users at once now.
2022-12-08 08:25:03 +01:00
Neil Alexander ba2ffb7da9
Repeatable reads for /sync (#2783)
This puts repeatable reads into all sync streams.

Co-authored-by: kegsay <kegan@matrix.org>
2022-12-06 18:16:17 +00:00
Tak Wai Wong 8f4d1828b4 matrix room Id does not resolve to channelId or spaceId correctly (#1010)
Issue: matrix room id does not always resolve to spaceId or channelId
correctly.

Root cause: The clientApi routing endpoint and the syncapi routing
endpoint uses different stores to query for the current room states.

One is correct, the other has incomplete events. Fix the issue by using
the correct store in both routing code paths.
2022-11-30 20:12:00 -08:00
Till f8d1dc521d
Fix m.receipts causing notifications (#2893)
Fixes https://github.com/matrix-org/dendrite/issues/2353
2022-11-29 15:46:28 +01:00
Erik Johnston 31f56ac3f4
Never filter out a user's own membership when using LL (#2887) 2022-11-22 21:38:27 +00:00
Neil Alexander 6650712a1c
Federation fixes for virtual hosting 2022-11-15 15:05:23 +00:00
Till d35a5642e8
Deny guest access on several endpoints (#2873)
Second part for guest access, this adds a `WithAllowGuests()` option to
`MakeAuthAPI`, allowing guests to access the specified endpoints.
Endpoints taken from the
[spec](https://spec.matrix.org/v1.4/client-server-api/#client-behaviour-14)
and by checking Synapse endpoints for `allow_guest=true`.
2022-11-11 10:52:08 +01:00
John Terzis 40830b8a37 Space,Channel soft deletion with dendrite gating, tests (#889)
Closes HNT-244.

The following PR implements Space,Channel soft deletion using on-chain
`disabled` flag scope to space, channel respectively. On message sync,
dendrite will now gate disabled rooms by performing a leave on the user
attempting to sync unless the user is the owner (more on this later). To
re-join, given rooms (spaces,channels) are created by default using
`invite` membership state, the owner will need to undo the on-chain
`disabled` flag, setting it false then re-invite users that left the
room as a side effect of it becoming disabled previously.

The owner does not leave the space, channel because if they did then
there would be no one left to invite users let alone themselves back in
if the action is ever undone.

What is not implemented in this PR:
1. **Transitive leaves on channels in a space** - If a space is
disabled, users will leave the space but not the channels within the
space. To allow for fully disabling a space and all its' channels, the
client can offer a view to the owner that iterates over the channels and
space to disable all on-chain. Furthermore, we could implement a batch
on-chain method that fully disables all channels within a space (plus
the space) in one on-chain call to save the owner gas.
2. **Data deletion** - No data is remove from the DAGs or on-chain.
Therefore deletion is soft and reversible.
3. **New hook to check if a room is disabled** - the client can leverage
existing on-chain public read only methods `getSpaceInfoBySpaceId`,
`getChannelInfoByChannelId` to read the state of each in order to remove
spaces, channels from a member's view that are disabled.
2022-11-09 17:07:51 -07:00
Tak Wai Wong 78a4a213d4 Refresh dendrite with latest contract changes (#836)
sync latest dendrite main & space manager contract changes from our
fork.

Co-authored-by: Tak Wai Wong <tak@hntlabs.com>
2022-11-04 11:23:15 -07:00
Tak Wai Wong 341bfe4fdd
sync upstream changes 2022-11-04 09:40:48 -07:00
Till efe28db631
Update latestPosition when getting reversed room delta (#2860)
Regression test added in
https://github.com/matrix-org/complement/pull/551
Should fix https://github.com/matrix-org/dendrite/issues/2514?
2022-11-04 15:39:09 +01:00
Tak Wai Wong a7ab7524da servers/dendrite Move calculation (#2856) commit 98d3f88bfb (#826) 2022-11-03 19:15:07 -07:00
Tak Wai Wong 1d633c0d5e
Force sync upstream commit 98d3f88bfb 2022-11-03 11:22:01 -07:00
Neil Alexander 98d3f88bfb
Move prev_batch calculation (#2856)
This might help #2847.
2022-11-03 16:56:21 +00:00
Neil Alexander 8704e84898
Tweak removeDuplicates calls to use events instead of recentEvents (#2853)
... since `events` is *after* history visibility filtering, not before
it.
2022-11-03 10:19:37 +00:00
Tak Wai Wong a9d3bdc058 subtree pull from dendrite fork for fix #2383 (#802)
Pull the fix for https://github.com/matrix-org/dendrite/issues/2838 into
the dendrite subtree. Includes latest dendrite main.

Signed-off-by: `Devon Hudson <devonhudson@librem.one>`
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Brian Meek <brian@hntlabs.com>
Signed-off-by: Austin Ellis <austin@hntlabs.com>
Signed-off-by: `Rubin Poster <rubinposter@gmail.com>`
Signed-off-by: `ash lea <example@thisismyactual.email>`
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: devonh <devon.dmytro@gmail.com>
Co-authored-by: Till <2353100+S7evinK@users.noreply.github.com>
Co-authored-by: Ashley Nelson <fant@shley.email>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
Co-authored-by: Jean Lucas <jean@4ray.co>
Co-authored-by: Kabir Kwatra <kabir@kwatra.me>
Co-authored-by: sergekh2 <gitgitgit@khorun.com>
Co-authored-by: texuf <texuf.eth@gmail.com>
Co-authored-by: kegsay <kegan@matrix.org>
Co-authored-by: Brian Meek <brian@here.video>
Co-authored-by: John Terzis <john.c.terzis@gmail.com>
Co-authored-by: John Terzis <john@hntlabs.com>
Co-authored-by: Kerem <kerem.kazan@gmail.com>
Co-authored-by: Neboer <43609792+Neboer@users.noreply.github.com>
Co-authored-by: X. Ding <dingsm@gmail.com>
Co-authored-by: dxl <dxl@plotbridge.com>
Co-authored-by: ash lea <ashkitten@users.noreply.github.com>
Co-authored-by: 0x1a8510f2 <admin@0x1a8510f2.space>
Co-authored-by: Till Faelligen <tfaelligen@gmail.com>
Co-authored-by: Till Faelligen <davidf@element.io>
2022-11-02 16:28:06 -07:00
Tak Wai Wong 063299248b
sync fix #2838 from main 2022-11-02 13:48:55 -07:00
Tak Wai Wong 86f9b5ca89
Fix issue where a member is forced to leave a room when the invite is marked deleted #2838 (#55)
* fix #2838 by querying db directly for membership info

* fix lint error

* revert changes and use SelectMembershipForUser directly

* Remove extra membership checks

* Add a comment to explain why the membership check is necessary
2022-11-02 12:34:07 -07:00
Tak Wai Wong 75a508cc27
Fix issue where a member is forced to leave a room when the invite is marked deleted (#2839)
Proposed fix for issue:
https://github.com/matrix-org/dendrite/issues/2838

Suppose bob received invites to spaceA and spaceB.
When Bob joins spaceA, we add an OutputEvent event to retire the invite.
This sets the invite to "deleted" in the database. This makes sense.

The bug is in stream_invites.go. Triggered when bob received a new
invite for spaceB, and does a client sync.

In the block (line 76)
`for roomID := range retiredInvites

   if _, ok := req.Response.Rooms.Invite[roomID]; ok {
	continue
   }

   if _, ok := req.Response.Rooms.Join[roomID]; ok {
	continue
  }
...
` 

Bob is not in either maps even though he had just accepted the invite
for spaceA. Consequently, the spaceA invite is treated as a retired
invite, and a membership Leave event is generated. What bob sees is that
after accepting the invite to spaceB, he lose access to spaceA.


### Pull Request Checklist

<!-- Please read
https://matrix-org.github.io/dendrite/development/contributing before
submitting your pull request -->

* [ ] I have added tests for PR _or_ I have justified why this PR
doesn't need tests.
* [x ] Pull request includes a [sign off below using a legally
identifiable
name](https://matrix-org.github.io/dendrite/development/contributing#sign-off)
_or_ I have already signed off privately

Signed-off-by: `Tak Wai Wong <tak@hntlabs.com>`

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-11-02 10:02:23 +00:00
Neil Alexander 3db9e98456
Don't limit "state" (#2849)
This is apparently some incorrect behaviour that we built as a result of
a spec bug (matrix-org/matrix-spec#1314) where we were applying a filter
to the `"state"` section of the `/sync` response incorrectly. The client
then has no way to know that the state was limited.

This PR removes the state limiting, which probably also helps #2842.
2022-11-02 09:34:19 +00:00
John Terzis b4223a1a18 Jterzis/sync dendrite (#793) 2022-11-01 23:17:48 -07:00
John Terzis 6059650464 message sync gating 2022-11-01 18:07:15 -07:00
Tak Wai Wong 424df14000 Sync dendrite fork changes for gating, and single chain support (#778)
* Latest dendrite main (8c7b274e4e)
* Gating implementation from John and Tak

Fixes for https://github.com/matrix-org/dendrite/issues/2838 and
https://github.com/matrix-org/dendrite/issues/2842

Co-authored-by: Tak Wai Wong <tak@hntlabs.com>
2022-11-01 11:09:34 -07:00
ash lea 5aaa60227a
return required room_id field in /members (#2846)
### Pull Request Checklist

<!-- Please read
https://matrix-org.github.io/dendrite/development/contributing before
submitting your pull request -->

* [ ] I have added tests for PR _or_ I have justified why this PR
doesn't need tests.
* [x] Pull request includes a [sign off below using a legally
identifiable
name](https://matrix-org.github.io/dendrite/development/contributing#sign-off)
_or_ I have already signed off privately

Signed-off-by: `ash lea <example@thisismyactual.email>`
2022-11-01 16:42:07 +00:00
Neil Alexander 0b21cb78aa
Try to fix a panic in the sync API PDU stream 2022-11-01 14:45:15 +00:00
Tak Wai Wong 1574b5e347
Fix #2842 - room state incorrectly filtered away for newly joined member (#52)
* fix room state regression when a new member joins the room

* disables sytests to workaround bug in dendrite

Co-authored-by: Tak Wai Wong <tak@hntlabs.com>
2022-10-31 16:56:15 -07:00
Tak Wai Wong 13f5aa6026
sync upstream changes 2022-10-31 10:49:06 -07:00
Neil Alexander 7307701a24
Tweak "state" and "timeline" filtering (#2844)
This should stop state events disappearing down a gap where we'd try to
separate out the sections *before* applying history visibility instead
of after.

This may be a better approach than #2843 but I hope @tak-hntlabs will
shout if it isn't.
2022-10-31 15:14:08 +00:00
Tak Wai Wong bcf4ac9c9d
fix stream_invite regression (#50) 2022-10-28 16:13:17 -07:00
Tak Wai Wong ed8b5d09eb refresh fork with upstream 2022-10-28 15:49:31 -07:00
Till 69aff372f3
Limit recent events when going backwards (#2840)
If we're going backwards, we were selecting potentially thousands of
events, which in turn were fed to history visibility checks, resulting
in bad sync performance.
2022-10-28 13:40:51 +02:00
Till Faelligen f6035822e7
Simplify error checking and check the correct error 2022-10-28 08:17:40 +02:00
Till a169a9121a
Fix /members (#2837)
Fixes a bug introduced in #2827, where the SyncAPI might not have all
requested eventIDs, resulting in too few members returned.
2022-10-27 14:18:22 +02:00
Tak Wai Wong 8d2033418e
Merge branch 'matrix-org:main' into main 2022-10-25 17:26:39 -07:00
Till c62ac3d6ad
Fix Current state appears in timeline in private history with many messages after (#2830)
The problem was that we weren't getting enough recent events, as most of
them were removed by the history visibility filter. Now we're getting
all events between the given input range and re-slice the returned
values after applying history visibility.
2022-10-25 15:15:24 +02:00
Till Faelligen 8b7bf5e7d7
Return forbidden if not a member anymore (fix #2802) 2022-10-25 15:00:52 +02:00
Till 313cb3fd19
Filter /members, return members at given point (#2827)
Makes the tests
```
Can get rooms/{roomId}/members at a given point
Can filter rooms/{roomId}/members
```
pass, by moving `/members` and `/joined_members` to the SyncAPI.
2022-10-25 12:39:10 +02:00
John Terzis c2d57b8679
merge latest changes from dendrite main (#42)
* merge latest changes from dendrite main

* fix go formatting

Co-authored-by: John Terzis <john@hntlabs.com>
Co-authored-by: Tak Wai Wong <64229756+tak-hntlabs@users.noreply.github.com>
2022-10-24 21:35:32 -07:00
texuf 1f2f42494c
Fix broken notification incremental sync
I was not seeing unread notifications in sync, even if they were written to the db

Notifications are in their own stream, but the code was trying to tack them onto the join room stream. If the offsets “happened” to line up, you might get a count here or there, but they would be totally wrong (jump from 1 to 0 to 2, etc)

To fix, put them in their own top level object, handle them on the client.

Signed-off-by: Austin Ellis <austin@hntlabs.com>
2022-10-24 21:15:42 -07:00