Commit graph

2405 commits

Author SHA1 Message Date
Neil Alexander bbb3ade4a2
Fix the Cat Overflow bug (update to matrix-org/gomatrixserverlib@2020bdc) 2022-08-30 11:44:26 +01:00
Till Faelligen 21ef487ff8
Log roomserver error 2022-08-29 14:18:33 +02:00
Brian Meek 704cc5c9f5
Race in keyserver intialization (#2619)
Signed-off-by: Brian Meek <brian@hntlabs.com>
2022-08-29 09:10:42 +02:00
Till Faelligen aba171d9bc
Default to limit of 20 state events 2022-08-29 07:23:46 +02:00
Neil Alexander a3eb4e5e98
Enforce type, sender, state_key and room_id lengths using codepoints rather than bytes (update to matrix-org/gomatrixserverlib@5f66df0, matrix-org/matrix-spec#1001) 2022-08-28 09:52:03 +01:00
Neil Alexander 38bed30b41
Restrict username length properly (#2676)
This fixes #2674 by fixing the username check to comply with the [spec appendices](https://spec.matrix.org/v1.3/appendices/#user-identifiers):

> The length of a user ID, including the @ sigil and the domain, MUST NOT exceed 255 characters.
2022-08-26 09:56:41 +01:00
Neil Alexander ed79e8626a
Version 0.9.5 (#2673)
Changelog and version bump.
2022-08-25 14:14:10 +01:00
Till 07dd9bd995
SyncAPI tweaks/fixes (#2671)
- Reverts 9dc57122d9 as it was causing issues https://github.com/matrix-org/dendrite/issues/2660
- Updates the GMSL `DefaultStateFilter` to use a limit of 20 events
- Uses the timeline events to determine the new position instead of the state events
2022-08-25 13:42:47 +01:00
Neil Alexander 8ff3f1a7c9
Remove a couple unnecessary Sentry captures from backfill 2022-08-25 11:01:07 +01:00
Neil Alexander cd7fa34595
Tweak logging and Sentry reporting for roomserver input 2022-08-25 10:57:27 +01:00
Neil Alexander 16156b0b09
Fix 500s on /state, /state_ids when state not known (#2672)
This was due to bad error bubbling.
2022-08-25 09:51:36 +01:00
Neil Alexander 522bd2999f
Allow un-rejecting events on reprocessing 2022-08-24 14:03:06 +01:00
Till 78e5d05efc
Only set backOffStarted to false if until is not zero (#2669) 2022-08-23 16:54:42 +02:00
Neil Alexander 14fea600bb
Detect types.MissingStateError in CheckServerAllowedToSeeEvent (#2667)
This will hopefully stop some 500 errors on `/event` where there is no state-before known.
2022-08-23 13:57:11 +01:00
Till 95a509757a
Complement QoL changes (#2663)
This PR does the following:
- adds a `keysize` parameter to `generate-keys`, so we can use lower sized keys when running in CI
- updates the Complement docker files to use BuildKit (requires Docker >18.09)
- uses `exec` when executing `dendrite-monotlith-server`, making it PID 1 inside docker, which results in Dendrite actually receiving the `SIGTERM` signal send by Docker. (Making it faster when running tests with Complement, as we don't take 10 seconds to timeout)
2022-08-23 13:10:29 +02:00
Till 33129c02f7
Add timeout parameter & trim URL (#2666)
A timeout of 10 seconds could cause issues with servers having a high `bcrypt_cost` configured in the config.
This adds a parameter to manually configure the timeout, defaults to 30 seconds.
2022-08-23 11:10:41 +02:00
Neil Alexander 2668050e53
Tweak soft-failure handling in roomserver
commit 1929b688e31987c46e0c8a546f0f9cb0a46bf9a3
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Aug 22 10:09:44 2022 +0100

    Still process state-before for soft-failed events

commit e83c0b701d40d78b92072c4643f6bc6f71b72800
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Aug 22 10:06:50 2022 +0100

    Improve logging

commit 29e26124bc27cb83d449de2a4214b253c594aa93
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Aug 22 09:58:13 2022 +0100

    Don't store soft-failed events as rejected
2022-08-22 10:34:07 +01:00
Till 9dc57122d9
Fetch more data for newly joined rooms in an incremental sync (#2657)
If we've joined a new room in an incremental sync, try fetching more data.
This deflakes the complement server notices test (at least in my tests).
2022-08-19 15:32:24 +02:00
Neil Alexander 56b55a28f5
Version 0.9.4 (#2658)
### Fixes

* A bug in the roomserver around handling rejected outliers has been fixed
* Backfilled events will now use the correct history visibility where possible
* The device list updater backoff has been fixed, which should reduce the number of outbound HTTP requests and `Failed to query device keys for some users` log entries for dead servers
* The `/sync` endpoint will no longer incorrectly return room entries for retired invites which could cause some rooms to show up in the client "Historical" section
* The `/createRoom` endpoint will now correctly populate `is_direct` in invite membership events, which may help clients to classify direct messages correctly
* The `create-account` tool will now log an error if the shared secret is not set in the Dendrite config
* A couple of minor bugs have been fixed in the membership lazy-loading
* Queued EDUs in the federation API are now cached properly
2022-08-19 12:46:14 +01:00
Neil Alexander 5513f182cc
Enforce device list backoffs (#2653)
This ensures that if the device list updater is already backing off a node, we don't try to call processServer again anyway for server just because the server name arrived in the channel. Otherwise we can keep trying to hit a remote server that is offline or not behaving every second and that spams the logs too.
2022-08-19 10:23:09 +01:00
Till 365da70a23
Set historyVisibility for backfilled events over federation (#2656)
This should hopefully deflake Backfill works correctly with history visibility set to joined as we were using the default shared visibility, even if the events are set to joined (or something else)
2022-08-19 11:04:26 +02:00
Till 5cacca92d2
Make SyncAPI unit tests more reliable (#2655)
This should hopefully make some SyncAPI tests more reliable
2022-08-19 11:03:55 +02:00
Winter a379d3e814
De-race TestExpireEDUs (#2654)
In some conditions (fast CPUs), this test would race the clock for EDU expiration when all we want to make sure of is that the expired EDUs are properly deleted. Given this, we set the expiry time to 0 so the specified EDUs are always deleted when DeleteExpiredEDUs is called.
Fixes #2650.

Signed-off-by: Winter <winter@winter.cafe>
2022-08-19 07:28:33 +02:00
Neil Alexander 6b48ce0d75
State handling tweaks (#2652)
This tweaks how rejected events are handled in room state and also to not apply checks we can't complete to outliers.
2022-08-18 17:06:13 +01:00
Neil Alexander 606cb67506
Enable workflow_dispatch in GHA 2022-08-18 13:50:58 +01:00
Till 7484689ad1
Actually store EDUs once we retrieved from the database (#2651)
We now actually cache the EDUs once we got them from the database and ensures we only evict them if we successfully deleted them.
2022-08-18 11:14:42 +01:00
Neil Alexander 59bc0a6f4e
Reprocess rejected input events (#2647)
* Reprocess outliers that were previously rejected

* Might as well do all events this way

* More useful errors

* Fix queries

* Tweak condition

* Don't wrap errors

* Report more useful error

* Flatten error on `r.Queryer.QueryStateAfterEvents`

* Some more debug logging

* Flatten error in `QueryRestrictedJoinAllowed`

* Revert "Flatten error in `QueryRestrictedJoinAllowed`"

This reverts commit 1238b4184c.

* Tweak `QueryStateAfterEvents`

* Handle MissingStateError too

* Scope to room

* Clean up

* Fix the error

* Only apply rejection check to outliers
2022-08-18 10:37:47 +01:00
Till Faelligen 8d9c8f11c5
Add a delay after sending events to the roomserver 2022-08-18 08:56:57 +02:00
Neil Alexander ad4ac2c016
Stop spamming the logs with StateBetween: ignoring deleted state event IDs 2022-08-16 14:42:35 +01:00
Neil Alexander ec16c944eb
Lazy-loading fixes (#2646)
* Use existing current room state if we have it

* Don't dedupe before applying the history vis filter

* Revert "Don't dedupe before applying the history vis filter"

This reverts commit d27c4a0874.

* Revert "Use existing current room state if we have it"

This reverts commit 5819b4a7ce.

* Tweaks
2022-08-16 14:42:00 +01:00
Till 804653e551
Verify a shared secret is set in create-account (#2645) 2022-08-16 13:21:22 +02:00
Neil Alexander 5424b88f30
Use is_direct flag from /createRoom, update stripped state (#2644)
* Use `is_direct` flag from `/createRoom`, update stripped state

* Add comment
2022-08-16 11:55:06 +01:00
Till 0642ffc0f6
Only return non-retired invites (#2643)
* Only return non-retired invites

* Revert "Only return non-retired invites"

This reverts commit 1150aa7f38.

* Check if we're doing an initial sync in the stream
2022-08-16 10:29:36 +02:00
Neil Alexander bcdbd5c00a
Version 0.9.3 2022-08-15 10:56:54 +01:00
Till Faelligen 65b7db633c
Fix FAQ question 2022-08-13 20:31:43 +02:00
timaeos da7d209b10
Admin endpoint documentation (#2640)
* Adding documentation for administration end points that aren't currently documented

* Additional information on using the administration API

* Fix inaccurate information on server notice documentation

* Fix typo in the curl command

* Add details about using the whois endpoint

* fix formatting in documentation for admin whois endpoint

* Fix more documentation formatting

* Additional formatting updates to match other formats

* Add a link to the server notice information in the Matrix Spec

* Fix broken link on server notices

* Adding access token information and admin info

* adding a warning regarding access_token

Co-authored-by: Timothy Arnold <Yourmove@gmail.com>
2022-08-13 20:25:39 +02:00
timaeos 30dbc5ac44
Adding FAQ Questions with common questions (#2641)
* Adding FAQ Questions with common questions

* changing order of the FAQ to reflect suggestions
2022-08-13 20:24:52 +02:00
Neil Alexander ef7262a7a2
Version 0.9.2 (#2638) 2022-08-12 15:18:31 +01:00
texuf e55cd6ea78
/hierarchy - return public and knockable rooms for authed users (#2578)
When requesting the room hierarchy with an authenticated user, return public and knockable rooms.

According to the spec, https://github.com/matrix-org/matrix-spec-proposals/blob/main/proposals/2946-spaces-summary.md

```
Any child room that the user is joined or is potentially joinable is included in the response.
```

This is currently not the case. See discussion here: https://matrix.to/#/!NasysSDfxKxZBzJJoE:matrix.org/$t2Csj-6y1PVsn8GOnFZfXzeQW13NfqvrFCxB-XI_uhA?via=matrix.org&via=libera.chat&via=element.io and here: https://matrix.to/#/!NasysSDfxKxZBzJJoE:matrix.org/$EHp1x1DY7tnYZtx_PVEb-sKB9lmJajqHx2uGlhrRh6k?via=matrix.org&via=libera.chat&via=element.io

Test Plan:
create and register clients bob and alice
have bob create a public space
have bob create a public room parented to the space
have alice join the space(room)
have alice sync the space
expect alice to see two rooms in the space hierarchy, the space and the child room

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-08-12 14:07:45 +01:00
Neil Alexander 6d0d7a0bc3
Update example in readme 2022-08-12 13:34:05 +01:00
Neil Alexander 9fa30f5d3c
Update NATS Server and nats.go libraries 2022-08-12 13:29:09 +01:00
Till Faelligen 20bf00b743
Update README 2022-08-12 13:55:45 +02:00
Neil Alexander f1e9108db8
Update create-account help text 2022-08-12 12:53:29 +01:00
Till b4647fbb7e
Show/hide users in user directory (#2637)
* CS API changes

* Query remote profiles

* Add passing tests

* Don't create a new FullyQualifiedProfile

* Handle sql.ErrNoRows

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-08-12 13:33:31 +02:00
Till 48600d5540
Use /admin/v1/register in create-account (#2484)
* Get all account data on CompleteSync

* Revert "Get all account data on CompleteSync"

This reverts commit 44a3e566d8.

* Use /_synapse/admin/v1/register to create account

* Linting

* Linter again :)

* Update docs

* Use HTTP API to reset password, add option to User API `PerformPasswordUpdate` to invalidate sessions

* Fix routing name

* Tell me more about what went wrong

* Deprecate the `-reset-password` flag, document the new API

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-08-12 12:00:07 +01:00
Tak Wai Wong fad3ac8e78
Protect user_interactive reads and writes with locks (#2635)
* Protect user_interactive reads and writes with locks

* Ignore golangci-lint false positive

* fix lint

Co-authored-by: Tak Wai Wong <tak@hntlabs.com>
2022-08-12 09:12:05 +01:00
Neil Alexander 2b352915a1
Update golangci-lint component in GHA workflow 2022-08-11 17:40:57 +01:00
Neil Alexander a01af55ec6
Restore the room version cache in the roomserver internal API HTTP client 2022-08-11 17:34:09 +01:00
Till 05cafbd197
Implement history visibility on /messages, /context, /sync (#2511)
* Add possibility to set history_visibility and user AccountType

* Add new DB queries

* Add actual history_visibility changes for /messages

* Add passing tests

* Extract check function

* Cleanup

* Cleanup

* Fix build on 386

* Move ApplyHistoryVisibilityFilter to internal

* Move queries to topology table

* Add filtering to /sync and /context
Some cleanup

* Add passing tests; Remove failing tests :(

* Re-add passing tests

* Move filtering to own function to avoid duplication

* Re-add passing test

* Use newly added GMSL HistoryVisibility

* Update gomatrixserverlib

* Set the visibility when creating events

* Default to shared history visibility

* Remove unused query

* Update history visibility checks to use gmsl
Update tests

* Remove unused statement

* Update migrations to set "correct" history visibility

* Add method to fetch the membership at a given event

* Tweaks and logging

* Use actual internal rsAPI, default to shared visibility in tests

* Revert "Move queries to topology table"

This reverts commit 4f0d41be9c.

* Remove noise/unneeded code

* More cleanup

* Try to optimize database requests

* Fix imports

* PR peview fixes/changes

* Move setting history visibility to own migration, be more restrictive

* Fix unit tests

* Lint

* Fix missing entries

* Tweaks for incremental syncs

* Adapt generic changes

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: kegsay <kegan@matrix.org>
2022-08-11 18:23:35 +02:00
Neil Alexander 371336c6b5
Set default room version to 9 2022-08-11 16:31:44 +01:00