Commit graph

2480 commits

Author SHA1 Message Date
Till Faelligen aba171d9bc
Default to limit of 20 state events 2022-08-29 07:23:46 +02:00
Neil Alexander a3eb4e5e98
Enforce type, sender, state_key and room_id lengths using codepoints rather than bytes (update to matrix-org/gomatrixserverlib@5f66df0, matrix-org/matrix-spec#1001) 2022-08-28 09:52:03 +01:00
Neil Alexander 38bed30b41
Restrict username length properly (#2676)
This fixes #2674 by fixing the username check to comply with the [spec appendices](https://spec.matrix.org/v1.3/appendices/#user-identifiers):

> The length of a user ID, including the @ sigil and the domain, MUST NOT exceed 255 characters.
2022-08-26 09:56:41 +01:00
Neil Alexander ed79e8626a
Version 0.9.5 (#2673)
Changelog and version bump.
2022-08-25 14:14:10 +01:00
Till 07dd9bd995
SyncAPI tweaks/fixes (#2671)
- Reverts 9dc57122d9 as it was causing issues https://github.com/matrix-org/dendrite/issues/2660
- Updates the GMSL `DefaultStateFilter` to use a limit of 20 events
- Uses the timeline events to determine the new position instead of the state events
2022-08-25 13:42:47 +01:00
Neil Alexander 8ff3f1a7c9
Remove a couple unnecessary Sentry captures from backfill 2022-08-25 11:01:07 +01:00
Neil Alexander cd7fa34595
Tweak logging and Sentry reporting for roomserver input 2022-08-25 10:57:27 +01:00
Neil Alexander 16156b0b09
Fix 500s on /state, /state_ids when state not known (#2672)
This was due to bad error bubbling.
2022-08-25 09:51:36 +01:00
Neil Alexander 522bd2999f
Allow un-rejecting events on reprocessing 2022-08-24 14:03:06 +01:00
Till 78e5d05efc
Only set backOffStarted to false if until is not zero (#2669) 2022-08-23 16:54:42 +02:00
Neil Alexander 14fea600bb
Detect types.MissingStateError in CheckServerAllowedToSeeEvent (#2667)
This will hopefully stop some 500 errors on `/event` where there is no state-before known.
2022-08-23 13:57:11 +01:00
Till 95a509757a
Complement QoL changes (#2663)
This PR does the following:
- adds a `keysize` parameter to `generate-keys`, so we can use lower sized keys when running in CI
- updates the Complement docker files to use BuildKit (requires Docker >18.09)
- uses `exec` when executing `dendrite-monotlith-server`, making it PID 1 inside docker, which results in Dendrite actually receiving the `SIGTERM` signal send by Docker. (Making it faster when running tests with Complement, as we don't take 10 seconds to timeout)
2022-08-23 13:10:29 +02:00
Till 33129c02f7
Add timeout parameter & trim URL (#2666)
A timeout of 10 seconds could cause issues with servers having a high `bcrypt_cost` configured in the config.
This adds a parameter to manually configure the timeout, defaults to 30 seconds.
2022-08-23 11:10:41 +02:00
Neil Alexander 2668050e53
Tweak soft-failure handling in roomserver
commit 1929b688e31987c46e0c8a546f0f9cb0a46bf9a3
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Aug 22 10:09:44 2022 +0100

    Still process state-before for soft-failed events

commit e83c0b701d40d78b92072c4643f6bc6f71b72800
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Aug 22 10:06:50 2022 +0100

    Improve logging

commit 29e26124bc27cb83d449de2a4214b253c594aa93
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Aug 22 09:58:13 2022 +0100

    Don't store soft-failed events as rejected
2022-08-22 10:34:07 +01:00
Till 9dc57122d9
Fetch more data for newly joined rooms in an incremental sync (#2657)
If we've joined a new room in an incremental sync, try fetching more data.
This deflakes the complement server notices test (at least in my tests).
2022-08-19 15:32:24 +02:00
Neil Alexander 56b55a28f5
Version 0.9.4 (#2658)
### Fixes

* A bug in the roomserver around handling rejected outliers has been fixed
* Backfilled events will now use the correct history visibility where possible
* The device list updater backoff has been fixed, which should reduce the number of outbound HTTP requests and `Failed to query device keys for some users` log entries for dead servers
* The `/sync` endpoint will no longer incorrectly return room entries for retired invites which could cause some rooms to show up in the client "Historical" section
* The `/createRoom` endpoint will now correctly populate `is_direct` in invite membership events, which may help clients to classify direct messages correctly
* The `create-account` tool will now log an error if the shared secret is not set in the Dendrite config
* A couple of minor bugs have been fixed in the membership lazy-loading
* Queued EDUs in the federation API are now cached properly
2022-08-19 12:46:14 +01:00
Neil Alexander 5513f182cc
Enforce device list backoffs (#2653)
This ensures that if the device list updater is already backing off a node, we don't try to call processServer again anyway for server just because the server name arrived in the channel. Otherwise we can keep trying to hit a remote server that is offline or not behaving every second and that spams the logs too.
2022-08-19 10:23:09 +01:00
Till 365da70a23
Set historyVisibility for backfilled events over federation (#2656)
This should hopefully deflake Backfill works correctly with history visibility set to joined as we were using the default shared visibility, even if the events are set to joined (or something else)
2022-08-19 11:04:26 +02:00
Till 5cacca92d2
Make SyncAPI unit tests more reliable (#2655)
This should hopefully make some SyncAPI tests more reliable
2022-08-19 11:03:55 +02:00
Winter a379d3e814
De-race TestExpireEDUs (#2654)
In some conditions (fast CPUs), this test would race the clock for EDU expiration when all we want to make sure of is that the expired EDUs are properly deleted. Given this, we set the expiry time to 0 so the specified EDUs are always deleted when DeleteExpiredEDUs is called.
Fixes #2650.

Signed-off-by: Winter <winter@winter.cafe>
2022-08-19 07:28:33 +02:00
Neil Alexander 6b48ce0d75
State handling tweaks (#2652)
This tweaks how rejected events are handled in room state and also to not apply checks we can't complete to outliers.
2022-08-18 17:06:13 +01:00
Neil Alexander 606cb67506
Enable workflow_dispatch in GHA 2022-08-18 13:50:58 +01:00
Till 7484689ad1
Actually store EDUs once we retrieved from the database (#2651)
We now actually cache the EDUs once we got them from the database and ensures we only evict them if we successfully deleted them.
2022-08-18 11:14:42 +01:00
Neil Alexander 59bc0a6f4e
Reprocess rejected input events (#2647)
* Reprocess outliers that were previously rejected

* Might as well do all events this way

* More useful errors

* Fix queries

* Tweak condition

* Don't wrap errors

* Report more useful error

* Flatten error on `r.Queryer.QueryStateAfterEvents`

* Some more debug logging

* Flatten error in `QueryRestrictedJoinAllowed`

* Revert "Flatten error in `QueryRestrictedJoinAllowed`"

This reverts commit 1238b4184c.

* Tweak `QueryStateAfterEvents`

* Handle MissingStateError too

* Scope to room

* Clean up

* Fix the error

* Only apply rejection check to outliers
2022-08-18 10:37:47 +01:00
Till Faelligen 8d9c8f11c5
Add a delay after sending events to the roomserver 2022-08-18 08:56:57 +02:00
Neil Alexander ad4ac2c016
Stop spamming the logs with StateBetween: ignoring deleted state event IDs 2022-08-16 14:42:35 +01:00
Neil Alexander ec16c944eb
Lazy-loading fixes (#2646)
* Use existing current room state if we have it

* Don't dedupe before applying the history vis filter

* Revert "Don't dedupe before applying the history vis filter"

This reverts commit d27c4a0874.

* Revert "Use existing current room state if we have it"

This reverts commit 5819b4a7ce.

* Tweaks
2022-08-16 14:42:00 +01:00
Till 804653e551
Verify a shared secret is set in create-account (#2645) 2022-08-16 13:21:22 +02:00
Neil Alexander 5424b88f30
Use is_direct flag from /createRoom, update stripped state (#2644)
* Use `is_direct` flag from `/createRoom`, update stripped state

* Add comment
2022-08-16 11:55:06 +01:00
Till 0642ffc0f6
Only return non-retired invites (#2643)
* Only return non-retired invites

* Revert "Only return non-retired invites"

This reverts commit 1150aa7f38.

* Check if we're doing an initial sync in the stream
2022-08-16 10:29:36 +02:00
Neil Alexander bcdbd5c00a
Version 0.9.3 2022-08-15 10:56:54 +01:00
Tak Wai Wong ea0a8804d2 Pull latest dendrite fork into harmony (#252)
Latest dendrite main has changes for knockable rooms, and the fix for login crash. Pulled into dendrite fork. Rebased dendrite fork from dendrite main.

Co-authored-by: Till <2353100+S7evinK@users.noreply.github.com>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: kegsay <kegan@matrix.org>
Co-authored-by: Tak Wai Wong <tak@hntlabs.com>
Co-authored-by: texuf <texuf.eth@gmail.com>
Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
2022-08-13 16:09:49 -07:00
Tak Wai Wong 4a9ebde5d4 Copy paste sytest-blacklist from main 2022-08-13 13:47:56 -07:00
Tak Wai Wong 305b805f3b revert unintended rebase changes. 2022-08-13 13:10:06 -07:00
Tak Wai Wong b4ff83bd96 Remove nonce generation for eip4361 signin (#25)
Co-authored-by: Tak Wai Wong <tak@hntlabs.com>
2022-08-13 13:01:21 -07:00
texuf 850788ba1a Fix lint errors
https://github.com/HereNotThere/dendrite/runs/7417009281?check_suite_focus=true
2022-08-13 13:01:21 -07:00
Tak Wai Wong a501c9b936 Login and Register tests for public key ethereum (#16)
* TestLoginPublicKeyNewSession

* use asserts

* setup, test, asserts

* TestLoginPublicKeyValidAuthTypeMissingSession

* invalid session id test

* create a helper newSession function

* TestLoginPublicKeyEthereumMissingUserId

* TestLoginPublicKeyEthereumAccountNotAvailable

* TestLoginPublicKeyEthereumInvalidUserId

* createEip4361TestMessage

* TestLoginPublicKeyEthereumMissingSignature

* TestLoginPublicKeyEthereum

* re-enable all publickey signin tests

* move common publickey test util to its own file

* register_public_key.go stub

* refactored common ethereum test helpers to its own folder

* refactor test helpers

* return error in test helpers

* fix regressions with ServerName

* TestRegistrationUnimplementedAlgo

* TestNewRegistration

* TestNewRegistrationSession

* verify new login session

* remove assert

* perform account creation

* TestRegisterEthereum

* Enable all tests

* move helper functions into test file

Co-authored-by: Tak Wai Wong <tak@hntlabs.com>
2022-08-13 12:59:33 -07:00
Tak Wai Wong 6f8881eadf merge latest changes from dendrite main (#15)
Co-authored-by: Tak Wai Wong <tak@hntlabs.com>
2022-08-13 12:57:17 -07:00
Till Faelligen 65b7db633c
Fix FAQ question 2022-08-13 20:31:43 +02:00
timaeos da7d209b10
Admin endpoint documentation (#2640)
* Adding documentation for administration end points that aren't currently documented

* Additional information on using the administration API

* Fix inaccurate information on server notice documentation

* Fix typo in the curl command

* Add details about using the whois endpoint

* fix formatting in documentation for admin whois endpoint

* Fix more documentation formatting

* Additional formatting updates to match other formats

* Add a link to the server notice information in the Matrix Spec

* Fix broken link on server notices

* Adding access token information and admin info

* adding a warning regarding access_token

Co-authored-by: Timothy Arnold <Yourmove@gmail.com>
2022-08-13 20:25:39 +02:00
timaeos 30dbc5ac44
Adding FAQ Questions with common questions (#2641)
* Adding FAQ Questions with common questions

* changing order of the FAQ to reflect suggestions
2022-08-13 20:24:52 +02:00
Tak Wai Wong ae011f8e54 Implement EIP-4361 sign in with Ethereum (#5)
* Blacklist some sytest tests that are failing in our environment

* Commenting out test that isn't reliably passing or failing, probably a race

* refresh latest dendrite main

* refresh latest dendrite main

* dendrite implementation of eip-4361

* simplify nonce generation

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
2022-08-12 18:06:39 -07:00
Tak Wai Wong b2717519f8 Takwaiw/dendrite publickey (#2)
* Implementation of MSC 3782 Add publickey login as a new auth type.

Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
2022-08-12 17:58:23 -07:00
Brian Meek 6c7d03c925 Test_UserStatistics Fix expected results to match observed results 2022-08-12 17:37:47 -07:00
Tak Wai Wong 0a99dac14b Takwaiw/fix concurrent registration bug (#12)
* fix concurrent registration bug. Rename decentralizedid

* remove unused module

* add regressed test to blacklist

Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
2022-08-12 17:37:40 -07:00
Tak Wai Wong 655df91021 Verify that the user ID for registration matches the spec, and the auth data (#10)
* Blacklist some sytest tests that are failing in our environment

* Commenting out test that isn't reliably passing or failing, probably a race

* refresh latest dendrite main

* pull latest from dendrite-fork subtree

* refresh latest dendrite main

* pull dendrite subtree and resolve merge conflicts

* check that userID matches the signed message

* verify that the user ID for registration is CAIP-10 compliant and MXID compliant

* removed space

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
2022-08-12 17:36:50 -07:00
Tak Wai Wong 7a882f31e6 Create session ID during registration (#8)
* Blacklist some sytest tests that are failing in our environment

* Commenting out test that isn't reliably passing or failing, probably a race

* refresh latest dendrite main

* pull latest from dendrite-fork subtree

* refresh latest dendrite main

* Create session ID during registration

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
2022-08-12 17:36:50 -07:00
Tak Wai Wong dd101b5281 Use rand.Seed to seed the random function generator (#6)
* Blacklist some sytest tests that are failing in our environment

* Commenting out test that isn't reliably passing or failing, probably a race

* refresh latest dendrite main

* use rand.Seed to seed the random function

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
2022-08-12 17:36:50 -07:00
Tak Wai Wong da9334831a Implement EIP-4361 sign in with Ethereum (#5)
* Blacklist some sytest tests that are failing in our environment

* Commenting out test that isn't reliably passing or failing, probably a race

* refresh latest dendrite main

* refresh latest dendrite main

* dendrite implementation of eip-4361

* simplify nonce generation

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
2022-08-12 17:36:47 -07:00
Tak Wai Wong 76e663a046 Refresh latest dendrite (#4)
* Blacklist some sytest tests that are failing in our environment

* Commenting out test that isn't reliably passing or failing, probably a race

* refresh latest dendrite main

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
2022-08-12 17:29:39 -07:00