Commit graph

2475 commits

Author SHA1 Message Date
Tak Wai Wong 6a5b223584
Pull the fix for user_interactive crash from dendrite fork (#249)
* Fix issues with migrations not getting executed (#2628)

* Fix issues with migrations not getting executed

* Check actual postgres error

* Return error if it's not "column does not exist"

* Send-to-device/sync tweaks (#2630)

* Always delete send to device messages

* Omit empty to_device

* Tweak /sync response to omit empty values

* Add housekeeping function to delete old/expired EDUs (#2399)

* Add housekeeping function to delete old/expired EDUs

* Add migrations

* Evict EDUs from cache

* Fix queries

* Fix upgrade

* Use map[string]time.Duration to specify different expiry times

* Fix copy & paste mistake

* Set expires_at to tomorrow

* Don't allow NULL

* Add comment

* Add tests

* Use new testrig package

* Fix migrations

* Never expire m.direct_to_device

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: kegsay <kegan@matrix.org>

* Remove nonce generation for eip4361 signin (#25)

Co-authored-by: Tak Wai Wong <tak@hntlabs.com>

* Fix user_interactive crash when multiple clients try to log in (#26)

* add rw locks

* lock / unlock protect other fields

Co-authored-by: Tak Wai Wong <tak@hntlabs.com>

Co-authored-by: Till <2353100+S7evinK@users.noreply.github.com>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: kegsay <kegan@matrix.org>
Co-authored-by: Tak Wai Wong <tak@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
2022-10-24 18:02:55 -07:00
Tak Wai Wong cc98ffb362
Tak/pull-dendrite-changes (#245)
* Fix issues with migrations not getting executed (#2628)

* Fix issues with migrations not getting executed

* Check actual postgres error

* Return error if it's not "column does not exist"

* Remove nonce generation for eip4361 signin (#25)

Co-authored-by: Tak Wai Wong <tak@hntlabs.com>
2022-10-24 17:58:05 -07:00
Brian Meek b41086e758
Hoist dendrite local test files out of dendrite directory as part of process of removing dendrite as a subtree from the harmony repo
Signed-off-by: Brian Meek <brian@hntlabs.com>
2022-10-24 17:56:49 -07:00
sergekh2 de5ca88d35
Add script for local testing to clean data, but keep existing config in place 2022-10-24 17:56:49 -07:00
sergekh2 25e1997986
Change local test deployment scripts to deploy multiple dendrites, use separate dirs 2022-10-24 17:56:49 -07:00
texuf ed906910cf
Fix lint errors
https://github.com/HereNotThere/dendrite/runs/7417009281?check_suite_focus=true
2022-10-24 17:56:49 -07:00
texuf e69af9405a
/hierarchy - return public and knockable rooms for authed users
When requesting the room hierarchy with an authenticated user, return public and knockable rooms.

According to the spec, https://github.com/matrix-org/matrix-spec-proposals/blob/main/proposals/2946-spaces-summary.md

```
Any child room that the user is joined or is potentially joinable is included in the response.
```

This is currently not the case. See discussion here: https://matrix.to/#/!NasysSDfxKxZBzJJoE:matrix.org/$t2Csj-6y1PVsn8GOnFZfXzeQW13NfqvrFCxB-XI_uhA?via=matrix.org&via=libera.chat&via=element.io and here: https://matrix.to/#/!NasysSDfxKxZBzJJoE:matrix.org/$EHp1x1DY7tnYZtx_PVEb-sKB9lmJajqHx2uGlhrRh6k?via=matrix.org&via=libera.chat&via=element.io

Test Plan:
create and register clients bob and alice
have bob create a public space
have bob create a public room parented to the space
have alice join the space(room)
have alice sync the space
expect alice to see two rooms in the space hierarchy, the space and the child room
2022-10-24 17:56:47 -07:00
Tak Wai Wong 5ac83189d9
Login and Register tests for public key ethereum (#16)
* TestLoginPublicKeyNewSession

* use asserts

* setup, test, asserts

* TestLoginPublicKeyValidAuthTypeMissingSession

* invalid session id test

* create a helper newSession function

* TestLoginPublicKeyEthereumMissingUserId

* TestLoginPublicKeyEthereumAccountNotAvailable

* TestLoginPublicKeyEthereumInvalidUserId

* createEip4361TestMessage

* TestLoginPublicKeyEthereumMissingSignature

* TestLoginPublicKeyEthereum

* re-enable all publickey signin tests

* move common publickey test util to its own file

* register_public_key.go stub

* refactored common ethereum test helpers to its own folder

* refactor test helpers

* return error in test helpers

* fix regressions with ServerName

* TestRegistrationUnimplementedAlgo

* TestNewRegistration

* TestNewRegistrationSession

* verify new login session

* remove assert

* perform account creation

* TestRegisterEthereum

* Enable all tests

* move helper functions into test file

Co-authored-by: Tak Wai Wong <tak@hntlabs.com>
2022-10-24 17:53:16 -07:00
Tak Wai Wong d5050c1d61
merge latest changes from dendrite main (#15)
Co-authored-by: Tak Wai Wong <tak@hntlabs.com>
2022-10-24 17:52:01 -07:00
Tak Wai Wong b5d5f57a1c
Implement EIP-4361 sign in with Ethereum (#5)
* Blacklist some sytest tests that are failing in our environment

* Commenting out test that isn't reliably passing or failing, probably a race

* refresh latest dendrite main

* refresh latest dendrite main

* dendrite implementation of eip-4361

* simplify nonce generation

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
2022-10-24 17:43:01 -07:00
Tak Wai Wong 8ab1737e7f
Takwaiw/dendrite publickey (#2)
* Implementation of MSC 3782 Add publickey login as a new auth type.

Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
2022-10-24 17:40:09 -07:00
Brian Meek 77e2c918fd
Test_UserStatistics Fix expected results to match observed results 2022-10-24 17:33:11 -07:00
Tak Wai Wong 916f17ceb7
Pull latest changes from dendrite fork (#193)
* Verify that the user ID for registration matches the spec, and the auth data (#10)

* Blacklist some sytest tests that are failing in our environment

* Commenting out test that isn't reliably passing or failing, probably a race

* refresh latest dendrite main

* pull latest from dendrite-fork subtree

* refresh latest dendrite main

* pull dendrite subtree and resolve merge conflicts

* check that userID matches the signed message

* verify that the user ID for registration is CAIP-10 compliant and MXID compliant

* removed space

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>

* Fix nats.go commit (#2540)

Signed-off-by: Jean Lucas <jean@4ray.co>

* Don't return `end` if there are not more messages (#2542)

* Be more spec compliant

* Move lazyLoadMembers to own method

* Return an error if trying to invite a malformed user ID (#2543)

* Add `evacuateUser` endpoint, use it when deactivating accounts (#2545)

* Add `evacuateUser` endpoint, use it when deactivating accounts

* Populate the API

* Clean up user devices when deactivating

* Include invites, delete pushers

* Silence presence logs (#2547)

* Blacklist `Guest users can join guest_access rooms` test until it can be investigated

* Disable WebAssembly builds for now

* Try to fix backfilling (#2548)

* Try to fix backfilling

* Return start/end to not confuse clients

* Update GMSL

* Update GMSL

* Roomserver producers package (#2546)

* Give the roomserver a producers package

* Change init point

* Populate ACLs API

* Fix build issues

* `RoomEventProducer` naming

* Version 0.8.9 (#2549)

* Version 0.8.9

* Update changelog

* Takwaiw/fix concurrent registration bug (#12)

* fix concurrent registration bug. Rename decentralizedid

* remove unused module

* add regressed test to blacklist

Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>

* Test_UserStatistics Fix expected results to match observed results

* Takwaiw/dendrite publickey (#2)

* Implementation of MSC 3782 Add publickey login as a new auth type.

Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>

* Implement EIP-4361 sign in with Ethereum (#5)

* Blacklist some sytest tests that are failing in our environment

* Commenting out test that isn't reliably passing or failing, probably a race

* refresh latest dendrite main

* refresh latest dendrite main

* dendrite implementation of eip-4361

* simplify nonce generation

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>

* Use rand.Seed to seed the random function generator (#6)

* Blacklist some sytest tests that are failing in our environment

* Commenting out test that isn't reliably passing or failing, probably a race

* refresh latest dendrite main

* use rand.Seed to seed the random function

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>

* Create session ID during registration (#8)

* Blacklist some sytest tests that are failing in our environment

* Commenting out test that isn't reliably passing or failing, probably a race

* refresh latest dendrite main

* pull latest from dendrite-fork subtree

* refresh latest dendrite main

* Create session ID during registration

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>

* Verify that the user ID for registration matches the spec, and the auth data (#10)

* Blacklist some sytest tests that are failing in our environment

* Commenting out test that isn't reliably passing or failing, probably a race

* refresh latest dendrite main

* pull latest from dendrite-fork subtree

* refresh latest dendrite main

* pull dendrite subtree and resolve merge conflicts

* check that userID matches the signed message

* verify that the user ID for registration is CAIP-10 compliant and MXID compliant

* removed space

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>

* Takwaiw/fix concurrent registration bug (#12)

* fix concurrent registration bug. Rename decentralizedid

* remove unused module

* add regressed test to blacklist

Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>

* removed unused module

* feat+fix: Ignore unknown keys and verify required fields are present in appservice registration files (#2550)

* fix: ignore unknown keys in appservice configs

fixes matrix-org/dendrite#1567

* feat: verify required fields in appservice configs

* Use new testrig for key changes tests (#2552)

* Use new testrig for tests

* Log the error message

* Fix QuerySharedUsers for the SyncAPI keychange consumer (#2554)

* Make more use of base.BaseDendrite

* Fix QuerySharedUsers if no UserIDs are supplied

* Return clearer error when no state NID exists for an event (#2555)

* Wrap error from `SnapshotNIDFromEventID`

* Hopefully fix read receipts timestamps (#2557)

This should avoid coercions between signed and unsigned ints which might fix problems like `sql: converting argument $5 type: uint64 values with high bit set are not supported`.

* fix concurrency issue when checking session ID (#14)

Co-authored-by: Tak Wai Wong <tak@hntlabs.com>

* merge latest changes from dendrite main (#15)

Co-authored-by: Tak Wai Wong <tak@hntlabs.com>

* Login and Register tests for public key ethereum (#16)

* TestLoginPublicKeyNewSession

* use asserts

* setup, test, asserts

* TestLoginPublicKeyValidAuthTypeMissingSession

* invalid session id test

* create a helper newSession function

* TestLoginPublicKeyEthereumMissingUserId

* TestLoginPublicKeyEthereumAccountNotAvailable

* TestLoginPublicKeyEthereumInvalidUserId

* createEip4361TestMessage

* TestLoginPublicKeyEthereumMissingSignature

* TestLoginPublicKeyEthereum

* re-enable all publickey signin tests

* move common publickey test util to its own file

* register_public_key.go stub

* refactored common ethereum test helpers to its own folder

* refactor test helpers

* return error in test helpers

* fix regressions with ServerName

* TestRegistrationUnimplementedAlgo

* TestNewRegistration

* TestNewRegistrationSession

* verify new login session

* remove assert

* perform account creation

* TestRegisterEthereum

* Enable all tests

* move helper functions into test file

Co-authored-by: Tak Wai Wong <tak@hntlabs.com>

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
Co-authored-by: Jean Lucas <jean@4ray.co>
Co-authored-by: Till <2353100+S7evinK@users.noreply.github.com>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: Tak Wai Wong <tak@hntlabs.com>
Co-authored-by: Kabir Kwatra <kabir@kwatra.me>
2022-10-24 17:26:35 -07:00
sergekh2 14d1a604ee
Config, certs and script to run local dendrite test server 2022-10-24 17:16:01 -07:00
Tak Wai Wong 1d906290cb
Takwaiw/pull latest dendrite fork changes (#137)
* Verify that the user ID for registration matches the spec, and the auth data (#10)

* Blacklist some sytest tests that are failing in our environment

* Commenting out test that isn't reliably passing or failing, probably a race

* refresh latest dendrite main

* pull latest from dendrite-fork subtree

* refresh latest dendrite main

* pull dendrite subtree and resolve merge conflicts

* check that userID matches the signed message

* verify that the user ID for registration is CAIP-10 compliant and MXID compliant

* removed space

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>

* Fix nats.go commit (#2540)

Signed-off-by: Jean Lucas <jean@4ray.co>

* Don't return `end` if there are not more messages (#2542)

* Be more spec compliant

* Move lazyLoadMembers to own method

* Return an error if trying to invite a malformed user ID (#2543)

* Add `evacuateUser` endpoint, use it when deactivating accounts (#2545)

* Add `evacuateUser` endpoint, use it when deactivating accounts

* Populate the API

* Clean up user devices when deactivating

* Include invites, delete pushers

* Silence presence logs (#2547)

* Blacklist `Guest users can join guest_access rooms` test until it can be investigated

* Disable WebAssembly builds for now

* Try to fix backfilling (#2548)

* Try to fix backfilling

* Return start/end to not confuse clients

* Update GMSL

* Update GMSL

* Roomserver producers package (#2546)

* Give the roomserver a producers package

* Change init point

* Populate ACLs API

* Fix build issues

* `RoomEventProducer` naming

* Version 0.8.9 (#2549)

* Version 0.8.9

* Update changelog

* Takwaiw/fix concurrent registration bug (#12)

* fix concurrent registration bug. Rename decentralizedid

* remove unused module

* add regressed test to blacklist

Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>

* Test_UserStatistics Fix expected results to match observed results

* Takwaiw/dendrite publickey (#2)

* Implementation of MSC 3782 Add publickey login as a new auth type.

Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>

* Implement EIP-4361 sign in with Ethereum (#5)

* Blacklist some sytest tests that are failing in our environment

* Commenting out test that isn't reliably passing or failing, probably a race

* refresh latest dendrite main

* refresh latest dendrite main

* dendrite implementation of eip-4361

* simplify nonce generation

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>

* Use rand.Seed to seed the random function generator (#6)

* Blacklist some sytest tests that are failing in our environment

* Commenting out test that isn't reliably passing or failing, probably a race

* refresh latest dendrite main

* use rand.Seed to seed the random function

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>

* Create session ID during registration (#8)

* Blacklist some sytest tests that are failing in our environment

* Commenting out test that isn't reliably passing or failing, probably a race

* refresh latest dendrite main

* pull latest from dendrite-fork subtree

* refresh latest dendrite main

* Create session ID during registration

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>

* Verify that the user ID for registration matches the spec, and the auth data (#10)

* Blacklist some sytest tests that are failing in our environment

* Commenting out test that isn't reliably passing or failing, probably a race

* refresh latest dendrite main

* pull latest from dendrite-fork subtree

* refresh latest dendrite main

* pull dendrite subtree and resolve merge conflicts

* check that userID matches the signed message

* verify that the user ID for registration is CAIP-10 compliant and MXID compliant

* removed space

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>

* Takwaiw/fix concurrent registration bug (#12)

* fix concurrent registration bug. Rename decentralizedid

* remove unused module

* add regressed test to blacklist

Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>

* removed unused module

* feat+fix: Ignore unknown keys and verify required fields are present in appservice registration files (#2550)

* fix: ignore unknown keys in appservice configs

fixes matrix-org/dendrite#1567

* feat: verify required fields in appservice configs

* Use new testrig for key changes tests (#2552)

* Use new testrig for tests

* Log the error message

* Fix QuerySharedUsers for the SyncAPI keychange consumer (#2554)

* Make more use of base.BaseDendrite

* Fix QuerySharedUsers if no UserIDs are supplied

* Return clearer error when no state NID exists for an event (#2555)

* Wrap error from `SnapshotNIDFromEventID`

* Hopefully fix read receipts timestamps (#2557)

This should avoid coercions between signed and unsigned ints which might fix problems like `sql: converting argument $5 type: uint64 values with high bit set are not supported`.

* fix concurrency issue when checking session ID (#14)

Co-authored-by: Tak Wai Wong <tak@hntlabs.com>

* resolve merge conflicts

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
Co-authored-by: Jean Lucas <jean@4ray.co>
Co-authored-by: Till <2353100+S7evinK@users.noreply.github.com>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: Tak Wai Wong <tak@hntlabs.com>
Co-authored-by: Kabir Kwatra <kabir@kwatra.me>
2022-10-24 17:15:58 -07:00
Tak Wai Wong f1d871fbea
Takwaiw/refresh dendrite fork (#132)
* Verify that the user ID for registration matches the spec, and the auth data (#10)

* Blacklist some sytest tests that are failing in our environment

* Commenting out test that isn't reliably passing or failing, probably a race

* refresh latest dendrite main

* pull latest from dendrite-fork subtree

* refresh latest dendrite main

* pull dendrite subtree and resolve merge conflicts

* check that userID matches the signed message

* verify that the user ID for registration is CAIP-10 compliant and MXID compliant

* removed space

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>

* Fix nats.go commit (#2540)

Signed-off-by: Jean Lucas <jean@4ray.co>

* Don't return `end` if there are not more messages (#2542)

* Be more spec compliant

* Move lazyLoadMembers to own method

* Return an error if trying to invite a malformed user ID (#2543)

* Add `evacuateUser` endpoint, use it when deactivating accounts (#2545)

* Add `evacuateUser` endpoint, use it when deactivating accounts

* Populate the API

* Clean up user devices when deactivating

* Include invites, delete pushers

* Silence presence logs (#2547)

* Takwaiw/fix concurrent registration bug (#12)

* fix concurrent registration bug. Rename decentralizedid

* remove unused module

* add regressed test to blacklist

Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
Co-authored-by: Jean Lucas <jean@4ray.co>
Co-authored-by: Till <2353100+S7evinK@users.noreply.github.com>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-10-24 16:56:03 -07:00
Tak Wai Wong 57cdd1ff65
Latest dendrite main refresh (#129)
* Verify that the user ID for registration matches the spec, and the auth data (#10)

* Blacklist some sytest tests that are failing in our environment

* Commenting out test that isn't reliably passing or failing, probably a race

* refresh latest dendrite main

* pull latest from dendrite-fork subtree

* refresh latest dendrite main

* pull dendrite subtree and resolve merge conflicts

* check that userID matches the signed message

* verify that the user ID for registration is CAIP-10 compliant and MXID compliant

* removed space

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>

* Fix nats.go commit (#2540)

Signed-off-by: Jean Lucas <jean@4ray.co>

* Don't return `end` if there are not more messages (#2542)

* Be more spec compliant

* Move lazyLoadMembers to own method

* Return an error if trying to invite a malformed user ID (#2543)

* Add `evacuateUser` endpoint, use it when deactivating accounts (#2545)

* Add `evacuateUser` endpoint, use it when deactivating accounts

* Populate the API

* Clean up user devices when deactivating

* Include invites, delete pushers

* Silence presence logs (#2547)

* update sytest blacklist

* Add new test that passed to the whitelist

* skip failed test

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
Co-authored-by: Jean Lucas <jean@4ray.co>
Co-authored-by: Till <2353100+S7evinK@users.noreply.github.com>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-10-24 16:54:16 -07:00
Tak Wai Wong c28715c5ff
Verify that the user ID for registration matches the spec, and the auth data (#10) (#104)
* Blacklist some sytest tests that are failing in our environment

* Commenting out test that isn't reliably passing or failing, probably a race

* refresh latest dendrite main

* pull latest from dendrite-fork subtree

* refresh latest dendrite main

* pull dendrite subtree and resolve merge conflicts

* check that userID matches the signed message

* verify that the user ID for registration is CAIP-10 compliant and MXID compliant

* removed space

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
2022-10-24 16:42:34 -07:00
Tak Wai Wong 3e014fb70c
pull dendrite subtree and resolve merge conflicts 2022-10-24 16:39:43 -07:00
Tak Wai Wong 1683a17dbf
refresh latest dendrite main 2022-10-24 16:39:40 -07:00
Tak Wai Wong 34342934cc
pull latest from dendrite-fork subtree 2022-10-24 16:35:54 -07:00
Tak Wai Wong 88bbaed247
refresh latest dendrite main 2022-10-24 16:32:29 -07:00
Brian Meek 87851ae919
Blacklist some sytest tests that are failing in our environment 2022-10-24 16:22:06 -07:00
Tak Wai Wong a6668726ac
Takwaiw/dendrite publickey (#2)
* Implementation of MSC 3782 Add publickey login as a new auth type.

Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
2022-10-24 16:21:40 -07:00
Brian Meek 0fa2f8ff60
Test_UserStatistics Fix expected results to match observed results 2022-10-24 16:17:55 -07:00
Till 7506e3303e
Get messages from before user left the room (#2824)
This is going to make `Can get rooms/{roomId}/messages for a departed
room (SPEC-216)` pass, since we now only grep events from before the
user left the room.
2022-10-24 17:03:04 +02:00
Neil Alexander a553fe7705
Fix slow querying of cross-signing signatures 2022-10-24 10:07:50 +01:00
Till Faelligen 0843bd776e
Fix wrong config key 2022-10-24 07:10:50 +02:00
Neil Alexander 411db6083b
Version 0.10.4 (#2822)
Changelog and version bump.
2022-10-21 15:00:51 +01:00
Till 3cf42a1d64
Add syncapi_memberships table tests (#2805) 2022-10-21 12:53:04 +02:00
Till 9e4c3171da
Optimize inserting pending PDUs/EDUs (#2821)
This optimizes the association of PDUs/EDUs to their destination by
inserting all destinations in one transaction.
2022-10-21 12:50:51 +02:00
Neil Alexander e98d75fd63
Verify room_id, type, sender and state_key field lengths using bytes rather than codepoints (update to matrix-org/gomatrixserverlib@7c772f1, reverts bbb3ade4a2) 2022-10-21 10:15:08 +01:00
Till e57b301722
Set display_name and/or avatar_url for server notices (#2820)
This should fix #2815 by making sure we actually set the `display_name`
and/or `avatar_url` and create the needed membership event.
To avoid creating a new membership event when starting Dendrite,
`SetAvatarURL` and `SetDisplayName` now return a `Changed` value, which
also makes the regular endpoints idempotent.
2022-10-21 10:48:25 +02:00
Till 40cfb9a4ea
Fix invite -> leave -> join dance when accepting invites (#2817)
As mentioned in
https://github.com/matrix-org/dendrite/issues/2361#issuecomment-1139394565
and observed by ourselves, this should fix the odd `invite -> leave ->
join` dance when accepting invites.
2022-10-21 09:26:22 +01:00
Neil Alexander 73e02463cf
Allow m.read.private to clear notifications (#2811)
Otherwise if a user switches to private read receipts, they may not be
able to clear notification counts.
2022-10-21 09:19:52 +01:00
devonh 9041491201
Mutex protect query keys response (#2812) 2022-10-20 15:54:18 +00:00
devonh b58c9bb094
Fix flakey queue test (#2818)
Ensure both events are added to the database, even if the destination is
already blacklisted.
2022-10-20 15:37:35 +00:00
Till Faelligen 539c61b3db
Remove test from blacklist 2022-10-20 12:34:53 +02:00
Till Faelligen 6a93858125
Fix race condition 2022-10-20 10:45:59 +02:00
Till e79bfd8fd5
Get state deltas without filters (#2810)
This makes the following changes:
- get state deltas without the user supplied filter, so we can actually
"calculate" state transitions
- closes `stmt` when using SQLite
- Adds presence for users who newly joined a room, even if the syncing
user already knows about the presence status (should fix
https://github.com/matrix-org/complement/pull/516)
2022-10-19 14:05:39 +02:00
Neil Alexander 8cbe14bd6d
Fix lock contention 2022-10-19 12:27:34 +01:00
Neil Alexander c1463db6c9
Fix concurrent map write in key server 2022-10-19 12:03:12 +01:00
dependabot[bot] f3dae0e749
Bump nokogiri from 1.13.6 to 1.13.9 in /docs (#2809)
Bumps [nokogiri](https://github.com/sparklemotion/nokogiri) from 1.13.6
to 1.13.9.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/sparklemotion/nokogiri/releases">nokogiri's
releases</a>.</em></p>
<blockquote>
<h2>1.13.9 / 2022-10-18</h2>
<h3>Security</h3>
<ul>
<li>[CRuby] Vendored libxml2 is updated to address <a
href="https://nvd.nist.gov/vuln/detail/CVE-2022-2309">CVE-2022-2309</a>,
<a
href="https://nvd.nist.gov/vuln/detail/CVE-2022-40304">CVE-2022-40304</a>,
and <a
href="https://nvd.nist.gov/vuln/detail/CVE-2022-40303">CVE-2022-40303</a>.
See <a
href="https://github.com/sparklemotion/nokogiri/security/advisories/GHSA-2qc6-mcvw-92cw">GHSA-2qc6-mcvw-92cw</a>
for more information.</li>
<li>[CRuby] Vendored zlib is updated to address <a
href="https://ubuntu.com/security/CVE-2022-37434">CVE-2022-37434</a>.
Nokogiri was not affected by this vulnerability, but this version of
zlib was being flagged up by some vulnerability scanners, see <a
href="https://github-redirect.dependabot.com/sparklemotion/nokogiri/issues/2626">#2626</a>
for more information.</li>
</ul>
<h3>Dependencies</h3>
<ul>
<li>[CRuby] Vendored libxml2 is updated to <a
href="https://gitlab.gnome.org/GNOME/libxml2/-/releases/v2.10.3">v2.10.3</a>
from v2.9.14.</li>
<li>[CRuby] Vendored libxslt is updated to <a
href="https://gitlab.gnome.org/GNOME/libxslt/-/releases/v1.1.37">v1.1.37</a>
from v1.1.35.</li>
<li>[CRuby] Vendored zlib is updated from 1.2.12 to 1.2.13. (See <a
href="https://github.com/sparklemotion/nokogiri/blob/v1.13.x/LICENSE-DEPENDENCIES.md#platform-releases">LICENSE-DEPENDENCIES.md</a>
for details on which packages redistribute this library.)</li>
</ul>
<h3>Fixed</h3>
<ul>
<li>[CRuby] <code>Nokogiri::XML::Namespace</code> objects, when
compacted, update their internal struct's reference to the Ruby object
wrapper. Previously, with GC compaction enabled, a segmentation fault
was possible after compaction was triggered. [<a
href="https://github-redirect.dependabot.com/sparklemotion/nokogiri/issues/2658">#2658</a>]
(Thanks, <a
href="https://github.com/eightbitraptor"><code>@​eightbitraptor</code></a>
and <a
href="https://github.com/peterzhu2118"><code>@​peterzhu2118</code></a>!)</li>
<li>[CRuby] <code>Document#remove_namespaces!</code> now defers freeing
the underlying <code>xmlNs</code> struct until the <code>Document</code>
is GCed. Previously, maintaining a reference to a <code>Namespace</code>
object that was removed in this way could lead to a segfault. [<a
href="https://github-redirect.dependabot.com/sparklemotion/nokogiri/issues/2658">#2658</a>]</li>
</ul>
<hr />
<p>sha256 checksums:</p>

<pre><code>9b69829561d30c4461ea803baeaf3460e8b145cff7a26ce397119577a4083a02
nokogiri-1.13.9-aarch64-linux.gem
e76ebb4b7b2e02c72b2d1541289f8b0679fb5984867cf199d89b8ef485764956
nokogiri-1.13.9-arm64-darwin.gem
15bae7d08bddeaa898d8e3f558723300137c26a2dc2632a1f89c8574c4467165
nokogiri-1.13.9-java.gem
f6a1dbc7229184357f3129503530af73cc59ceba4932c700a458a561edbe04b9
nokogiri-1.13.9-x64-mingw-ucrt.gem
36d935d799baa4dc488024f71881ff0bc8b172cecdfc54781169c40ec02cbdb3
nokogiri-1.13.9-x64-mingw32.gem
ebaf82aa9a11b8fafb67873d19ee48efb565040f04c898cdce8ca0cd53ff1a12
nokogiri-1.13.9-x86-linux.gem
11789a2a11b28bc028ee111f23311461104d8c4468d5b901ab7536b282504154
nokogiri-1.13.9-x86-mingw32.gem
01830e1646803ff91c0fe94bc768ff40082c6de8cfa563dafd01b3f7d5f9d795
nokogiri-1.13.9-x86_64-darwin.gem
8e93b8adec22958013799c8690d81c2cdf8a90b6f6e8150ab22e11895844d781
nokogiri-1.13.9-x86_64-linux.gem
96f37c1baf0234d3ae54c2c89aef7220d4a8a1b03d2675ff7723565b0a095531
nokogiri-1.13.9.gem
</code></pre>
<h2>1.13.8 / 2022-07-23</h2>
<h3>Deprecated</h3>
<ul>
<li><code>XML::Reader#attribute_nodes</code> is deprecated due to
incompatibility between libxml2's <code>xmlReader</code> memory
semantics and Ruby's garbage collector. Although this method continues
to exist for backwards compatibility, it is unsafe to call and may
segfault. This method will be removed in a future version of Nokogiri,
and callers should use <code>#attribute_hash</code> instead. [<a
href="https://github-redirect.dependabot.com/sparklemotion/nokogiri/issues/2598">#2598</a>]</li>
</ul>
<h3>Improvements</h3>
<ul>
<li><code>XML::Reader#attribute_hash</code> is a new method to safely
retrieve the attributes of a node from <code>XML::Reader</code>. [<a
href="https://github-redirect.dependabot.com/sparklemotion/nokogiri/issues/2598">#2598</a>,
<a
href="https://github-redirect.dependabot.com/sparklemotion/nokogiri/issues/2599">#2599</a>]</li>
</ul>
<h3>Fixed</h3>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/sparklemotion/nokogiri/blob/main/CHANGELOG.md">nokogiri's
changelog</a>.</em></p>
<blockquote>
<h2>1.13.9 / 2022-10-18</h2>
<h3>Security</h3>
<ul>
<li>[CRuby] Vendored libxml2 is updated to address <a
href="https://nvd.nist.gov/vuln/detail/CVE-2022-2309">CVE-2022-2309</a>,
<a
href="https://nvd.nist.gov/vuln/detail/CVE-2022-40304">CVE-2022-40304</a>,
and <a
href="https://nvd.nist.gov/vuln/detail/CVE-2022-40303">CVE-2022-40303</a>.
See <a
href="https://github.com/sparklemotion/nokogiri/security/advisories/GHSA-2qc6-mcvw-92cw">GHSA-2qc6-mcvw-92cw</a>
for more information.</li>
<li>[CRuby] Vendored zlib is updated to address <a
href="https://ubuntu.com/security/CVE-2022-37434">CVE-2022-37434</a>.
Nokogiri was not affected by this vulnerability, but this version of
zlib was being flagged up by some vulnerability scanners, see <a
href="https://github-redirect.dependabot.com/sparklemotion/nokogiri/issues/2626">#2626</a>
for more information.</li>
</ul>
<h3>Dependencies</h3>
<ul>
<li>[CRuby] Vendored libxml2 is updated to <a
href="https://gitlab.gnome.org/GNOME/libxml2/-/releases/v2.10.3">v2.10.3</a>
from v2.9.14.</li>
<li>[CRuby] Vendored libxslt is updated to <a
href="https://gitlab.gnome.org/GNOME/libxslt/-/releases/v1.1.37">v1.1.37</a>
from v1.1.35.</li>
<li>[CRuby] Vendored zlib is updated from 1.2.12 to 1.2.13. (See <a
href="https://github.com/sparklemotion/nokogiri/blob/v1.13.x/LICENSE-DEPENDENCIES.md#platform-releases">LICENSE-DEPENDENCIES.md</a>
for details on which packages redistribute this library.)</li>
</ul>
<h3>Fixed</h3>
<ul>
<li>[CRuby] <code>Nokogiri::XML::Namespace</code> objects, when
compacted, update their internal struct's reference to the Ruby object
wrapper. Previously, with GC compaction enabled, a segmentation fault
was possible after compaction was triggered. [<a
href="https://github-redirect.dependabot.com/sparklemotion/nokogiri/issues/2658">#2658</a>]
(Thanks, <a
href="https://github.com/eightbitraptor"><code>@​eightbitraptor</code></a>
and <a
href="https://github.com/peterzhu2118"><code>@​peterzhu2118</code></a>!)</li>
<li>[CRuby] <code>Document#remove_namespaces!</code> now defers freeing
the underlying <code>xmlNs</code> struct until the <code>Document</code>
is GCed. Previously, maintaining a reference to a <code>Namespace</code>
object that was removed in this way could lead to a segfault. [<a
href="https://github-redirect.dependabot.com/sparklemotion/nokogiri/issues/2658">#2658</a>]</li>
</ul>
<h2>1.13.8 / 2022-07-23</h2>
<h3>Deprecated</h3>
<ul>
<li><code>XML::Reader#attribute_nodes</code> is deprecated due to
incompatibility between libxml2's <code>xmlReader</code> memory
semantics and Ruby's garbage collector. Although this method continues
to exist for backwards compatibility, it is unsafe to call and may
segfault. This method will be removed in a future version of Nokogiri,
and callers should use <code>#attribute_hash</code> instead. [<a
href="https://github-redirect.dependabot.com/sparklemotion/nokogiri/issues/2598">#2598</a>]</li>
</ul>
<h3>Improvements</h3>
<ul>
<li><code>XML::Reader#attribute_hash</code> is a new method to safely
retrieve the attributes of a node from <code>XML::Reader</code>. [<a
href="https://github-redirect.dependabot.com/sparklemotion/nokogiri/issues/2598">#2598</a>,
<a
href="https://github-redirect.dependabot.com/sparklemotion/nokogiri/issues/2599">#2599</a>]</li>
</ul>
<h3>Fixed</h3>
<ul>
<li>[CRuby] Calling <code>XML::Reader#attributes</code> is now safe to
call. In Nokogiri &lt;= 1.13.7 this method may segfault. [<a
href="https://github-redirect.dependabot.com/sparklemotion/nokogiri/issues/2598">#2598</a>,
<a
href="https://github-redirect.dependabot.com/sparklemotion/nokogiri/issues/2599">#2599</a>]</li>
</ul>
<h2>1.13.7 / 2022-07-12</h2>
<h3>Fixed</h3>
<p><code>XML::Node</code> objects, when compacted, update their internal
struct's reference to the Ruby object wrapper. Previously, with GC
compaction enabled, a segmentation fault was possible after compaction
was triggered. [<a
href="https://github-redirect.dependabot.com/sparklemotion/nokogiri/issues/2578">#2578</a>]
(Thanks, <a
href="https://github.com/eightbitraptor"><code>@​eightbitraptor</code></a>!)</p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="897759cc25"><code>897759c</code></a>
version bump to v1.13.9</li>
<li><a
href="aeb1ac3283"><code>aeb1ac3</code></a>
doc: update CHANGELOG</li>
<li><a
href="c663e4905a"><code>c663e49</code></a>
Merge pull request <a
href="https://github-redirect.dependabot.com/sparklemotion/nokogiri/issues/2671">#2671</a>
from sparklemotion/flavorjones-update-zlib-1.2.13_v1...</li>
<li><a
href="212e07da28"><code>212e07d</code></a>
ext: hack to cross-compile zlib v1.2.13 on darwin</li>
<li><a
href="76dbc8c5be"><code>76dbc8c</code></a>
dep: update zlib to v1.2.13</li>
<li><a
href="24e3a9c414"><code>24e3a9c</code></a>
doc: update CHANGELOG</li>
<li><a
href="4db3b4daa9"><code>4db3b4d</code></a>
Merge pull request <a
href="https://github-redirect.dependabot.com/sparklemotion/nokogiri/issues/2668">#2668</a>
from sparklemotion/flavorjones-namespace-scopes-comp...</li>
<li><a
href="73d73d6e43"><code>73d73d6</code></a>
fix: Document#remove_namespaces! use-after-free bug</li>
<li><a
href="5f58b34724"><code>5f58b34</code></a>
fix: namespace nodes behave properly when compacted</li>
<li><a
href="b08a8586c7"><code>b08a858</code></a>
test: repro namespace_scopes compaction issue</li>
<li>Additional commits viewable in <a
href="https://github.com/sparklemotion/nokogiri/compare/v1.13.6...v1.13.9">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=nokogiri&package-manager=bundler&previous-version=1.13.6&new-version=1.13.9)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the
default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as
the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as
the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the
default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/matrix-org/dendrite/network/alerts).

</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-19 11:40:38 +01:00
devonh 241d5c47df
Refactor Federation Destination Queues (#2807)
This is a refactor of the federation destination queues.
It fixes a few things, namely:
- actually retry outgoing events with backoff behaviour
- obtain enough events from the database to fill messages as much as
possible
- minimize the amount of running goroutines
  - use pure timers for backoff
  - don't restart queue unless necessary
  - close the background task when backing off
- increase max edus in a transaction to match the spec
- cleanup timers more aggresively to reduce memory usage
- add jitter to backoff timers to reduce resource spikes
- add a bunch of tests (with real and fake databases) to ensure
everything is working
2022-10-19 11:03:16 +01:00
Neil Alexander 3aa92efaa3
Namespace user API tables (#2806)
This migrates all the various user API tables, indices and sequences to
be `userapi_`-namespaced, rather than the mess they are all now.
2022-10-18 15:59:08 +01:00
Neil Alexander 9c189b1b80
Try to make AddEvent less expensive (update to matrix-org/gomatrixserverlib@a72a83f) 2022-10-18 09:51:31 +01:00
Till 07bfb791ca
Scope transactions to endpoints (#2799)
To avoid returning results from e.g. `/redact` on `/sendToDevice`
requests.
Takes the raw URL path and uses `filepath.Dir` to remove the `txnID`
(file) from it.

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-10-17 14:48:35 +02:00
Neil Alexander d72d4f8d5d
Set org.matrix.msc2285.stable in /versions 2022-10-17 10:38:22 +01:00
Till Faelligen 83c9dde219
Return error if we fail to read the response body 2022-10-17 07:27:11 +02:00
Neil Alexander 81dbad39a3
Dendrite 0.10.3
commit c6e18c18e93b54c006c6b4d0044aa53a0735a906
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Fri Oct 14 15:42:58 2022 +0100

    Changelog and version bump
2022-10-14 15:53:58 +01:00