* Verify that the user ID for registration matches the spec, and the auth data (#10)
* Blacklist some sytest tests that are failing in our environment
* Commenting out test that isn't reliably passing or failing, probably a race
* refresh latest dendrite main
* pull latest from dendrite-fork subtree
* refresh latest dendrite main
* pull dendrite subtree and resolve merge conflicts
* check that userID matches the signed message
* verify that the user ID for registration is CAIP-10 compliant and MXID compliant
* removed space
Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
* Fix nats.go commit (#2540)
Signed-off-by: Jean Lucas <jean@4ray.co>
* Don't return `end` if there are not more messages (#2542)
* Be more spec compliant
* Move lazyLoadMembers to own method
* Return an error if trying to invite a malformed user ID (#2543)
* Add `evacuateUser` endpoint, use it when deactivating accounts (#2545)
* Add `evacuateUser` endpoint, use it when deactivating accounts
* Populate the API
* Clean up user devices when deactivating
* Include invites, delete pushers
* Silence presence logs (#2547)
* Blacklist `Guest users can join guest_access rooms` test until it can be investigated
* Disable WebAssembly builds for now
* Try to fix backfilling (#2548)
* Try to fix backfilling
* Return start/end to not confuse clients
* Update GMSL
* Update GMSL
* Roomserver producers package (#2546)
* Give the roomserver a producers package
* Change init point
* Populate ACLs API
* Fix build issues
* `RoomEventProducer` naming
* Version 0.8.9 (#2549)
* Version 0.8.9
* Update changelog
* Takwaiw/fix concurrent registration bug (#12)
* fix concurrent registration bug. Rename decentralizedid
* remove unused module
* add regressed test to blacklist
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
* Test_UserStatistics Fix expected results to match observed results
* Takwaiw/dendrite publickey (#2)
* Implementation of MSC 3782 Add publickey login as a new auth type.
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
* Implement EIP-4361 sign in with Ethereum (#5)
* Blacklist some sytest tests that are failing in our environment
* Commenting out test that isn't reliably passing or failing, probably a race
* refresh latest dendrite main
* refresh latest dendrite main
* dendrite implementation of eip-4361
* simplify nonce generation
Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
* Use rand.Seed to seed the random function generator (#6)
* Blacklist some sytest tests that are failing in our environment
* Commenting out test that isn't reliably passing or failing, probably a race
* refresh latest dendrite main
* use rand.Seed to seed the random function
Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
* Create session ID during registration (#8)
* Blacklist some sytest tests that are failing in our environment
* Commenting out test that isn't reliably passing or failing, probably a race
* refresh latest dendrite main
* pull latest from dendrite-fork subtree
* refresh latest dendrite main
* Create session ID during registration
Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
* Verify that the user ID for registration matches the spec, and the auth data (#10)
* Blacklist some sytest tests that are failing in our environment
* Commenting out test that isn't reliably passing or failing, probably a race
* refresh latest dendrite main
* pull latest from dendrite-fork subtree
* refresh latest dendrite main
* pull dendrite subtree and resolve merge conflicts
* check that userID matches the signed message
* verify that the user ID for registration is CAIP-10 compliant and MXID compliant
* removed space
Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
* Takwaiw/fix concurrent registration bug (#12)
* fix concurrent registration bug. Rename decentralizedid
* remove unused module
* add regressed test to blacklist
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
* removed unused module
* feat+fix: Ignore unknown keys and verify required fields are present in appservice registration files (#2550)
* fix: ignore unknown keys in appservice configs
fixesmatrix-org/dendrite#1567
* feat: verify required fields in appservice configs
* Use new testrig for key changes tests (#2552)
* Use new testrig for tests
* Log the error message
* Fix QuerySharedUsers for the SyncAPI keychange consumer (#2554)
* Make more use of base.BaseDendrite
* Fix QuerySharedUsers if no UserIDs are supplied
* Return clearer error when no state NID exists for an event (#2555)
* Wrap error from `SnapshotNIDFromEventID`
* Hopefully fix read receipts timestamps (#2557)
This should avoid coercions between signed and unsigned ints which might fix problems like `sql: converting argument $5 type: uint64 values with high bit set are not supported`.
* fix concurrency issue when checking session ID (#14)
Co-authored-by: Tak Wai Wong <tak@hntlabs.com>
* merge latest changes from dendrite main (#15)
Co-authored-by: Tak Wai Wong <tak@hntlabs.com>
* Login and Register tests for public key ethereum (#16)
* TestLoginPublicKeyNewSession
* use asserts
* setup, test, asserts
* TestLoginPublicKeyValidAuthTypeMissingSession
* invalid session id test
* create a helper newSession function
* TestLoginPublicKeyEthereumMissingUserId
* TestLoginPublicKeyEthereumAccountNotAvailable
* TestLoginPublicKeyEthereumInvalidUserId
* createEip4361TestMessage
* TestLoginPublicKeyEthereumMissingSignature
* TestLoginPublicKeyEthereum
* re-enable all publickey signin tests
* move common publickey test util to its own file
* register_public_key.go stub
* refactored common ethereum test helpers to its own folder
* refactor test helpers
* return error in test helpers
* fix regressions with ServerName
* TestRegistrationUnimplementedAlgo
* TestNewRegistration
* TestNewRegistrationSession
* verify new login session
* remove assert
* perform account creation
* TestRegisterEthereum
* Enable all tests
* move helper functions into test file
Co-authored-by: Tak Wai Wong <tak@hntlabs.com>
Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
Co-authored-by: Jean Lucas <jean@4ray.co>
Co-authored-by: Till <2353100+S7evinK@users.noreply.github.com>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: Tak Wai Wong <tak@hntlabs.com>
Co-authored-by: Kabir Kwatra <kabir@kwatra.me>
* Put federation client functions into their own file
* Look for missing auth events in RS input
* Remove retrieveMissingAuthEvents from federation API
* Logging
* Sorta transplanted the code over
* Use event origin failing all else
* Don't get stuck on mutexes:
* Add verifier
* Don't mark state events with zero snapshot NID as not existing
* Check missing state if not an outlier before storing the event
* Reject instead of soft-fail, don't copy roominfo so much
* Use synchronous contexts, limit time to fetch missing events
* Clean up some commented out bits
* Simplify `/send` endpoint significantly
* Submit async
* Report errors on sending to RS input
* Set max payload in NATS to 16MB
* Tweak metrics
* Add `workerForRoom` for tidiness
* Try skipping unmarshalling errors for RespMissingEvents
* Track missing prev events separately to avoid calculating state when not possible
* Tweak logic around checking missing state
* Care about state when checking missing prev events
* Don't check missing state for create events
* Try that again
* Handle create events better
* Send create room events as new
* Use given event kind when sending auth/state events
* Revert "Use given event kind when sending auth/state events"
This reverts commit 089d64d271.
* Only search for missing prev events or state for new events
* Tweaks
* We only have missing prev if we don't supply state
* Room version tweaks
* Allow async inputs again
* Apply backpressure to consumers/synchronous requests to hopefully stop things being overwhelmed
* Set timeouts on roomserver input tasks (need to decide what timeout makes sense)
* Use work queue policy, deliver all on restart
* Reduce chance of duplicates being sent by NATS
* Limit the number of servers we attempt to reduce backpressure
* Some review comment fixes
* Tidy up a couple things
* Don't limit servers, randomise order using map
* Some context refactoring
* Update gmsl
* Don't resend create events
* Set stateIDs length correctly or else the roomserver thinks there are missing events when there aren't
* Exclude our own servername
* Try backing off servers
* Make excluding self behaviour optional
* Exclude self from g_m_e
* Update sytest-whitelist
* Update consumers for the roomserver output stream
* Remember to send outliers for state returned from /gme
* Make full HTTP tests less upsetti
* Remove 'If a device list update goes missing, the server resyncs on the next one' from the sytest blacklist
* Remove debugging test
* Fix blacklist again, remove unnecessary duplicate context
* Clearer contexts, don't use background in case there's something happening there
* Don't queue up events more than once in memory
* Correctly identify create events when checking for state
* Fill in gaps again in /gme code
* Remove `AuthEventIDs` from `InputRoomEvent`
* Remove stray field
Co-authored-by: Kegan Dougal <kegan@matrix.org>
* Add NATS JetStream support
Update shopify/sarama
* Fix addresses
* Don't change Addresses in Defaults
* Update saramajetstream
* Add missing error check
Keep typing events for at least one minute
* Use all configured NATS addresses
* Update saramajetstream
* Try setting up with NATS
* Make sure NATS uses own persistent directory (TODO: make this configurable)
* Update go.mod/go.sum
* Jetstream package
* Various other refactoring
* Build fixes
* Config tweaks, make random jetstream storage path for CI
* Disable interest policies
* Try to sane default on jetstream base path
* Try to use in-memory for CI
* Restore storage/retention
* Update nats.go dependency
* Adapt changes to config
* Remove unneeded TopicFor
* Dep update
* Revert "Remove unneeded TopicFor"
This reverts commit f5a4e4a339.
* Revert changes made to streams
* Fix build problems
* Update nats-server
* Update go.mod/go.sum
* Roomserver input API queuing using NATS
* Fix topic naming
* Prometheus metrics
* More refactoring to remove saramajetstream
* Add missing topic
* Don't try to populate map that doesn't exist
* Roomserver output topic
* Update go.mod/go.sum
* Message acknowledgements
* Ack tweaks
* Try to resume transaction re-sends
* Try to resume transaction re-sends
* Update to matrix-org/gomatrixserverlib@91dadfb
* Remove internal.PartitionStorer from components that don't consume keychanges
* Try to reduce re-allocations a bit in resolveConflictsV2
* Tweak delivery options on RS input
* Publish send-to-device messages into correct JetStream subject
* Async and sync roomserver input
* Update dendrite-config.yaml
* Remove roomserver tests for now (they need rewriting)
* Remove roomserver test again (was merged back in)
* Update documentation
* Docker updates
* More Docker updates
* Update Docker readme again
* Fix lint issues
* Send final event in `processEvent` synchronously (since this might stop Sytest from being so upset)
* Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that
* Go 1.16 instead of Go 1.13 for upgrade tests and Complement
* Revert "Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that"
This reverts commit 368675283f.
* Don't report any errors on `/send` to see what fun that creates
* Fix panics on closed channel sends
* Enforce state key matches sender
* Do the same for leave
* Various tweaks to make tests happier
Squashed commit of the following:
commit 13f9028e7a
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Tue Jan 4 15:47:14 2022 +0000
Do the same for leave
commit e6be7f05c3
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Tue Jan 4 15:33:42 2022 +0000
Enforce state key matches sender
commit 85ede6d64b
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Tue Jan 4 14:07:04 2022 +0000
Fix panics on closed channel sends
commit 9755494a98
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Tue Jan 4 13:38:22 2022 +0000
Don't report any errors on `/send` to see what fun that creates
commit 3bb4f87b5d
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Tue Jan 4 13:00:26 2022 +0000
Revert "Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that"
This reverts commit 368675283f.
commit fe2673ed7b
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Tue Jan 4 12:09:34 2022 +0000
Go 1.16 instead of Go 1.13 for upgrade tests and Complement
commit 368675283f
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Tue Jan 4 11:51:45 2022 +0000
Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that
commit b028dfc085
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Tue Jan 4 10:29:08 2022 +0000
Send final event in `processEvent` synchronously (since this might stop Sytest from being so upset)
* Merge in NATS Server v2.6.6 and nats.go v1.13 into the in-process connection fork
* Add `jetstream.WithJetStreamMessage` to make ack/nak-ing less messy, use process context in consumers
* Fix consumer component name in federation API
* Add comment explaining where streams are defined
* Tweaks to roomserver input with comments
* Finish that sentence that I apparently forgot to finish in INSTALL.md
* Bump version number of config to 2
* Add comments around asynchronous sends to roomserver in processEventWithMissingState
* More useful error message when the config version does not match
* Set version in generate-config
* Fix version in config.Defaults
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Check for missing state keys to avoid panicking
* Check for not allowed errors on send_leave
* More logging
* handle send_join errors too
* Additional send_join checks
* s/join/gmsl.json/
* Add KindOld
* Don't process latest events/memberships for old events
* Allow federationsender to ignore duplicate key entries when LatestEventIDs is duplicated by RS output events
* Signal to downstream components if an event has become a forward extremity
* Don't exclude from sync
* Soft-fail checks on KindNew
* Don't run the latest events updater at all for KindOld
* Don't make federation sender change after all
* Kind in federation sender join
* Don't send isForwardExtremity
* Fix syncapi
* Update comments
* Fix SendEventWithState
* Update sytest-whitelist
* Generate old output events
* Sync API consumes old room events
* Update comments
* Initial FIFOing of roomserver inputs
* Remove EventID response from api.InputRoomEventsResponse
* Don't send back event ID unnecessarily
* Fix ordering hopefully
* Reduce copies, use buffered task channel to reduce contention on other rooms
* Fix error handling
* Add Queryer and use embedded structs
* Add Inputer and factor out more RS API stuff
This neatly splits up the RS API based on the functionality it provides,
whilst providing a useful place for code sharing via the `helpers` package.
* Initial pass at refactoring config (not finished)
* Don't forget current state and EDU servers
* More shifting around
* Update server key API tests
* Fix roomserver test
* Fix more tests
* Further tweaks
* Fix current state server test (sort of)
* Maybe fix appservices
* Fix client API test
* Include database connection string in database options
* Fix sync API build
* Update config test
* Fix unit tests
* Fix federation sender build
* Fix gobind build
* Set Listen address for all services in HTTP monolith mode
* Validate config, reinstate appservice derived in directory, tweaks
* Tweak federation API test
* Set MaxOpenConnections/MaxIdleConnections to previous values
* Update generate-config
* Fix rooms v3 url paths for good - with tests
- Add a test rig around `federationapi` to test routing.
- Use `JSONVerifier` over `KeyRing` so we can stub things out more easily.
- Add `test.NopJSONVerifier` which verifies nothing.
- Add `base.BaseMux` which is the original `mux.Router` used to spawn public/internal routers.
- Listen on `base.BaseMux` and not the default serve mux as it cleans paths which we don't want.
- Factor out `ListenAndServe` to `test.ListenAndServe` and add flag for listening on TLS.
* Fix comments
* Linting
* Remove clientapi producers which aren't actually producers
They are actually just convenience wrappers around the internal APIs
for roomserver/eduserver. Move their logic to their respective `api`
packages and call them directly.
* Remove TODO
* unbreak ygg
* Return bad request on CS API /send if bad JSON
* Return some more M_BAD_JSON in the right places
* nolint because damnit gocyclo all I added was a type check for an error
* Update gomatrixserverlib
* Update gomatrixserverlib
* Update sytest-whitelist
* Update gomatrixserverlib
* Update sytest-whitelist
* NotJSON -> BadJSON
* Consolidation of roomserver APIs
* Comment out alias tests for now, they are broken
* Wire AS API into roomserver again
* Roomserver didn't take asAPI param before so return to that
* Prevent roomserver asking AS API for alias info
* Rename some files
* Remove alias_test, incoherent tests and unwanted appservice integration
* Remove FS API inject on syncapi component
* Update gomatrixserverlib
* Test matrix.org as perspective key server
* Base64 decode better
* Optional strict validity checking in gmsl
* Update gomatrixserverlib
* Attempt to find missing auth events over federation (this shouldn't happen but I am guessing there is a synapse bug involved where we don't get all of the auth events)
* Update gomatrixserverlib, debug logging
* Remove debugging output
* More verbose debugging
* Print outliers
* Increase timeouts for testing, observe contexts before trying to join over more servers
* Don't block on roomserver (experimental)
* Don't block on roomserver
* Update gomatrixserverlib
* Update gomatrixserverlib
* Configurable perspective key fetchers
* Output number of configured keys for perspective
* Example perspective config included
* Undo debug stack trace
* Undo debug stack trace
* Restore original HTTP listener in monolith
* Fix lint
* Review comments
* Set default HTTP server timeout to 5 minutes now, block again when joining
* Don't use HTTP address for HTTPS whoops
* Update gomatrixserverlib
* Update gomatrixserverlib
* Update gomatrixserverlib
* Actually add perspectives
* Actually add perspectives
* Update gomatrixserverlib
* Room version 2 by default, other wiring updates, update gomatrixserverlib
* Fix nil pointer exception
* Fix some more nil pointer exceptions hopefully
* Update gomatrixserverlib
* Send all room versions when joining, not just stable ones
* Remove room version cquery
* Get room version when getting events from the roomserver database
* Reset default back to room version 2
* Don't generate event IDs unless needed
* Revert "Remove room version cquery"
This reverts commit a170d58733.
* Query room version in federation API, client API as needed
* Improvements to make_join send_join dance
* Make room server producers use headered events
* Lint tweaks
* Update gomatrixserverlib
* Versioned SendJoin
* Query room version in syncapi backfill
* Handle transaction marshalling/unmarshalling within Dendrite
* Sorta fix federation (kinda)
* whoops commit federation API too
* Use NewEventFromTrustedJSON when getting events from the database
* Update gomatrixserverlib
* Strip headers on federationapi endpoints
* Fix bug in clientapi profile room version query
* Update gomatrixserverlib
* Return more useful error if room version query doesn't find the room
* Update gomatrixserverlib
* Update gomatrixserverlib
* Maybe fix federation
* Fix formatting directive
* Update sytest whitelist and blacklist
* Temporarily disable room versions 3 and 4 until gmsl is fixed
* Fix count of EDUs in logging
* Update gomatrixserverlib
* Update gomatrixserverlib
* Update gomatrixserverlib
* Rely on EventBuilder in gmsl to generate the event IDs for us
* Some review comments fixed
* Move function out of common and into gmsl
* Comment in federationsender destinationqueue
* Update gomatrixserverlib
* Implement gomatrixserverlib.HeaderedEvent, which should allow us to store room version headers along with the event across API boundaries and consumers/producers, and intercept unmarshalling to get the event structure right
* Add federationsender to previous
* Pass cfg by reference around the codebase
* Merge branch 'master' into pass-cfg-by-ref
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
