* Implement /_synapse/admin/v1/register
This is implemented identically to Synapse, so scripts which work
with Synapse should work with Dendrite.
```
Test 27 POST /_synapse/admin/v1/register with shared secret... OK
Test 28 POST /_synapse/admin/v1/register admin with shared secret... OK
Test 29 POST /_synapse/admin/v1/register with shared secret downcases capitals... OK
Test 30 POST /_synapse/admin/v1/register with shared secret disallows symbols... OK
```
Sytest however has `implementation_specific => "synapse"` which stops these
tests from running.
* Add missing muxes to gobind
* Linting
* Implement OpenID module (#599)
- Unrelated: change Riot references to Element in client API routing
Signed-off-by: Bruce MacDonald <contact@bruce-macdonald.com>
* OpenID module tweaks (#599)
- specify expiry is ms rather than vague ts
- add OpenID token lifetime to configuration
- use Go naming conventions for the path params
- store plaintext token rather than hash
- remove openid table sqllite mutex
* Add default OpenID token lifetime (#599)
* Update dendrite-config.yaml
Co-authored-by: Kegsay <kegsay@gmail.com>
Co-authored-by: Kegsay <kegan@matrix.org>
* Tweak AS registration check
* Check appservice usernames using correct function
* Update sytest-whitelist
* Use gomatrixserverlib.Client since that allows us to disable TLS validation using the config
* Add appservice-specific client and ability to control TLS validation for appservices only
* Set timeout on appservice client
* Review comments
* Remove dead code
* Enforce LoginTypeApplicationService after all
* Check correct auth type field
* Check membership of room
* Use QueryStateAfterEventsResponse
* Fix complexity
* Add field ShouldHitAppservice to GetRoomIDForAlias
* Hit appservice when trying to join a non-existent alias
* remove unused
* Changes that I made a long time ago
* Rename to appserviceJoinedAtEvent
* Check membership in GetMemberships
* Update QueryMembershipsForRoom
* Tweaks in client API
* Update appserviceJoinedAtEvent
* Comments
* Try QueryMembershipForUser instead
* Undo some changes to client API that shouldn't be needed
* More /event tweaks
* Refactor /event bit
* Go back to QueryMembershipsForRoom because appservices are hard
* Fix bugs in onMessage
* Add comments
* More logical naming, clean up a bit
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Check membership of room
* Use QueryStateAfterEventsResponse
* Fix complexity
* Changes that I made a long time ago
* Rename to appserviceJoinedAtEvent
* Check membership in GetMemberships
* Update QueryMembershipsForRoom
* Tweaks in client API
* Update appserviceJoinedAtEvent
* Comments
* Try QueryMembershipForUser instead
* Undo some changes to client API that shouldn't be needed
* More /event tweaks
* Refactor /event bit
* Go back to QueryMembershipsForRoom because appservices are hard
* Fix bugs in onMessage
* Add comments
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Always return OTK counts
* Fix parameter ordering
* Send IDs over to keyserver internal API
* Review comments
* Fix syntax error
* Fix panic, hopefully
* Require user ID to be set
* Fix user API call
* Mention unstable features in /_matrix/client/versions
* list enabled mscs
* Don't update go.mod/go.sum
* goimports
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Do not check if a username is exclusive if the request is for an appservice in /register
* remove useless comment
* Move statements
* fix broken test
* Also fix the senderLocalpart problem
* Check domain name is ours
* Handle accessTokenErr
* Return unauthorised instead of forbidden
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Squashed commit of the following:
commit c38c39107b6dda0c8e6320d61da2365c47eea4e9
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Thu Dec 17 10:13:09 2020 +0000
Update buckets
commit 5a3bcc8bd4167150374827b0e42a0dea0366beff
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Thu Dec 17 09:41:43 2020 +0000
Update buckets
commit 78dff8165ddf596e28af04faf56466752ebc17af
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Thu Dec 17 09:37:27 2020 +0000
Register sendEventDuration
commit 612b0f3d84f1547ff30131f7b084baf609edab52
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Thu Dec 17 09:35:59 2020 +0000
sendevent metrics
While I was breaking through all the TDD bureaucracy to finally push this fix,
it turned out that it already got fixed in #1578. Still I push the rest of the
changes (basically, everything except of actually fixing the bug ;)).
`=` is crucial for matrix-bifrost which turns XMPP @'s into =40.
Signed-off-by: Bohdan Horbeshko <bodqhrohro@gmail.com>
* fix conversion from int to string yields a string of one rune, not a string of digits
* Add receipts table to syncapi
* Use StreamingToken as the since value
* Add required method to testEDUProducer
* Make receipt json creation "easier" to read
* Add receipts api to the eduserver
* Add receipts endpoint
* Add eduserver kafka consumer
* Add missing kafka config
* Add passing tests to whitelist
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
* Fix copy & paste error
* Fix column count error
* Make outbound federation receipts pass
* Make "Inbound federation rejects receipts from wrong remote" pass
* Don't use errors package
* - Add TODO for batching requests
- Rename variable
* Return a better error message
* - Use OutputReceiptEvent instead of InputReceiptEvent as result
- Don't use the errors package for errors
- Defer CloseAndLogIfError to close rows
- Fix Copyright
* Better creation/usage of JoinResponse
* Query all joined rooms instead of just one
* Update gomatrixserverlib
* Add sqlite3 migration
* Add postgres migration
* Ensure required sequence exists before running migrations
* Clarification on comment
* - Fix a bug when creating client receipts
- Use concrete types instead of interface{}
* Remove dead code
Use key for timestamp
* Fix postgres query...
* Remove single purpose struct
* Use key/value directly
* Only apply receipts on initial sync or if edu positions differ,
otherwise we'll be sending the same receipts over and over again.
* Actually update the id, so it is correctly send in syncs
* Set receipt on request to /read_markers
* Fix issue with receipts getting overwritten
* Use fmt.Errorf instead of pkg/errors
* Revert "Add postgres migration"
This reverts commit 722fe5a046.
* Revert "Add sqlite3 migration"
This reverts commit d113b03f64.
* Fix selectRoomReceipts query
* Make golangci-lint happy
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Add basic storage methods
* Add internal api handler
* Add check for forgotten room
* Add /rooms/{roomID}/forget endpoint
* Add missing rsAPI method
* Remove unused parameters
* Add passing tests
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
* Add missing file
* Add postgres migration
* Add sqlite migration
* Use Forgetter to forget room
* Remove empty line
* Update HTTP status codes
It looks like the spec calls for these to be 400, rather than 403: https://matrix.org/docs/spec/client_server/r0.6.1#post-matrix-client-r0-rooms-roomid-forget
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Add KindOld
* Don't process latest events/memberships for old events
* Allow federationsender to ignore duplicate key entries when LatestEventIDs is duplicated by RS output events
* Signal to downstream components if an event has become a forward extremity
* Don't exclude from sync
* Soft-fail checks on KindNew
* Don't run the latest events updater at all for KindOld
* Don't make federation sender change after all
* Kind in federation sender join
* Don't send isForwardExtremity
* Fix syncapi
* Update comments
* Fix SendEventWithState
* Update sytest-whitelist
* Generate old output events
* Sync API consumes old room events
* Update comments
* Remove QueryBulkStateContent from current state server
Expected fail due to db impl not existing
* Implement query bulk state content
* Fix up rejecting invites over federation
* Fix bulk content marshalling
* User API support for password changes
* Password changes in client API
* Update sytest-whitelist
* Remove debug logging
* Default logout_devices to true
* Fix deleting devices by local part
* Initial FIFOing of roomserver inputs
* Remove EventID response from api.InputRoomEventsResponse
* Don't send back event ID unnecessarily
* Fix ordering hopefully
* Reduce copies, use buffered task channel to reduce contention on other rooms
* Fix error handling
* Add Queryer and use embedded structs
* Add Inputer and factor out more RS API stuff
This neatly splits up the RS API based on the functionality it provides,
whilst providing a useful place for code sharing via the `helpers` package.
* Enforce history visibility etc for /rooms/{roomID}/state
* Deduplicate OnIncomingStateRequest and OnIncomingStateTypeRequest
* Revert "Deduplicate OnIncomingStateRequest and OnIncomingStateTypeRequest"
This reverts commit 335035d66e.
* Client API: mutex on (user_id, room_id)
* Client API: mutex on (user_id, room_id)
Changed variable name used for the mutexes map
Changed the place where the mutex is locked
Changed unlock to a defered call instead of manually calling it
* Make PerformJoin send input membership event
* Invite input room events in separate goroutine
* Don't limit roomserver input events using request context
* Synchronous input room events
* Nope, that didn't work
* oops send state key to GetMembership
* Don't generate stripped state in client API more times than necessary, generate output events on receiving end of federated invite
* Commit membership updater changes
* Tweaks
* Initial pass at refactoring config (not finished)
* Don't forget current state and EDU servers
* More shifting around
* Update server key API tests
* Fix roomserver test
* Fix more tests
* Further tweaks
* Fix current state server test (sort of)
* Maybe fix appservices
* Fix client API test
* Include database connection string in database options
* Fix sync API build
* Update config test
* Fix unit tests
* Fix federation sender build
* Fix gobind build
* Set Listen address for all services in HTTP monolith mode
* Validate config, reinstate appservice derived in directory, tweaks
* Tweak federation API test
* Set MaxOpenConnections/MaxIdleConnections to previous values
* Update generate-config
* Fix New users appear in /keys/changes
* Create blank device keys when logging in on a new device
* Add PerformDeviceUpdate and fix a few bugs
- Correct device deletion query on sqlite
- Return no keys on /keys/query rather than an empty key
* Unbreak sqlite properly
* Use a real DB for currentstateserver integration tests
* Race fix
* Recheck device lists when join/leave events come in
* Add PerformDeviceDeletion
* Notify clients when devices are deleted
* Unbreak things
* Remove debug logging
* User directory
* Fix syncapi unit test
* Make user directory only show remote users you know about from your joined rooms
* Update sytest-whitelist
* Review comments
* Modify /state/{eventType}/{stateKey} to return the event at the time the user left
Or live, depending on their current state. Hopefully fixes some sytests!
* Linting
* Set HasBeenInRoom
* Fix cases for world-readable history visibility
* Fix bug in finding the requested state event
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Add storage layer for postgres/sqlite
* Return OTK counts when inserting new keys
* Hook up the key DB and make a test pass
* Convert postgres queries to be sqlite queries
* Blacklist test due to requiring rejected events
* Unbreak tests
* Update blacklist
* Add User-Interactive Authentication
And use it when deleting a device. With tests.
* Make remaining sytest pass
* Linting
* 403 not 401 on wrong user/pass
* Don't return null to public directory request
* Initial support for finding public rooms in Yggdrasil demo (incomplete)
* Increase QUIC idle time to 15 minutes
* Use content_value instead of membership
* Fix build
* Replace publicroomsapi with a combination of clientapi/roomserver/currentstateserver
- All public rooms paths are now handled by clientapi
- Requests to (un)publish rooms are sent to the roomserver via `PerformPublish`
which are stored in a new `published_table.go`
- Requests for public rooms are handled in clientapi by:
* Fetch all room IDs which are published using `QueryPublishedRooms` on the roomserver.
* Apply pagination parameters to the slice.
* Do a `QueryBulkStateContent` request to the currentstateserver to pull out
required state event *content* (not entire events).
* Aggregate and return the chunk.
Mostly but not fully implemented (DB queries on currentstateserver are missing)
* Fix pq query
* Make postgres work
* Make sqlite work
* Fix tests
* Unbreak pagination tests
* Linting
* Remove membership table from account DB
And make code which needs that data use the currentstate server
* Unbreak tests; use a membership enum for space
* Move filter table to syncapi where it is used
* Implement /sync `limited` and read timeline limit from stored filters
We now fully handle `room.timeline.limit` filters (in-line + stored) and
return the right value for `limited` syncs.
* Update whitelist
* Default to the default timeline limit if it's unset, also strip the extra event correctly
* Update whitelist
* Add PerformInvite and refactor how errors get handled
- Rename `JoinError` to `PerformError`
- Remove `error` from the API function signature entirely. This forces
errors to be bundled into `PerformError` which makes it easier for callers
to detect and handle errors. On network errors, HTTP clients will make a
`PerformError`.
* Unbreak everything; thanks Go!
* Send back JSONResponse according to the PerformError
* Update federation invite code too
* Pass join errors through internal API boundaries
Required for certain invite sytests. We will need to think of a
better way of handling this going forwards.
* Include m.room.avatar in stripped state; handle trailing slashes when GETing state events
* Update whitelist
* Update whitelist
* Join room body is optional
* Support deprecated login by user/password
* Implement dummy key upload endpoint
* Make a very determinate end to /messages if we hit the create event in back-pagination
* Linting
* Make userapi responsible for checking access tokens
There's still plenty of dependencies on account/device DBs, but this
is a start. This is a breaking change as it adds a required config
value `listen.user_api`.
* Cleanup
* Review comments and test fix
* Remove clientapi producers which aren't actually producers
They are actually just convenience wrappers around the internal APIs
for roomserver/eduserver. Move their logic to their respective `api`
packages and call them directly.
* Remove TODO
* unbreak ygg
* Return bad request on CS API /send if bad JSON
* Return some more M_BAD_JSON in the right places
* nolint because damnit gocyclo all I added was a type check for an error
* Update gomatrixserverlib
* Update gomatrixserverlib
* Update sytest-whitelist
* Update gomatrixserverlib
* Update sytest-whitelist
* NotJSON -> BadJSON
* Return newly created error if user already exists (#1002)
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
* Rename variable
* Remove check for account and use returned error
* Return ErrUserExists
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
* State that CreateAccount will return err ErrUserExists if the user exists
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
* Also check sqlite for constraint error
* Revert "Also check sqlite for constraint error"
This reverts commit 7d310514
* Check for sqlite3 constraint error
* Add documentation to CreateAccount
* Move ErrUserExists to accounts package
* Revert "Move ErrUserExists to accounts package"
Import Cycle..
This reverts commit be3d4cda
Co-authored-by: Kegsay <kegan@matrix.org>
* Groundwork for send-to-device messaging
* Update sample config
* Add unstable routing for now
* Send to device consumer in sync API
* Start the send-to-device consumer
* fix indentation in dendrite-config.yaml
* Create send-to-device database tables, other tweaks
* Add some logic for send-to-device messages, add them into sync stream
* Handle incoming send-to-device messages, count them with EDU stream pos
* Undo changes to test
* pq.Array
* Fix sync
* Logging
* Fix a couple of transaction things, fix client API
* Add send-to-device test, hopefully fix bugs
* Comments
* Refactor a bit
* Fix schema
* Fix queries
* Debug logging
* Fix storing and retrieving of send-to-device messages
* Try to avoid database locks
* Update sync position
* Use latest sync position
* Jiggle about sync a bit
* Fix tests
* Break out the retrieval from the update/delete behaviour
* Comments
* nolint on getResponseWithPDUsForCompleteSync
* Try to line up sync tokens again
* Implement wildcard
* Add all send-to-device tests to whitelist, what could possibly go wrong?
* Only care about wildcard when targeted locally
* Deduplicate transactions
* Handle tokens properly, return immediately if waiting send-to-device messages
* Fix sync
* Update sytest-whitelist
* Fix copyright notice (need to do more of this)
* Comments, copyrights
* Return errors from Do, fix dendritejs
* Review comments
* Comments
* Constructor for TransactionWriter
* defletions
* Update gomatrixserverlib, sytest-blacklist
