* Use content_value instead of membership
* Fix build
* Replace publicroomsapi with a combination of clientapi/roomserver/currentstateserver
- All public rooms paths are now handled by clientapi
- Requests to (un)publish rooms are sent to the roomserver via `PerformPublish`
which are stored in a new `published_table.go`
- Requests for public rooms are handled in clientapi by:
* Fetch all room IDs which are published using `QueryPublishedRooms` on the roomserver.
* Apply pagination parameters to the slice.
* Do a `QueryBulkStateContent` request to the currentstateserver to pull out
required state event *content* (not entire events).
* Aggregate and return the chunk.
Mostly but not fully implemented (DB queries on currentstateserver are missing)
* Fix pq query
* Make postgres work
* Make sqlite work
* Fix tests
* Unbreak pagination tests
* Linting
* Return remote errors from FS.PerformJoin
Follows the same pattern as PerformJoin on roomserver (no error return).
Also return the right format for incompatible room version errors.
Makes a bunch of tests pass!
* Handle network errors better when returning remote HTTP errors
* Linting
* Fix tests
* Update whitelist, pass network errors through in API=1 mode
* More key tweaks
* Start testing stuff
* Move responsibility for generating local keys into server key API, don't register prom in caches unless needed, start tests
* Don't store our own keys in the database
* Don't store our own keys in the database
* Don't run tests for now
* Tweak caching behaviour, update tests
* Update comments, add fixes from forward-merge
* Debug logging
* Debug logging
* Perform final comparison against original set of requests
* oops
* Fetcher timeouts
* Fetcher timeouts
* missing func
* Tweaks
* Update gomatrixserverlib
* Fix Federation API test
* Break up FetchKeys
* Add comments to caching
* Add URL check in test
* Partially revert "Move responsibility for generating local keys into server key API, don't register prom in caches unless needed, start tests"
This reverts commit d7eb54c5b3.
* Fix federation API test
* Fix internal cache stuff again
* Fix server key API test
* Update comments
* Update comments from review
* Fix lint
* Use MissingAuthEventHandler on performjoin to try and speed up cases where we have missing events
* Update gomatrixserverlib
* Use supplied room version
* Use AuthChainProvider
* Tweaks
* Update gomatrixserverlib
* Signature checks
* Return bad request on CS API /send if bad JSON
* Return some more M_BAD_JSON in the right places
* nolint because damnit gocyclo all I added was a type check for an error
* Update gomatrixserverlib
* Update gomatrixserverlib
* Update sytest-whitelist
* Update gomatrixserverlib
* Update sytest-whitelist
* NotJSON -> BadJSON
* Groundwork for send-to-device messaging
* Update sample config
* Add unstable routing for now
* Send to device consumer in sync API
* Start the send-to-device consumer
* fix indentation in dendrite-config.yaml
* Create send-to-device database tables, other tweaks
* Add some logic for send-to-device messages, add them into sync stream
* Handle incoming send-to-device messages, count them with EDU stream pos
* Undo changes to test
* pq.Array
* Fix sync
* Logging
* Fix a couple of transaction things, fix client API
* Add send-to-device test, hopefully fix bugs
* Comments
* Refactor a bit
* Fix schema
* Fix queries
* Debug logging
* Fix storing and retrieving of send-to-device messages
* Try to avoid database locks
* Update sync position
* Use latest sync position
* Jiggle about sync a bit
* Fix tests
* Break out the retrieval from the update/delete behaviour
* Comments
* nolint on getResponseWithPDUsForCompleteSync
* Try to line up sync tokens again
* Implement wildcard
* Add all send-to-device tests to whitelist, what could possibly go wrong?
* Only care about wildcard when targeted locally
* Deduplicate transactions
* Handle tokens properly, return immediately if waiting send-to-device messages
* Fix sync
* Update sytest-whitelist
* Fix copyright notice (need to do more of this)
* Comments, copyrights
* Return errors from Do, fix dendritejs
* Review comments
* Comments
* Constructor for TransactionWriter
* defletions
* Update gomatrixserverlib, sytest-blacklist
* Move Updater structs to shared and use it for postgres
* Add constructors for NewXXXUpdater and a useTxns flag
In sqlite, we set useTxns=false and comment why.
* Handle nil txn
* Handle nil in transaction
* Missed one
* Close the txn at the right time
* Don't close the transaction as we reuse it between calls
* Server key API (works for monolith but not for polylith yet)
* Re-enable caching on server key API component
* Groundwork for HTTP APIs for server key API
* Hopefully implement HTTP for server key API
* Simplify public key request marshalling from map keys
* Update gomatrixserverlib
* go mod tidy
* Common -> internal
* remove keyring.go
* Update Docker Hub for server key API
* YAML is funny about indentation
* Wire in new server key API into hybrid monolith mode
* Create maps
* Route server key API endpoints on internal API mux
* Fix server key API URLs
* Add fetcher behaviour into server key API implementation
* Return error if we failed to fetch some keys
* Return results anyway
* Move things about a bit
* Remove unused code
* Fix comments, don't use federation sender URL in polylith mode
* Add server_key_api to sample config
* Review comments
* HTTP API to cache keys that have been requested
* Overwrite server_key_api listen in monolith hybrid mode
* Make backfill work for shared history visibility
* fetch missing state on backfill to remember snapshots correctly
* Fix gmsl to not mux in auth events into room state
* Whoops
* Linting
* Initial cut for backfilling
The syncserver now asks the roomserver via QueryBackfill (which already
existed to *handle* backfill requests) which then makes federation requests
via gomatrixserverlib.RequestBackfill.
Currently, tests fail on subsequent /messages requests because we don't know
which servers are in the room, because we are unable to get state snapshots
from a backfilled event because that code doesn't exist yet.
* WIP backfill, doesn't work
* Make initial backfill pass checks
* Persist backfilled events with state snapshots
* Remove debug lines
* Linting
* Review comments
* Update gomatixserverlib
* Try to build invite stripped state if not given to us
* SendInvite improvements
* Transpose invite_room_state into invite_state.events for sync API
* Remove syncapi debugging output
* Use RespInviteV2
* Update gomatrixserverlib
* Send the invite event as a normal roomserver event too, for incorporating into room (should this be done by the roomserver automatically for invite inputs?)
* Federation sender use invite_room_state, room server try to insert membership state
* Check supported room versions on the invite endpoint
* Prevent roomserver query API from trying to handle requests for stub rooms
* Adding a nolint
* Replace IsRoomStub with RoomNIDExcludingStubs, fix query API to use that instead
* Review comments
* Update gomatrixserverlib
* Test matrix.org as perspective key server
* Base64 decode better
* Optional strict validity checking in gmsl
* Update gomatrixserverlib
* Attempt to find missing auth events over federation (this shouldn't happen but I am guessing there is a synapse bug involved where we don't get all of the auth events)
* Update gomatrixserverlib, debug logging
* Remove debugging output
* More verbose debugging
* Print outliers
* Increase timeouts for testing, observe contexts before trying to join over more servers
* Don't block on roomserver (experimental)
* Don't block on roomserver
* Update gomatrixserverlib
* Update gomatrixserverlib
* Configurable perspective key fetchers
* Output number of configured keys for perspective
* Example perspective config included
* Undo debug stack trace
* Undo debug stack trace
* Restore original HTTP listener in monolith
* Fix lint
* Review comments
* Set default HTTP server timeout to 5 minutes now, block again when joining
* Don't use HTTP address for HTTPS whoops
* Update gomatrixserverlib
* Update gomatrixserverlib
* Update gomatrixserverlib
* Actually add perspectives
* Actually add perspectives
* Update gomatrixserverlib
* Improve error handling in federation /send endpoint a bit
* Remove unknownRoomError, use unmarshalError when unable to get room ID
* Swap out a couple more internal server errors
* Update gomatrixserverlib
* Update gomatrixserverlib
* Update gomatrixserverlib
* Update gomatrixserverlib
* Update gomatrixserverlib
* Update gomatrixserverlib
* Return bad limit in error
* Same with domain/userid
* Set default room version to 4
* Default to v1 when room_version key missing
Signed-off-by: Alex Chen <minecnly@gmail.com>
* Squashed commit of the following:
commit c1bca95adb
Author: Kegsay <kegan@matrix.org>
Date: Thu Apr 16 10:06:55 2020 +0100
Add SQL tracing via DENDRITE_TRACE_SQL (#968)
* Add SQL tracing via DENDRITE_TRACE_SQL
Add this to `internal/sqlutil` in preparation for #897
* Not entirely
commit c2ea961909
Author: Kegsay <kegan@matrix.org>
Date: Wed Apr 15 17:48:40 2020 +0100
Add HTTP trace logging (#965)
* Dump all requests/response server-side
* Wrap in DENDRITE_TRACE
* DENDRITE_TRACE_HTTP is better
* Bugfix for response body and linting
* False is true and true is false
* Linting
* How did this get missed
* More linting
* Update gomatrixserverlib
* Remove unneeded imports
Co-authored-by: Cnly <minecnly@gmail.com>
* Dump all requests/response server-side
* Wrap in DENDRITE_TRACE
* DENDRITE_TRACE_HTTP is better
* Bugfix for response body and linting
* False is true and true is false
* Linting
* How did this get missed
* More linting
