* SendEventWithState events as new
* Use cumulative state IDs for final event
* Error wrapping in calculateAndSetState
* Handle overwriting same event type and state key
* Hacky way to spot historical events
* Don't exclude from sync
* Don't generate output events when rewriting forward extremities
* Update output event check
* Historical output events
* Define output room event type
* Notify key changes on state
* Don't send our membership event twice
* Deduplicate state entries
* Tweaks
* Remove unnecessary nolint
* Fix current state upsert in sync API
* Send auth events as outliers, state events as rewrite
* Sync API don't consume state events
* Process events actually
* Improve outlier check
* Fix local room check
* Remove extra room check, it seems to break the whole damn world
* Fix federated join check
* Fix nil pointer exception
* Better comments on DeduplicateStateEntries
* Reflow forced federated joins
* Don't force federated join for possibly even local invites
* Comment SendEventWithState better
* Rewrite room state in sync API storage
* Add TODO
* Clean up all room data when receiving create event
* Don't generate output events for rewrites, but instead notify that state is rewritten on the final new event
* Rename to PurgeRoom
* Exclude backfilled messages from /sync
* Split out rewriting state from updating state from state res
Co-authored-by: Kegan Dougal <kegan@matrix.org>
* Use state from RespSendJoin post-check
* Don't create input events for invalid events
* Let's try this again
* Update gomatrixserverlib
* Update gomatrixserverlib to matrix-org/gomatrixserverlib@38f437f
* Use federation sender for backfill and getting missing events
* Fix internal URL paths
* Update go.mod/go.sum for matrix-org/gomatrixserverlib#218
* Add missing server implementations in HTTP interface
* Offset updates take place using TransactionWriter
* Refactor TransactionWriter in current state server
* Refactor TransactionWriter in federation sender
* Refactor TransactionWriter in key server
* Refactor TransactionWriter in media API
* Refactor TransactionWriter in server key API
* Refactor TransactionWriter in sync API
* Refactor TransactionWriter in user API
* Fix deadlocking Sync API tests
* Un-deadlock device database
* Fix appservice API
* Rename TransactionWriters to Writers
* Move writers up a layer in sync API
* Document sqlutil.Writer interface
* Add note to Writer documentation
* Add FederationClient interface to federationsender
- Use a shim struct in HTTP mode to keep the same API as `FederationClient`.
- Use `federationsender` instead of `FederationClient` in `keyserver`.
* Pointers not values
* Review comments
* Fix unit tests
* Rejig backoff
* Unbreak test
* Remove debug logs
* Review comments and linting
* Updated TransactionWriters, moved locks in roomserver, various other tweaks
* Fix redaction deadlocks
* Fix lint issue
* Rename SQLiteTransactionWriter to ExclusiveTransactionWriter
* Fix us not sending transactions through in latest events updater
* Initial pass at refactoring config (not finished)
* Don't forget current state and EDU servers
* More shifting around
* Update server key API tests
* Fix roomserver test
* Fix more tests
* Further tweaks
* Fix current state server test (sort of)
* Maybe fix appservices
* Fix client API test
* Include database connection string in database options
* Fix sync API build
* Update config test
* Fix unit tests
* Fix federation sender build
* Fix gobind build
* Set Listen address for all services in HTTP monolith mode
* Validate config, reinstate appservice derived in directory, tweaks
* Tweak federation API test
* Set MaxOpenConnections/MaxIdleConnections to previous values
* Update generate-config
* Add QueryDeviceMessages to serve up device keys and stream IDs
* Consume key change events in fedsender
Don't yet send them to destinations as we haven't worked them out yet
* Send device list updates to all required servers
* Glue it all together
* Initial persistence of blacklists
* Move statistics folder
* Make MaxFederationRetries configurable
* Set lower failure thresholds for Yggdrasil demos
* Still write events into database for blacklisted hosts (they can be tidied up later)
* Review comments
* Deduplicate FS database, add some EDU persistence groundwork
* Extend TransactionWriter to use optional existing transaction, use that for FS SQLite database writes
* Fix build due to bad keyserver import
* Working EDU persistence
* gocyclo, unsurprisingly
* Remove unused
* Update copyright notices
Squashed commit of the following:
commit 86c2388e13ffdbabdd50cea205652dccc40e1860
Merge: b0a3ee6c f5e7e751
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Thu Jul 16 13:47:10 2020 +0100
Merge branch 'master' into neilalexander/yggbarequic
commit b0a3ee6c5c063962384bb91c59ec753ddc8cfe5f
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Thu Jul 16 13:42:22 2020 +0100
Add support for broadcasting wake-up EDUs to known hosts
commit 8a5c2020b3a4b705b5d5686a9e71990a49e6d471
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Thu Jul 16 13:42:10 2020 +0100
Bare QUIC demo working
commit d3939b3d6568cf4262c0391486a5203873b68bfc
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Wed Jul 15 11:42:43 2020 +0100
Support bare Yggdrasil sessions with encrypted QUIC
* Initial work to send send-to-device messages over federation
* Wire up send-to-device consumer, message formatting
* Generate random message ID
* Review comments, update sytest whitelist
* Add a bit more logging to the fedsender
* bugfix: continue sending PDUs if ones are added whilst sending another PDU
Without this, the queue goes back to sleep on `<-oq.notifyPDUs` which won't
fire because `pendingPDUs` is already > 0. This should fix a flakey sytest.
* Break if no txn is sent
* Tweak federation sender wake-ups
* Update comments
* Remove break or that'll kill the parent loop
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Initial QUIC work
* Update Yggdrasil demo
* Make sure that the federation sender knows how many pending events are in the database when the worker starts
* QUIC tunables
* pprof
* Don't spin
* Set build info for Yggdrasil
* Initial work on persistent queues
* Update index for event ID and server name
* Put things into database (postgres for now)
* Duplicate postgres code into sqlite for now just to stop build errors, will fix SQLite soon
* Fix table name
* Fix index
* Fix table name
* Use RETURNING because LastInsertID is not supported by postgres
* Use functions
* Marshal headered event
* Don't error on now rows
* Don't block if there are PDUs waiting
* Try to tidy up JSON
* Debug logging
* Fix query, use transactions in postgres
* Clean up
* Rehydrate more opportunistically
* Fix SQLite
* remove unused types
* Review comments
* Shuffle things around a bit
* Clean up transaction properly
* Don't send empty transactions
* Reduce unnecessary retries
* Count PDUs to make more resilient
* Don't stop when there is work to be done
* Try to limit wakeups
* well this is tedious
* Fix race in incomplete transactions
* Thread safety on transaction ID/count
* Return remote errors from FS.PerformJoin
Follows the same pattern as PerformJoin on roomserver (no error return).
Also return the right format for incompatible room version errors.
Makes a bunch of tests pass!
* Handle network errors better when returning remote HTTP errors
* Linting
* Fix tests
* Update whitelist, pass network errors through in API=1 mode
* Minor perf/debugging improvements
- publicroomsapi: Don't call QueryEventsByID with no event IDs
- appservice: Consume only if there are 1 or more ASes
- roomserver: don't keep a copy of the request "for debugging" - we trace now
* fedsender: return early if we have no destinations
* Unbreak tests
This is a wrapper around whatever impl we have which then logs
the function name/request/response/error.
Also tweak when we log on kafka streams: only log on the producer
side not the consumer side: we've never had issues with comms and
having 1 message rather than N would be nice.
* s/QueryBackfill/PerformBackfill/g
* OutputEvent now includes AddStateEvents which contain the full event of extra state events
* Only include adds not the current event
* Get adding state right
