Commit graph

145 commits

Author SHA1 Message Date
Neil Alexander 3ba3530685
Merge branch 'master' into matthew/peeking-over-fed 2020-12-02 15:44:09 +00:00
Neil Alexander bdf6490375
Add ability to disable federation (#1604)
* Allow disabling federation

* Don't start federation queues if disabled

* Fix for Go 1.13
2020-12-02 15:10:03 +00:00
Neil Alexander a2a5c7eb53
Merge branch 'master' into matthew/peeking-over-fed 2020-12-02 15:01:05 +00:00
Kegsay 6353b0b7e4
MSC2836: Threading - part one (#1589)
* Add mscs/hooks package, begin work for msc2836

* Flesh out hooks and add SQL schema

* Begin implementing core msc2836 logic

* Add test harness

* Linting

* Implement visibility checks; stub out APIs for tests

* Flesh out testing

* Flesh out walkThread a bit

* Persist the origin_server_ts as well

* Edges table instead of relationships

* Add nodes table for event metadata

* LEFT JOIN to extract origin_server_ts for children

* Add graph walking structs

* Implement walking algorithm

* Add more graph walking tests

* Add auto_join for local rooms

* Fix create table syntax on postgres

* Add relationship_room_id|servers to the unsigned section of events

* Persist the parent room_id/servers in edge metadata

Other events cannot assert the true room_id/servers for the
parent event, only make claims to them, hence why this is
edge metadata.

* guts to pass through room_id/servers

* Refactor msc2836 to allow handling from federation

* Add JoinedVia to PerformJoin responses

* Fix tests; review comments
2020-11-19 11:34:59 +00:00
Neil Alexander 20a01bceb2
Pass pointers to events — reloaded (#1583)
* Pass events as pointers

* Fix lint errors

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update to matrix-org/gomatrixserverlib#240
2020-11-16 15:44:53 +00:00
S7evinK bcb89ada5e
Implement read receipts (#1528)
* fix conversion from int to string yields a string of one rune, not a string of digits

* Add receipts table to syncapi

* Use StreamingToken as the since value

* Add required method to testEDUProducer

* Make receipt json creation "easier" to read

* Add receipts api to the eduserver

* Add receipts endpoint

* Add eduserver kafka consumer

* Add missing kafka config

* Add passing tests to whitelist

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Fix copy & paste error

* Fix column count error

* Make outbound federation receipts pass

* Make "Inbound federation rejects receipts from wrong remote" pass

* Don't use errors package

* - Add TODO for batching requests
- Rename variable

* Return a better error message

* - Use OutputReceiptEvent instead of InputReceiptEvent as result
- Don't use the errors package for errors
- Defer CloseAndLogIfError to close rows
- Fix Copyright

* Better creation/usage of JoinResponse

* Query all joined rooms instead of just one

* Update gomatrixserverlib

* Add sqlite3 migration

* Add postgres migration

* Ensure required sequence exists before running migrations

* Clarification on comment

* - Fix a bug when creating client receipts
- Use concrete types instead of interface{}

* Remove dead code
Use key for timestamp

* Fix postgres query...

* Remove single purpose struct

* Use key/value directly

* Only apply receipts on initial sync or if edu positions differ,
otherwise we'll be sending the same receipts over and over again.

* Actually update the id, so it is correctly send in syncs

* Set receipt on request to /read_markers

* Fix issue with receipts getting overwritten

* Use fmt.Errorf instead of pkg/errors

* Revert "Add postgres migration"

This reverts commit 722fe5a046.

* Revert "Add sqlite3 migration"

This reverts commit d113b03f64.

* Fix selectRoomReceipts query

* Make golangci-lint happy

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-11-09 18:46:11 +00:00
Kegsay 0fd9e960bd
Merge branch 'master' into matthew/peeking-over-fed 2020-10-22 12:06:43 +01:00
Neil Alexander 3afc623098
Fix RewritesState bug (#1557)
* Set RewritesState once

* Check if any new state provided

* Obey rewritesState

* Don't nuke everything the sync API knows when purging state

* Fix panic from duplicate insert

* Consistency

* Use HasState

* Remove nolint

* Clean up joined rooms on state rewrite
2020-10-22 10:39:16 +01:00
Neil Alexander be5a4e6adc
Fix performOutboundPeekUsingServer 2020-10-20 17:32:41 +01:00
Neil Alexander 797085f1b7
Merge branch 'master' into matthew/peeking-over-fed 2020-10-20 17:18:01 +01:00
Neil Alexander 6e63df1d9a
KindOld (#1531)
* Add KindOld

* Don't process latest events/memberships for old events

* Allow federationsender to ignore duplicate key entries when LatestEventIDs is duplicated by RS output events

* Signal to downstream components if an event has become a forward extremity

* Don't exclude from sync

* Soft-fail checks on KindNew

* Don't run the latest events updater at all for KindOld

* Don't make federation sender change after all

* Kind in federation sender join

* Don't send isForwardExtremity

* Fix syncapi

* Update comments

* Fix SendEventWithState

* Update sytest-whitelist

* Generate old output events

* Sync API consumes old room events

* Update comments
2020-10-19 14:59:13 +01:00
Neil Alexander 49abe359e6
Start Kafka connections for each component that needs them (#1527)
* Start Kafka connection for each component that needs one

* Fix roomserver unit tests

* Rename to naffkaInstance (@Kegsay review comment)

* Fix import cycle
2020-10-15 13:27:13 +01:00
Neil Alexander 9d6b77c58a
Try to retrieve missing auth events from multiple servers (#1516)
* Recursively fetch auth events if needed

* Fix processEvent call

* Ask more servers in lookupEvent

* Don't panic!

* Panic at the Disco

* Find servers more aggressively

* Add getServers

* Fix number of servers to 5, don't bail making RespState if auth events missing

* Fix panic

* Ignore missing state events too

* Report number of servers correctly

* Don't reuse request context for /send_join

* Update federation API tests

* Don't recurse processEvents

* Implement getEvents differently
2020-10-13 11:53:20 +01:00
Kegsay 9096bfcee8
Validate m.room.create events in send_join responses (#1505)
* Validate m.room.create events in send_join responses

For sytest compliance, refs #1315 and #1317

Fixes #1317

* Linting
2020-10-10 00:21:15 +01:00
Neil Alexander fe5d1400bf
Update federation timeouts (#1504)
* Update to matrix-org/gomatrixserverlib#234

* Update gomatrixserverlib

* Update federation timeouts

* Fix dendritejs

* Increase /send context time in destination queue
2020-10-09 17:08:32 +01:00
Neil Alexander bf90db5b60
Remove KindRewrite (#1481)
* Don't send rewrite events

* Remove final traces of rewrite events

* Remove test that is no longer needed

* Revert "Remove test that is no longer needed"

This reverts commit 9a45babff6.

* Update test to use KindOutlier
2020-10-06 11:05:00 +01:00
Neil Alexander d63d7c5640
Tweak log level of a fairly common log line 2020-09-29 17:08:47 +01:00
Matthew Hodgson ebeff8ddb1 go fmt 2020-09-27 21:03:00 +01:00
Matthew Hodgson efd8656302 support postgres for federated peeking 2020-09-27 20:59:04 +01:00
Matthew Hodgson fd90849802 fix comment 2020-09-27 00:31:56 +01:00
Matthew Hodgson 75c3f2df8d merge master 2020-09-26 23:27:39 +01:00
Matthew Hodgson 0ab4bc9e8e document the peek send stream race better 2020-09-23 00:11:58 +01:00
Matthew Hodgson 3202c7e76f go fmt 2020-09-23 00:08:23 +01:00
Matthew Hodgson 20e2cb4b7e track latestevent on /peek 2020-09-22 23:42:19 +01:00
Neil Alexander a854e3aa18
Fix backoff bug 2020-09-22 14:53:36 +01:00
Neil Alexander a14b29b526
Initial notary support (#1436)
* Initial work on notary support

* Somewhat working (but not properly filtered) notary support, other tweaks

* Update gomatrixserverlib
2020-09-22 14:40:54 +01:00
Neil Alexander a7563ede3d
Process federated joins in background context (#1434)
* Return early from federated room join

* Synchronous perform-join as long as possible

* Don't allow multiple federated joins to the same room by the same user
2020-09-22 11:05:45 +01:00
Neil Alexander 880b164490
Refactor backoff again (#1431)
* Tweak backoffs

* Refactor backoff some more, remove BackoffIfRequired as it adds unnecessary complexity

* Ignore 404s
2020-09-21 13:30:37 +01:00
Matthew Hodgson 41b9b663a5 Merge branch 'master' into matthew/peeking-over-fed 2020-09-19 17:28:12 +01:00
Neil Alexander 965f068d1a
Handle state with input event as new events (#1415)
* SendEventWithState events as new

* Use cumulative state IDs for final event

* Error wrapping in calculateAndSetState

* Handle overwriting same event type and state key

* Hacky way to spot historical events

* Don't exclude from sync

* Don't generate output events when rewriting forward extremities

* Update output event check

* Historical output events

* Define output room event type

* Notify key changes on state

* Don't send our membership event twice

* Deduplicate state entries

* Tweaks

* Remove unnecessary nolint

* Fix current state upsert in sync API

* Send auth events as outliers, state events as rewrite

* Sync API don't consume state events

* Process events actually

* Improve outlier check

* Fix local room check

* Remove extra room check, it seems to break the whole damn world

* Fix federated join check

* Fix nil pointer exception

* Better comments on DeduplicateStateEntries

* Reflow forced federated joins

* Don't force federated join for possibly even local invites

* Comment SendEventWithState better

* Rewrite room state in sync API storage

* Add TODO

* Clean up all room data when receiving create event

* Don't generate output events for rewrites, but instead notify that state is rewritten on the final new event

* Rename to PurgeRoom

* Exclude backfilled messages from /sync

* Split out rewriting state from updating state from state res

Co-authored-by: Kegan Dougal <kegan@matrix.org>
2020-09-15 11:17:46 +01:00
Matthew Hodgson 32f898d668 fix api.SendEventWithState with no event 2020-09-13 03:32:01 +01:00
Matthew Hodgson a160c074e8 setup inbound peeks db correctly 2020-09-13 03:31:11 +01:00
Matthew Hodgson 3caae7913b rename FSAPI's PerformPeek as PerformOutboundPeek 2020-09-13 00:14:24 +01:00
Matthew Hodgson 8f203febc1 rename ambiguous RemotePeek as InboundPeek 2020-09-13 00:04:02 +01:00
Matthew Hodgson 71732f2c28 add inbound peeks table and hook it up 2020-09-12 23:54:46 +01:00
Matthew Hodgson 59e2be7113 rename remote_peeks_table as outbound_peeks_table 2020-09-12 23:01:06 +01:00
Matthew Hodgson 4e96e62923 rename RemotePeeks as OutboundPeeks 2020-09-12 22:59:25 +01:00
Matthew Hodgson 0ae0d11446 reimplement SS /peek to prod the RS to tell the FS about the peek 2020-09-12 22:45:00 +01:00
Matthew Hodgson fff18454b1 todo for authenticating state returned by /peek 2020-09-12 01:23:32 +01:00
Matthew Hodgson f8bb4487df fix sql thinko 2020-09-12 00:24:22 +01:00
Matthew Hodgson df29509e7e fix build & lint 2020-09-11 22:39:50 +01:00
Matthew Hodgson baee97bff7 (almost) make it build 2020-09-11 21:04:33 +01:00
Matthew Hodgson f236e8290d Merge branch 'master' into matthew/peeking-over-fed 2020-09-11 19:44:40 +01:00
Matthew Hodgson 39507bacc3
Peeking via MSC2753 (#1370)
Initial implementation of MSC2753, as tested by https://github.com/matrix-org/sytest/pull/944.
Doesn't yet handle unpeeks, peeked EDUs, or history viz changing during a peek - these will follow.
https://github.com/matrix-org/dendrite/pull/1370 has full details.
2020-09-10 14:39:18 +01:00
Neil Alexander 668a722ee0
Backoff for 401s (#1410)
* Backoff for 401s

* Human-readable retry_after in logs
2020-09-08 13:41:08 +01:00
Neil Alexander 1602df8752
Ignore state events with invalid signatures when joining rooms (#1407)
* Use state from RespSendJoin post-check

* Don't create input events for invalid events

* Let's try this again

* Update gomatrixserverlib

* Update gomatrixserverlib to matrix-org/gomatrixserverlib@38f437f
2020-09-07 16:54:51 +01:00
Kegsay 088294ee65
Remove QueryRoomsForUser from current state server (#1398) 2020-09-04 15:58:30 +01:00
Kegsay 2570418f42
Remove ServerACLs from the current state server (#1390)
* Remove ServerACLs from the current state server

Functionality moved to roomserver

* Nothing to see here, move along
2020-09-04 10:40:58 +01:00
Matthew Hodgson 4ca2cf49f8 typo 2020-09-04 01:44:38 +01:00
Matthew Hodgson c1f1fcdc67 WIP of implementing MSC2444 2020-09-04 01:38:22 +01:00