Neil Alexander
cae335bc74
Check for duplicates when pulling things out of the database
2020-12-07 13:14:25 +00:00
Neil Alexander
147b3eddad
Fix a couple of bugs
2020-12-07 12:58:36 +00:00
Neil Alexander
d4bd44ebe6
Don't queue in memory for blacklisted destinations
2020-12-07 12:36:07 +00:00
Neil Alexander
558bfa8b06
Don't queue in memory for blacklisted destinations
2020-12-07 12:22:24 +00:00
Neil Alexander
68a3b3e3fa
Don't try to rehydrate twice
2020-12-07 12:11:30 +00:00
Neil Alexander
880e151439
Bug-fixing
2020-12-07 12:08:23 +00:00
Neil Alexander
7207ad53fe
Comments
2020-12-07 11:58:41 +00:00
Neil Alexander
949ada103b
Less spam about sleeping queues
2020-12-07 11:46:19 +00:00
Neil Alexander
8861f907ed
Don't hold lock while sending transaction
2020-12-07 11:43:35 +00:00
Neil Alexander
36356ac8d1
Remove channels as they add extra complexity and possibly will deadlock
2020-12-07 11:38:32 +00:00
Neil Alexander
613a47354f
Check nils more
2020-12-07 10:49:06 +00:00
Neil Alexander
7246580d54
Update comments
2020-12-07 10:28:34 +00:00
Neil Alexander
26cc2d480b
Fix transaction coalescing
2020-12-07 09:54:51 +00:00
Neil Alexander
91ac6e9880
Tweak logging
2020-12-07 09:40:23 +00:00
Neil Alexander
6029c79dfa
nil PDUs/EDUs shouldn't happen but guard against them for safety
2020-12-07 09:29:59 +00:00
Neil Alexander
7faef15339
Comments
2020-12-04 17:37:48 +00:00
Neil Alexander
e0fde0c24e
Log less
2020-12-04 17:22:58 +00:00
Neil Alexander
f8825e0c12
Try to get overflowed events from database
2020-12-04 17:18:23 +00:00
Neil Alexander
9700f0b31c
Missing operators
2020-12-04 16:24:36 +00:00
Neil Alexander
44187311b6
Clean up better
2020-12-04 16:23:58 +00:00
Neil Alexander
bfcf1307e8
Refactor federation sender, again
2020-12-04 16:22:01 +00:00
Ronnie Ebrin
a677a288bd
federationsender/roomserver: don't panic while federation is disabled ( #1615 )
2020-12-04 14:08:17 +00:00
Neil Alexander
bdf6490375
Add ability to disable federation ( #1604 )
...
* Allow disabling federation
* Don't start federation queues if disabled
* Fix for Go 1.13
2020-12-02 15:10:03 +00:00
Neil Alexander
fe5d1400bf
Update federation timeouts ( #1504 )
...
* Update to matrix-org/gomatrixserverlib#234
* Update gomatrixserverlib
* Update federation timeouts
* Fix dendritejs
* Increase /send context time in destination queue
2020-10-09 17:08:32 +01:00
Neil Alexander
d63d7c5640
Tweak log level of a fairly common log line
2020-09-29 17:08:47 +01:00
Neil Alexander
a854e3aa18
Fix backoff bug
2020-09-22 14:53:36 +01:00
Neil Alexander
880b164490
Refactor backoff again ( #1431 )
...
* Tweak backoffs
* Refactor backoff some more, remove BackoffIfRequired as it adds unnecessary complexity
* Ignore 404s
2020-09-21 13:30:37 +01:00
Kegsay
2570418f42
Remove ServerACLs from the current state server ( #1390 )
...
* Remove ServerACLs from the current state server
Functionality moved to roomserver
* Nothing to see here, move along
2020-09-04 10:40:58 +01:00
Neil Alexander
04bc09f591
Defer keyserver and federationsender wakeups to give HTTP listeners time to start ( #1389 )
2020-09-03 21:17:55 +01:00
Neil Alexander
6cb1a65809
Synchronous invites ( #1273 )
...
* Refactor invites to be synchronous
* Fix synchronous invites
* Fix client API return type for send invite error
* Linter
* Restore PerformError on rsAPI.PerformInvite
* Update sytest-whitelist
* Don't override PerformError with normal errors
* Fix error passing
* Un-whitelist a couple of tests
* Update sytest-whitelist
* Try to handle multiple invite rejections better
* nolint
* Update gomatrixserverlib
* Fix /v1/invite test
* Remove replace from go.mod
2020-08-17 11:40:49 +01:00
Neil Alexander
4c4732a9c9
Don't send to ACL'd servers ( #1267 )
...
* Don't send to ACL'd servers
* Use gjson to look for room_id in EDU
2020-08-13 14:23:37 +01:00
Neil Alexander
58998e9874
Backoff fixes ( #1250 )
...
* Backoff fixes
* Update comments
* Fix destination queue
* Log why we're blacklisting
* Fix logic fail
* Logging level
* Fix bug
* Maybe fix that bug after all
* Fix debug output
* Fix tests
2020-08-07 18:50:29 +01:00
Neil Alexander
5dd5a41119
Tweak log levels of some federation logging ( #1248 )
...
* Tweak log levels of some federation logging
* Update go.mod/go.sum for matrix-org/util#22 and matrix-org/gomatrixserverlib#215
2020-08-07 15:00:23 +01:00
Neil Alexander
1e71fd645e
Persistent federation sender blacklist ( #1214 )
...
* Initial persistence of blacklists
* Move statistics folder
* Make MaxFederationRetries configurable
* Set lower failure thresholds for Yggdrasil demos
* Still write events into database for blacklisted hosts (they can be tidied up later)
* Review comments
2020-07-22 17:01:29 +01:00
Neil Alexander
11a39fe3b5
Deduplicate FS database, EDU persistence table ( #1207 )
...
* Deduplicate FS database, add some EDU persistence groundwork
* Extend TransactionWriter to use optional existing transaction, use that for FS SQLite database writes
* Fix build due to bad keyserver import
* Working EDU persistence
* gocyclo, unsurprisingly
* Remove unused
* Update copyright notices
2020-07-20 16:55:20 +01:00
Neil Alexander
72b3160776
Send-to-device messages over federation ( #1198 )
...
* Initial work to send send-to-device messages over federation
* Wire up send-to-device consumer, message formatting
* Generate random message ID
* Review comments, update sytest whitelist
2020-07-14 12:33:37 +01:00
Neil Alexander
08e9d996b6
Yggdrasil demo updates
...
Squashed commit of the following:
commit 6c2c48f862c1b6f8e741c57804282eceffe02487
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Fri Jul 10 16:28:09 2020 +0100
Add README.md
commit 5eeefdadf8e3881dd7a32559a92be49bd7ddaf47
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Fri Jul 10 10:18:50 2020 +0100
Fix wedge in federation sender
commit e2ebffbfba25cf82378393940a613ec32bfb909f
Merge: 0883ef88 abf26c12aee933f878478374e5d1
2020-07-10 16:28:18 +01:00
Neil Alexander
99b50f30a0
Reduce federation sender wedges ( #1191 )
...
* Reduce federation sender wedges
* Un-goroutine the goroutines
2020-07-09 15:39:35 +01:00
Neil Alexander
2bb580c1b0
Handle case where pendingPDUs might get out of sync for some reason
2020-07-08 15:42:36 +01:00
Neil Alexander
af6bc47f16
Squashed commit of the following:
...
commit b4cb47aa1329d2ada10ae6426fd9d2a69f47536a
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Wed Jul 8 14:13:27 2020 +0100
Restrict transaction send context time
commit 7c28205cdb5d842071d46b1ec599d09cca708e57
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Wed Jul 8 14:00:06 2020 +0100
Add to gobind build
commit d9e2c72e0576a2eb0ce6ac48eed6cc9d4761a0ea
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Wed Jul 8 13:43:21 2020 +0100
Wake up destination queues for new sessions/links
commit 21766c6c52bd00511d28981457e9034358c32a8d
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Wed Jul 8 13:17:18 2020 +0100
Tweak QUIC parameters
2020-07-08 14:52:48 +01:00
Neil Alexander
de0f427ddc
Fix build
2020-07-07 16:54:14 +01:00
Neil Alexander
51fd532940
Fix error handling in federationsender
2020-07-07 16:53:10 +01:00
Kegsay
8e9580852d
bugfix: continue sending PDUs if ones are added whilst sending another PDU ( #1187 )
...
* Add a bit more logging to the fedsender
* bugfix: continue sending PDUs if ones are added whilst sending another PDU
Without this, the queue goes back to sleep on `<-oq.notifyPDUs` which won't
fire because `pendingPDUs` is already > 0. This should fix a flakey sytest.
* Break if no txn is sent
* Tweak federation sender wake-ups
* Update comments
* Remove break or that'll kill the parent loop
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-07-07 16:36:10 +01:00
Neil Alexander
46dbc46f84
Wake up destination queues more aggressively ( #1183 )
...
* Wake up destination queues more aggressively
* We don't really need Ch here do we
2020-07-03 16:31:56 +01:00
Neil Alexander
1773fd84b7
Hydrate destination queues at startup ( #1179 )
...
* Hydrate destination queues at startup
* Review comments
2020-07-03 11:49:49 +01:00
Neil Alexander
38caf8e5b7
Yggdrasil+QUIC demo, federation sender tweaks ( #1177 )
...
* Initial QUIC work
* Update Yggdrasil demo
* Make sure that the federation sender knows how many pending events are in the database when the worker starts
* QUIC tunables
* pprof
* Don't spin
* Set build info for Yggdrasil
2020-07-02 17:43:07 +01:00
Neil Alexander
42dd962425
Persistent federation sender queues (PDUs) ( #1173 )
...
* Initial work on persistent queues
* Update index for event ID and server name
* Put things into database (postgres for now)
* Duplicate postgres code into sqlite for now just to stop build errors, will fix SQLite soon
* Fix table name
* Fix index
* Fix table name
* Use RETURNING because LastInsertID is not supported by postgres
* Use functions
* Marshal headered event
* Don't error on now rows
* Don't block if there are PDUs waiting
* Try to tidy up JSON
* Debug logging
* Fix query, use transactions in postgres
* Clean up
* Rehydrate more opportunistically
* Fix SQLite
* remove unused types
* Review comments
* Shuffle things around a bit
* Clean up transaction properly
* Don't send empty transactions
* Reduce unnecessary retries
* Count PDUs to make more resilient
* Don't stop when there is work to be done
* Try to limit wakeups
* well this is tedious
* Fix race in incomplete transactions
* Thread safety on transaction ID/count
2020-07-01 11:46:38 +01:00
Kegsay
0dc4ceaa2d
Minor perf/debugging improvements ( #1121 )
...
* Minor perf/debugging improvements
- publicroomsapi: Don't call QueryEventsByID with no event IDs
- appservice: Consume only if there are 1 or more ASes
- roomserver: don't keep a copy of the request "for debugging" - we trace now
* fedsender: return early if we have no destinations
* Unbreak tests
2020-06-12 15:11:33 +01:00
Kegsay
399b6ae334
Remove federationsender producer, which in fact was not a producer ( #1115 )
...
* Remove federationsender producer, which in fact was not a producer
* Set the signing struct
2020-06-10 16:54:43 +01:00
Neil Alexander
225b72bd42
Don't reset counters before successful outgoing federation request ( #1089 )
...
* Don't reset counters before successful outgoing federation request on incoming federation request
* Comments
2020-06-04 10:54:10 +01:00
