Commit graph

2098 commits

Author SHA1 Message Date
Kegsay 61963a74ae
Add logic for determining when device lists have changed due to membership changes (#1220)
* Flesh out structure for handling device list updates for room membership changes

* First cut untested algorithm

* Add tests for determining changed/left device lists

* Linting

* Unbreak tests

* Sigh.. linting
2020-07-27 09:19:55 +01:00
Kegan Dougal abef9bc04f Put this in the right category 2020-07-24 17:11:16 +01:00
Kegan Dougal b63fa7b880 Update AWSY list 2020-07-24 16:17:33 +01:00
Kegsay af5b4d1f6b
Modify QuerySharedUsers to handle counts/include/exclude (#1219)
* Modify QuerySharedUsers to handle counts/include/exclude

We will need this functionality when working out whether to
send device list changes to users who have joined/left a room.

* Linting
2020-07-24 10:33:41 +01:00
Kegsay 98f2f09bb4
keyserver: produce key change events (#1218)
* Produce kafka events when keys are added

* Consume key changes in syncapi with TODO markers for handling them and catching up

* unbreak tests

* Linting
2020-07-23 16:41:36 +01:00
Kegsay 7b862384a7
currentstate: Add QuerySharedUsers (#1217)
This will be used to determine who to send device list updates to. It
can also be used to determine who to send presence info to.
2020-07-23 12:26:31 +01:00
Neil Alexander cfeb1b2f42
Add UNIQUE constraint to blacklist table (#1216) 2020-07-23 10:22:23 +01:00
Kegan Dougal 9684efebd7 Fix gobind 2020-07-22 17:06:19 +01:00
Kegsay 541a23f712
Handle inbound federation E2E key queries/claims (#1215)
* Handle inbound /keys/claim and /keys/query requests

* Add display names to device key responses

* Linting
2020-07-22 17:04:57 +01:00
Neil Alexander 1e71fd645e
Persistent federation sender blacklist (#1214)
* Initial persistence of blacklists

* Move statistics folder

* Make MaxFederationRetries configurable

* Set lower failure thresholds for Yggdrasil demos

* Still write events into database for blacklisted hosts (they can be tidied up later)

* Review comments
2020-07-22 17:01:29 +01:00
Kegsay 470933789b
Perform outbound federation hits for querying/claiming E2E keys (#1212)
* Perform outbound federation hits for querying/claiming E2E keys

Untested currently because we need the receiving end to work
before sytest will be happy.

* Linting
2020-07-21 17:46:47 +01:00
Neil Alexander f54689f862
Fix lint error 2020-07-21 17:21:48 +01:00
Neil Alexander b6bc132485
Use TransactionWriter in other component SQLite (#1209)
* Use TransactionWriter on other component SQLites

* Fix sync API tests

* Fix panic in media API

* Fix a couple of transactions

* Fix wrong query, add some logging output

* Add debug logging into StoreEvent

* Adjust InsertRoomNID

* Update logging
2020-07-21 15:48:21 +01:00
Kegsay 1d72ce8b7a
Implement claiming one-time keys locally (#1210)
* Add API shape for claiming keys

* Implement claiming one-time keys locally

Fairly boring, nothing too special going on.
2020-07-21 14:47:53 +01:00
Neil Alexander d76eb1b994
Use TransactionWriter in roomserver SQLite (#1208) 2020-07-21 10:48:49 +01:00
Neil Alexander 489f34fed7
Remove debug lines 2020-07-20 17:03:20 +01:00
Neil Alexander 11a39fe3b5
Deduplicate FS database, EDU persistence table (#1207)
* Deduplicate FS database, add some EDU persistence groundwork

* Extend TransactionWriter to use optional existing transaction, use that for FS SQLite database writes

* Fix build due to bad keyserver import

* Working EDU persistence

* gocyclo, unsurprisingly

* Remove unused

* Update copyright notices
2020-07-20 16:55:20 +01:00
Creed Haymond f3c482b078
Adding the e2ekey database to the create_db script and docs (#1206)
Signed-off-by: Creed Haymond <creedasaurus@gmail.com>
2020-07-20 09:18:48 +01:00
Neil Alexander 986498f5a5
Don't block Yggdrasil switch 2020-07-16 15:19:26 +01:00
Neil Alexander fbf139f20e
Small fix to gobind build 2020-07-16 14:26:04 +01:00
Neil Alexander e5208c2ec9
Yggdrasil demo updates ("Bare QUIC")
Squashed commit of the following:

commit 86c2388e13ffdbabdd50cea205652dccc40e1860
Merge: b0a3ee6c f5e7e751
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Jul 16 13:47:10 2020 +0100

    Merge branch 'master' into neilalexander/yggbarequic

commit b0a3ee6c5c063962384bb91c59ec753ddc8cfe5f
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Jul 16 13:42:22 2020 +0100

    Add support for broadcasting wake-up EDUs to known hosts

commit 8a5c2020b3a4b705b5d5686a9e71990a49e6d471
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Jul 16 13:42:10 2020 +0100

    Bare QUIC demo working

commit d3939b3d6568cf4262c0391486a5203873b68bfc
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Jul 15 11:42:43 2020 +0100

    Support bare Yggdrasil sessions with encrypted QUIC
2020-07-16 13:52:08 +01:00
Kegsay f5e7e7513c
Implement /keys/query locally (#1204)
* Implement /keys/query locally

* Fix sqlite tests and close rows
2020-07-15 18:40:41 +01:00
Kegsay df8d6823ee
Fix panic when uploading keys (#1203)
We don't return OTK counts when the client doesn't upload OTKs.
2020-07-15 15:26:04 +01:00
Neil Alexander 84387e5249
Protect User API SQLite calls with mutexes (#1201)
* Protect SQLite calls with mutexes (replaces #1200)

* Remove locks on reads
2020-07-15 15:14:59 +01:00
Kegsay 9dd2ed7f65
Implement key uploads (#1202)
* Add storage layer for postgres/sqlite

* Return OTK counts when inserting new keys

* Hook up the key DB and make a test pass

* Convert postgres queries to be sqlite queries

* Blacklist test due to requiring rejected events

* Unbreak tests

* Update blacklist
2020-07-15 12:02:34 +01:00
Neil Alexander b4c07995d6
Create FUNDING.yml 2020-07-15 11:30:32 +01:00
Kegsay 7daa3bf098
Implement logic for key uploads (#1197)
* begin work on storing keys

* Finish rough impl of the internal key API

* Linting
2020-07-14 12:59:07 +01:00
Kegsay 37db60f4d4
Update README/docs (#1199)
* Update README

* Update docs
2020-07-14 12:58:00 +01:00
Neil Alexander 72b3160776
Send-to-device messages over federation (#1198)
* Initial work to send send-to-device messages over federation

* Wire up send-to-device consumer, message formatting

* Generate random message ID

* Review comments, update sytest whitelist
2020-07-14 12:33:37 +01:00
Kegan Dougal 5355c289b9 Bump GMSL version to fix sytest 2020-07-14 10:47:34 +01:00
Kegsay 396219ef53
Add boilerplate for key server APIs (#1196)
Also add a README which outilnes how things will work.
2020-07-13 16:02:35 +01:00
Neil Alexander 3178afde2c
Update go.mod/go.sum 2020-07-13 09:38:40 +01:00
Neil Alexander 08e9d996b6
Yggdrasil demo updates
Squashed commit of the following:

commit 6c2c48f862c1b6f8e741c57804282eceffe02487
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Fri Jul 10 16:28:09 2020 +0100

    Add README.md

commit 5eeefdadf8e3881dd7a32559a92be49bd7ddaf47
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Fri Jul 10 10:18:50 2020 +0100

    Fix wedge in federation sender

commit e2ebffbfba25cf82378393940a613ec32bfb909f
Merge: 0883ef88 abf26c12
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Fri Jul 10 09:51:23 2020 +0100

    Merge branch 'master' into neilalexander/yggdrasil

commit 0883ef8870e340f2ae9a0c37ed939dc2ab9911f6
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Fri Jul 10 09:51:06 2020 +0100

    Adjust timeouts

commit ba2d53199910f13b60cc892debe96a962e8c9acb
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Jul 9 16:34:40 2020 +0100

    Try to wake up from peers/sessions properly

commit 73f42eb494741ba5b0e0cef43654708e3c8eb399
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Jul 9 15:43:38 2020 +0100

    Use TransactionWriter to reduce database lock issues on SQLite

commit 08bfe63241a18c58c539c91b9f52edccda63a611
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Jul 9 12:38:02 2020 +0100

    Un-wedge federation

    Squashed commit of the following:

    commit aee933f878
    Author: Neil Alexander <neilalexander@users.noreply.github.com>
    Date:   Thu Jul 9 12:22:41 2020 +0100

        Un-goroutine the goroutines

    commit 478374e5d1
    Author: Neil Alexander <neilalexander@users.noreply.github.com>
    Date:   Thu Jul 9 12:09:31 2020 +0100

        Reduce federation sender wedges

commit 40cc62c54d9e3a863868214c48b7c18e522a4772
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Jul 9 10:02:52 2020 +0100

    Handle switching in/out background more reliably
2020-07-10 16:28:18 +01:00
Kegsay abf26c12f1
Add User-Interactive Authentication (#1193)
* Add User-Interactive Authentication

And use it when deleting a device. With tests.

* Make remaining sytest pass

* Linting

* 403 not 401 on wrong user/pass
2020-07-10 00:39:44 +01:00
Neil Alexander 9cc52f47f3
Use TransactionWriter to reduce database lock issues on SQLite (#1192) 2020-07-09 17:48:56 +01:00
Neil Alexander 99b50f30a0
Reduce federation sender wedges (#1191)
* Reduce federation sender wedges

* Un-goroutine the goroutines
2020-07-09 15:39:35 +01:00
Kegsay d9648b0615
Finish implementing redactions (#1189)
* Add a bit more logging to the fedsender

* bugfix: continue sending PDUs if ones are added whilst sending another PDU

Without this, the queue goes back to sleep on `<-oq.notifyPDUs` which won't
fire because `pendingPDUs` is already > 0. This should fix a flakey sytest.

* Break if no txn is sent

* WIP syncapi work

* More debugging

* Bump GMSL version to pull in working Event.Redact

* Remove logging

* Make redactions work on v3+

* Fix more tests
2020-07-08 17:45:39 +01:00
Neil Alexander a5a51b4141
linter 2020-07-08 17:28:16 +01:00
Neil Alexander 6bee6e77d9
Yet another v1 vs r0 media endpoint (#1190) 2020-07-08 16:39:50 +01:00
Neil Alexander 4370a808e3
Fix media path for iOS gobind build 2020-07-08 16:24:31 +01:00
Neil Alexander fbd9c11c3c
Check error when setting static peer 2020-07-08 15:48:10 +01:00
Neil Alexander 689c3df02d
Set QUIC keepalive again 2020-07-08 15:44:37 +01:00
Neil Alexander 2bb580c1b0
Handle case where pendingPDUs might get out of sync for some reason 2020-07-08 15:42:36 +01:00
Neil Alexander af6bc47f16
Squashed commit of the following:
commit b4cb47aa1329d2ada10ae6426fd9d2a69f47536a
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Jul 8 14:13:27 2020 +0100

    Restrict transaction send context time

commit 7c28205cdb5d842071d46b1ec599d09cca708e57
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Jul 8 14:00:06 2020 +0100

    Add to gobind build

commit d9e2c72e0576a2eb0ce6ac48eed6cc9d4761a0ea
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Jul 8 13:43:21 2020 +0100

    Wake up destination queues for new sessions/links

commit 21766c6c52bd00511d28981457e9034358c32a8d
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Jul 8 13:17:18 2020 +0100

    Tweak QUIC parameters
2020-07-08 14:52:48 +01:00
Neil Alexander de0f427ddc Fix build 2020-07-07 16:54:14 +01:00
Neil Alexander 51fd532940 Fix error handling in federationsender 2020-07-07 16:53:10 +01:00
Kegsay 8e9580852d
bugfix: continue sending PDUs if ones are added whilst sending another PDU (#1187)
* Add a bit more logging to the fedsender

* bugfix: continue sending PDUs if ones are added whilst sending another PDU

Without this, the queue goes back to sleep on `<-oq.notifyPDUs` which won't
fire because `pendingPDUs` is already > 0. This should fix a flakey sytest.

* Break if no txn is sent

* Tweak federation sender wake-ups

* Update comments

* Remove break or that'll kill the parent loop

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-07-07 16:36:10 +01:00
Neil Alexander d03f6ac725 Fix static peer on Ygg demo build 2020-07-07 14:38:59 +01:00
Neil Alexander b21fad92e5 Tweaks to Dendrite Yggdrasil demo setup 2020-07-07 14:18:58 +01:00
Kegsay 99ea1f9b48
Emit redacted_event from the roomserver when redactions are validated (#1186)
* Emit redacted_event from the roomserver when redactions are validated

- Consume them in the currentstateserver and act accordingly.
- Add integration test for the roomserver to check that injecting
  `m.room.redaction` events result in `redacted_event` being emitted.

* Linting

* Ignore events that redact themselves
2020-07-07 12:51:55 +01:00