Commit graph

868 commits

Author SHA1 Message Date
Neil Alexander d7eb54c5b3 Move responsibility for generating local keys into server key API, don't register prom in caches unless needed, start tests 2020-06-15 14:05:57 +01:00
Neil Alexander 81f4ef5e09 Start testing stuff 2020-06-11 13:02:11 +01:00
Neil Alexander 687326933e More key tweaks 2020-06-11 10:50:48 +01:00
Neil Alexander ce2517b9fb Support embedding Riot Web (#1113)
* Embed

    * Support for embedding is better now

    * HTTP on localhost for Ygg demo

    * Move embed into Yggdrasil demo
2020-06-11 10:16:46 +01:00
Neil Alexander b7d0ca6855 Update Yggdrasil demo timeouts again 2020-06-11 09:50:54 +01:00
Neil Alexander 63a24e81c4 Yggdrasil demo tweaks 2020-06-10 17:18:37 +01:00
Kegsay 399b6ae334
Remove federationsender producer, which in fact was not a producer (#1115)
* Remove federationsender producer, which in fact was not a producer

* Set the signing struct
2020-06-10 16:54:43 +01:00
Neil Alexander 3b4be90000
Improvements to Yggdrasil demo (#1114)
* Improvements to Yggdrasil demo

* Fix missing copyright

* Fix tie-break
2020-06-10 16:29:02 +01:00
Kegsay 90a0aa9b3e
Remove public appservice routes (#1112) 2020-06-10 13:28:32 +01:00
Kegsay b7187a9a35
Remove clientapi producers which aren't actually producers (#1111)
* Remove clientapi producers which aren't actually producers

They are actually just convenience wrappers around the internal APIs
for roomserver/eduserver. Move their logic to their respective `api`
packages and call them directly.

* Remove TODO

* unbreak ygg
2020-06-10 12:17:54 +01:00
Neil Alexander d9d6f4568c
Use ServerKeyAPI as keyring (#1110)
* Use ServerKeyAPI as keyring

* Update fetcher name for debug logs
2020-06-10 11:02:03 +01:00
Neil Alexander 464718c3e6
Yggdrasil-based P2P demo (#1108)
* Initial work on Yggdrasil demo

* Muxing?

* Yamux

* Updates to yamux

* Updates

* Comments

* Update to use monolith stuff

* Update go.mod/go.sum

* Set defaults

* Tweaks

* Update yggdrasil

* Update config

* MarshalIndent

* Change default instance name/port

* add -peer switch

* gocyclo, for a change

* Determinate yamux roles

* Fix copyright notices

* Remove HTTP API checks as always false, remove unused topic
2020-06-10 10:54:06 +01:00
Kegsay 98cb0705ea
Remove unused UserUpdates producer (#1109) 2020-06-10 10:34:22 +01:00
Kegsay 85ac8a3f5b
Factor out how monolith routes get added (#1107)
Previously we had 3 monoliths:
 - dendrite-monolith-server
 - dendrite-demo-libp2p
 - dendritejs

which all had their own of setting up public routes. Factor this
out into a new `setup.Monolith` struct which gets all dependencies
set as fields. This is different to `basecomponent.Base` which
doesn't provide any way to set configured deps (e.g public rooms db)

Part of a larger process to clean up how we initialise Dendrite.
2020-06-09 12:07:33 +01:00
Kegsay 4f171c56a8
Split out SetupFooComponent (#1106)
* Split out adding HTTP routes from making internal APIs for clarity

* Split out more components

* Split out more things

* Finish converting

* internal mux for internal routes
2020-06-08 15:51:07 +01:00
Neil Alexander cdb9a11571 Update gomatrixserverlib 2020-06-08 13:56:03 +01:00
Neil Alexander 67784ecb56
Nuke old Docker stuff (#1104)
* Remove old Docker stuff

* Move hub stuff to upper level docker directory

* Build monolith images

* Update README.md

* Update paths

* Update README.md
2020-06-05 19:00:30 +01:00
Neil Alexander e7b19d2c70
More flexible caching (#1101) 2020-06-05 16:42:01 +01:00
Neil Alexander 76ff47c052
Use AuthChainProvider to try and speed up federated joins (#1100)
* Use MissingAuthEventHandler on performjoin to try and speed up cases where we have missing events

* Update gomatrixserverlib

* Use supplied room version

* Use AuthChainProvider

* Tweaks

* Update gomatrixserverlib

* Signature checks
2020-06-05 11:48:52 +01:00
Kegsay 29a20d1da7
General cleanup when making components (#1098)
* Remove ParseMonolith/LoadMonolith

* cleanup which components need to be made
2020-06-05 09:28:15 +01:00
Kegsay 2bd12f635c
Convert serverkeys to inthttp (#1097) 2020-06-04 16:26:35 +01:00
Kegsay 9834ac97db
Convert everything but serverkeyapi to inthttp (#1096)
* Convert roomserver to new inthttp format

* Convert eduserver to new inthttp format

* Convert appservice to new inthttp format
2020-06-04 15:43:07 +01:00
Neil Alexander d785ad82b9
Add key validity fetching to server key API (#1094)
* Add key validity checks

* Store fetched keys

* Don't double-cache key results

* Perform server key API operations using new context

* Revert "Perform server key API operations using new context"

This reverts commit 02172223f5.

* Perform server key API operations using new context
2020-06-04 15:40:23 +01:00
Kegsay f4c676ccdd
Refactor how federationsender gets created (#1095)
* Refactor how federationsender gets created

* s/httpint/inthttp/ for alphabetical niceness with internal package
2020-06-04 14:27:10 +01:00
Neil Alexander f7025d3499 Update gomatrixserverlib 2020-06-04 13:30:08 +01:00
Kegsay 097cdf879c
s/Base64String/Base64Bytes/g (#1093)
* s/Base64String/Base64Bytes/g

Requires https://github.com/matrix-org/gomatrixserverlib/pull/203 to land first

* update gmsl
2020-06-04 11:50:57 +01:00
Kegsay e7d1ac84c3
Add ParseFileURI and use it when dealing with file URIs (#1088)
* Add ParseFileURI and use it when dealing with file URIs

Fixes #1059

* Missing file

* Linting
2020-06-04 11:18:08 +01:00
Kegsay d4f9a4bb97
Fix #632 and send spec-compliant transactions to the AS (#1091) 2020-06-04 11:17:37 +01:00
Kegsay feb32ba365
Encode v3 event IDs correctly (#1090) 2020-06-04 11:14:08 +01:00
Neil Alexander 8c3f51d624
Update are-we-synapse-yet (#1092) 2020-06-04 11:13:40 +01:00
Neil Alexander 225b72bd42
Don't reset counters before successful outgoing federation request (#1089)
* Don't reset counters before successful outgoing federation request on incoming federation request

* Comments
2020-06-04 10:54:10 +01:00
Neil Alexander 8a6152ca70
Enable room version 6 (#1087)
* Return bad request on CS API /send if bad JSON

* Return some more M_BAD_JSON in the right places

* nolint because damnit gocyclo all I added was a type check for an error

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update sytest-whitelist

* Update gomatrixserverlib

* Update sytest-whitelist

* NotJSON -> BadJSON
2020-06-04 10:53:39 +01:00
Kegsay e21d7d4baf
Update DendriteJS.Dockerfile 2020-06-03 16:03:54 +01:00
Aiden McClelland 17c92ad10e
Adds support for adding a proxy to the HTTP Client from the config (#1055)
* adds support for defining an proxy for the http client within the config

* alphabetize imports

* goimports

* comments
2020-06-02 21:02:24 +01:00
Kegsay dc3338d1f2
Remove ENV to avoid confusion 2020-06-02 16:35:07 +01:00
Neil Alexander e598e80d76
Delegate responsibility for marking room versions as supported/stable to gomatrixserverlib (#1082) 2020-06-02 16:20:50 +01:00
Kegsay e37720be44
Sort public rooms again by member count (#1083) 2020-06-02 16:18:01 +01:00
Kegsay 02b150fd13
Only store our own aliases in publicroomsapi (#1081)
Otherwise we just store the latest aliases submitted from a server,
which is not what we want.
2020-06-02 15:01:13 +01:00
Neil Alexander 794c63e757
Reset backoff on incoming federation (#1080)
* Reset backoffs in response to incoming federation requests

* Federation wakeups no more than once per minute per origin
2020-06-02 12:42:36 +01:00
Neil Alexander 484b6f694c
Use gomatrixserverlib device structs (#1079) 2020-06-02 11:29:47 +01:00
Kegan Dougal 353a5d6fc2 Remove p2p log lines 2020-06-02 11:03:01 +01:00
Neil Alexander a07fc0ef35 Return correct error to LookupRoomAlias 2020-06-02 09:34:36 +01:00
Kegan Dougal 42e797de5e Unbreak dendritejs 2020-06-01 18:41:58 +01:00
S7evinK 895c8f03c0
Fix pg user already exists (#1076)
* Return newly created error if user already exists (#1002)

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Rename variable

* Remove check for account and use returned error

* Return ErrUserExists

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* State that CreateAccount will return err ErrUserExists if the user exists

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Also check sqlite for constraint error

* Revert "Also check sqlite for constraint error"

This reverts commit 7d310514

* Check for sqlite3 constraint error

* Add documentation to CreateAccount

* Move ErrUserExists to accounts package

* Revert "Move ErrUserExists to accounts package"
Import Cycle..

This reverts commit be3d4cda

Co-authored-by: Kegsay <kegan@matrix.org>
2020-06-01 18:34:29 +01:00
Kegsay cfc137652e
Add a way to force federationsender to retry sending transactions (#1077)
* Add a way to force federationsender to retry sending transactions

And use it in P2P mode when we pick up new nodes.

* Linting

* Use atomic bool to stop us blocking on the channel
2020-06-01 18:34:08 +01:00
Neil Alexander a5d822004d
Send-to-device support (#1072)
* Groundwork for send-to-device messaging

* Update sample config

* Add unstable routing for now

* Send to device consumer in sync API

* Start the send-to-device consumer

* fix indentation in dendrite-config.yaml

* Create send-to-device database tables, other tweaks

* Add some logic for send-to-device messages, add them into sync stream

* Handle incoming send-to-device messages, count them with EDU stream pos

* Undo changes to test

* pq.Array

* Fix sync

* Logging

* Fix a couple of transaction things, fix client API

* Add send-to-device test, hopefully fix bugs

* Comments

* Refactor a bit

* Fix schema

* Fix queries

* Debug logging

* Fix storing and retrieving of send-to-device messages

* Try to avoid database locks

* Update sync position

* Use latest sync position

* Jiggle about sync a bit

* Fix tests

* Break out the retrieval from the update/delete behaviour

* Comments

* nolint on getResponseWithPDUsForCompleteSync

* Try to line up sync tokens again

* Implement wildcard

* Add all send-to-device tests to whitelist, what could possibly go wrong?

* Only care about wildcard when targeted locally

* Deduplicate transactions

* Handle tokens properly, return immediately if waiting send-to-device messages

* Fix sync

* Update sytest-whitelist

* Fix copyright notice (need to do more of this)

* Comments, copyrights

* Return errors from Do, fix dendritejs

* Review comments

* Comments

* Constructor for TransactionWriter

* defletions

* Update gomatrixserverlib, sytest-blacklist
2020-06-01 17:50:19 +01:00
Kegan Dougal 1f43c24f86 Add Dockerfile for building p2p riot
$ docker build -t dendritejs -f DendriteJS.Dockerfile .
$ docker run --rm -p 8090:80 dendritejs
2020-05-29 21:01:58 +01:00
Kegsay 97c64bdb6d
p2p: de-dupe and shuffle public rooms; implement keydb (#1074)
- We don't want a serverKeyAPI as fetching keys doesn't need a DB.
- De-dupe rooms so we don't see them multiple times, but shuffle the
  alias we join via so we don't all flood a single server.
2020-05-29 15:34:15 +01:00
Kegsay fe5cf6f880
fedsender: de-duplicate without sorting server names (#1073) 2020-05-29 13:50:06 +01:00
Neil Alexander 5307c499fe
Update gomatrixserverlib (#1071) 2020-05-28 13:44:16 +01:00