Commit graph

2080 commits

Author SHA1 Message Date
Neil Alexander 794c63e757
Reset backoff on incoming federation (#1080)
* Reset backoffs in response to incoming federation requests

* Federation wakeups no more than once per minute per origin
2020-06-02 12:42:36 +01:00
Neil Alexander 484b6f694c
Use gomatrixserverlib device structs (#1079) 2020-06-02 11:29:47 +01:00
Kegan Dougal 353a5d6fc2 Remove p2p log lines 2020-06-02 11:03:01 +01:00
Neil Alexander a07fc0ef35 Return correct error to LookupRoomAlias 2020-06-02 09:34:36 +01:00
Kegan Dougal 42e797de5e Unbreak dendritejs 2020-06-01 18:41:58 +01:00
S7evinK 895c8f03c0
Fix pg user already exists (#1076)
* Return newly created error if user already exists (#1002)

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Rename variable

* Remove check for account and use returned error

* Return ErrUserExists

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* State that CreateAccount will return err ErrUserExists if the user exists

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Also check sqlite for constraint error

* Revert "Also check sqlite for constraint error"

This reverts commit 7d310514

* Check for sqlite3 constraint error

* Add documentation to CreateAccount

* Move ErrUserExists to accounts package

* Revert "Move ErrUserExists to accounts package"
Import Cycle..

This reverts commit be3d4cda

Co-authored-by: Kegsay <kegan@matrix.org>
2020-06-01 18:34:29 +01:00
Kegsay cfc137652e
Add a way to force federationsender to retry sending transactions (#1077)
* Add a way to force federationsender to retry sending transactions

And use it in P2P mode when we pick up new nodes.

* Linting

* Use atomic bool to stop us blocking on the channel
2020-06-01 18:34:08 +01:00
Neil Alexander a5d822004d
Send-to-device support (#1072)
* Groundwork for send-to-device messaging

* Update sample config

* Add unstable routing for now

* Send to device consumer in sync API

* Start the send-to-device consumer

* fix indentation in dendrite-config.yaml

* Create send-to-device database tables, other tweaks

* Add some logic for send-to-device messages, add them into sync stream

* Handle incoming send-to-device messages, count them with EDU stream pos

* Undo changes to test

* pq.Array

* Fix sync

* Logging

* Fix a couple of transaction things, fix client API

* Add send-to-device test, hopefully fix bugs

* Comments

* Refactor a bit

* Fix schema

* Fix queries

* Debug logging

* Fix storing and retrieving of send-to-device messages

* Try to avoid database locks

* Update sync position

* Use latest sync position

* Jiggle about sync a bit

* Fix tests

* Break out the retrieval from the update/delete behaviour

* Comments

* nolint on getResponseWithPDUsForCompleteSync

* Try to line up sync tokens again

* Implement wildcard

* Add all send-to-device tests to whitelist, what could possibly go wrong?

* Only care about wildcard when targeted locally

* Deduplicate transactions

* Handle tokens properly, return immediately if waiting send-to-device messages

* Fix sync

* Update sytest-whitelist

* Fix copyright notice (need to do more of this)

* Comments, copyrights

* Return errors from Do, fix dendritejs

* Review comments

* Comments

* Constructor for TransactionWriter

* defletions

* Update gomatrixserverlib, sytest-blacklist
2020-06-01 17:50:19 +01:00
Kegan Dougal 1f43c24f86 Add Dockerfile for building p2p riot
$ docker build -t dendritejs -f DendriteJS.Dockerfile .
$ docker run --rm -p 8090:80 dendritejs
2020-05-29 21:01:58 +01:00
Kegsay 97c64bdb6d
p2p: de-dupe and shuffle public rooms; implement keydb (#1074)
- We don't want a serverKeyAPI as fetching keys doesn't need a DB.
- De-dupe rooms so we don't see them multiple times, but shuffle the
  alias we join via so we don't all flood a single server.
2020-05-29 15:34:15 +01:00
Kegsay fe5cf6f880
fedsender: de-duplicate without sorting server names (#1073) 2020-05-29 13:50:06 +01:00
Neil Alexander 5307c499fe
Update gomatrixserverlib (#1071) 2020-05-28 13:44:16 +01:00
Neil Alexander f123d7d65b
Hopefully fix get_missing_events (#1070) 2020-05-28 12:44:34 +01:00
Kegan Dougal 2411007c4b Parse URIs correctly 2020-05-28 12:28:21 +01:00
Kegsay a6f995eb45
Merge Updater structs (#1069)
* Move Updater structs to shared and use it for postgres

* Add constructors for NewXXXUpdater and a useTxns flag

In sqlite, we set useTxns=false and comment why.

* Handle nil txn

* Handle nil in transaction

* Missed one

* Close the txn at the right time

* Don't close the transaction as we reuse it between calls
2020-05-28 11:15:21 +01:00
Neil Alexander 02fe38e1f7
Per-user-per-device sync streams (#1068)
* Per-user-per-device sync streams

* Tweaks

* Tweaks

* Pass full device into CompleteSync

* Set user IDs and device IDs properly in tests

* Add new test, fix TestNewEventAndWasPreviouslyJoinedToRoom

* nolint a function that is not used yet

* Add test for waking up single device

* Hopefully unstick test

* Try to ensure that TestCorrectStreamWakeup doesn't block forever

* Update tests
2020-05-28 10:05:04 +01:00
Neil Alexander 57841fc35e
Read batches from incoming channels (#1067) 2020-05-27 12:16:53 +01:00
Neil Alexander 406b47267e
Return 500 when processing a transaction fails fatally (#1066) 2020-05-27 11:16:27 +01:00
Kegsay a7f12bce79
Convert remaining roomserver tables (#1065)
* Convert invites table

* Convert membership table

* Factor out remaining functions except for *Updater structs

* Remove 'implements' comments from long-forgotten interfaces. Move those comments to storage.Database
2020-05-27 11:03:47 +01:00
Neil Alexander 7d6461dd3c
Server key component (#1050)
* Server key API (works for monolith but not for polylith yet)

* Re-enable caching on server key API component

* Groundwork for HTTP APIs for server key API

* Hopefully implement HTTP for server key API

* Simplify public key request marshalling from map keys

* Update gomatrixserverlib

* go mod tidy

* Common -> internal

* remove keyring.go

* Update Docker Hub for server key API

* YAML is funny about indentation

* Wire in new server key API into hybrid monolith mode

* Create maps

* Route server key API endpoints on internal API mux

* Fix server key API URLs

* Add fetcher behaviour into server key API implementation

* Return error if we failed to fetch some keys

* Return results anyway

* Move things about a bit

* Remove unused code

* Fix comments, don't use federation sender URL in polylith mode

* Add server_key_api to sample config

* Review comments

* HTTP API to cache keys that have been requested

* Overwrite server_key_api listen in monolith hybrid mode
2020-05-27 10:19:24 +01:00
Kegan Dougal 267a4d1823 Update p2p docs 2020-05-27 09:39:26 +01:00
Kegsay c0c5d9452a Convert room_aliases previous_events state_block and state_snapshot tables (#1064)
* Convert state_snapshot and state_block tables

* Convert room_aliases and previous_events tables

* Add missing table
2020-05-27 09:36:09 +01:00
Kegsay 19aa44ecae
Convert transactions/rooms table to share more code (#1063)
* Convert rooms table

* Convert transactions table

* Convert rooms table and factor out lots of functions

* I think you'll be needing this..
2020-05-26 18:23:39 +01:00
Kegsay 803af87dc4
Convert events/event_json tables to share code (#1062)
* Convert event_json table

* Convert the events table
2020-05-26 16:45:28 +01:00
Kegsay 737c83e0ae
roomserver: start refactoring storage layer (#1061)
* Begin refactoring out roomserver/storage/*/storage.go

* Convert event_types table
2020-05-26 15:42:42 +01:00
Neil Alexander 6d50212f29
Miscellaneous fixes (#1060)
* Add missing routing for PerformDirectoryLookupRequest

* Tweak output

* Fix some bugs in devices

* Don't default to federated room joins in response to invite

* Update sytest-whitelist

* Update comments

* Return correct room ID from PerformJoin

* Fix appservice and EDU server API setup, update sytest-whitelist

* Update sytest-whitelist
2020-05-26 14:41:16 +01:00
Kegan Dougal 492af0f2ec Use Opaque in addition to Path to set naffka DB names 2020-05-25 15:29:49 +01:00
Neil Alexander 3c3e014901 Define path prefixes in a package that doesn't create import cycles 2020-05-22 14:18:41 +01:00
Neil Alexander 06d5f1e6dc Fix API paths 2020-05-22 14:14:39 +01:00
Neil Alexander 0978630b55 Fix monolith room server-federation sender connection 2020-05-22 14:11:06 +01:00
Neil Alexander 3d06fe91f2 Fix internal HTTP API calls 2020-05-22 13:54:04 +01:00
Kegsay 3daa2327ed
dendritejs tweaks for persisting sqlite DBs (#1058)
* Use uri.path so we don't have file: in the filename

* New go-sqlite-js version
2020-05-22 12:28:48 +01:00
Neil Alexander fbdcfdd256
Use HTTP APIs when -api specified (#1057) 2020-05-22 12:28:36 +01:00
Neil Alexander fe82e1f725
Separate muxes for public and internal APIs (#1056)
* Separate muxes for public and internal APIs

* Update client-api-proxy and federation-api-proxy so they don't add /api to the path

* Tidy up

* Consistent HTTP setup

* Set up prefixes properly
2020-05-22 11:43:17 +01:00
Neil Alexander f223da2f35 Fix URL in compose file 2020-05-21 16:22:25 +01:00
Kegsay 24d8df664c
Fix #897 and shuffle directory around (#1054)
* Fix #897 and shuffle directory around

* Update find-lint

* goimports

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-05-21 14:40:13 +01:00
Neil Alexander 3fdb045116
Docker Hub (#1053)
* Initial Docker Hub test

* Change upstream from for now

* Working polylith setup and build tools

* Some readme and other bits

* Remove the readme for now

* Add readme, rename a couple of things

* COPY instead of ADD

* Simplify component builds

* Fix the docker-compose things

* Build the base image again

* Update readme, add .dockerignore

* Indentation

* Whitespace

* Update images-push.sh

* Update images-push.sh
2020-05-21 13:02:28 +01:00
Kegan Dougal 91902df95d Add tests due to https://github.com/matrix-org/sytest/pull/876 2020-05-21 11:51:15 +01:00
Kegan Dougal 7d91ef0616 This now passes on sytest/develop 2020-05-20 18:31:02 +01:00
Neil Alexander f2c07437fe
Use memberships to determine whether to reset latest events/state on room join (#1047)
* Track local/remote memberships, re-scope some input stuff

* Check if we're in the room already before resetting latest events/state

* Fix postgres, fix lint

* Review comments
2020-05-20 18:03:06 +01:00
Kegsay 6091bf044f
sytest: Add remaining backfill tests (#1052)
One failed because of `null` instead of `[]` in HTTP responses.

One failed because we hadn't implemented in-line filter limits!
2020-05-20 17:30:03 +01:00
Kegsay 1414922026
sytest: Make 'Inbound federation can backfill events' pass (#1051)
* sytest: Make 'Inbound federation can backfill events' pass

This breaks 'Outbound federation can backfill events' because now
we are returning the right number of events, which the previous
test was relying on.

Previously, /messages was backfilling the membership event, causing
the test to pass. Now we are no longer backfilling the membership
event due to the change in this commit, causing the test to fail.

The test should instead be returning the membership event locally
from synacpis database, but it doesn't do it fast enough, resulting
in a no-op /sync response with a next_batch=s0_0 which will never
pick up the local membership event when it rolls in. The test
does attempt to retry, but doesn't take the new next_batch=s1_0
resulting in it missing from the /messages response.

* Linting
2020-05-20 16:04:31 +01:00
Kegsay 260e69d138
Make "Outbound federation can backfill events" pass sytest (#1049)
- Use a backfill limit of 100 regardless of what was asked.
- Special case the create event for `StateIDsBeforeEvent`
- Trim to the limit in `syncapi`
2020-05-19 18:42:55 +01:00
Kegsay 5faecdac82
Bake in git commit into dendritejs binary (#1048) 2020-05-19 16:42:30 +01:00
Kegsay 8b3100935c
dendritejs: Persist ed25519 keys in localforage (#1046) 2020-05-19 12:09:52 +01:00
Kegan Dougal b730e21892 bump go-http-js-libp2p 2020-05-18 18:12:11 +01:00
Neil Alexander 9ef30bb13b
Tweaks to latest events updater (#1045)
* Comment out updaters a bit, add overwrite flag to latest events

* Make sure we don't send fast-forwarded state changes over federation, start with empty set when overwriting

* Remove redundant check for overwrite
2020-05-18 17:49:24 +01:00
Kegan Dougal 5c221f0655 Latest go-http-js-libp2p 2020-05-18 15:51:49 +01:00
Neil Alexander dce4f436f7
Add -api flag to monolith (#1044)
* Add flag for enabling HTTP APIs in monolith mode

* Flag -api

* Only start HTTP APIs if needed
2020-05-18 10:56:43 +01:00
Kegan Dougal bfb954519b Only keep keys valid for 100 years rather than the heat death of the universe
WASM builds really dislike thinking that far into the future, causing
WASM to trap with "float unrepresentable in integer range", which corrupts
the Go stack in wasm_exec which then leads to a segfault and the program
exiting.
2020-05-15 17:39:20 +01:00