Commit graph

835 commits

Author SHA1 Message Date
Neil Alexander bf5df4f438 Logging 2020-05-28 16:45:54 +01:00
Neil Alexander 587108fe31 Fix sync 2020-05-28 15:52:41 +01:00
Neil Alexander 581e5929e1 pq.Array 2020-05-28 15:43:01 +01:00
Neil Alexander e3267cf12a Undo changes to test 2020-05-28 15:39:29 +01:00
Neil Alexander 4d6347b21a Handle incoming send-to-device messages, count them with EDU stream pos 2020-05-28 15:35:49 +01:00
Neil Alexander d3bf9cb31b Add some logic for send-to-device messages, add them into sync stream 2020-05-28 15:11:44 +01:00
Neil Alexander 8b7e81b423 Create send-to-device database tables, other tweaks 2020-05-28 14:34:17 +01:00
Neil Alexander 9694907ccc fix indentation in dendrite-config.yaml 2020-05-28 13:46:21 +01:00
Neil Alexander d4d4915194 Merge branch 'master' into neilalexander/sendtodevice 2020-05-28 13:45:34 +01:00
Neil Alexander 5307c499fe
Update gomatrixserverlib (#1071) 2020-05-28 13:44:16 +01:00
Neil Alexander 1b7ad3e4e0 Merge branch 'master' into neilalexander/sendtodevice 2020-05-28 12:48:02 +01:00
Neil Alexander f123d7d65b
Hopefully fix get_missing_events (#1070) 2020-05-28 12:44:34 +01:00
Kegan Dougal 2411007c4b Parse URIs correctly 2020-05-28 12:28:21 +01:00
Kegsay a6f995eb45
Merge Updater structs (#1069)
* Move Updater structs to shared and use it for postgres

* Add constructors for NewXXXUpdater and a useTxns flag

In sqlite, we set useTxns=false and comment why.

* Handle nil txn

* Handle nil in transaction

* Missed one

* Close the txn at the right time

* Don't close the transaction as we reuse it between calls
2020-05-28 11:15:21 +01:00
Neil Alexander 23a16d1a57 Merge branch 'master' into neilalexander/sendtodevice 2020-05-28 10:13:08 +01:00
Neil Alexander 02fe38e1f7
Per-user-per-device sync streams (#1068)
* Per-user-per-device sync streams

* Tweaks

* Tweaks

* Pass full device into CompleteSync

* Set user IDs and device IDs properly in tests

* Add new test, fix TestNewEventAndWasPreviouslyJoinedToRoom

* nolint a function that is not used yet

* Add test for waking up single device

* Hopefully unstick test

* Try to ensure that TestCorrectStreamWakeup doesn't block forever

* Update tests
2020-05-28 10:05:04 +01:00
Neil Alexander ba6e55ecf5 Start the send-to-device consumer 2020-05-27 15:04:06 +01:00
Neil Alexander 062325b6c6 Send to device consumer in sync API 2020-05-27 14:37:13 +01:00
Neil Alexander f91f91dcb1 Add unstable routing for now 2020-05-27 14:17:43 +01:00
Neil Alexander 74fb1daa52 Update sample config 2020-05-27 13:42:43 +01:00
Neil Alexander a90ea4c320 Groundwork for send-to-device messaging 2020-05-27 13:33:56 +01:00
Neil Alexander 57841fc35e
Read batches from incoming channels (#1067) 2020-05-27 12:16:53 +01:00
Neil Alexander 406b47267e
Return 500 when processing a transaction fails fatally (#1066) 2020-05-27 11:16:27 +01:00
Kegsay a7f12bce79
Convert remaining roomserver tables (#1065)
* Convert invites table

* Convert membership table

* Factor out remaining functions except for *Updater structs

* Remove 'implements' comments from long-forgotten interfaces. Move those comments to storage.Database
2020-05-27 11:03:47 +01:00
Neil Alexander 7d6461dd3c
Server key component (#1050)
* Server key API (works for monolith but not for polylith yet)

* Re-enable caching on server key API component

* Groundwork for HTTP APIs for server key API

* Hopefully implement HTTP for server key API

* Simplify public key request marshalling from map keys

* Update gomatrixserverlib

* go mod tidy

* Common -> internal

* remove keyring.go

* Update Docker Hub for server key API

* YAML is funny about indentation

* Wire in new server key API into hybrid monolith mode

* Create maps

* Route server key API endpoints on internal API mux

* Fix server key API URLs

* Add fetcher behaviour into server key API implementation

* Return error if we failed to fetch some keys

* Return results anyway

* Move things about a bit

* Remove unused code

* Fix comments, don't use federation sender URL in polylith mode

* Add server_key_api to sample config

* Review comments

* HTTP API to cache keys that have been requested

* Overwrite server_key_api listen in monolith hybrid mode
2020-05-27 10:19:24 +01:00
Kegan Dougal 267a4d1823 Update p2p docs 2020-05-27 09:39:26 +01:00
Kegsay c0c5d9452a Convert room_aliases previous_events state_block and state_snapshot tables (#1064)
* Convert state_snapshot and state_block tables

* Convert room_aliases and previous_events tables

* Add missing table
2020-05-27 09:36:09 +01:00
Kegsay 19aa44ecae
Convert transactions/rooms table to share more code (#1063)
* Convert rooms table

* Convert transactions table

* Convert rooms table and factor out lots of functions

* I think you'll be needing this..
2020-05-26 18:23:39 +01:00
Kegsay 803af87dc4
Convert events/event_json tables to share code (#1062)
* Convert event_json table

* Convert the events table
2020-05-26 16:45:28 +01:00
Kegsay 737c83e0ae
roomserver: start refactoring storage layer (#1061)
* Begin refactoring out roomserver/storage/*/storage.go

* Convert event_types table
2020-05-26 15:42:42 +01:00
Neil Alexander 6d50212f29
Miscellaneous fixes (#1060)
* Add missing routing for PerformDirectoryLookupRequest

* Tweak output

* Fix some bugs in devices

* Don't default to federated room joins in response to invite

* Update sytest-whitelist

* Update comments

* Return correct room ID from PerformJoin

* Fix appservice and EDU server API setup, update sytest-whitelist

* Update sytest-whitelist
2020-05-26 14:41:16 +01:00
Kegan Dougal 492af0f2ec Use Opaque in addition to Path to set naffka DB names 2020-05-25 15:29:49 +01:00
Neil Alexander 3c3e014901 Define path prefixes in a package that doesn't create import cycles 2020-05-22 14:18:41 +01:00
Neil Alexander 06d5f1e6dc Fix API paths 2020-05-22 14:14:39 +01:00
Neil Alexander 0978630b55 Fix monolith room server-federation sender connection 2020-05-22 14:11:06 +01:00
Neil Alexander 3d06fe91f2 Fix internal HTTP API calls 2020-05-22 13:54:04 +01:00
Kegsay 3daa2327ed
dendritejs tweaks for persisting sqlite DBs (#1058)
* Use uri.path so we don't have file: in the filename

* New go-sqlite-js version
2020-05-22 12:28:48 +01:00
Neil Alexander fbdcfdd256
Use HTTP APIs when -api specified (#1057) 2020-05-22 12:28:36 +01:00
Neil Alexander fe82e1f725
Separate muxes for public and internal APIs (#1056)
* Separate muxes for public and internal APIs

* Update client-api-proxy and federation-api-proxy so they don't add /api to the path

* Tidy up

* Consistent HTTP setup

* Set up prefixes properly
2020-05-22 11:43:17 +01:00
Neil Alexander f223da2f35 Fix URL in compose file 2020-05-21 16:22:25 +01:00
Kegsay 24d8df664c
Fix #897 and shuffle directory around (#1054)
* Fix #897 and shuffle directory around

* Update find-lint

* goimports

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-05-21 14:40:13 +01:00
Neil Alexander 3fdb045116
Docker Hub (#1053)
* Initial Docker Hub test

* Change upstream from for now

* Working polylith setup and build tools

* Some readme and other bits

* Remove the readme for now

* Add readme, rename a couple of things

* COPY instead of ADD

* Simplify component builds

* Fix the docker-compose things

* Build the base image again

* Update readme, add .dockerignore

* Indentation

* Whitespace

* Update images-push.sh

* Update images-push.sh
2020-05-21 13:02:28 +01:00
Kegan Dougal 91902df95d Add tests due to https://github.com/matrix-org/sytest/pull/876 2020-05-21 11:51:15 +01:00
Kegan Dougal 7d91ef0616 This now passes on sytest/develop 2020-05-20 18:31:02 +01:00
Neil Alexander f2c07437fe
Use memberships to determine whether to reset latest events/state on room join (#1047)
* Track local/remote memberships, re-scope some input stuff

* Check if we're in the room already before resetting latest events/state

* Fix postgres, fix lint

* Review comments
2020-05-20 18:03:06 +01:00
Kegsay 6091bf044f
sytest: Add remaining backfill tests (#1052)
One failed because of `null` instead of `[]` in HTTP responses.

One failed because we hadn't implemented in-line filter limits!
2020-05-20 17:30:03 +01:00
Kegsay 1414922026
sytest: Make 'Inbound federation can backfill events' pass (#1051)
* sytest: Make 'Inbound federation can backfill events' pass

This breaks 'Outbound federation can backfill events' because now
we are returning the right number of events, which the previous
test was relying on.

Previously, /messages was backfilling the membership event, causing
the test to pass. Now we are no longer backfilling the membership
event due to the change in this commit, causing the test to fail.

The test should instead be returning the membership event locally
from synacpis database, but it doesn't do it fast enough, resulting
in a no-op /sync response with a next_batch=s0_0 which will never
pick up the local membership event when it rolls in. The test
does attempt to retry, but doesn't take the new next_batch=s1_0
resulting in it missing from the /messages response.

* Linting
2020-05-20 16:04:31 +01:00
Kegsay 260e69d138
Make "Outbound federation can backfill events" pass sytest (#1049)
- Use a backfill limit of 100 regardless of what was asked.
- Special case the create event for `StateIDsBeforeEvent`
- Trim to the limit in `syncapi`
2020-05-19 18:42:55 +01:00
Kegsay 5faecdac82
Bake in git commit into dendritejs binary (#1048) 2020-05-19 16:42:30 +01:00
Kegsay 8b3100935c
dendritejs: Persist ed25519 keys in localforage (#1046) 2020-05-19 12:09:52 +01:00