Commit graph

110 commits

Author SHA1 Message Date
Neil Alexander 9f834726cd Constructor for TransactionWriter 2020-06-01 17:02:48 +01:00
Neil Alexander 44b25a7f9e Comments 2020-06-01 16:58:37 +01:00
Neil Alexander fa74ded4d6 Review comments 2020-06-01 16:51:30 +01:00
Neil Alexander 6158a8dcc5 Return errors from Do, fix dendritejs 2020-06-01 14:46:32 +01:00
Neil Alexander e323bba3fd Comments, copyrights 2020-06-01 14:14:48 +01:00
Neil Alexander e1095211a0 Fix copyright notice (need to do more of this) 2020-06-01 12:33:07 +01:00
Neil Alexander 642283936c Fix sync 2020-06-01 11:33:22 +01:00
Neil Alexander 2e40e92ed1 Handle tokens properly, return immediately if waiting send-to-device messages 2020-06-01 11:19:43 +01:00
Neil Alexander dddc9efe3e Deduplicate transactions 2020-06-01 09:25:57 +01:00
Neil Alexander 5cf900428c Only care about wildcard when targeted locally 2020-05-29 17:37:13 +01:00
Neil Alexander 4664f02b4d Implement wildcard 2020-05-29 16:59:32 +01:00
Neil Alexander 43a83d375e Try to line up sync tokens again 2020-05-29 16:22:52 +01:00
Neil Alexander 2ef53ff726 nolint on getResponseWithPDUsForCompleteSync 2020-05-29 15:51:36 +01:00
Neil Alexander 0177ea04c4 Comments 2020-05-29 15:50:34 +01:00
Neil Alexander ded130f548 Break out the retrieval from the update/delete behaviour 2020-05-29 15:41:04 +01:00
Neil Alexander 3ca5d8e21b Fix tests 2020-05-29 13:50:56 +01:00
Neil Alexander fc1303bbbe Jiggle about sync a bit 2020-05-29 13:43:37 +01:00
Neil Alexander cc456240fa Update sync position 2020-05-29 11:48:42 +01:00
Neil Alexander 5487edf1d6 Try to avoid database locks 2020-05-29 11:38:19 +01:00
Neil Alexander 005c4c0168 Fix storing and retrieving of send-to-device messages 2020-05-29 10:07:57 +01:00
Neil Alexander 4fda6c65e6 Debug logging 2020-05-29 09:39:17 +01:00
Neil Alexander 49498fb794 Fix queries 2020-05-28 18:40:55 +01:00
Neil Alexander 544f3b0a61 Fix schema 2020-05-28 18:40:26 +01:00
Neil Alexander e4eba873f5 Refactor a bit 2020-05-28 18:38:23 +01:00
Neil Alexander f4376c4582 Comments 2020-05-28 17:53:40 +01:00
Neil Alexander 5389a62952 Add send-to-device test, hopefully fix bugs 2020-05-28 17:49:56 +01:00
Neil Alexander 1e3ad88172 Fix a couple of transaction things, fix client API 2020-05-28 17:25:28 +01:00
Neil Alexander bf5df4f438 Logging 2020-05-28 16:45:54 +01:00
Neil Alexander 587108fe31 Fix sync 2020-05-28 15:52:41 +01:00
Neil Alexander 581e5929e1 pq.Array 2020-05-28 15:43:01 +01:00
Neil Alexander e3267cf12a Undo changes to test 2020-05-28 15:39:29 +01:00
Neil Alexander 4d6347b21a Handle incoming send-to-device messages, count them with EDU stream pos 2020-05-28 15:35:49 +01:00
Neil Alexander d3bf9cb31b Add some logic for send-to-device messages, add them into sync stream 2020-05-28 15:11:44 +01:00
Neil Alexander 8b7e81b423 Create send-to-device database tables, other tweaks 2020-05-28 14:34:17 +01:00
Neil Alexander 23a16d1a57 Merge branch 'master' into neilalexander/sendtodevice 2020-05-28 10:13:08 +01:00
Neil Alexander 02fe38e1f7
Per-user-per-device sync streams (#1068)
* Per-user-per-device sync streams

* Tweaks

* Tweaks

* Pass full device into CompleteSync

* Set user IDs and device IDs properly in tests

* Add new test, fix TestNewEventAndWasPreviouslyJoinedToRoom

* nolint a function that is not used yet

* Add test for waking up single device

* Hopefully unstick test

* Try to ensure that TestCorrectStreamWakeup doesn't block forever

* Update tests
2020-05-28 10:05:04 +01:00
Neil Alexander ba6e55ecf5 Start the send-to-device consumer 2020-05-27 15:04:06 +01:00
Neil Alexander 062325b6c6 Send to device consumer in sync API 2020-05-27 14:37:13 +01:00
Neil Alexander 06d5f1e6dc Fix API paths 2020-05-22 14:14:39 +01:00
Neil Alexander fe82e1f725
Separate muxes for public and internal APIs (#1056)
* Separate muxes for public and internal APIs

* Update client-api-proxy and federation-api-proxy so they don't add /api to the path

* Tidy up

* Consistent HTTP setup

* Set up prefixes properly
2020-05-22 11:43:17 +01:00
Kegsay 24d8df664c
Fix #897 and shuffle directory around (#1054)
* Fix #897 and shuffle directory around

* Update find-lint

* goimports

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-05-21 14:40:13 +01:00
Kegsay 6091bf044f
sytest: Add remaining backfill tests (#1052)
One failed because of `null` instead of `[]` in HTTP responses.

One failed because we hadn't implemented in-line filter limits!
2020-05-20 17:30:03 +01:00
Kegsay 1414922026
sytest: Make 'Inbound federation can backfill events' pass (#1051)
* sytest: Make 'Inbound federation can backfill events' pass

This breaks 'Outbound federation can backfill events' because now
we are returning the right number of events, which the previous
test was relying on.

Previously, /messages was backfilling the membership event, causing
the test to pass. Now we are no longer backfilling the membership
event due to the change in this commit, causing the test to fail.

The test should instead be returning the membership event locally
from synacpis database, but it doesn't do it fast enough, resulting
in a no-op /sync response with a next_batch=s0_0 which will never
pick up the local membership event when it rolls in. The test
does attempt to retry, but doesn't take the new next_batch=s1_0
resulting in it missing from the /messages response.

* Linting
2020-05-20 16:04:31 +01:00
Kegsay 260e69d138
Make "Outbound federation can backfill events" pass sytest (#1049)
- Use a backfill limit of 100 regardless of what was asked.
- Special case the create event for `StateIDsBeforeEvent`
- Trim to the limit in `syncapi`
2020-05-19 18:42:55 +01:00
Kegsay 8db60c90bb
Fix a bug whereby backfilling could leak events across rooms (#1043)
* Fix a bug whereby backfilling could leak events across rooms

Caused by a faulty SQL query. With tests now.

* comment
2020-05-15 16:27:34 +01:00
Kegsay 2b5052eccf
Add Range (#1037)
* Add Range

* Use Range
2020-05-15 09:41:12 +01:00
Kegsay 7ca230e931
Cleanup syncapi topology logic (#1035)
* Cleanup syncapi topology logic

* Variable renaming

* comments
2020-05-14 17:30:16 +01:00
Kegsay 1b34130a5b
Finish merging syncserver.go (#1033)
* Refactor all postgres tables; start work on sqlite

* wip sqlite merges; database is locked errors to investigate and failing tests

* Revert "wip sqlite merges; database is locked errors to investigate and failing tests"

This reverts commit 26cbfc5b75.

* convert current room state table

* port over sqlite topology table

* remove a few functions

* remove more functions

* Share more code

* factor out completesync and a bit more

* Remove remaining code
2020-05-14 16:11:37 +01:00
Kegsay 9ed68a3125
Factor out account data and events table (#1031)
* Factor out account data

* Factor out events table and EDU cache

* linting

* fix npe
2020-05-14 09:53:55 +01:00
Kegsay a25d477cdb
Initial syncapi storage refactor to share pq/sqlite code (#1030)
* Initial syncapi storage refactor to share pq/sqlite code

This goes down a different route than https://github.com/matrix-org/dendrite/pull/985
which tried to even reduce the boilerplate of `ExecContext` etc. The previous pattern
fails badly when there are subtle differences in parameters and hence the shared
boilerplate to read from `QueryContext` breaks. Rather than attacking it at that level,
the main place where we want to reuse code is for the `syncserver.go` itself - the
database implementation which has lots of complex logic. So instead, this commit:
 - Makes `invites_table.go` an interface.
 - Makes `SyncServerDatasource` use that interface
 - This means some functions are now identical for pq/sqlite, so factor them out
   to a temporary `shared.Database` struct which will grow until it replaces all of
   `SyncServerDatasource`.

* Missing files
2020-05-13 17:28:42 +01:00