Commit graph

2656 commits

Author SHA1 Message Date
Kegsay 737c83e0ae
roomserver: start refactoring storage layer (#1061)
* Begin refactoring out roomserver/storage/*/storage.go

* Convert event_types table
2020-05-26 15:42:42 +01:00
Neil Alexander 6d50212f29
Miscellaneous fixes (#1060)
* Add missing routing for PerformDirectoryLookupRequest

* Tweak output

* Fix some bugs in devices

* Don't default to federated room joins in response to invite

* Update sytest-whitelist

* Update comments

* Return correct room ID from PerformJoin

* Fix appservice and EDU server API setup, update sytest-whitelist

* Update sytest-whitelist
2020-05-26 14:41:16 +01:00
Kegan Dougal 492af0f2ec Use Opaque in addition to Path to set naffka DB names 2020-05-25 15:29:49 +01:00
Neil Alexander 3c3e014901 Define path prefixes in a package that doesn't create import cycles 2020-05-22 14:18:41 +01:00
Neil Alexander 06d5f1e6dc Fix API paths 2020-05-22 14:14:39 +01:00
Neil Alexander 0978630b55 Fix monolith room server-federation sender connection 2020-05-22 14:11:06 +01:00
Neil Alexander 3d06fe91f2 Fix internal HTTP API calls 2020-05-22 13:54:04 +01:00
Kegsay 3daa2327ed
dendritejs tweaks for persisting sqlite DBs (#1058)
* Use uri.path so we don't have file: in the filename

* New go-sqlite-js version
2020-05-22 12:28:48 +01:00
Neil Alexander fbdcfdd256
Use HTTP APIs when -api specified (#1057) 2020-05-22 12:28:36 +01:00
Neil Alexander fe82e1f725
Separate muxes for public and internal APIs (#1056)
* Separate muxes for public and internal APIs

* Update client-api-proxy and federation-api-proxy so they don't add /api to the path

* Tidy up

* Consistent HTTP setup

* Set up prefixes properly
2020-05-22 11:43:17 +01:00
Neil Alexander f223da2f35 Fix URL in compose file 2020-05-21 16:22:25 +01:00
Kegsay 24d8df664c
Fix #897 and shuffle directory around (#1054)
* Fix #897 and shuffle directory around

* Update find-lint

* goimports

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-05-21 14:40:13 +01:00
Neil Alexander 3fdb045116
Docker Hub (#1053)
* Initial Docker Hub test

* Change upstream from for now

* Working polylith setup and build tools

* Some readme and other bits

* Remove the readme for now

* Add readme, rename a couple of things

* COPY instead of ADD

* Simplify component builds

* Fix the docker-compose things

* Build the base image again

* Update readme, add .dockerignore

* Indentation

* Whitespace

* Update images-push.sh

* Update images-push.sh
2020-05-21 13:02:28 +01:00
Kegan Dougal 91902df95d Add tests due to https://github.com/matrix-org/sytest/pull/876 2020-05-21 11:51:15 +01:00
Kegan Dougal 7d91ef0616 This now passes on sytest/develop 2020-05-20 18:31:02 +01:00
Neil Alexander f2c07437fe
Use memberships to determine whether to reset latest events/state on room join (#1047)
* Track local/remote memberships, re-scope some input stuff

* Check if we're in the room already before resetting latest events/state

* Fix postgres, fix lint

* Review comments
2020-05-20 18:03:06 +01:00
Kegsay 6091bf044f
sytest: Add remaining backfill tests (#1052)
One failed because of `null` instead of `[]` in HTTP responses.

One failed because we hadn't implemented in-line filter limits!
2020-05-20 17:30:03 +01:00
Kegsay 1414922026
sytest: Make 'Inbound federation can backfill events' pass (#1051)
* sytest: Make 'Inbound federation can backfill events' pass

This breaks 'Outbound federation can backfill events' because now
we are returning the right number of events, which the previous
test was relying on.

Previously, /messages was backfilling the membership event, causing
the test to pass. Now we are no longer backfilling the membership
event due to the change in this commit, causing the test to fail.

The test should instead be returning the membership event locally
from synacpis database, but it doesn't do it fast enough, resulting
in a no-op /sync response with a next_batch=s0_0 which will never
pick up the local membership event when it rolls in. The test
does attempt to retry, but doesn't take the new next_batch=s1_0
resulting in it missing from the /messages response.

* Linting
2020-05-20 16:04:31 +01:00
Kegsay 260e69d138
Make "Outbound federation can backfill events" pass sytest (#1049)
- Use a backfill limit of 100 regardless of what was asked.
- Special case the create event for `StateIDsBeforeEvent`
- Trim to the limit in `syncapi`
2020-05-19 18:42:55 +01:00
Kegsay 5faecdac82
Bake in git commit into dendritejs binary (#1048) 2020-05-19 16:42:30 +01:00
Kegsay 8b3100935c
dendritejs: Persist ed25519 keys in localforage (#1046) 2020-05-19 12:09:52 +01:00
Kegan Dougal b730e21892 bump go-http-js-libp2p 2020-05-18 18:12:11 +01:00
Neil Alexander 9ef30bb13b
Tweaks to latest events updater (#1045)
* Comment out updaters a bit, add overwrite flag to latest events

* Make sure we don't send fast-forwarded state changes over federation, start with empty set when overwriting

* Remove redundant check for overwrite
2020-05-18 17:49:24 +01:00
Kegan Dougal 5c221f0655 Latest go-http-js-libp2p 2020-05-18 15:51:49 +01:00
Neil Alexander dce4f436f7
Add -api flag to monolith (#1044)
* Add flag for enabling HTTP APIs in monolith mode

* Flag -api

* Only start HTTP APIs if needed
2020-05-18 10:56:43 +01:00
Kegan Dougal bfb954519b Only keep keys valid for 100 years rather than the heat death of the universe
WASM builds really dislike thinking that far into the future, causing
WASM to trap with "float unrepresentable in integer range", which corrupts
the Go stack in wasm_exec which then leads to a segfault and the program
exiting.
2020-05-15 17:39:20 +01:00
Kegsay 8db60c90bb
Fix a bug whereby backfilling could leak events across rooms (#1043)
* Fix a bug whereby backfilling could leak events across rooms

Caused by a faulty SQL query. With tests now.

* comment
2020-05-15 16:27:34 +01:00
Neil Alexander 5f6f8adaa5
Don't prematurely stop trying to join using servers (#1041)
* Don't prematurely stop trying to join using servers

* Factor out performJoinUsingServer
2020-05-15 13:55:14 +01:00
Kegsay f4f032381b
Set canonical alias on room creation (#1042) 2020-05-15 13:39:01 +01:00
Neil Alexander 773d5bb9f9
Return user_id and stream_id in federated devices query (#1040) 2020-05-15 12:07:51 +01:00
Neil Alexander f0e0a6668f
Prometheus metrics for LRU cache (#1039)
* Add prom metrics for the in-memory LRU cache

* Increase cache sizes
2020-05-15 11:27:10 +01:00
Kegsay 2b5052eccf
Add Range (#1037)
* Add Range

* Use Range
2020-05-15 09:41:12 +01:00
Neil Alexander 419ff150d4
Implement key caching directly (#1038)
* Use gomatrixserverlib key caching

* Implement key caching wrapper

* Add caching wrapper in BaseComponent

* Review comments
2020-05-15 09:32:40 +01:00
Kegsay 7ca230e931
Cleanup syncapi topology logic (#1035)
* Cleanup syncapi topology logic

* Variable renaming

* comments
2020-05-14 17:30:16 +01:00
Neil Alexander 3cb04e8004
Update INSTALL.md, move docs (#1034)
* Update INSTALL.md

* Move some things

* Clean up

* Move some more things

* Don't build all the things for the monolith

* Update INSTALL.md

* Nuke hooks
2020-05-14 16:49:18 +01:00
Kegsay 1b34130a5b
Finish merging syncserver.go (#1033)
* Refactor all postgres tables; start work on sqlite

* wip sqlite merges; database is locked errors to investigate and failing tests

* Revert "wip sqlite merges; database is locked errors to investigate and failing tests"

This reverts commit 26cbfc5b75.

* convert current room state table

* port over sqlite topology table

* remove a few functions

* remove more functions

* Share more code

* factor out completesync and a bit more

* Remove remaining code
2020-05-14 16:11:37 +01:00
Neil Alexander 640a0265df
Fix some cases where accepting invites over federation doesn't work (#1028)
* Handle cases where accepting invites doesn't work for historic rooms

* Rewrite pairUpChanges

* Review comments
2020-05-14 14:58:47 +01:00
Neil Alexander 8adc128225
Keyserver skeleton (#1032)
* Keyserver skeleton

* Indentation
2020-05-14 14:05:14 +01:00
Kegsay 9ed68a3125
Factor out account data and events table (#1031)
* Factor out account data

* Factor out events table and EDU cache

* linting

* fix npe
2020-05-14 09:53:55 +01:00
Kegsay a25d477cdb
Initial syncapi storage refactor to share pq/sqlite code (#1030)
* Initial syncapi storage refactor to share pq/sqlite code

This goes down a different route than https://github.com/matrix-org/dendrite/pull/985
which tried to even reduce the boilerplate of `ExecContext` etc. The previous pattern
fails badly when there are subtle differences in parameters and hence the shared
boilerplate to read from `QueryContext` breaks. Rather than attacking it at that level,
the main place where we want to reuse code is for the `syncserver.go` itself - the
database implementation which has lots of complex logic. So instead, this commit:
 - Makes `invites_table.go` an interface.
 - Makes `SyncServerDatasource` use that interface
 - This means some functions are now identical for pq/sqlite, so factor them out
   to a temporary `shared.Database` struct which will grow until it replaces all of
   `SyncServerDatasource`.

* Missing files
2020-05-13 17:28:42 +01:00
Kegan Dougal bdddd83753 Spurious logging 2020-05-13 15:00:47 +01:00
Neil Alexander 334680a216
Get profile content for join request again (#1027) 2020-05-13 14:53:25 +01:00
Kegsay 37b2299b71
Fix #1015 (#1026)
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-05-13 13:02:34 +01:00
Neil Alexander ee140c9d6a
Reduce 500s (#1017)
* Try to avoid returning 500s on /send

* Don't return 500s from media API download requests

* Don't 500 on context errors

* Update sytest-whitelist

* Fix lint, add comments
2020-05-13 13:01:45 +01:00
Kegsay 5e9dce1c0c
syncapi: Rename and split out tokens (#1025)
* syncapi: Rename and split out tokens

Previously we used the badly named `PaginationToken` which was
used for both `/sync` and `/messages` requests. This quickly
became confusing because named fields like `PDUPosition` meant
different things depending on the token type. Instead, we now have
two token types: `TopologyToken` and `StreamingToken`, both of
which have fields which make more sense for their specific situations.

Updated the codebase to use one or the other. `PaginationToken` still
lives on as `syncToken`, an unexported type which both tokens rely on.
This allows us to guarantee that the specific mappings of positions
to a string remain solely under the control of the `types` package.
This enables us to move high-level conceptual things like
"decrement this topological token" to function calls e.g
`TopologicalToken.Decrement()`.

Currently broken because `/messages` seemingly used both stream and
topological tokens, though I need to confirm this.

* final tweaks/hacks

* spurious logging

* Review comments and linting
2020-05-13 12:14:50 +01:00
Sid Karunaratne 31e6a7f193
Enforce mediaIDRegex to be only valid mediaIDCharacters (#1020)
Error messages indicate that:
> mediaId must be a non-empty string using only characters in `mediaIDCharacters`

However the regex used only required that some characters in the filename match
the restriction, not that the entire filename does. This commit ensures that
the filename must entirely fullfill the `mediaIDCharacters` restriction

Signed-off-by: Sid Karunaratne <sid@karunaratne.net>

Co-authored-by: Kegsay <kegan@matrix.org>
2020-05-13 12:04:54 +01:00
Neil Alexander 106a335b7d Update sytest-whitelist 2020-05-12 17:49:04 +01:00
Kegsay ce5dfbebf9
Implement /get_missing_events (#1022)
* WIP get_missing_events work

* More WIP get_missing_events work

* First working /get_missing_events implementation

Flakey currently due to racing between /sync and /send

* Final tweaks

* Remove log lines

* Linting

* go mod tidy

* Clamp min depth to 0

* sort events by depth because sytest makes me sad

Specifically I think it's
4172585c25/lib/SyTest/Federation/Client.pm (L265)
to blame here.
2020-05-12 16:24:28 +01:00
Neil Alexander 32624697fd
Add PPROFLISTEN (#1019)
* Add PPROFLISTEN env var

* Direct logging to more useful places

* Space
2020-05-11 18:21:39 +01:00
Neil Alexander 0c892d59fa
Prevent panic in membership updater (#1021) 2020-05-11 18:21:25 +01:00