Commit graph

1887 commits

Author SHA1 Message Date
Neil Alexander 1773fd84b7
Hydrate destination queues at startup (#1179)
* Hydrate destination queues at startup

* Review comments
2020-07-03 11:49:49 +01:00
Neil Alexander 9dffeb9b67
Fix joins to rooms that we know about that have room IDs with our server part (#1178) 2020-07-03 10:25:26 +01:00
Neil Alexander 38caf8e5b7
Yggdrasil+QUIC demo, federation sender tweaks (#1177)
* Initial QUIC work

* Update Yggdrasil demo

* Make sure that the federation sender knows how many pending events are in the database when the worker starts

* QUIC tunables

* pprof

* Don't spin

* Set build info for Yggdrasil
2020-07-02 17:43:07 +01:00
Kegsay 9c1f38621c
Remove publicroomsapi (#1176)
* Remove all of publicroomsapi

* Remove references to publicroomsapi

* Remove doc references to publicroomsapi
2020-07-02 17:11:33 +01:00
Kegsay 4c1e6597c0
Replace publicroomsapi with a combination of clientapi/roomserver/currentstateserver (#1174)
* Use content_value instead of membership

* Fix build

* Replace publicroomsapi with a combination of clientapi/roomserver/currentstateserver

- All public rooms paths are now handled by clientapi
- Requests to (un)publish rooms are sent to the roomserver via `PerformPublish`
  which are stored in a new `published_table.go`
- Requests for public rooms are handled in clientapi by:
    * Fetch all room IDs which are published using `QueryPublishedRooms` on the roomserver.
    * Apply pagination parameters to the slice.
    * Do a `QueryBulkStateContent` request to the currentstateserver to pull out
      required state event *content* (not entire events).
    * Aggregate and return the chunk.

Mostly but not fully implemented (DB queries on currentstateserver are missing)

* Fix pq query

* Make postgres work

* Make sqlite work

* Fix tests

* Unbreak pagination tests

* Linting
2020-07-02 15:41:18 +01:00
Neil Alexander 55bc82c439 Update Yggdrasil demo peer count 2020-07-01 13:47:16 +01:00
Neil Alexander 8e7947926c Fix Yggdrasil gobind build, set display name at registration 2020-07-01 13:35:58 +01:00
Neil Alexander 42dd962425
Persistent federation sender queues (PDUs) (#1173)
* Initial work on persistent queues

* Update index for event ID and server name

* Put things into database (postgres for now)

* Duplicate postgres code into sqlite for now just to stop build errors, will fix SQLite soon

* Fix table name

* Fix index

* Fix table name

* Use RETURNING because LastInsertID is not supported by postgres

* Use functions

* Marshal headered event

* Don't error on now rows

* Don't block if there are PDUs waiting

* Try to tidy up JSON

* Debug logging

* Fix query, use transactions in postgres

* Clean up

* Rehydrate more opportunistically

* Fix SQLite

* remove unused types

* Review comments

* Shuffle things around a bit

* Clean up transaction properly

* Don't send empty transactions

* Reduce unnecessary retries

* Count PDUs to make more resilient

* Don't stop when there is work to be done

* Try to limit wakeups

* well this is tedious

* Fix race in incomplete transactions

* Thread safety on transaction ID/count
2020-07-01 11:46:38 +01:00
Kegsay 6f49758b90
Remove membership table from account DB (#1172)
* Remove membership table from account DB

And make code which needs that data use the currentstate server

* Unbreak tests; use a membership enum for space
2020-06-30 13:34:59 +01:00
Kegsay ca5bbffd8d
Add a new component: currentstateserver (#1171)
* Add a new component: currentstateserver

- Add a skeleton for it, with databases and a single query method.
- Add integration tests for it.
- Add listen/address fields in the config (breaking as this will force people to specify this to validate)

Not currently hooked up to anything yet.

* Unbreak config tests

* Add current_state to sample config

* comments
2020-06-30 10:37:21 +01:00
Kegan Dougal 3a18b7fc78 Update awsy list 2020-06-30 10:26:26 +01:00
Neil Alexander 4cf45d1ce9
Don't include current state in processEventWithMissingState (#1126)
* Don't include current state in processEventWithMissingState

* Remove lookupCurrentState as not needed

Co-authored-by: Kegsay <kegan@matrix.org>
2020-06-29 14:39:21 +01:00
Kegsay 1ad7219e4b
Implement /sync limited and read timeline limit from stored filters (#1168)
* Move filter table to syncapi where it is used

* Implement /sync `limited` and read timeline limit from stored filters

We now fully handle `room.timeline.limit` filters (in-line + stored) and
return the right value for `limited` syncs.

* Update whitelist

* Default to the default timeline limit if it's unset, also strip the extra event correctly

* Update whitelist
2020-06-26 15:34:41 +01:00
Kegsay 164057a3be
Honour event size limits and return 413 (#1167) 2020-06-26 12:51:54 +01:00
Kegan Dougal 9592d53364 Fix div 0 error and add new tests to list 2020-06-26 11:34:25 +01:00
Kegsay 4897beabee
Finish implementing retiring invites (#1166)
* Pass retired invites to the syncapi with the event ID of the invite

* Implement retire invite streaming

* Update whitelist
2020-06-26 11:07:52 +01:00
Kegan Dougal c1d2382e6d Reject non-numeric ports (done in GMSL) 2020-06-25 18:05:41 +01:00
Kegsay 7a8282fccf
Handle invite v1 (#1165)
* Implement invite v1 for sytest mainly

* Bump gmsl version which falls back to invite v1 if v2 404s

* Update whitelist
2020-06-25 17:07:40 +01:00
Kegan Dougal 67f7a53f12 Add missing typing test 2020-06-25 15:06:46 +01:00
Kegsay 43cddfe00f
Return remote errors from FS.PerformJoin (#1164)
* Return remote errors from FS.PerformJoin

Follows the same pattern as PerformJoin on roomserver (no error return).

Also return the right format for incompatible room version errors.

Makes a bunch of tests pass!

* Handle network errors better when returning remote HTTP errors

* Linting

* Fix tests

* Update whitelist, pass network errors through in API=1 mode
2020-06-25 15:04:48 +01:00
Ashley Nelson c2d34422d6
Remove trailing slash in client api proxy (#1163)
Signed-off-by: Ashley Nelson <amn@fantashley.com>
2020-06-25 12:27:09 +01:00
Kegsay 46de400aa0
Hopefully fix databased is locked errors on sqlite account creation (#1162) 2020-06-24 20:46:28 +01:00
Kegsay e560619f76
Refactor SendMembership - make ban test pass (#1160)
* Refactor SendMembership - make ban test pass

* Only check invite auth events for local invites
2020-06-24 18:19:54 +01:00
Kegsay a06d0921c9
Make same membership transitions 403, add sytests (#1161)
* Make same membership transitions 403, add sytests

* Update blacklist
2020-06-24 18:19:14 +01:00
Kegsay 002fe05a20
Add PerformInvite and refactor how errors get handled (#1158)
* Add PerformInvite and refactor how errors get handled

- Rename `JoinError` to `PerformError`
- Remove `error` from the API function signature entirely. This forces
  errors to be bundled into `PerformError` which makes it easier for callers
  to detect and handle errors. On network errors, HTTP clients will make a
  `PerformError`.

* Unbreak everything; thanks Go!

* Send back JSONResponse according to the PerformError

* Update federation invite code too
2020-06-24 15:06:14 +01:00
Kegan Dougal ebaaf65c54 This doesn't pass 2020-06-24 13:40:50 +01:00
Kegan Dougal c72d23c8eb Update whitelist 2020-06-24 10:28:03 +01:00
Kegsay 0577bfca55
Pass join errors through internal API boundaries (#1157)
* Pass join errors through internal API boundaries

Required for certain invite sytests. We will need to think of a
better way of handling this going forwards.

* Include m.room.avatar in stripped state; handle trailing slashes when GETing state events

* Update whitelist

* Update whitelist
2020-06-24 09:59:59 +01:00
Kegan Dougal 1f93427ed9 Linting 2020-06-23 18:44:34 +01:00
Kegan Dougal 81beab8002 Add instrumented main for coverage 2020-06-23 18:32:22 +01:00
Kegsay 914f6cadce
Add /send restrictions and return correct error codes (#1156)
* Add /send restrictions and return correct error codes

- Max 50 PDUs / 100 EDUs
- Fail the transaction when PDUs contain bad JSON

* Update whitelist

* Unbreak test

* Linting
2020-06-23 13:15:15 +01:00
Kegsay 4220a374ca
Fix room checks for /state and /state_ids (#1155)
We would return a 403 first (as the server would not be allowed to
see this event) and only then return a 404 if the event is not in
the given room. We now invert those checks for /state and /state_ids
to make the tests pass.
2020-06-23 11:47:48 +01:00
Kegsay 02565c37aa
/send auth errors are silent (#1149)
* /send auth errors are silent

* Fix test
2020-06-23 10:31:17 +01:00
Martin Honermeyer a1352cbd9e
Push serverkeyapi docker image to registry as well (#1154)
Signed-off-by: Martin Honermeyer <maze@strahlungsfrei.de>
2020-06-23 09:51:38 +01:00
Ashley Nelson 61e0482fef
Add appservices component to docker scripts (#1153)
Signed-off-by: Ashley Nelson <amn@fantashley.com>
2020-06-20 21:28:30 +01:00
Neil Alexander 7f26b0cd13
Bind build support, further Yggdrasil demo updates (#1152)
* Add gobind builds for Yggdrasil demo

* Massage client API a bit

* Fix build

* Fix gobind build

* Fix gobind client API setup

* Tweaks

* Tweaks

* Update sytest-whitelist, add comment

* Default to sending push rules on initial sync
2020-06-19 13:29:27 +01:00
Neil Alexander 72444e4a4f
User API polylith component (#1151)
* User API polylith component

* Add docker-pull.sh
2020-06-19 09:37:19 +01:00
Neil Alexander 9e3d771a32 Fix comment in InputAccountDataRequest 2020-06-19 09:18:09 +01:00
Neil Alexander dc0bac85d5
Refactor account data (#1150)
* Refactor account data

* Tweak database fetching

* Tweaks

* Restore syncProducer notification

* Various tweaks, update tag behaviour

* Fix initial sync
2020-06-18 18:36:03 +01:00
Neil Alexander 3547a1768c Fix embed Riot Web into Yggdrasil demo 2020-06-18 13:48:47 +01:00
Neil Alexander ddf1c8adf1
Hacks for supporting Riot iOS (#1148)
* Join room body is optional

* Support deprecated login by user/password

* Implement dummy key upload endpoint

* Make a very determinate end to /messages if we hit the create event in back-pagination

* Linting
2020-06-17 17:41:45 +01:00
Kegsay 84a7881468
Make account data sytests pass (#1147) 2020-06-17 17:01:03 +01:00
Kegan Dougal 9b408c19fb Missing sytests 2020-06-17 16:47:21 +01:00
Kegsay 8efeb8eb3b
Return the correct /joined_members response and allow ?format=event (#1146) 2020-06-17 16:21:42 +01:00
Kegsay 8e7c1eda05
Enable more sytests (#1145) 2020-06-17 15:40:37 +01:00
Kegsay c7f7ae69eb
Are we synapse yet: Accept tests without a group rather than dying (#1142)
Produces output like:
```
Non-Spec APIs: 0% (0/52 tests)
--------------
  Non-Spec API             :   0% (0/50 tests)
  Unknown API (no group specified):   0% (0/2 tests)
```

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-06-17 15:12:09 +01:00
Neil Alexander 23bed196e6
Fallback parsing for Content-Disposition minefield (#1144) 2020-06-17 14:26:45 +01:00
Kegsay 38053a5bb7
Do not wrap send_join errors on /v1/send_join (#1143)
* Do not wrap v1 send_join errors in [code, body]

* Don't wrap errors
2020-06-17 13:55:55 +01:00
Kegan Dougal c4d9b37492 add test 2020-06-17 13:54:47 +01:00
Kegsay e09d24e732
Move account/device DBs to userapi (#1141) 2020-06-17 12:05:56 +01:00