Commit graph

309 commits

Author SHA1 Message Date
Neil Alexander 9cd8e9d4b9
Use /v3 to request media from remote servers (update to matrix-org/gomatrixserverlib#324) 2022-07-13 09:32:03 +01:00
Neil Alexander 3ea21273bc
Ristretto cache (#2563)
* Try Ristretto cache

* Tweak

* It's beautiful

* Update GMSL

* More strict keyable interface

* Fix that some more

* Make less panicky

* Don't enforce mutability checks for now

* Determine mutability using deep equality

* Tweaks

* Namespace keys

* Make federation caches mutable

* Update cost estimation, add metric

* Update GMSL

* Estimate cost for metrics better

* Reduce counters a bit

* Try caching events

* Some guards

* Try again

* Try this

* Use separate caches for hopefully better hash distribution

* Fix bug with admitting events into cache

* Try to fix bugs

* Check nil

* Try that again

* Preserve order jeezo this is messy

* thanks VS Code for doing exactly the wrong thing

* Try this again

* Be more specific

* aaaaargh

* One more time

* That might be better

* Stronger sorting

* Cache expiries, async publishing of EDUs

* Put it back

* Use a shared cache again

* Cost estimation fixes

* Update ristretto

* Reduce counters a bit

* Clean up a bit

* Update GMSL

* 1GB

* Configurable cache sizees

* Tweaks

* Add `config.DataUnit` for specifying friendly cache sizes

* Various tweaks

* Update GMSL

* Add back some lazy loading caching

* Include key in cost

* Include key in cost

* Tweak max age handling, config key name

* Only register prometheus metrics if requested

* Review comments @S7evinK

* Don't return errors when creating caches (it is better just to crash since otherwise we'll `nil`-pointer exception everywhere)

* Review comments

* Update sample configs

* Update GHA Workflow

* Update Complement images to Go 1.18

* Remove the cache test from the federation API as we no longer guarantee immediate cache admission

* Don't check the caches in the renewal test

* Possibly fix the upgrade tests

* Update to matrix-org/gomatrixserverlib#322

* Update documentation to refer to Go 1.18
2022-07-11 14:31:31 +01:00
Neil Alexander eb8dc50a97
Update Pinecone to matrix-org/pinecone@1ce778f 2022-07-08 14:54:08 +01:00
Till 89cd0e8fc1
Try to fix backfilling (#2548)
* Try to fix backfilling

* Return start/end to not confuse clients

* Update GMSL

* Update GMSL
2022-07-01 11:49:26 +02:00
Jean Lucas 920a20821b
Fix nats.go commit (#2540)
Signed-off-by: Jean Lucas <jean@4ray.co>
2022-06-27 09:15:19 +01:00
Neil Alexander 4c2a10f1a6
Handle state before, send history visibility in output (#2532)
* Check state before event

* Tweaks

* Refactor a bit, include in output events

* Don't waste time if soft failed either

* Tweak control flow, comments, use GMSL history visibility type
2022-06-13 15:11:10 +01:00
Neil Alexander c4df6d7723
Optimise state res v2 by only updating the allower when necessary (update to matrix-org/gomatrixserverlib@e55d796) 2022-06-07 15:34:58 +01:00
Neil Alexander b21a2223ef
Optimise state res v2 by parsing power level content less often (update to matrix-org/gomatrixserverlib@e938c07) 2022-06-07 15:04:00 +01:00
Neil Alexander 3d9fe20748
Fix bugs related to state resolution (#2507)
* Fix bugs related to state resolution

* Clean up `resolve-state`

* Don't panic when entries can't be found

* Ensure we have state entries for the auth events

* Revert "Ensure we have state entries for the auth events"

This reverts commit 9b13b7ed37.

* Revert "Revert "Ensure we have state entries for the auth events""

This reverts commit d86db197e3.

* Fix bug

* Try that again

* Update gomatrixserverlib

* Remove recursion from `loadAuthEvents`
2022-06-01 09:46:21 +01:00
Neil Alexander 500124dd18
Add support for room version 10 (update to matrix-org/gomatrixserverlib#315) 2022-05-30 09:50:23 +01:00
Neil Alexander b541f3043f
Add support for MSC3787 and org.matrix.msc3787 room version (update to matrix-org/gomatrixserverlib#310) 2022-05-26 15:08:17 +01:00
Neil Alexander 9e46d5025a
Upgrade various dependencies (#2496)
* Upgrade various dependencies

* Yaml dependency too
2022-05-26 12:34:43 +01:00
Neil Alexander 5a878b6e14
Update to matrix-org/gomatrixserverlib#314 2022-05-26 11:52:39 +01:00
Neil Alexander ff53398635
Mark room versions 8 and 9 as stable 2022-05-25 16:39:51 +01:00
Neil Alexander ac4d0072cf
Fix using the remote provided event from /send_join (#2490) 2022-05-25 14:31:07 +01:00
Neil Alexander d621dd2986
Allow authing invited user in rooms with "restricted" join rule (update to matrix-org/gomatrixserverlib#313) 2022-05-24 11:08:28 +01:00
Neil Alexander 8a7567c652
Fix control flow related to joining restricted v8/v9 rooms (update to matrix-org/gomatrixserverlib#312) 2022-05-23 17:49:48 +01:00
Neil Alexander 028963af1c
Update to matrix-org/gomatrixserverlib#311 (#2479)
* Update to matrix-org/gomatrixserverlib#311

* Update to squash commit
2022-05-23 14:26:56 +01:00
Neil Alexander be9be2553f
Resolve over old and new extremities (#2457)
* Feed existing state into state res when calculating state from new extremities

* Remove duplicates

* Fix bug

* Sort and unique

* Update to matrix-org/gomatrixserverlib#308

* Trim the slice properly

* Update gomatrixserverlib again

* Update to matrix-org/gomatrixserverlib#308
2022-05-13 11:52:04 +01:00
Neil Alexander cafc2d2c10
Update NATS Server to version 2.8.2 (#2460) 2022-05-13 11:36:04 +01:00
Till f69ebc6af2
Add roomserver tests (1/?) (#2434)
* Add EventJSONTable tests

* Add eventJSON tests

* Add EventStateKeysTable tests

* Add EventTypesTable tests

* Add Events Table tests
Move variable declaration outside loops
Switch to testify/assert for tests

* Move variable declaration outside loop

* Remove random data

* Fix issue where the EventReferenceSHA256 is not set

* Add more tests

* Revert "Fix issue where the EventReferenceSHA256 is not set"

This reverts commit 8ae34c4e5f.

* Update GMSL

* Add tests for duplicate entries

* Test what happens if we select non-existing NIDs

* Add test for non-existing eventType

* Really update GMSL
2022-05-09 15:30:32 +02:00
Neil Alexander 79e2fbc663
Update to matrix-org/gomatrixserverlib#307 2022-05-09 13:53:51 +01:00
Neil Alexander 85c00208c5
Fix power level event auth bugs (update to matrix-org/gomatrixserverlib#306) 2022-05-06 15:41:16 +01:00
Neil Alexander a1a5357f79
Produce more useful event auth errors (update to matrix-org/gomatrixserverlib#305) 2022-05-06 12:46:01 +01:00
Neil Alexander e4da04e75b
Update to matrix-org/gomatrixserverlib#303 2022-05-05 14:06:05 +01:00
Neil Alexander d9e71b93b6
Use gomatrixserverlib.Client instead of http.Client (#2421)
* Update to matrix-org/gomatrixserverlib#303

* Use `gomatrixserverlib.Client` for phone-home stats

* Use `gomatrixserverlib.Client` for push notifications

* Use `gomatrixserverlib.Client` for appservices

* Use `gomatrixserverlib.Client` for three-PID invites
2022-05-05 11:33:16 +01:00
Till f023cdf8c4
Add UserAPI storage tests (#2384)
* Add tests for parts of the userapi storage

* Add tests for keybackup

* Add LoginToken tests

* Add OpenID tests

* Add profile tests

* Add pusher tests

* Add ThreePID tests

* Add notification tests

* Add more device tests, fix numeric localpart query

* Fix failing CI

* Fix numeric local part query
2022-04-27 15:05:49 +02:00
Neil Alexander d7cc187ec0
Prevent JetStream from handling OS signals, allow running as a Windows service (#2385)
* Prevent JetStream from handling OS signals, allow running as a Windows service (fixes #2374)

* Remove double import
2022-04-27 13:36:40 +01:00
Neil Alexander 144c060fa7
Remove libp2p demos (#2337)
* Remove libp2p demos

* Put it back
2022-04-21 13:31:43 +01:00
Neil Alexander a9f0a390c6
Update to NATS Server v2.8.0 and nats.go v1.14.0 (#2359) 2022-04-20 15:13:04 +01:00
Neil Alexander 711e377b9c
Update go-sqlite3-js to matrix-org/go-sqlite3-js#2 (SQLite 3.36.0) 2022-04-19 10:34:33 +01:00
Till 69f2ff7c82
Correctly use provided filters (#2339)
* Apply filters correctly

* Fix issues; Use prepareWithFilters

* Update gmsl & tests

* go.mod..

* PR comments
2022-04-11 09:05:23 +02:00
Neil Alexander 986d27a128
Update to matrix-org/pinecone@2999ea2 2022-04-08 16:39:09 +01:00
Neil Alexander 9bd9f2beba
Update to matrix-org/pinecone@9b3963248c 2022-04-08 16:23:50 +01:00
Neil Alexander 4312b87ab0
Update dependencies (#2336) 2022-04-08 12:24:49 +01:00
Neil Alexander 4d9d9cc9b1
Update to matrix-org/gomatrixserverlib#300 2022-04-05 14:43:44 +01:00
Neil Alexander 562d742240
Update to matrix-org/gomatrixserverlib#299 2022-04-04 19:04:17 +01:00
Neil Alexander c69159bda7
Update to matrix-org/pinecone@e526fa8 2022-04-04 15:16:02 +01:00
Neil Alexander 8213b2ba30
Update Pinecone P2P demo 2022-03-30 15:01:22 +01:00
Neil Alexander 6c95f9b535
Update to matrix-org/pinecone@6fb0773 2022-03-23 14:45:22 +00:00
Neil Alexander c54d88aecf
Update to matrix-org/pinecone@8687b0d 2022-03-18 15:32:14 +00:00
Neil Alexander 0fb94fc781
Update to matrix-org/gomatrixserverlib@0980b7f 2022-03-17 16:48:41 +00:00
Neil Alexander 1894f34868
Update to matrix-org/gomatrixserverlib@a62d9f3 2022-03-17 09:58:44 +00:00
Neil Alexander 047147a3af
Update Yggdrasil P2P demo (also remove QUIC dependency) 2022-03-16 16:49:10 +00:00
Neil Alexander 0ad665fabc
Update to NATS Server 2.7.4 (#2282)
* Update to NATS Server v2.7.4-beta2

* Bump to release v2.7.4

* Listen for TCP

* Revert "Listen for TCP"

This reverts commit 28dff0ab4d.
2022-03-16 10:47:18 +00:00
Neil Alexander 2696f36738
Federation client improvements (HTTP/2, reaping federation trippers, TLS client session cache, keepalive tweaks, update to matrix-org/gomatrixserverlib#297) 2022-03-16 10:27:16 +00:00
Neil Alexander e1881627d1
Implement MSC3383 (update to matrix-org/gomatrixserverlib#296) 2022-03-10 12:44:31 +00:00
Neil Alexander 030b995636
Update to matrix-org/pinecone@cfde1f8 2022-03-09 10:19:49 +00:00
Neil Alexander 0297929b76
Revert NATS server upgrade in 00b3545b14 2022-03-04 15:56:31 +00:00
Dan f05ce478f0
Implement Push Notifications (#1842)
* Add Pushserver component with Pushers API

Co-authored-by: Tommie Gannert <tommie@gannert.se>
Co-authored-by: Dan Peleg <dan@globekeeper.com>

* Wire Pushserver component

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>

* Add PushGatewayClient.

The full event format is required for Sytest.

* Add a pushrules module.

* Change user API account creation to use the new pushrules module's defaults.

Introduces "scope" as required by client API, and some small field
tweaks to make some 61push Sytests pass.

* Add push rules query/put API in Pushserver.

This manipulates account data over User API, and fires sync messages
for changes. Those sync messages should, according to an existing TODO
in clientapi, be moved to userapi.

Forks clientapi/producers/syncapi.go to pushserver/ for later extension.

* Add clientapi routes for push rules to Pushserver.

A cleanup would be to move more of the name-splitting logic into
pushrules.go, to depollute routing.go.

* Output rooms.join.unread_notifications in /sync.

This is the read-side. Pushserver will be the write-side.

* Implement pushserver/storage for notifications.

* Use PushGatewayClient and the pushrules module in Pushserver's room consumer.

* Use one goroutine per user to avoid locking up the entire server for
  one bad push gateway.
* Split pushing by format.
* Send one device per push. Sytest does not support coalescing
  multiple devices into one push. Matches Synapse. Either we change
  Sytest, or remove the group-by-url-and-format logic.
* Write OutputNotificationData from push server. Sync API is already
  the consumer.

* Implement read receipt consumers in Pushserver.

Supports m.read and m.fully_read receipts.

* Add clientapi route for /unstable/notifications.

* Rename to UpsertPusher for clarity and handle pusher update

* Fix linter errors

* Ignore body.Close() error check

* Fix push server internal http wiring

* Add 40 newly passing 61push tests to whitelist

* Add next 12 newly passing 61push tests to whitelist

* Send notification data before notifying users in EDU server consumer

* NATS JetStream

* Goodbye sarama

* Fix `NewStreamTokenFromString`

* Consume on the correct topic for the roomserver

* Don't panic, NAK instead

* Move push notifications into the User API

* Don't set null values since that apparently causes Element upsetti

* Also set omitempty on conditions

* Fix bug so that we don't override the push rules unnecessarily

* Tweak defaults

* Update defaults

* More tweaks

* Move `/notifications` onto `r0`/`v3` mux

* User API will consume events and read/fully read markers from the sync API with stream positions, instead of consuming directly

Co-authored-by: Piotr Kozimor <p1996k@gmail.com>
Co-authored-by: Tommie Gannert <tommie@gannert.se>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-03-03 11:40:53 +00:00