Commit graph

2301 commits

Author SHA1 Message Date
Neil Alexander 9cd8e9d4b9
Use /v3 to request media from remote servers (update to matrix-org/gomatrixserverlib#324) 2022-07-13 09:32:03 +01:00
Till 09f0ff14c8
Minor SendToDevice fix (#2565)
* Avoid unnecessary marshalling if sending to the local server

* Fix ordering of ToDevice messages

* Revive SendToDevice test
2022-07-12 08:23:58 +02:00
Neil Alexander 3ea21273bc
Ristretto cache (#2563)
* Try Ristretto cache

* Tweak

* It's beautiful

* Update GMSL

* More strict keyable interface

* Fix that some more

* Make less panicky

* Don't enforce mutability checks for now

* Determine mutability using deep equality

* Tweaks

* Namespace keys

* Make federation caches mutable

* Update cost estimation, add metric

* Update GMSL

* Estimate cost for metrics better

* Reduce counters a bit

* Try caching events

* Some guards

* Try again

* Try this

* Use separate caches for hopefully better hash distribution

* Fix bug with admitting events into cache

* Try to fix bugs

* Check nil

* Try that again

* Preserve order jeezo this is messy

* thanks VS Code for doing exactly the wrong thing

* Try this again

* Be more specific

* aaaaargh

* One more time

* That might be better

* Stronger sorting

* Cache expiries, async publishing of EDUs

* Put it back

* Use a shared cache again

* Cost estimation fixes

* Update ristretto

* Reduce counters a bit

* Clean up a bit

* Update GMSL

* 1GB

* Configurable cache sizees

* Tweaks

* Add `config.DataUnit` for specifying friendly cache sizes

* Various tweaks

* Update GMSL

* Add back some lazy loading caching

* Include key in cost

* Include key in cost

* Tweak max age handling, config key name

* Only register prometheus metrics if requested

* Review comments @S7evinK

* Don't return errors when creating caches (it is better just to crash since otherwise we'll `nil`-pointer exception everywhere)

* Review comments

* Update sample configs

* Update GHA Workflow

* Update Complement images to Go 1.18

* Remove the cache test from the federation API as we no longer guarantee immediate cache admission

* Don't check the caches in the renewal test

* Possibly fix the upgrade tests

* Update to matrix-org/gomatrixserverlib#322

* Update documentation to refer to Go 1.18
2022-07-11 14:31:31 +01:00
Neil Alexander eb8dc50a97
Update Pinecone to matrix-org/pinecone@1ce778f 2022-07-08 14:54:08 +01:00
Till f76f28e6db
Fix issue uint64 values with high bit are not supported in presence (#2562)
* Fix issue #2528

* Use gomatrixserverlib.Timestamp

* Use ParseUint instead of ParseInt
2022-07-07 16:29:25 +02:00
Till f3e8a9a4cb
Fix nil pointer access when redacting events (#2560) 2022-07-07 11:40:53 +02:00
Neil Alexander 460dccf93d
Hopefully fix read receipts timestamps (#2557)
This should avoid coercions between signed and unsigned ints which might fix problems like `sql: converting argument $5 type: uint64 values with high bit set are not supported`.
2022-07-05 17:13:26 +01:00
Neil Alexander c0f824d437
Wrap error from SnapshotNIDFromEventID 2022-07-05 15:06:10 +01:00
Neil Alexander d4341a2d97
Return clearer error when no state NID exists for an event (#2555) 2022-07-05 15:01:34 +01:00
Till 5087b36af0
Fix QuerySharedUsers for the SyncAPI keychange consumer (#2554)
* Make more use of base.BaseDendrite

* Fix QuerySharedUsers if no UserIDs are supplied
2022-07-05 14:50:56 +02:00
Till f29cdb26f6
Use new testrig for key changes tests (#2552)
* Use new testrig for tests

* Log the error message
2022-07-05 14:50:24 +02:00
Kabir Kwatra 43147bd654
feat+fix: Ignore unknown keys and verify required fields are present in appservice registration files (#2550)
* fix: ignore unknown keys in appservice configs

fixes matrix-org/dendrite#1567

* feat: verify required fields in appservice configs
2022-07-05 12:53:51 +01:00
PiotrKozimor 1214c6137a
CD with GCP cloud builds (#12)
* Setup CD on GCP

* Squashed commit of the following:

commit 7823481a0e
Author: PiotrKozimor <37144818+PiotrKozimor@users.noreply.github.com>
Date:   Mon Jul 4 17:39:22 2022 +0200

    Merge v0.8.9 (#13)

    Squashed commit of the following:

    commit b5c55faf98
    Author: Neil Alexander <neilalexander@users.noreply.github.com>
    Date:   Fri Jul 1 12:00:32 2022 +0100

        Version 0.8.9 (#2549)

        * Version 0.8.9

        * Update changelog

    commit b50a24c666
    Author: Neil Alexander <neilalexander@users.noreply.github.com>
    Date:   Fri Jul 1 10:54:07 2022 +0100

        Roomserver producers package (#2546)

        * Give the roomserver a producers package

        * Change init point

        * Populate ACLs API

        * Fix build issues

        * `RoomEventProducer` naming

    commit 89cd0e8fc1
    Author: Till <2353100+S7evinK@users.noreply.github.com>
    Date:   Fri Jul 1 11:49:26 2022 +0200

        Try to fix backfilling (#2548)

        * Try to fix backfilling

        * Return start/end to not confuse clients

        * Update GMSL

        * Update GMSL

    commit 086f182e24
    Author: Neil Alexander <neilalexander@users.noreply.github.com>
    Date:   Fri Jul 1 09:50:06 2022 +0100

        Disable WebAssembly builds for now

    commit 54bed4c593
    Author: Neil Alexander <neilalexander@users.noreply.github.com>
    Date:   Fri Jul 1 09:37:54 2022 +0100

        Blacklist `Guest users can join guest_access rooms` test until it can be investigated

    commit 561c159ad7
    Author: Till <2353100+S7evinK@users.noreply.github.com>
    Date:   Thu Jun 30 12:34:37 2022 +0200

        Silence presence logs (#2547)

    commit 519bc1124b
    Author: Neil Alexander <neilalexander@users.noreply.github.com>
    Date:   Wed Jun 29 15:29:39 2022 +0100

        Add `evacuateUser` endpoint, use it when deactivating accounts (#2545)

        * Add `evacuateUser` endpoint, use it when deactivating accounts

        * Populate the API

        * Clean up user devices when deactivating

        * Include invites, delete pushers

    commit 2dea466685
    Author: Neil Alexander <neilalexander@users.noreply.github.com>
    Date:   Wed Jun 29 12:32:24 2022 +0100

        Return an error if trying to invite a malformed user ID (#2543)

    commit 2086992caf
    Author: Till <2353100+S7evinK@users.noreply.github.com>
    Date:   Wed Jun 29 10:49:12 2022 +0200

        Don't return `end` if there are not more messages (#2542)

        * Be more spec compliant

        * Move lazyLoadMembers to own method

    commit 920a20821b
    Author: Jean Lucas <jean@4ray.co>
    Date:   Mon Jun 27 04:15:19 2022 -0400

        Fix nats.go commit (#2540)

        Signed-off-by: Jean Lucas <jean@4ray.co>

    commit 7120eb6bc9
    Author: Neil Alexander <neilalexander@users.noreply.github.com>
    Date:   Wed Jun 15 14:27:07 2022 +0100

        Add `InputDeviceListUpdate` to the keyserver, remove old input API (#2536)

        * Add `InputDeviceListUpdate` to the keyserver, remove old input API

        * Fix copyright

        * Log more information when a device list update fails

    commit 1b90cc9536
    Author: Till <2353100+S7evinK@users.noreply.github.com>
    Date:   Wed Jun 15 12:50:02 2022 +0200

        Fix rare panic when returning user devices over federation (#2534)

    commit 4c2a10f1a6
    Author: Neil Alexander <neilalexander@users.noreply.github.com>
    Date:   Mon Jun 13 15:11:10 2022 +0100

        Handle state before, send history visibility in output (#2532)

        * Check state before event

        * Tweaks

        * Refactor a bit, include in output events

        * Don't waste time if soft failed either

        * Tweak control flow, comments, use GMSL history visibility type

    commit c500958583
    Author: Emanuele Aliberti <dev@mtka.eu>
    Date:   Mon Jun 13 13:08:46 2022 +0200

        generic CaddyFile in front of Dendrite (monolith) (#2531)

        for Caddy 2.5.x

        Co-authored-by: emanuele.aliberti <emanuele.aliberti@mtka.eu>

    commit e1136f4d3e
    Author: Till Faelligen <davidf@element.io>
    Date:   Mon Jun 13 11:46:59 2022 +0200

        Make the linter happy again

    commit 0a7f7dc716
    Author: Neil Alexander <neilalexander@users.noreply.github.com>
    Date:   Mon Jun 13 10:16:30 2022 +0100

        Add `--difference` to `resolve-state` tool

    commit 89d2adadbd
    Author: Neil Alexander <neilalexander@users.noreply.github.com>
    Date:   Fri Jun 10 10:58:04 2022 +0100

        Attempt to raise the file descriptor limit at startup (#2527)

    commit 1030072285
    Author: Neil Alexander <neilalexander@users.noreply.github.com>
    Date:   Fri Jun 10 10:18:32 2022 +0100

        Rename the page to "Optimise your installation"

    commit 16ed1633b6
    Author: Neil Alexander <neilalexander@users.noreply.github.com>
    Date:   Fri Jun 10 10:15:14 2022 +0100

        Highlighting in docs

    commit e2a64773ce
    Author: Neil Alexander <neilalexander@users.noreply.github.com>
    Date:   Fri Jun 10 10:14:15 2022 +0100

        Add new next steps page to the documentation

    commit 660f7839f5
    Author: Till <2353100+S7evinK@users.noreply.github.com>
    Date:   Thu Jun 9 18:38:07 2022 +0200

        Correctly redact events over federation (#2526)

        * Ensure we check powerlevel/origin before redacting an event

        * Add passing test

        * Use pl.UserLevel

        * Make check more readable, also check for the sender
2022-07-04 18:07:55 +02:00
PiotrKozimor 7823481a0e
Merge v0.8.9 (#13)
Squashed commit of the following:

commit b5c55faf98
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Fri Jul 1 12:00:32 2022 +0100

    Version 0.8.9 (#2549)

    * Version 0.8.9

    * Update changelog

commit b50a24c666
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Fri Jul 1 10:54:07 2022 +0100

    Roomserver producers package (#2546)

    * Give the roomserver a producers package

    * Change init point

    * Populate ACLs API

    * Fix build issues

    * `RoomEventProducer` naming

commit 89cd0e8fc1
Author: Till <2353100+S7evinK@users.noreply.github.com>
Date:   Fri Jul 1 11:49:26 2022 +0200

    Try to fix backfilling (#2548)

    * Try to fix backfilling

    * Return start/end to not confuse clients

    * Update GMSL

    * Update GMSL

commit 086f182e24
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Fri Jul 1 09:50:06 2022 +0100

    Disable WebAssembly builds for now

commit 54bed4c593
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Fri Jul 1 09:37:54 2022 +0100

    Blacklist `Guest users can join guest_access rooms` test until it can be investigated

commit 561c159ad7
Author: Till <2353100+S7evinK@users.noreply.github.com>
Date:   Thu Jun 30 12:34:37 2022 +0200

    Silence presence logs (#2547)

commit 519bc1124b
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Jun 29 15:29:39 2022 +0100

    Add `evacuateUser` endpoint, use it when deactivating accounts (#2545)

    * Add `evacuateUser` endpoint, use it when deactivating accounts

    * Populate the API

    * Clean up user devices when deactivating

    * Include invites, delete pushers

commit 2dea466685
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Jun 29 12:32:24 2022 +0100

    Return an error if trying to invite a malformed user ID (#2543)

commit 2086992caf
Author: Till <2353100+S7evinK@users.noreply.github.com>
Date:   Wed Jun 29 10:49:12 2022 +0200

    Don't return `end` if there are not more messages (#2542)

    * Be more spec compliant

    * Move lazyLoadMembers to own method

commit 920a20821b
Author: Jean Lucas <jean@4ray.co>
Date:   Mon Jun 27 04:15:19 2022 -0400

    Fix nats.go commit (#2540)

    Signed-off-by: Jean Lucas <jean@4ray.co>

commit 7120eb6bc9
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Jun 15 14:27:07 2022 +0100

    Add `InputDeviceListUpdate` to the keyserver, remove old input API (#2536)

    * Add `InputDeviceListUpdate` to the keyserver, remove old input API

    * Fix copyright

    * Log more information when a device list update fails

commit 1b90cc9536
Author: Till <2353100+S7evinK@users.noreply.github.com>
Date:   Wed Jun 15 12:50:02 2022 +0200

    Fix rare panic when returning user devices over federation (#2534)

commit 4c2a10f1a6
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 13 15:11:10 2022 +0100

    Handle state before, send history visibility in output (#2532)

    * Check state before event

    * Tweaks

    * Refactor a bit, include in output events

    * Don't waste time if soft failed either

    * Tweak control flow, comments, use GMSL history visibility type

commit c500958583
Author: Emanuele Aliberti <dev@mtka.eu>
Date:   Mon Jun 13 13:08:46 2022 +0200

    generic CaddyFile in front of Dendrite (monolith) (#2531)

    for Caddy 2.5.x

    Co-authored-by: emanuele.aliberti <emanuele.aliberti@mtka.eu>

commit e1136f4d3e
Author: Till Faelligen <davidf@element.io>
Date:   Mon Jun 13 11:46:59 2022 +0200

    Make the linter happy again

commit 0a7f7dc716
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 13 10:16:30 2022 +0100

    Add `--difference` to `resolve-state` tool

commit 89d2adadbd
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Fri Jun 10 10:58:04 2022 +0100

    Attempt to raise the file descriptor limit at startup (#2527)

commit 1030072285
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Fri Jun 10 10:18:32 2022 +0100

    Rename the page to "Optimise your installation"

commit 16ed1633b6
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Fri Jun 10 10:15:14 2022 +0100

    Highlighting in docs

commit e2a64773ce
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Fri Jun 10 10:14:15 2022 +0100

    Add new next steps page to the documentation

commit 660f7839f5
Author: Till <2353100+S7evinK@users.noreply.github.com>
Date:   Thu Jun 9 18:38:07 2022 +0200

    Correctly redact events over federation (#2526)

    * Ensure we check powerlevel/origin before redacting an event

    * Add passing test

    * Use pl.UserLevel

    * Make check more readable, also check for the sender
2022-07-04 17:39:22 +02:00
Neil Alexander b5c55faf98
Version 0.8.9 (#2549)
* Version 0.8.9

* Update changelog
2022-07-01 12:00:32 +01:00
Neil Alexander b50a24c666
Roomserver producers package (#2546)
* Give the roomserver a producers package

* Change init point

* Populate ACLs API

* Fix build issues

* `RoomEventProducer` naming
2022-07-01 10:54:07 +01:00
Till 89cd0e8fc1
Try to fix backfilling (#2548)
* Try to fix backfilling

* Return start/end to not confuse clients

* Update GMSL

* Update GMSL
2022-07-01 11:49:26 +02:00
Neil Alexander 086f182e24
Disable WebAssembly builds for now 2022-07-01 09:50:06 +01:00
Neil Alexander 54bed4c593
Blacklist Guest users can join guest_access rooms test until it can be investigated 2022-07-01 09:37:54 +01:00
PiotrKozimor 374b77a3df
Entry improvements (#11)
* Refactor ApplicationServiceWorkerState to be more robust

* Add launch.json to VS Code

* Implement login with JWT, registering with email, failed login rate limiting and reset password with m.login.email.identity auth type

* Log errors when JWT parsing failed

* Development build script

* Fix linter errors

* Use golangci-lint as a linter in VS Code

* Fix tests with RtFailedLogin

* Pass config load tests - parse JWT public key only if enabled

* Reduce CI steps

Do not support 386 arch and go 1.16, 1.17

* Fix linter errors

* Change RtFailedLogin logic - nil pointer can be provided

* Respect access token in query

* Fix typos

* Use only one mutex in RtFailedLogin

* Remove eventsRemaining across appservice component

* Push dendrite to production registry as well

* Rafactor TestRtFailedLogin
2022-06-30 14:56:45 +02:00
Till 561c159ad7
Silence presence logs (#2547) 2022-06-30 12:34:37 +02:00
Neil Alexander 519bc1124b
Add evacuateUser endpoint, use it when deactivating accounts (#2545)
* Add `evacuateUser` endpoint, use it when deactivating accounts

* Populate the API

* Clean up user devices when deactivating

* Include invites, delete pushers
2022-06-29 15:29:39 +01:00
Neil Alexander 2dea466685
Return an error if trying to invite a malformed user ID (#2543) 2022-06-29 12:32:24 +01:00
Till 2086992caf
Don't return end if there are not more messages (#2542)
* Be more spec compliant

* Move lazyLoadMembers to own method
2022-06-29 10:49:12 +02:00
Jean Lucas 920a20821b
Fix nats.go commit (#2540)
Signed-off-by: Jean Lucas <jean@4ray.co>
2022-06-27 09:15:19 +01:00
Neil Alexander 7120eb6bc9
Add InputDeviceListUpdate to the keyserver, remove old input API (#2536)
* Add `InputDeviceListUpdate` to the keyserver, remove old input API

* Fix copyright

* Log more information when a device list update fails
2022-06-15 14:27:07 +01:00
Till 1b90cc9536
Fix rare panic when returning user devices over federation (#2534) 2022-06-15 12:50:02 +02:00
Neil Alexander 4c2a10f1a6
Handle state before, send history visibility in output (#2532)
* Check state before event

* Tweaks

* Refactor a bit, include in output events

* Don't waste time if soft failed either

* Tweak control flow, comments, use GMSL history visibility type
2022-06-13 15:11:10 +01:00
Emanuele Aliberti c500958583
generic CaddyFile in front of Dendrite (monolith) (#2531)
for Caddy 2.5.x

Co-authored-by: emanuele.aliberti <emanuele.aliberti@mtka.eu>
2022-06-13 12:08:46 +01:00
Till Faelligen e1136f4d3e Make the linter happy again 2022-06-13 11:46:59 +02:00
Neil Alexander 0a7f7dc716
Add --difference to resolve-state tool 2022-06-13 10:16:30 +01:00
Neil Alexander 89d2adadbd
Attempt to raise the file descriptor limit at startup (#2527) 2022-06-10 10:58:04 +01:00
Neil Alexander 1030072285
Rename the page to "Optimise your installation" 2022-06-10 10:18:32 +01:00
Neil Alexander 16ed1633b6
Highlighting in docs 2022-06-10 10:15:14 +01:00
Neil Alexander e2a64773ce
Add new next steps page to the documentation 2022-06-10 10:14:15 +01:00
Till 660f7839f5
Correctly redact events over federation (#2526)
* Ensure we check powerlevel/origin before redacting an event

* Add passing test

* Use pl.UserLevel

* Make check more readable, also check for the sender
2022-06-09 18:38:07 +02:00
Neil Alexander 83797573be
Version 0.8.8 (#2525) 2022-06-09 12:18:50 +01:00
Till 289b3c5608
Allow "registration is idempotent, with username specified" to pass (#2488)
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-06-09 11:26:48 +01:00
Neil Alexander 3cdefcf765
Update sample configs for exempt_user_ids 2022-06-08 09:13:25 +01:00
Neil Alexander c4df6d7723
Optimise state res v2 by only updating the allower when necessary (update to matrix-org/gomatrixserverlib@e55d796) 2022-06-07 15:34:58 +01:00
Neil Alexander b21a2223ef
Optimise state res v2 by parsing power level content less often (update to matrix-org/gomatrixserverlib@e938c07) 2022-06-07 15:04:00 +01:00
Neil Alexander 6d4bd5d890
Rate limiting changes (#2519)
* Rate limiting changes

This makes the following changes:

* For logged in users, the rate limiting now applies to the device session rather than the remote IP address;
* For non-logged in users, the rate limiting continues to apply to remote address as it does today;
* It is now possible to add user IDs to the `exempt_user_ids` option under `rate_limiting` to exclude bots from rate limiting;
* Admin and appservice users are now exempt from rate limiting by default.

* Fix build with media API
2022-06-07 14:24:04 +01:00
Neil Alexander 27948fb304
Optimise loadAuthEvents, add roomserver tracing 2022-06-07 14:23:26 +01:00
Neil Alexander aafb7bf120
Avoid panicking unnecessarily when shutting down the gobind P2P demos (#2520) 2022-06-07 10:46:21 +01:00
Neil Alexander 0d7020fbaf
Send tombstone to other servers when upgrading rooms 2022-06-06 17:27:50 +01:00
Neil Alexander 2cb609c428
Room upgrade tweaks
Squashed commit of the following:

commit 7a1568c716866594af6d0b1d561c58c96de29b20
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 6 15:17:49 2022 +0100

    Make errors more useful

commit 64befe7c9a901b00650442171660c2dc4ea575fa
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 6 15:02:40 2022 +0100

    Tweak ordering a bit
2022-06-06 15:18:02 +01:00
Neil Alexander 02597f15f0
Fix panic in QueryRestrictedJoinAllowed 2022-06-06 08:56:06 +01:00
Till 3e9c734da5
Make setting state idempotent (#2512)
* Make Setting state twice is idempotent pass

* Add passing tests

* PR comment & comments
2022-06-03 06:43:51 +02:00
Neil Alexander f41931b566
Version 0.8.7 (#2509) 2022-06-01 11:40:37 +01:00
Neil Alexander 02e5c74101
Revert #2457
Squashed commit of the following:

commit 2bd0daf4d61376d2dd56628eaff267b0bc63e116
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Jun 1 09:55:54 2022 +0100

    Revert resolving old extremities as well as new

    This may no longer be needed with the new state fixes and probably just burns more CPU time than is strictly necessary.
2022-06-01 10:09:27 +01:00