Commit graph

1832 commits

Author SHA1 Message Date
sumitks866 dd1e31bee7
Mention unstable features in /_matrix/client/versions (#1710)
* Mention unstable features in /_matrix/client/versions

* list enabled mscs

* Don't update go.mod/go.sum

* goimports

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-01-18 13:09:28 +00:00
Neil Alexander 534c29ab02
Log event ID on consumer errors (fixes #1714) 2021-01-18 12:58:48 +00:00
Caleb Xavier Berger efc91146f0
Use go build instead of go install (#1716)
* Use go build instead of go install

go install doesn't like to cross-compile things. (Try running build.sh with GOARCH set to something other than what it "should" be.)

With go build, it appears that cross-compilation is really, really straightforward. Simply install a compiler for your target platform and set `GOARCH` and `CC` accordingly.

* Use shell expansion instead of loop

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-01-18 12:44:44 +00:00
Loïck Bonniot 940577cd3c
Fix integer overflow in device_list_update.go (#1717)
Fix #1511
On 32-bits systems, int(hash.Sum32()) can be negative.
This makes the computation of array indices using modulo invalid, crashing dendrite.
Signed-off-by: Loïck Bonniot <git@lesterpig.com>
2021-01-18 12:43:15 +00:00
Neil Alexander cf82e08096
Update GHA Docker Hub builds
Squashed commit of the following:

commit 4a61aa711473deece2adf415cfd65501dbca63b2
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jan 18 12:19:24 2021 +0000

    Set back to matrixdotorg on published releases

commit 6d1ac53f2c0c9b30e1e70c0bb1559e1b8ec874a2
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jan 18 11:55:28 2021 +0000

    Rename

commit 258999f7fb7b655b3a02a06a7ea05e66fb7740fb
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jan 18 11:52:26 2021 +0000

    Refactor multi-stage builds

commit c7ab8e476939899571e7b5668860dec372b9b60f
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jan 18 11:13:19 2021 +0000

    Let's try this again
2021-01-18 12:24:23 +00:00
Neil Alexander df4386f764
Build docker branch only 2021-01-18 11:03:50 +00:00
Neil Alexander 833e01a612
Update docker-build-and-push.yml 2021-01-18 10:26:36 +00:00
TR_SLimey ff94490384
GitHub action for automatic multiarch Docker build (#1613)
* Create docker-build-and-push.yml

* Switched to using official Docker buildx action

* Added comment to docker-build-and-push.yml
In case something needs to be tweaked in the future, the link
contains some examples and explanations which would be useful

* Run only on release (and produce release tags)

As this workflow takes quite a lot of time, and [pushing to master happens frequently](https://github.com/matrix-org/dendrite/pull/1613#issuecomment-746086980), the container will now only be built when a release is created, and the builds will also be correctly tagged.

* Add latest tag, test at neilalexander/dendrite*

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-01-18 10:23:53 +00:00
Neil Alexander 6dadb1c06b
Fix lint error in generate-keys 2021-01-18 09:34:49 +00:00
Kegan Dougal fdd534f86a Escape the create event content key so it can be extracted correctly 2021-01-15 18:49:21 +00:00
Kegan Dougal 2113da5332 Enable MSC 2946 in CI 2021-01-15 17:56:50 +00:00
Kegsay 5366c33885
MSC2946: Use new MSC1772 format for space parents (#1713) 2021-01-15 12:30:41 +00:00
Kegsay 2626525c65
MSC2946: Allow redactions/updates for space state events (#1712) 2021-01-15 11:44:36 +00:00
rany ef24ea2678
Add madvdontneed=1 to decrease RAM consumption for systemd service (#1690)
* Add madvdontneed=1 to decrease RAM consumption for systemd service

* Update monolith-example.service

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-01-14 10:31:45 +00:00
Kegan Dougal e30c523089 Per request cache for now as we don't do batching correclty 2021-01-13 23:00:25 +00:00
Kegan Dougal 19392c8c6b Decode uri values correctly 2021-01-13 22:27:04 +00:00
Kegsay 3183f75aed
MSC2946: Spaces Summary (#1700)
* Add stub functions for MSC2946

* Implement core space walking algorithm

* Flesh out stub functions; add test stubs

* Implement storage bits and add sanity check test

* Implement world_readable auth with test

* Linting
2021-01-13 18:00:38 +00:00
Neil Alexander 3ac693c7a5
Add dendrite_roomserver_processroomevent_duration_millis to prometheus
Squashed commit of the following:

commit e5e2d793119733ecbcf9b85f966e018ab0318741
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Jan 13 17:28:12 2021 +0000

    Add dendrite_roomserver_processroomevent_duration_millis to prometheus
2021-01-13 17:31:46 +00:00
Neil Alexander 266f9c4abd
Update to matrix-org/gomatrixserverlib#247 - cache server well-known/DNS addresses 2021-01-13 17:31:13 +00:00
Neil Alexander d8fba52e97
Simplify send-to-device messaging (#1702)
* Simplify send-to-device messaging

* Don't return error if there's no work to do

* Remove SQLite migrations for now

* Tweak Postgres migrations

* Tweaks

* Fixes

* Cleanup separately

* Fix SQLite migration
2021-01-13 17:29:46 +00:00
Neil Alexander bb9e6a1281
Sync fixes (#1709)
* omitempty some fields in sync

* Add a few more

* Don't send push rules over and over again in incremental sync

* Further tweaks
2021-01-13 14:32:49 +00:00
Neil Alexander 55cfe391f7
Update /messages pagination token behaviour (#1708)
* Tweak pagination tokens

* start should be the specified from

* Don't reverse start and end

* Tweak getStartEnd again

* Update sytest-whitelist

* NOTSPEC: Re-add iOS end of topology
2021-01-13 12:59:29 +00:00
Neil Alexander fa65c40bae
Reduce device list GetUserDevices timeout (#1704) 2021-01-12 16:13:21 +00:00
Neil Alexander ec1b017906
Version 0.3.5 2021-01-11 13:55:40 +00:00
Neil Alexander e1e34b8994
Deep-checking forward extremities (#1698) 2021-01-11 12:47:25 +00:00
Neil Alexander 0fbebecd41
Tweak ApplyUpdates (#1691) 2021-01-09 11:25:09 +00:00
Neil Alexander b5a8935042
Sync refactor — Part 1 (#1688)
* It's half-alive

* Wakeups largely working

* Other tweaks, typing works

* Fix bugs, add receipt stream

* Delete notifier, other tweaks

* Dedupe a bit, add a template for the invite stream

* Clean up, add templates for other streams

* Don't leak channels

* Bring forward some more PDU logic, clean up other places

* Add some more wakeups

* Use addRoomDeltaToResponse

* Log tweaks, typing fixed?

* Fix timed out syncs

* Don't reset next batch position on timeout

* Add account data stream/position

* End of day

* Fix complete sync for receipt, typing

* Streams package

* Clean up a bit

* Complete sync send-to-device

* Don't drop errors

* More lightweight notifications

* Fix typing positions

* Don't advance position on remove again unless needed

* Device list updates

* Advance account data position

* Use limit for incremental sync

* Limit fixes, amongst other things

* Remove some fmt.Println

* Tweaks

* Re-add notifier

* Fix invite position

* Fixes

* Notify account data without advancing PDU position in notifier

* Apply account data position

* Get initial position for account data

* Fix position update

* Fix complete sync positions

* Review comments @Kegsay

* Room consumer parameters
2021-01-08 16:59:06 +00:00
Jan Pechek 56a7839aed
Fixed links to Docker Hub (#1686) 2021-01-07 13:02:15 +00:00
Jason Robinson e3ce6a924f
Add MSC's section to example config (#1675)
The config section for MSC's was missing from the example config. Even though these are experimental features, it might be good to add the section into the example config for visibility.

Signed-off-by: Jason Robinson <mail@jasonrobinson.me>
2021-01-05 11:31:10 +00:00
Neil Alexander 05324b6861
Send/state tweaks (#1681)
* Check missing event count

* Don't use request context for /send
2021-01-04 13:47:48 +00:00
Jason Robinson 597350a67f
Ensure appservices can auth as users in their namespaces (#1672)
* Ensure appservices can auth as users in their namespaces

Currently in Dendrite appservices can only auth as a user if the user was created by said appservice. This does not align with the appservices spec which specifically says:

> The application service may specify the virtual user to act as through use of a user_id query string parameter on the request. The user specified in the query string must be covered by one of the application service’s user namespaces.

https://matrix.org/docs/spec/application_service/r0.1.2#identity-assertion

In the case that a user has been created for example via manual registration but belongs to an appservice namespace, the current functionality does not allow appservices to auth as them. This PR fixes that by replacing the appservice ID check with a check against the appservice namespace.

This also matches Synapse functionality, which I confirmed to allow appservices to auth as a user in their namespace, irregardless of how the user was registered.

* Also allow appservice itself to auth with user_id

Appservice user_id + access token check needs to work both when user_id is the appservice and when appservice has the user in their user namespace.

Signed-off-by: Jason Robinson <mail@jasonrobinson.me>
2021-01-04 10:43:23 +00:00
Will Hunt e7f2d770df
Update docker-compose.deps.yml to use postgres:11 (#1677) 2021-01-04 10:36:36 +00:00
6543 a85051ca9d
fix imports (#1665)
* fix imports

Signed-off-by: 6543 <6543@obermui.de>

* add sqlite driver import back

Signed-off-by: 6543 <6543@obermui.de>

* rm import of userapi/storage/accounts/sqlite3/storage.go
2020-12-21 14:27:01 +00:00
Neil Alexander 4a0461378a
Version 0.3.4 2020-12-18 14:53:31 +00:00
Neil Alexander 932ff6b166
Fix /register 2020-12-18 14:51:30 +00:00
Will Hunt e4d1f0958f
Do not check if a username is exclusive if the request is for an appservice in /register (#1652)
* Do not check if a username is exclusive if the request is for an appservice in /register

* remove useless comment

* Move statements

* fix broken test

* Also fix the senderLocalpart problem

* Check domain name is ours

* Handle accessTokenErr

* Return unauthorised instead of forbidden

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-12-18 14:36:59 +00:00
Neil Alexander 4fe4c180e6
Ensure we wake for our own device list updates (#1661)
* Make sure we wake up for our own key changes

* Whitelist 'Users receive device_list updates for their own devices'
2020-12-18 14:04:17 +00:00
Neil Alexander fac71edc62
Fix #1655 by re-adding the appservice alias query (#1660) 2020-12-18 13:33:28 +00:00
Neil Alexander 65ff5c9a2c
Don't send null in joined_rooms response (#1659) 2020-12-18 13:33:08 +00:00
Neil Alexander 50963b724b
More sane next batch handling, typing notification tweaks, give invites their own stream position, device list fix (#1641)
* Update sync responses

* Fix positions, add ApplyUpdates

* Fix MarshalText as non-pointer, PrevBatch is optional

* Increment by number of read receipts

* Merge branch 'master' into neilalexander/devicelist

* Tweak typing

* Include keyserver position tweak

* Fix typing next position in all cases

* Tweaks

* Fix typo

* Tweaks, restore StreamingToken.MarshalText which somehow went missing?

* Rely on positions from notifier rather than manually advancing them

* Revert "Rely on positions from notifier rather than manually advancing them"

This reverts commit 53112a62cc.

* Give invites their own position, fix other things

* Fix test

* Fix invites maybe

* Un-whitelist tests that look to be genuinely wrong

* Use real receipt positions

* Ensure send-to-device uses real positions too
2020-12-18 11:11:21 +00:00
Kegsay a518e2971a
Fix flakey complement (#1653) 2020-12-17 18:25:51 +00:00
Neil Alexander 9a1ef81fa2
Add CS sendevent metrics
Squashed commit of the following:

commit c38c39107b6dda0c8e6320d61da2365c47eea4e9
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Dec 17 10:13:09 2020 +0000

    Update buckets

commit 5a3bcc8bd4167150374827b0e42a0dea0366beff
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Dec 17 09:41:43 2020 +0000

    Update buckets

commit 78dff8165ddf596e28af04faf56466752ebc17af
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Dec 17 09:37:27 2020 +0000

    Register sendEventDuration

commit 612b0f3d84f1547ff30131f7b084baf609edab52
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Dec 17 09:35:59 2020 +0000

    sendevent metrics
2020-12-17 10:52:31 +00:00
Neil Alexander e1ace7e44a
Add event ID index on current state table (helps performance) (#1649) 2020-12-16 18:16:39 +00:00
Kegsay 76becac003
Add start_stream to /messages (#1648) 2020-12-16 18:10:39 +00:00
Kegsay 42e9cbf342
NOTSPEC: Make ?from= optional in /messages (#1647) 2020-12-16 17:31:03 +00:00
Neil Alexander 56b5847c74
Add prometheus metrics for destination queues, sync requests
Squashed commit of the following:

commit 7ed1c6cfe67429dbe378a763d832c150eb0f781d
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Dec 16 14:53:27 2020 +0000

    Updates

commit 8442099d08760b8d086e6d58f9f30284e378a2cd
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Dec 16 14:43:18 2020 +0000

    Add some sync statistics

commit ffe2a11644ed3d5297d1775a680886c574143fdb
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Dec 16 14:37:00 2020 +0000

    Fix backing off display

commit 27443a93855aa60a49806ecabbf9b09f818301bd
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Dec 16 14:28:43 2020 +0000

    Add some destination queue metrics
2020-12-16 15:02:39 +00:00
Neil Alexander b891c00b09
Add RoomInfo cache, remove RoomServerRoomNIDsCache (#1646)
* Add RoomInfo cache, remove RoomServerRoomNID cache, ensure caches are thread-safe

* Don't panic if the roomInfo isn't known yet

* LRU package is already threadsafe

* Use RoomInfo cache to find room version if possible in Events()

* Adding comments about RoomInfoCache safety
2020-12-16 12:15:12 +00:00
Neil Alexander 9057143033
Hit the database far less in Events to find room NIDs and room versions (#1643)
* Hit the database far less to find room NIDs for event NIDs

* Close the rows

* Fix SQLite selectRoomNIDsForEventNIDsSQL

* Give same treatment to room version lookups
2020-12-16 10:33:28 +00:00
Neil Alexander d2bcc5f746
Fix main.wasm output path (fixes #1645) 2020-12-16 09:33:48 +00:00
Neil Alexander 38318b0f16
De-map device list positions in streaming tokens (#1642)
* De-map device list positions in streaming tokens

* Fix lint error

* Tweak toOffset
2020-12-15 15:09:10 +00:00