Commit graph

2813 commits

Author SHA1 Message Date
Neil Alexander 02e6d89cc2
Fix crash in membership updater (#1753)
* Fix nil pointer exception in membership updater

* goimports
2021-02-06 11:49:18 +00:00
Neil Alexander 397b158c05
Version 0.3.9 2021-02-04 12:39:43 +00:00
Neil Alexander bd72ed50d4
Reduce log level of 'Failed to send transaction' log line, since quite often it is flooding logs for dead servers 2021-02-04 12:25:31 +00:00
Neil Alexander 6e44450cc9
Don't re-request state events that are already in the timeline (#1739)
* Don't request state events if we already have the timeline events (Postgres only)

* Rename variable

* nocyclo

* Add SQLite

* Tweaks

* Revert query change

* Don't dedupe if asking for full state

* Update query
2021-02-04 12:20:37 +00:00
Neil Alexander 6099379ea4
Remove rooms table from federation sender (#1751)
* Remove last sent event ID column from federation sender

* Remove EventIDMismatchError

* Remove the federationsender rooms table altogether, it's useless

* Add migration

* Fix migrations

* Fix migrations
2021-02-04 11:52:49 +00:00
Neil Alexander b7e3b81a22
Fix ON CONFLICT on sync API account data (#1745) (#1750) 2021-02-04 11:45:49 +00:00
Neil Alexander de5f22a469
Remove redundant check (#1748) 2021-02-04 11:12:52 +00:00
Neil Alexander 369d3939fd
Drop state events we can't auth instead of failing altogether 2021-01-29 16:33:59 +00:00
Neil Alexander 62a325ded8
Complete sync performance (#1741)
* Parallelise PDU stream fetching for complete sync

* Fixes

* Fixes

* Worker queue

* Workers

* Don't populate device list changes on complete sync

* Don't fast-forward typing notifications either on complete sync

* Revert "Don't fast-forward typing notifications either on complete sync"
This reverts commit 01471f7843.

* Comments
2021-01-29 16:32:54 +00:00
Kegsay 6d1c6f29e0
Add m.room.create to invite stripped state (#1740)
MSC1772 needs this because the create event contains info on if
the room is a space or not. The create event itself isn't sensitive
so other people may find this useful too.
2021-01-29 11:36:26 +00:00
Neil Alexander 61090cbe35
Version 0.3.8 2021-01-28 13:37:50 +00:00
Neil Alexander d56a5a6509
Fix well-known lookups 2021-01-28 13:25:45 +00:00
Neil Alexander e14e9a1606
Version 0.3.7 2021-01-26 13:13:48 +00:00
Neil Alexander 9f443317bc
Graceful shutdowns (#1734)
* Initial graceful stop

* Fix dendritejs

* Use process context for outbound federation requests in destination queues

* Reduce logging

* Fix log level
2021-01-26 12:56:20 +00:00
Neil Alexander 64fb6de6d4
Don't retrieve same state events over and over again (#1737) 2021-01-26 09:12:17 +00:00
Kegsay ef9d5ad4fe
Check peek state response and refactor checking send_join response (#1732) 2021-01-22 17:16:35 +00:00
Neil Alexander 6757b67a32
NewClient and NewFederationClient updates (#1730)
* Use matrix-org/gomatrixserverlib#252

* Add missing WithSkipVerify to test

* Functions instead

* Update gomatrixserverlib to matrix-org/gomatrixserverlib#252

* Fix disabling TLS validation
2021-01-22 16:09:05 +00:00
Kegsay 93942f8ab6
Gate peeking behind msc flags (#1731) 2021-01-22 16:08:47 +00:00
Matthew Hodgson 0571d395b5
Peeking over federation via MSC2444 (#1391)
* a very very WIP first cut of peeking via MSC2753.

doesn't yet compile or work.
needs to actually add the peeking block into the sync response.
checking in now before it gets any bigger, and to gather any initial feedback on the vague shape of it.

* make PeekingDeviceSet private

* add server_name param

* blind stab at adding a `peek` section to /sync

* make it build

* make it launch

* add peeking to getResponseWithPDUsForCompleteSync

* cancel any peeks when we join a room

* spell out how to runoutside of docker if you want speed

* fix SQL

* remove unnecessary txn for SelectPeeks

* fix s/join/peek/ cargocult fail

* HACK: Track goroutine IDs to determine when we write by the wrong thread

To use: set `DENDRITE_TRACE_SQL=1` then grep for `unsafe`

* Track partition offsets and only log unsafe for non-selects

* Put redactions in the writer goroutine

* Update filters on writer goroutine

* wrap peek storage in goid hack

* use exclusive writer, and MarkPeeksAsOld more efficiently

* don't log ascii in binary at sql trace...

* strip out empty roomd deltas

* re-add txn to SelectPeeks

* re-add accidentally deleted field

* reject peeks for non-worldreadable rooms

* move perform_peek

* fix package

* correctly refactor perform_peek

* WIP of implementing MSC2444

* typo

* Revert "Merge branch 'kegan/HACK-goid-sqlite-db-is-locked' into matthew/peeking"

This reverts commit 3cebd8dbfb, reversing
changes made to ed4b3a58a7.

* (almost) make it build

* clean up bad merge

* support SendEventWithState with optional event

* fix build & lint

* fix build & lint

* reinstate federated peeks in the roomserver (doh)

* fix sql thinko

* todo for authenticating state returned by /peek

* support returning current state from QueryStateAndAuthChain

* handle SS /peek

* reimplement SS /peek to prod the RS to tell the FS about the peek

* rename RemotePeeks as OutboundPeeks

* rename remote_peeks_table as outbound_peeks_table

* add perform_handle_remote_peek.go

* flesh out federation doc

* add inbound peeks table and hook it up

* rename ambiguous RemotePeek as InboundPeek

* rename FSAPI's PerformPeek as PerformOutboundPeek

* setup inbound peeks db correctly

* fix api.SendEventWithState with no event

* track latestevent on /peek

* go fmt

* document the peek send stream race better

* fix SendEventWithRewrite not to bail if handed a non-state event

* add fixme

* switch SS /peek to use SendEventWithRewrite

* fix comment

* use reverse topo ordering to find latest extrem

* support postgres for federated peeking

* go fmt

* back out bogus go.mod change

* Fix performOutboundPeekUsingServer

* Fix getAuthChain -> GetAuthChain

* Fix build issues

* Fix build again

* Fix getAuthChain -> GetAuthChain

* Don't repeat outbound peeks for the same room ID to the same servers

* Fix lint

* Don't omitempty to appease sytest

Co-authored-by: Kegan Dougal <kegan@matrix.org>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-01-22 14:55:08 +00:00
Neil Alexander 5d8ec0ff1a
Fix build error 2021-01-22 14:18:36 +00:00
Neil Alexander 805a74892e
DNS caching (#1728)
* Allow configuring DNS cache

* Update sample configs

* Fix build errors

* Fix time resolution

* Default 5m

* In seconds

* Use WithDNScache

* Correct field name

* Update go.mod/go.sum to matrix-org/gomatrixserverlib#251
2021-01-22 14:16:59 +00:00
Neil Alexander 745ee20b90
Update sytest-whitelist after matrix-org/sytest#991 2021-01-22 12:56:20 +00:00
Neil Alexander 58f78e08af
Limit well-known response sizes (matrix-org/gomatrixserverlib#250) 2021-01-21 13:57:41 +00:00
Neil Alexander 8fe51019ad
Add sync API memberships table (#1726) 2021-01-20 20:43:20 +00:00
Kegsay c08e38df2c
MSC2946: Treat federation responses the same way as local responses (#1724)
* Start treating fed rooms/events the same as local rooms/events

* Share more code
2021-01-20 17:03:35 +00:00
Neil Alexander b70238f2d5
Basic sync filtering (#1721)
* Add some filtering (postgres only for now)

* Fix build error

* Try to use request filter

* Use default filter as a template when retrieving from the database

* Remove unused strut

* Update sytest-whitelist

* Add filtering to SelectEarlyEvents

* Fix Postgres selectEarlyEvents query

* Attempt filtering on SQLite

* Test limit, set field for limit/order in prepareWithFilters

* Remove debug logging, add comments

* Tweaks, debug logging

* Separate SQLite stream IDs

* Fix filtering in current state table

* Fix lock issues

* More tweaks

* Current state requires room ID

* Review comments
2021-01-19 18:00:42 +00:00
Kegsay 80aa9aa8b0
Implement MSC2946 over federation (#1722)
* Add fedsender dep on msc2946

* Add MSC2946Spaces to fsAPI

* Add exclude_rooms impl

* Implement fed spaces handler

* Use stripped state not room version

* Call federated spaces at the right time
2021-01-19 17:14:25 +00:00
Neil Alexander ccfcb2d280
Create bin dir 2021-01-18 15:09:54 +00:00
Neil Alexander f6151b6290
Use dendritegithub for Docker Hub images 2021-01-18 14:00:47 +00:00
Neil Alexander 67c89b3c5e
Version 0.3.6 2021-01-18 13:37:49 +00:00
Neil Alexander 244ff0dccb
Don't create so many state snapshots when updating forward extremities (#1718)
* Light-weight checking of state changes when updating forward extremities

* Only do this for non-state events, since state events will always result in state change at extremities
2021-01-18 13:21:33 +00:00
sumitks866 dd1e31bee7
Mention unstable features in /_matrix/client/versions (#1710)
* Mention unstable features in /_matrix/client/versions

* list enabled mscs

* Don't update go.mod/go.sum

* goimports

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-01-18 13:09:28 +00:00
Neil Alexander 534c29ab02
Log event ID on consumer errors (fixes #1714) 2021-01-18 12:58:48 +00:00
Caleb Xavier Berger efc91146f0
Use go build instead of go install (#1716)
* Use go build instead of go install

go install doesn't like to cross-compile things. (Try running build.sh with GOARCH set to something other than what it "should" be.)

With go build, it appears that cross-compilation is really, really straightforward. Simply install a compiler for your target platform and set `GOARCH` and `CC` accordingly.

* Use shell expansion instead of loop

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-01-18 12:44:44 +00:00
Loïck Bonniot 940577cd3c
Fix integer overflow in device_list_update.go (#1717)
Fix #1511
On 32-bits systems, int(hash.Sum32()) can be negative.
This makes the computation of array indices using modulo invalid, crashing dendrite.
Signed-off-by: Loïck Bonniot <git@lesterpig.com>
2021-01-18 12:43:15 +00:00
Neil Alexander cf82e08096
Update GHA Docker Hub builds
Squashed commit of the following:

commit 4a61aa711473deece2adf415cfd65501dbca63b2
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jan 18 12:19:24 2021 +0000

    Set back to matrixdotorg on published releases

commit 6d1ac53f2c0c9b30e1e70c0bb1559e1b8ec874a2
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jan 18 11:55:28 2021 +0000

    Rename

commit 258999f7fb7b655b3a02a06a7ea05e66fb7740fb
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jan 18 11:52:26 2021 +0000

    Refactor multi-stage builds

commit c7ab8e476939899571e7b5668860dec372b9b60f
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jan 18 11:13:19 2021 +0000

    Let's try this again
2021-01-18 12:24:23 +00:00
Neil Alexander df4386f764
Build docker branch only 2021-01-18 11:03:50 +00:00
Neil Alexander 833e01a612
Update docker-build-and-push.yml 2021-01-18 10:26:36 +00:00
TR_SLimey ff94490384
GitHub action for automatic multiarch Docker build (#1613)
* Create docker-build-and-push.yml

* Switched to using official Docker buildx action

* Added comment to docker-build-and-push.yml
In case something needs to be tweaked in the future, the link
contains some examples and explanations which would be useful

* Run only on release (and produce release tags)

As this workflow takes quite a lot of time, and [pushing to master happens frequently](https://github.com/matrix-org/dendrite/pull/1613#issuecomment-746086980), the container will now only be built when a release is created, and the builds will also be correctly tagged.

* Add latest tag, test at neilalexander/dendrite*

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-01-18 10:23:53 +00:00
Neil Alexander 6dadb1c06b
Fix lint error in generate-keys 2021-01-18 09:34:49 +00:00
Kegan Dougal fdd534f86a Escape the create event content key so it can be extracted correctly 2021-01-15 18:49:21 +00:00
Kegan Dougal 2113da5332 Enable MSC 2946 in CI 2021-01-15 17:56:50 +00:00
Kegsay 5366c33885
MSC2946: Use new MSC1772 format for space parents (#1713) 2021-01-15 12:30:41 +00:00
Kegsay 2626525c65
MSC2946: Allow redactions/updates for space state events (#1712) 2021-01-15 11:44:36 +00:00
rany ef24ea2678
Add madvdontneed=1 to decrease RAM consumption for systemd service (#1690)
* Add madvdontneed=1 to decrease RAM consumption for systemd service

* Update monolith-example.service

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-01-14 10:31:45 +00:00
Kegan Dougal e30c523089 Per request cache for now as we don't do batching correclty 2021-01-13 23:00:25 +00:00
Kegan Dougal 19392c8c6b Decode uri values correctly 2021-01-13 22:27:04 +00:00
Kegsay 3183f75aed
MSC2946: Spaces Summary (#1700)
* Add stub functions for MSC2946

* Implement core space walking algorithm

* Flesh out stub functions; add test stubs

* Implement storage bits and add sanity check test

* Implement world_readable auth with test

* Linting
2021-01-13 18:00:38 +00:00
Neil Alexander 3ac693c7a5
Add dendrite_roomserver_processroomevent_duration_millis to prometheus
Squashed commit of the following:

commit e5e2d793119733ecbcf9b85f966e018ab0318741
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Jan 13 17:28:12 2021 +0000

    Add dendrite_roomserver_processroomevent_duration_millis to prometheus
2021-01-13 17:31:46 +00:00
Neil Alexander 266f9c4abd
Update to matrix-org/gomatrixserverlib#247 - cache server well-known/DNS addresses 2021-01-13 17:31:13 +00:00