Commit graph

2532 commits

Author SHA1 Message Date
Neil Alexander 6a35d9f1b1
Version 0.3.11 2021-03-02 11:46:35 +00:00
Neil Alexander 81312b8a78
Return the current OTK count on an empty upload request (#1774)
* Always return OTK counts

* Fix parameter ordering

* Send IDs over to keyserver internal API

* Review comments

* Fix syntax error

* Fix panic, hopefully

* Require user ID to be set

* Fix user API call
2021-03-02 11:40:20 +00:00
Neil Alexander f5cf241877
Fix user registration bug (#1777) 2021-03-02 10:43:25 +00:00
Neil Alexander 3069079e37
Revert quic-go dependency update 2021-02-18 10:18:14 +00:00
Neil Alexander db637515a5
Update libp2p dependencies 2021-02-18 10:14:24 +00:00
Neil Alexander 11830fd4af
Fix transitive dependency for P2P demo 2021-02-18 09:45:55 +00:00
Neil Alexander 4c0103a2d5
Don't close channels when clearing queue (we might race and panic, when the GC will still clean it up for us anyway) 2021-02-18 09:26:40 +00:00
Neil Alexander 2fdc318f2c
Version 0.3.10 2021-02-17 16:05:04 +00:00
Kegan Dougal bf9c530fdb Unbreak DendriteJS.Dockerfile; Riot is now Element 2021-02-17 15:45:50 +00:00
Marco Kundt d1496793b9
fix database names to reflect renaming (#1636)
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-02-17 15:20:06 +00:00
Neil Alexander c9f305f254
Don't exclude an event from sync if it was previously not excluded (#1767) 2021-02-17 15:18:41 +00:00
Neil Alexander 8b5cd256cb
Don't hold destination queues in memory forever (#1769)
* Don't hold destination queues in memory forever

* Close channels

* Fix ordering

* Clear more aggressively

* clearQueue only called by defer so should be safe to delete queue in any case

* Wake queue when created, otherwise cleanup doesn't get called in all cases

* Clean up periodically, we hit a race condition otherwise

* Tweaks

* Don't create queues for blacklisted hosts

* Check blacklist properly
2021-02-17 15:16:35 +00:00
darkgallium da797c7998
Retrieve remote file size without the Content-Size header (#1537)
* Get downloaded file size regardless of the Content-Size header

Signed-off-by: Florian Le Minoux <darkgallium+git@deuxfleurs.fr>

* fixing lint issues

* Don't exhaust memory for large files, don't limit more than necessary

* Don't use errors.Wrap in download.go

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-02-17 13:54:53 +00:00
Neil Alexander 2386e0c7af
Gradually evict oldest cache entries (#1768)
* Gradually evict oldest cache entries

* Keep the remaining 10% of cached entries
2021-02-17 13:50:27 +00:00
Neil Alexander 5d74a1757f
Don't query for servers so often in /send (#1766)
* Look up servers less often, don't hit API for missing auth events unless there are actually missing auth events

* Remove ResolveConflictsAdhoc (since it is already in GMSL), other tweaks

* Update gomatrixserverlib to matrix-org/gomatrixserverlib#254

* Fix resolve-state

* Initialise t.servers on first use
2021-02-16 17:12:17 +00:00
Michael Telatynski f448e8972a
Implement /createRoom power_level_content_override (#1761)
* Implement /createRoom power_level_content_override

* delint

* Skip having to re-marshal PowerLevelContentOverride by making use of `json.RawMessage`
2021-02-12 09:43:31 +00:00
David Florness 85aaaf9bb9
Fix a few indentation mistakes (#1752)
Signed-off-by: David Florness <david@florness.com>
2021-02-12 09:25:35 +00:00
Kegan Dougal 9a199ba179 Remove 100 default 2021-02-06 17:05:00 +00:00
Kegan Dougal 82df194858 Increase limit 2021-02-06 16:56:55 +00:00
Neil Alexander 02e6d89cc2
Fix crash in membership updater (#1753)
* Fix nil pointer exception in membership updater

* goimports
2021-02-06 11:49:18 +00:00
Neil Alexander 397b158c05
Version 0.3.9 2021-02-04 12:39:43 +00:00
Neil Alexander bd72ed50d4
Reduce log level of 'Failed to send transaction' log line, since quite often it is flooding logs for dead servers 2021-02-04 12:25:31 +00:00
Neil Alexander 6e44450cc9
Don't re-request state events that are already in the timeline (#1739)
* Don't request state events if we already have the timeline events (Postgres only)

* Rename variable

* nocyclo

* Add SQLite

* Tweaks

* Revert query change

* Don't dedupe if asking for full state

* Update query
2021-02-04 12:20:37 +00:00
Neil Alexander 6099379ea4
Remove rooms table from federation sender (#1751)
* Remove last sent event ID column from federation sender

* Remove EventIDMismatchError

* Remove the federationsender rooms table altogether, it's useless

* Add migration

* Fix migrations

* Fix migrations
2021-02-04 11:52:49 +00:00
Neil Alexander b7e3b81a22
Fix ON CONFLICT on sync API account data (#1745) (#1750) 2021-02-04 11:45:49 +00:00
Neil Alexander de5f22a469
Remove redundant check (#1748) 2021-02-04 11:12:52 +00:00
Neil Alexander 369d3939fd
Drop state events we can't auth instead of failing altogether 2021-01-29 16:33:59 +00:00
Neil Alexander 62a325ded8
Complete sync performance (#1741)
* Parallelise PDU stream fetching for complete sync

* Fixes

* Fixes

* Worker queue

* Workers

* Don't populate device list changes on complete sync

* Don't fast-forward typing notifications either on complete sync

* Revert "Don't fast-forward typing notifications either on complete sync"
This reverts commit 01471f7843.

* Comments
2021-01-29 16:32:54 +00:00
Kegsay 6d1c6f29e0
Add m.room.create to invite stripped state (#1740)
MSC1772 needs this because the create event contains info on if
the room is a space or not. The create event itself isn't sensitive
so other people may find this useful too.
2021-01-29 11:36:26 +00:00
Neil Alexander 61090cbe35
Version 0.3.8 2021-01-28 13:37:50 +00:00
Neil Alexander d56a5a6509
Fix well-known lookups 2021-01-28 13:25:45 +00:00
Neil Alexander e14e9a1606
Version 0.3.7 2021-01-26 13:13:48 +00:00
Neil Alexander 9f443317bc
Graceful shutdowns (#1734)
* Initial graceful stop

* Fix dendritejs

* Use process context for outbound federation requests in destination queues

* Reduce logging

* Fix log level
2021-01-26 12:56:20 +00:00
Neil Alexander 64fb6de6d4
Don't retrieve same state events over and over again (#1737) 2021-01-26 09:12:17 +00:00
Kegsay ef9d5ad4fe
Check peek state response and refactor checking send_join response (#1732) 2021-01-22 17:16:35 +00:00
Neil Alexander 6757b67a32
NewClient and NewFederationClient updates (#1730)
* Use matrix-org/gomatrixserverlib#252

* Add missing WithSkipVerify to test

* Functions instead

* Update gomatrixserverlib to matrix-org/gomatrixserverlib#252

* Fix disabling TLS validation
2021-01-22 16:09:05 +00:00
Kegsay 93942f8ab6
Gate peeking behind msc flags (#1731) 2021-01-22 16:08:47 +00:00
Matthew Hodgson 0571d395b5
Peeking over federation via MSC2444 (#1391)
* a very very WIP first cut of peeking via MSC2753.

doesn't yet compile or work.
needs to actually add the peeking block into the sync response.
checking in now before it gets any bigger, and to gather any initial feedback on the vague shape of it.

* make PeekingDeviceSet private

* add server_name param

* blind stab at adding a `peek` section to /sync

* make it build

* make it launch

* add peeking to getResponseWithPDUsForCompleteSync

* cancel any peeks when we join a room

* spell out how to runoutside of docker if you want speed

* fix SQL

* remove unnecessary txn for SelectPeeks

* fix s/join/peek/ cargocult fail

* HACK: Track goroutine IDs to determine when we write by the wrong thread

To use: set `DENDRITE_TRACE_SQL=1` then grep for `unsafe`

* Track partition offsets and only log unsafe for non-selects

* Put redactions in the writer goroutine

* Update filters on writer goroutine

* wrap peek storage in goid hack

* use exclusive writer, and MarkPeeksAsOld more efficiently

* don't log ascii in binary at sql trace...

* strip out empty roomd deltas

* re-add txn to SelectPeeks

* re-add accidentally deleted field

* reject peeks for non-worldreadable rooms

* move perform_peek

* fix package

* correctly refactor perform_peek

* WIP of implementing MSC2444

* typo

* Revert "Merge branch 'kegan/HACK-goid-sqlite-db-is-locked' into matthew/peeking"

This reverts commit 3cebd8dbfb, reversing
changes made to ed4b3a58a7.

* (almost) make it build

* clean up bad merge

* support SendEventWithState with optional event

* fix build & lint

* fix build & lint

* reinstate federated peeks in the roomserver (doh)

* fix sql thinko

* todo for authenticating state returned by /peek

* support returning current state from QueryStateAndAuthChain

* handle SS /peek

* reimplement SS /peek to prod the RS to tell the FS about the peek

* rename RemotePeeks as OutboundPeeks

* rename remote_peeks_table as outbound_peeks_table

* add perform_handle_remote_peek.go

* flesh out federation doc

* add inbound peeks table and hook it up

* rename ambiguous RemotePeek as InboundPeek

* rename FSAPI's PerformPeek as PerformOutboundPeek

* setup inbound peeks db correctly

* fix api.SendEventWithState with no event

* track latestevent on /peek

* go fmt

* document the peek send stream race better

* fix SendEventWithRewrite not to bail if handed a non-state event

* add fixme

* switch SS /peek to use SendEventWithRewrite

* fix comment

* use reverse topo ordering to find latest extrem

* support postgres for federated peeking

* go fmt

* back out bogus go.mod change

* Fix performOutboundPeekUsingServer

* Fix getAuthChain -> GetAuthChain

* Fix build issues

* Fix build again

* Fix getAuthChain -> GetAuthChain

* Don't repeat outbound peeks for the same room ID to the same servers

* Fix lint

* Don't omitempty to appease sytest

Co-authored-by: Kegan Dougal <kegan@matrix.org>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-01-22 14:55:08 +00:00
Neil Alexander 5d8ec0ff1a
Fix build error 2021-01-22 14:18:36 +00:00
Neil Alexander 805a74892e
DNS caching (#1728)
* Allow configuring DNS cache

* Update sample configs

* Fix build errors

* Fix time resolution

* Default 5m

* In seconds

* Use WithDNScache

* Correct field name

* Update go.mod/go.sum to matrix-org/gomatrixserverlib#251
2021-01-22 14:16:59 +00:00
Neil Alexander 745ee20b90
Update sytest-whitelist after matrix-org/sytest#991 2021-01-22 12:56:20 +00:00
Neil Alexander 58f78e08af
Limit well-known response sizes (matrix-org/gomatrixserverlib#250) 2021-01-21 13:57:41 +00:00
Neil Alexander 8fe51019ad
Add sync API memberships table (#1726) 2021-01-20 20:43:20 +00:00
Kegsay c08e38df2c
MSC2946: Treat federation responses the same way as local responses (#1724)
* Start treating fed rooms/events the same as local rooms/events

* Share more code
2021-01-20 17:03:35 +00:00
Neil Alexander b70238f2d5
Basic sync filtering (#1721)
* Add some filtering (postgres only for now)

* Fix build error

* Try to use request filter

* Use default filter as a template when retrieving from the database

* Remove unused strut

* Update sytest-whitelist

* Add filtering to SelectEarlyEvents

* Fix Postgres selectEarlyEvents query

* Attempt filtering on SQLite

* Test limit, set field for limit/order in prepareWithFilters

* Remove debug logging, add comments

* Tweaks, debug logging

* Separate SQLite stream IDs

* Fix filtering in current state table

* Fix lock issues

* More tweaks

* Current state requires room ID

* Review comments
2021-01-19 18:00:42 +00:00
Kegsay 80aa9aa8b0
Implement MSC2946 over federation (#1722)
* Add fedsender dep on msc2946

* Add MSC2946Spaces to fsAPI

* Add exclude_rooms impl

* Implement fed spaces handler

* Use stripped state not room version

* Call federated spaces at the right time
2021-01-19 17:14:25 +00:00
Neil Alexander ccfcb2d280
Create bin dir 2021-01-18 15:09:54 +00:00
Neil Alexander f6151b6290
Use dendritegithub for Docker Hub images 2021-01-18 14:00:47 +00:00
Neil Alexander 67c89b3c5e
Version 0.3.6 2021-01-18 13:37:49 +00:00
Neil Alexander 244ff0dccb
Don't create so many state snapshots when updating forward extremities (#1718)
* Light-weight checking of state changes when updating forward extremities

* Only do this for non-state events, since state events will always result in state change at extremities
2021-01-18 13:21:33 +00:00