Commit graph

2301 commits

Author SHA1 Message Date
Till 29f2168789
Make /messages filterable (#2347)
* Make /messages filterable
Fix bug when determining if an event contains an URL

* Add newly passing test

* Fix test
2022-04-13 13:16:02 +02:00
kegsay ea92f80c12
Add database namespacing for unit tests (#2340)
* Add database namespacing for unit tests

Background: Running `go test ./...` will run tests in different packages concurrently.
This can be stopped or limited by using `-p 1` (no concurrency). We want concurrency,
but this causes problems when running Postgres DBs in CI. The problem is that, in CI,
we have 1x postgres server exposing 1x postgres DB, which we wipe clean at the end of
each test via `defer close()`. When tests run concurrently, calls to `close()` will
delete data/tables which other tests are currently using, causing havoc.

Fix this by:
 - Creating a database per package.
 - Namespacing the database name by a hash of the current working directory (the directory containing those `_test.go` files)

This is exactly what SQLite does, quite unintentionally, via the use of `file:dendrite_test.db`,
which dumps the file into the current working directory which is the package running the tests,
hence deleting the file is safe when running concurrently.

* Linting

* Don't create the database in a txn

* dupe db is not an error
2022-04-11 10:23:01 +01:00
Till 69f2ff7c82
Correctly use provided filters (#2339)
* Apply filters correctly

* Fix issues; Use prepareWithFilters

* Update gmsl & tests

* go.mod..

* PR comments
2022-04-11 09:05:23 +02:00
kegsay b4b2fbc36b Remove dead code in the sync api (#2341) 2022-04-09 00:37:50 +01:00
kegsay 6d25bd6ca5
syncapi: add more tests; fix more bugs (#2338)
* syncapi: add more tests; fix more bugs

bugfixes:
 - The postgres impl of TopologyTable.SelectEventIDsInRange did not use the provided txn
 - The postgres impl of EventsTable.SelectEvents did not preserve the ordering of the input event IDs in the output events slice
 - The sqlite impl of EventsTable.SelectEvents did not use a bulk `IN ($1)` query.

Added tests:
 - `TestGetEventsInRangeWithTopologyToken`
 - `TestOutputRoomEventsTable`
 - `TestTopologyTable`

* -p 1 for now
2022-04-08 17:53:24 +01:00
Neil Alexander 986d27a128
Update to matrix-org/pinecone@2999ea2 2022-04-08 16:39:09 +01:00
Neil Alexander 9bd9f2beba
Update to matrix-org/pinecone@9b3963248c 2022-04-08 16:23:50 +01:00
Neil Alexander 6907e42167
Use connection manager in Pinecone demos 2022-04-08 13:55:46 +01:00
Neil Alexander 4312b87ab0
Update dependencies (#2336) 2022-04-08 12:24:49 +01:00
Till e8dd37d533
Add metrics for internal API requests (#2310)
* Add response size and requests total to internal handler

* Move MustRegister calls to New* funcs

* Move MustRegister back to init

* Init at some place, minimize changes
2022-04-08 12:24:40 +02:00
Neil Alexander 25d99c4473
Remove client-api-proxy and federation-api-proxy (closes #928) (#2335) 2022-04-08 10:59:51 +01:00
Neil Alexander f299f97e0a
Fix data race in TestCorrectStreamWakeup (#2334) 2022-04-08 10:46:36 +01:00
Neil Alexander b8c97431b9
Prevent setting push rules via account data endpoint (#2333) 2022-04-08 10:46:23 +01:00
PiotrKozimor 63b7c3da46
Handle QueryAccountAvailabilityPath in internal user API (#2332) 2022-04-08 10:18:27 +01:00
kegsay 7499147550
Add test infrastructure code for dendrite unit/integ tests (#2331)
* Add test infrastructure code for dendrite unit/integ tests

Start re-enabling some syncapi storage tests in the process.

* Linting

* Add postgres service to unit tests

* dendrite not syncv3

* Skip test which doesn't work

* Linting

* Add `jetstream.PrepareForTests`

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-08 10:12:30 +01:00
Neil Alexander 955e6eb307
Dendrite 0.8.1 2022-04-07 17:13:14 +01:00
Neil Alexander 7902859bc1
Fix lock contention in sync notifier 2022-04-07 17:08:15 +01:00
Neil Alexander 87b1ed1338
Version 0.8.0 (#2330)
* Version 0.8.0

* Update changelog again
2022-04-07 16:09:40 +01:00
Neil Alexander 2dc35403d1
Send avatar updates asynchronously, same as display name updates 2022-04-07 15:50:42 +01:00
Till 60ee7eef4c
Add possibility to ignore users (#2329)
* Add ignore users

* Ignore users in pushrules
Add passing tests

* Update sytest lists

* Store ignore knowledge in the sync API

* Fix copyrights

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-07 15:08:19 +01:00
Neil Alexander 99ef547295
Simplify presence stringification (should help with vector-im/element-android#5712) 2022-04-07 10:10:28 +01:00
Neil Alexander 99f2007eb6
Fix bug where state events could be requested twice in appservices (#2328)
* Fix bug where state events could be requested twice (should fix #2305)

* Avoid a pointless HTTP request if possible
2022-04-06 16:10:31 +01:00
Neil Alexander 850932c7a7
Update README.md 2022-04-06 15:23:09 +01:00
Neil Alexander c28f00bf96
Sync notifier tweaks (#2327)
* Micro-optimisations, lock fixes

* Refactor `SharedUsers`

* Reuse map to reduce allocations/GC pressure

* oh yeah, initialise it

* Leave room for the user ID we'll no doubt append afterward
2022-04-06 15:04:41 +01:00
Neil Alexander 602818460d
Reduce allocations in /sync presence stream (#2326)
* Reduce allocations on presence

* Try to reduce allocations further

* Tweak `IsSharedUser` some more

* Take map lock
2022-04-06 13:31:44 +01:00
Till e5e3350ce1
Add presence module V2 (#2312)
* Syncapi presence

* Clientapi http presence handler

* Why is this here?

* Missing files

* FederationAPI presence implementation

* Add new presence stream

* Pinecone update

* Pinecone update

* Add passing tests

* Make linter happy

* Add presence producer

* Add presence config option

* Set user to unavailable after x minutes

* Only set currently_active if online
Avoid unneeded presence updates when syncing

* Tweaks

* Query devices for last_active_ts
Fixes & tweaks

* Export SharedUsers/SharedUsers

* Presence stream in MemoryStorage

* Remove status_msg_nil

* Fix sytest crashes

* Make presence types const and use stringer for it

* Change options to allow inbound/outbound presence

* Fix option & typo

* Update configs

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-06 13:11:19 +02:00
Neil Alexander 16e2d243fc
Reduce the number of allocations made by localRoomMembers when consuming stream events for push notifications (#2324) 2022-04-06 10:43:54 +01:00
Neil Alexander f7109de500
Nuke a couple of tests for now 2022-04-05 16:45:01 +01:00
Neil Alexander b49a6757c8
Update the "Are We Synapse Yet?" list 2022-04-05 16:43:29 +01:00
Neil Alexander c874d0b2ce
Merge branch 'main' of github.com:matrix-org/dendrite 2022-04-05 14:43:58 +01:00
Neil Alexander 4d9d9cc9b1
Update to matrix-org/gomatrixserverlib#300 2022-04-05 14:43:44 +01:00
Till c84937b852
Add Are We Synapse Yet to GHA (#2321)
* Add Are We Synapse Yet to GHA

* Better output & add comments
2022-04-05 15:32:30 +02:00
Neil Alexander 47be39c18e
Update sytest-whitelist again 2022-04-05 11:49:29 +01:00
Emanuele Aliberti fb10633a80
build.sh to build.cmd (#2319)
* build.sh to build.cmd

convert and adapt for Windows users

* remove hardwired GO ARCH

* silence cmd.exe

* update directions for Windows

* Update INSTALL.md

Co-authored-by: emanuele.aliberti <emanuele.aliberti@mtka.eu>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-05 11:27:29 +01:00
David Spenler 2defc4249d
Added /upgrade endpoint (#2307)
* Added /upgrade endpoint

* fix

* Fix lints

* More lint lifex

* Move room upgrading to the roomserver

* Remove extraneous arg

* Fix HTTP API for `PerformUpgrade`

* Reduce number of API calls in `generateInitialEvents`, preserve membership fields

* Refactor `generateInitialEvents` to preserve old state events for all but the essential room setup events

* Handle ban events in the state transfer

* Refactor and comment `createTemporaryPowerLevels`

* Only send two power levels if we needed to override the levels, preserve miscellaneous fields in the create event

* Fix copyrights

* Review comments @S7evinK

* Update sytest whitelist

* Specify empty state keys, use `EventLevel`, remove unnecessary check on state copy

* Add comment to `restrictOldRoomPowerLevels`

* Ensure canonical aliases exist before clearing

* Copy invites as well as bans

* Fix return error on `m.room.tombstone` handling in client API

* Relax checks for well-formedness of join rules, membership event etc

Co-authored-by: Alex Kursell <alex@awk.run>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: kegsay <kegan@matrix.org>
2022-04-05 10:04:08 +01:00
Neil Alexander 562d742240
Update to matrix-org/gomatrixserverlib#299 2022-04-04 19:04:17 +01:00
Sebastian Spaeth 1783496423
remove obsolete config entry "federation_certificates" (#2318)
* Remove all federation_certificates occurencs

This configuration value has not been used since 2019 apparently, and indeed
it is never really used in the code base. So remove all traces of it from
the various configuration files.

Also remove the unused variable FederationCertificatePaths

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>

* setup/config/config_test.go: remove federation_sender config snippet

The federation_sender: section was folded into the federation_api some
time ago, and this seems to be the only leftover in the code base.
So remove it.
2022-04-04 17:31:41 +01:00
Neil Alexander f2f9d2916b
Fix dendritejs-pinecone build 2022-04-04 17:26:01 +01:00
Neil Alexander cee12a7ab0
Enhanced calm at startup of Pinecone demos 2022-04-04 15:35:47 +01:00
Neil Alexander c69159bda7
Update to matrix-org/pinecone@e526fa8 2022-04-04 15:16:02 +01:00
Neil Alexander 9b316ac64c
Slower federation warm-up (#2320)
* Wake destination queues gradually, rather than all at once

* Delay device list updates too

* Maximum two minute warmup period
2022-04-04 15:14:10 +01:00
Neil Alexander 6748a2a823
Fix sign-off link in CONTRIBUTING.md 2022-04-04 11:02:16 +01:00
S7evinK a0f5d8e1a5
Fix unlimitedSize uploads (#2317) 2022-04-04 10:32:53 +02:00
Neil Alexander cd8fac152e
Include joined and invite member counts in room summary (#2315)
* Include joined and invite member counts in room summary

This should fix #2314 and also fix the problem where some clients like Element Android, Fluffychat etc would display the wrong member count for a given room.

* Improve SQLite query precision

* Check existence of state key for membership events
2022-04-01 16:14:38 +01:00
Neil Alexander 8213b2ba30
Update Pinecone P2P demo 2022-03-30 15:01:22 +01:00
S7evinK 49dc49b232
Remove eduserver (#2306)
* Move receipt sending to own JetStream producer

* Move SendToDevice to producer

* Remove most parts of the EDU server

* Fix SendToDevice & copyrights

* Move structs, cleanup EDU Server traces

* Use HeadersOnly subscription

* Missing file

* Fix linter issues

* Move consumers to own files

* Rename durable consumer; Consumer cleanup

* Docs/config cleanup
2022-03-29 14:14:35 +02:00
Neil Alexander 7972915806
User directory for nearby Pinecone peers (P2P demo) (#2311)
* User directory for nearby Pinecone peers

* Fix mux routing

* Use config to determine which server notices user to exclude
2022-03-28 16:25:26 +01:00
Neil Alexander 0692be44d9
Fix account availability on register 2022-03-28 13:31:17 +01:00
Neil Alexander 8099bcbc8b
P2P demo tweaks 2022-03-28 11:44:25 +01:00
Neil Alexander 34b9c8c670
Ensure Dendrite has stopped in Pinecone demo Stop() 2022-03-28 10:55:53 +01:00