Commit graph

2491 commits

Author SHA1 Message Date
Neil Alexander 3f82bceb70
Don't try to talk to ourselves when finding missing events 2022-10-06 10:51:06 +01:00
Neil Alexander e53dcb25a9
Tweak logging for federated room joins 2022-10-06 10:07:13 +01:00
Neil Alexander 9ba3103f88
Document database connection limits 2022-10-05 14:54:19 +01:00
Till 0f777d421c
Remove empty fields from /sync response (#2755)
First attempt at removing empty fields from `/sync` responses. Needs
https://github.com/matrix-org/sytest/pull/1298 to keep Sytest happy.

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-10-05 13:47:13 +01:00
Neil Alexander c85bc3434f
Optimise QuerySharedUsers so that we can only work on local users (#2766)
Otherwise the sync API key change consumer wastes a lot of time trying
to wake up the notifiers for non-local users.
2022-10-05 12:47:53 +01:00
Neil Alexander 6f602bb096
Demote Failed to query device keys for some users warning to level=debug
Many of these warnings are due to dead servers and are quite annoying when they fill up the logs.
2022-10-05 11:16:05 +01:00
Till 8c0c3441d8
Add RoomEventType nats.Header to avoid unneeded unmarshalling (#2765) 2022-10-05 12:12:42 +02:00
Neil Alexander ebd137cf6b
Check PostgreSQL connection count (#2760)
This PR queries PostgreSQL for the `max_connections` and
`superuser_reserved_connections` settings and then ensures that
Dendrite's `max_open_conns` doesn't exceed the allowed value.

This is a really common source of configuration problems and can either
result in blocking queries or deadlocks, so it seems reasonable that we
complain as loudly as possible when it happens.
2022-10-05 11:07:17 +01:00
Till e070352293
Side effect import bleve analyzer languages (#2763)
... to actually allow different languages. Fixes #2761 
Binary size increases by ~1MB.
2022-10-05 11:14:33 +02:00
Neil Alexander 21f8881985
Add indexes that optimise selectStateInRangeSQL (#2764)
This gets rid of some expensive scans on `add_state_ids` and
`remove_state_ids`, turning them into much cheaper and faster index
scans instead.
2022-10-04 16:43:10 +01:00
Neil Alexander ae10aac456
Don't perform a federated join after invite if we are already joined to the room (#2762)
If we are already joined to the room then it shouldn't matter if you
were invited or not, so this looks like a bug.
2022-10-04 15:40:04 +01:00
Neil Alexander 3da182212e
Track reasons why the process is in a degraded state 2022-10-04 13:02:41 +01:00
Neil Alexander a767102f8a
Reduce max_open_conns in monolith sample config 2022-10-04 11:34:52 +01:00
Neil Alexander 085bf5e28b
Revert Docker changes 2022-10-04 11:33:05 +01:00
Neil Alexander 98b73652e0
Try to populate -ldflags in Docker builds 2022-10-04 11:07:54 +01:00
Neil Alexander ede4632835
Fix Docker GHA 2022-10-04 10:43:58 +01:00
Till e6c992ba8b
Update Dockerfile (#2342)
Updates/adds a new multistage (build-kit) Dockerfile. (if accepted,
could make `Dockerfile.monolith` and `Dockerfile.polylith` in
`build/docker` obsolete)
There's no huge difference between the dockerfiles, except this uses a
non-root user when running the container, also doesn't copy the working
directory to the image when building.
Also adds vulnerabilities scans using
[Trivy](https://github.com/aquasecurity/trivy) for the created docker
images. (untested)

Building images is done using 
```
docker build . --target image-monolith -t dendrite-monolith
docker build . --target image-polylith -t dendrite-polylith
```

As noted in the comments, only adds `dendrite-polylith-multi` to the
polylith image and all required binaries to the monolith image.
Probably needs some docs updating, if this is accepted.

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-10-04 10:41:06 +01:00
Neil Alexander 34ed316584
Fix docs 2022-10-03 19:35:26 +01:00
Neil Alexander ba66b5a3b9
Allow multiple static peers in Pinecone iOS/Android demos 2022-10-03 14:43:38 +01:00
Neil Alexander 34451d21b8
P2P demo tweaks 2022-10-03 14:35:10 +01:00
Neil Alexander 50fa50a343
Update P2P base directories 2022-10-03 13:05:58 +01:00
Neil Alexander fec3ee384b
Stop CPU burn in PerformMarkAsStaleIfNeeded 2022-10-03 12:59:56 +01:00
Ashley Nelson c1e16fd41e
Fix fragility of selectEventsWithEventIDsSQL queries (#2757)
This fixes a temporary workaround with the `selectEventsWithEventIDsSQL`
queries where fields need to be artificially added to the queries so the
row results match the format of the `syncapi_output_room_events` table.
I made similar functions that accept row results from the
`syncapi_current_room_state` table and convert them into StreamEvents
without the fields that are specific to output room events.

There is also a unit test in the first commit to ensure the resulting
behavior doesn't change from the modified queries and functions.

Fixes #601.

### Pull Request Checklist

<!-- Please read docs/CONTRIBUTING.md before submitting your pull
request -->

* [x] I have added tests for PR _or_ I have justified why this PR
doesn't need tests.
* [x] Pull request includes a [sign
off](https://github.com/matrix-org/dendrite/blob/main/docs/CONTRIBUTING.md#sign-off)

Signed-off-by: `Ashley Nelson <fant@shley.email>`

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-10-03 11:57:21 +01:00
Neil Alexander b7f4bab358
Hopefully fix P2P --config error (re. #2756) 2022-10-03 11:38:31 +01:00
Neil Alexander d32f60249d
Modify sync transaction behaviour (#2758)
This now uses a transaction per stream, so that errors in one stream
don't propagate to another, and we therefore no longer need to do hacks
to reopen a new transaction after aborting a failed one.
2022-10-03 11:38:20 +01:00
Till Faelligen d4710217f8
Use non-HTTPS as default URL, as most people will be running behind a
reverse proxy
2022-10-02 11:31:40 +02:00
Neil Alexander a050503d8d
Version 0.10.1 2022-09-30 17:15:43 +01:00
Neil Alexander 0116db79c6
Reset transaction after a failure 2022-09-30 17:07:37 +01:00
Neil Alexander 16048be236
Handle case when applying history visibility failed 2022-09-30 16:55:10 +01:00
Neil Alexander 7d9545ceea
Another /sync fix 2022-09-30 16:34:06 +01:00
Till Faelligen 3617d5a0ff
Remove SendAccountData, since InputAccountData is already doing that 2022-09-30 17:31:41 +02:00
Neil Alexander 60ec9180e6
Update NATS Server to v2.9.2 2022-09-30 16:10:44 +01:00
Neil Alexander ee40a29e55
Fix broken /sync due to transaction error 2022-09-30 16:07:18 +01:00
Neil Alexander aa8ec1acbf
Update 2_domainname.md 2022-09-30 15:39:05 +01:00
Neil Alexander 0e2fb63b4f
Version 0.10.0 (#2753)
Changelog and version bump.
2022-09-30 13:31:05 +01:00
Neil Alexander 6348486a13
Transactional isolation for /sync (#2745)
This should transactional snapshot isolation for `/sync` etc requests.

For now we don't use repeatable read due to some odd test failures with
invites.
2022-09-30 12:48:10 +01:00
Neil Alexander 8a82f10046
Allow more time for device list updates (#2749)
This updates the device list updater so that it has a context
per-request, rather than a global 30 seconds for the entire server. This
could mean that talking to a slow remote server or requesting a lot of
user IDs was pretty much guaranteed to fail.

It also uses the process context to allow correct cancellation when
Dendrite wants to shut down cleanly.
2022-09-30 09:41:16 +01:00
Till 9005e5b4a8
Add /_dendrite/admin/refreshDevices/{userID} (#2746)
Allows to immediately query `/devices/{userID}` over federation to
(hopefully) resolve E2EE issues.
2022-09-30 09:32:31 +01:00
Neil Alexander e45ba35e97
Enable knocking on room versions 8 and 9 (update to matrix-org/gomatrixserverlib@91d455c) 2022-09-29 20:05:05 +01:00
devonh 90f1985bf3
Add network interface registration to pinecone demo (#2750)
### Pull Request Checklist

<!-- Please read docs/CONTRIBUTING.md before submitting your pull
request -->

No additional tests have been added as this simply extends the API for
the embedded pinecone demo.

Signed-off-by: `Devon Hudson <devonhudson@librem.one>`
2022-09-29 16:05:16 +00:00
Neil Alexander b28bbadeb0
Update to matrix-org/gomatrixserverlib@377b320 2022-09-29 16:55:03 +01:00
Neil Alexander 1b65c97ad1
Remove Suppressing send-to-device log line
The behaviour is correct and I have no idea why we are logging it.
2022-09-29 16:18:42 +01:00
Neil Alexander e1bf709eb3
Improve device list update parsing (update to matrix-org/gomatrixserverlib#342) 2022-09-29 16:08:41 +01:00
Neil Alexander 6f3dbee5f4
Fix Go 1.18 2022-09-29 16:08:00 +01:00
Neil Alexander 68d6eb0a6f
P2P demo fixes 2022-09-29 13:06:55 +01:00
Neil Alexander 3f9e38e80a
Consistent *sql.Tx usage across sync API (#2744)
This tidies up the `storage` package so that everything takes a
transaction parameter instead of something things that do and some that
don't.
2022-09-28 10:18:03 +01:00
texuf a574ed5369
Fix for `sql: converting argument $1 type: unsupported type []interfa… (#2743)
…ce {}, a slice of interface` in new notifications select

The sqlite3 version was just not working, original pr here:
https://github.com/matrix-org/dendrite/pull/2688

signed off by: austin ellis <austin@hntlabs.com>

This doesn't fix the notification counts, they still only work about 1
out of every 5 times in my tests. I will stick with my other fix locally
for reliable notification delivery:
https://github.com/matrix-org/dendrite/pull/2701
2022-09-28 06:19:34 +02:00
Neil Alexander 083ae01520
Promote reindexing log level 2022-09-27 17:30:40 +01:00
Neil Alexander 34993717fd
Update search docs 2022-09-27 17:10:47 +01:00
Till 87be32ca26
Fulltext implementation using Bleve (#2675)
Based on #2480

This actually indexes events based on their event type. They are removed
from the index if we receive a `m.room.redaction` event on the
`OutputRoomEvent` stream.
An admin endpoint is added to reindex all existing events.


Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-09-27 18:06:49 +02:00