Commit graph

2394 commits

Author SHA1 Message Date
Till 8c5b166784
Use the stream positions of the notifier (#2768)
Use the stream positions of the notifier, which might have advanced
since setting it at the beginning of the loop. This possibly helps in
reducing roundtrips to the SyncAPI, just because we didn't fetch the
latest data.
Also fixes a minor oversight in the receipts stream.
2022-10-06 11:57:13 +01:00
Neil Alexander d605d928bc
Allow specifying old signing keys with the public key and key ID only (#2770)
If the private key file is lost, it's often possible to retrieve the
public key from another server elsewhere, so we should make it possible
to configure it in that way.
2022-10-06 11:56:00 +01:00
Till ec5d1d681d
Always return one_time_key_counts on /keys/upload (#2769)
The OTK count is
[required](https://spec.matrix.org/v1.4/client-server-api/#post_matrixclientv3keysupload)
in responses to `/keys/upload`, so return those.
2022-10-06 12:30:24 +02:00
Neil Alexander 3f82bceb70
Don't try to talk to ourselves when finding missing events 2022-10-06 10:51:06 +01:00
Neil Alexander e53dcb25a9
Tweak logging for federated room joins 2022-10-06 10:07:13 +01:00
Neil Alexander 9ba3103f88
Document database connection limits 2022-10-05 14:54:19 +01:00
Till 0f777d421c
Remove empty fields from /sync response (#2755)
First attempt at removing empty fields from `/sync` responses. Needs
https://github.com/matrix-org/sytest/pull/1298 to keep Sytest happy.

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-10-05 13:47:13 +01:00
Neil Alexander c85bc3434f
Optimise QuerySharedUsers so that we can only work on local users (#2766)
Otherwise the sync API key change consumer wastes a lot of time trying
to wake up the notifiers for non-local users.
2022-10-05 12:47:53 +01:00
Neil Alexander 6f602bb096
Demote Failed to query device keys for some users warning to level=debug
Many of these warnings are due to dead servers and are quite annoying when they fill up the logs.
2022-10-05 11:16:05 +01:00
Till 8c0c3441d8
Add RoomEventType nats.Header to avoid unneeded unmarshalling (#2765) 2022-10-05 12:12:42 +02:00
Neil Alexander ebd137cf6b
Check PostgreSQL connection count (#2760)
This PR queries PostgreSQL for the `max_connections` and
`superuser_reserved_connections` settings and then ensures that
Dendrite's `max_open_conns` doesn't exceed the allowed value.

This is a really common source of configuration problems and can either
result in blocking queries or deadlocks, so it seems reasonable that we
complain as loudly as possible when it happens.
2022-10-05 11:07:17 +01:00
Till e070352293
Side effect import bleve analyzer languages (#2763)
... to actually allow different languages. Fixes #2761 
Binary size increases by ~1MB.
2022-10-05 11:14:33 +02:00
Neil Alexander 21f8881985
Add indexes that optimise selectStateInRangeSQL (#2764)
This gets rid of some expensive scans on `add_state_ids` and
`remove_state_ids`, turning them into much cheaper and faster index
scans instead.
2022-10-04 16:43:10 +01:00
Neil Alexander ae10aac456
Don't perform a federated join after invite if we are already joined to the room (#2762)
If we are already joined to the room then it shouldn't matter if you
were invited or not, so this looks like a bug.
2022-10-04 15:40:04 +01:00
Neil Alexander 3da182212e
Track reasons why the process is in a degraded state 2022-10-04 13:02:41 +01:00
Neil Alexander a767102f8a
Reduce max_open_conns in monolith sample config 2022-10-04 11:34:52 +01:00
Neil Alexander 085bf5e28b
Revert Docker changes 2022-10-04 11:33:05 +01:00
Neil Alexander 98b73652e0
Try to populate -ldflags in Docker builds 2022-10-04 11:07:54 +01:00
Neil Alexander ede4632835
Fix Docker GHA 2022-10-04 10:43:58 +01:00
Till e6c992ba8b
Update Dockerfile (#2342)
Updates/adds a new multistage (build-kit) Dockerfile. (if accepted,
could make `Dockerfile.monolith` and `Dockerfile.polylith` in
`build/docker` obsolete)
There's no huge difference between the dockerfiles, except this uses a
non-root user when running the container, also doesn't copy the working
directory to the image when building.
Also adds vulnerabilities scans using
[Trivy](https://github.com/aquasecurity/trivy) for the created docker
images. (untested)

Building images is done using 
```
docker build . --target image-monolith -t dendrite-monolith
docker build . --target image-polylith -t dendrite-polylith
```

As noted in the comments, only adds `dendrite-polylith-multi` to the
polylith image and all required binaries to the monolith image.
Probably needs some docs updating, if this is accepted.

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-10-04 10:41:06 +01:00
Neil Alexander 34ed316584
Fix docs 2022-10-03 19:35:26 +01:00
Neil Alexander ba66b5a3b9
Allow multiple static peers in Pinecone iOS/Android demos 2022-10-03 14:43:38 +01:00
Neil Alexander 34451d21b8
P2P demo tweaks 2022-10-03 14:35:10 +01:00
Neil Alexander 50fa50a343
Update P2P base directories 2022-10-03 13:05:58 +01:00
Neil Alexander fec3ee384b
Stop CPU burn in PerformMarkAsStaleIfNeeded 2022-10-03 12:59:56 +01:00
Ashley Nelson c1e16fd41e
Fix fragility of selectEventsWithEventIDsSQL queries (#2757)
This fixes a temporary workaround with the `selectEventsWithEventIDsSQL`
queries where fields need to be artificially added to the queries so the
row results match the format of the `syncapi_output_room_events` table.
I made similar functions that accept row results from the
`syncapi_current_room_state` table and convert them into StreamEvents
without the fields that are specific to output room events.

There is also a unit test in the first commit to ensure the resulting
behavior doesn't change from the modified queries and functions.

Fixes #601.

### Pull Request Checklist

<!-- Please read docs/CONTRIBUTING.md before submitting your pull
request -->

* [x] I have added tests for PR _or_ I have justified why this PR
doesn't need tests.
* [x] Pull request includes a [sign
off](https://github.com/matrix-org/dendrite/blob/main/docs/CONTRIBUTING.md#sign-off)

Signed-off-by: `Ashley Nelson <fant@shley.email>`

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-10-03 11:57:21 +01:00
Neil Alexander b7f4bab358
Hopefully fix P2P --config error (re. #2756) 2022-10-03 11:38:31 +01:00
Neil Alexander d32f60249d
Modify sync transaction behaviour (#2758)
This now uses a transaction per stream, so that errors in one stream
don't propagate to another, and we therefore no longer need to do hacks
to reopen a new transaction after aborting a failed one.
2022-10-03 11:38:20 +01:00
Till Faelligen d4710217f8
Use non-HTTPS as default URL, as most people will be running behind a
reverse proxy
2022-10-02 11:31:40 +02:00
Neil Alexander a050503d8d
Version 0.10.1 2022-09-30 17:15:43 +01:00
Neil Alexander 0116db79c6
Reset transaction after a failure 2022-09-30 17:07:37 +01:00
Neil Alexander 16048be236
Handle case when applying history visibility failed 2022-09-30 16:55:10 +01:00
Neil Alexander 7d9545ceea
Another /sync fix 2022-09-30 16:34:06 +01:00
Till Faelligen 3617d5a0ff
Remove SendAccountData, since InputAccountData is already doing that 2022-09-30 17:31:41 +02:00
Neil Alexander 60ec9180e6
Update NATS Server to v2.9.2 2022-09-30 16:10:44 +01:00
Neil Alexander ee40a29e55
Fix broken /sync due to transaction error 2022-09-30 16:07:18 +01:00
Neil Alexander aa8ec1acbf
Update 2_domainname.md 2022-09-30 15:39:05 +01:00
Neil Alexander 0e2fb63b4f
Version 0.10.0 (#2753)
Changelog and version bump.
2022-09-30 13:31:05 +01:00
Neil Alexander 6348486a13
Transactional isolation for /sync (#2745)
This should transactional snapshot isolation for `/sync` etc requests.

For now we don't use repeatable read due to some odd test failures with
invites.
2022-09-30 12:48:10 +01:00
Neil Alexander 8a82f10046
Allow more time for device list updates (#2749)
This updates the device list updater so that it has a context
per-request, rather than a global 30 seconds for the entire server. This
could mean that talking to a slow remote server or requesting a lot of
user IDs was pretty much guaranteed to fail.

It also uses the process context to allow correct cancellation when
Dendrite wants to shut down cleanly.
2022-09-30 09:41:16 +01:00
Till 9005e5b4a8
Add /_dendrite/admin/refreshDevices/{userID} (#2746)
Allows to immediately query `/devices/{userID}` over federation to
(hopefully) resolve E2EE issues.
2022-09-30 09:32:31 +01:00
Neil Alexander e45ba35e97
Enable knocking on room versions 8 and 9 (update to matrix-org/gomatrixserverlib@91d455c) 2022-09-29 20:05:05 +01:00
devonh 90f1985bf3
Add network interface registration to pinecone demo (#2750)
### Pull Request Checklist

<!-- Please read docs/CONTRIBUTING.md before submitting your pull
request -->

No additional tests have been added as this simply extends the API for
the embedded pinecone demo.

Signed-off-by: `Devon Hudson <devonhudson@librem.one>`
2022-09-29 16:05:16 +00:00
Neil Alexander b28bbadeb0
Update to matrix-org/gomatrixserverlib@377b320 2022-09-29 16:55:03 +01:00
Neil Alexander 1b65c97ad1
Remove Suppressing send-to-device log line
The behaviour is correct and I have no idea why we are logging it.
2022-09-29 16:18:42 +01:00
Neil Alexander e1bf709eb3
Improve device list update parsing (update to matrix-org/gomatrixserverlib#342) 2022-09-29 16:08:41 +01:00
Neil Alexander 6f3dbee5f4
Fix Go 1.18 2022-09-29 16:08:00 +01:00
Neil Alexander 68d6eb0a6f
P2P demo fixes 2022-09-29 13:06:55 +01:00
Neil Alexander 3f9e38e80a
Consistent *sql.Tx usage across sync API (#2744)
This tidies up the `storage` package so that everything takes a
transaction parameter instead of something things that do and some that
don't.
2022-09-28 10:18:03 +01:00
texuf a574ed5369
Fix for `sql: converting argument $1 type: unsupported type []interfa… (#2743)
…ce {}, a slice of interface` in new notifications select

The sqlite3 version was just not working, original pr here:
https://github.com/matrix-org/dendrite/pull/2688

signed off by: austin ellis <austin@hntlabs.com>

This doesn't fix the notification counts, they still only work about 1
out of every 5 times in my tests. I will stick with my other fix locally
for reliable notification delivery:
https://github.com/matrix-org/dendrite/pull/2701
2022-09-28 06:19:34 +02:00