Commit graph

331 commits

Author SHA1 Message Date
Till Faelligen 638300efd0
Sort the events and skip if eventID is equal beforeID 2023-03-07 11:25:03 +01:00
Till Faelligen 26375dd56a
Update GMSL 2023-03-07 10:20:19 +01:00
Till Faelligen 78076aff3a
Merge branch 'main' of github.com:matrix-org/dendrite into s7evink/backfilltofile 2023-03-07 09:15:22 +01:00
Robin Westerik 7cde99a7a7
Updated instructions and references to monolith to their new names (#2994)
Currently, the documentation makes use of the old names for the binary
and configuration files. This updates the documentation so that users
can follow the guide without issues again.
These changes don't require any go unit tests because it does not modify
any golang code.

Signed-off-by: `Robin Westerik <gh@westerik.me>`
2023-03-03 10:20:53 +01:00
Boris Rybalkin 6b1c9eafa9
unix socket support (#2974)
### Pull Request Checklist

<!-- Please read
https://matrix-org.github.io/dendrite/development/contributing before
submitting your pull request -->

* [x] I have added Go unit tests or [Complement integration
tests](https://github.com/matrix-org/complement) for this PR _or_ I have
justified why this PR doesn't need tests
* [x] Pull request includes a [sign off below using a legally
identifiable
name](https://matrix-org.github.io/dendrite/development/contributing#sign-off)
_or_ I have already signed off privately

Signed-off-by: `Boris Rybalkin <ribalkin@gmail.com>`

I need this for Syncloud project (https://github.com/syncloud/platform)
where I run multiple apps behind an nginx on the same RPi like device so
unix socket is very convenient to not have port conflicts between apps.
Also someone opened this Issue:
https://github.com/matrix-org/dendrite/issues/2924

---------

Co-authored-by: kegsay <kegan@matrix.org>
Co-authored-by: Till <2353100+S7evinK@users.noreply.github.com>
2023-03-01 22:57:30 +01:00
Till 6c20f8f742
Refactor StoreEvent, add MaybeRedactEvent, create an EventDatabase (#2989)
This PR changes the following:
- `StoreEvent` now only stores an event (and possibly prev event),
instead of also doing redactions
- Adds a `MaybeRedactEvent` (pulled out from `StoreEvent`), which should
be called after storing events
- a few other things
2023-03-01 17:06:47 +01:00
Till Faelligen d60d7f9837
Add basic tool to download a rooms state to file 2023-03-01 16:07:08 +01:00
Devon Hudson eddf31f915
Fix lint error 2023-02-24 15:49:51 -07:00
Devon Hudson b28406c7d0
Tweaks to pinecone demo to shutdown more cleanly 2023-02-24 15:41:47 -07:00
Till ad07b169b8
Refactor StoreEvent and create a new RoomDatabase interface (#2985)
This PR changes a few things:
- It pulls out the creation of several NIDs from the `StoreEvent`
function to make the functions more reusable
- Uses more caching when using those NIDs to avoid DB round trips
2023-02-24 09:40:20 +01:00
Till 4594233f89
Merge keyserver & userapi (#2972)
As discussed yesterday, a first draft of merging the keyserver and the
userapi.
2023-02-20 14:58:03 +01:00
Till 11d9b9db0e
Remove polylith/API mode (#2967)
This removes most of the code used for polylith/API mode.

This removes the `/api` internal endpoints entirely. 

Binary size change roughly 5%: 
```
51437560 Feb 13 10:15 dendrite-monolith-server # old
48759008 Feb 13 10:15 dendrite-monolith-server # new
```
2023-02-14 12:47:47 +01:00
Devon Hudson a666c06da1
Consolidate pinecone demo http server variations 2023-02-01 14:11:48 -07:00
Devon Hudson 048e35026c
Refactor common pinecone demo code to remove major duplication 2023-02-01 13:41:38 -07:00
Devon Hudson dbc2869cbd
Refactor pinecone demo to remove duplicate pinecone setup 2023-02-01 13:41:38 -07:00
Devon Hudson d4f64f91ca
Refactor pinecone demo to remove duplicate key setup 2023-02-01 13:41:37 -07:00
Devon Hudson 529feb07ee
Refactor conduit type from pinecone demo into its own package 2023-02-01 13:41:37 -07:00
Devon Hudson be43b9c0ea
Refactor common relay sync struct to remove duplication 2023-02-01 13:41:36 -07:00
Devon Hudson f98003c030
Add cmd line option to pinecone demo for enabling relaying 2023-01-29 18:13:39 -07:00
Devon Hudson 0f998e3af3
Add pinecone demo toggle for dis/enabling relaying for other nodes 2023-01-29 12:26:16 -07:00
Devon Hudson 24a865aeb7
Move relay arch into relayapi and add docs for new endpoints 2023-01-26 10:15:53 -07:00
devonh 5b73592f5a
Initial Store & Forward Implementation (#2917)
This adds store & forward relays into dendrite for p2p.
A few things have changed:
- new relay api serves new http endpoints for s&f federation
- updated outbound federation queueing which will attempt to forward
using s&f if appropriate
- database entries to track s&f relays for other nodes
2023-01-23 17:55:12 +00:00
Till 48fa869fa3
Use t.TempDir for SQLite databases, so tests don't rip out each others databases (#2950)
This should hopefully finally fix issues about `disk I/O error` as seen
[here](https://gitlab.alpinelinux.org/alpine/aports/-/jobs/955030/raw)

Hopefully this will also fix `SSL accept attempt failed` issues by
disabling HTTP keep alives when generating a config for CI.
2023-01-23 13:17:15 +01:00
Bernhard Feichtinger a2b4860912
Fix oversight in cmd/generate-config (#2946)
The -dir argument was ignored for media_api->base_path.
Signed-off-by: `Bernhard Feichtinger
<43303168+BieHDC@users.noreply.github.com>`
2023-01-20 13:13:36 +01:00
Till f762ce1050
Add clientapi tests (#2916)
This PR
- adds several tests for the clientapi, mostly around `/register` and
auth fallback.
- removes the now deprecated `homeserver` field from responses to
`/register` and `/login`
- slightly refactors auth fallback handling
2022-12-23 14:11:11 +01:00
Till 7d2344049d
Cleanup stale device lists for users we don't share a room with anymore (#2857)
The stale device lists table might contain entries for users we don't
share a room with anymore. This now asks the roomserver about left users
and removes those entries from the table.

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-12-12 08:20:59 +01:00
Till e245a26f6b
Enable/Disable internal metrics (#2899)
Basically enables us to use `test.WithAllDatabases` when testing
internal HTTP APIs, as this would otherwise result in Prometheus
complaining about already registered metric names.
2022-12-05 13:53:36 +01:00
Till 9a46d8d95c
Test and CI related changes (#2896)
In an attempt to:
- make on-boarding a bit easier (`go test ./...` should now not need
additional postgres setup)
- get code coverage faster, not only scheduled at night
- test the `create-account` binary
2022-12-02 11:44:20 +01:00
Neil Alexander 934056f21f
Fix dendrite-demo-pinecone, /_dendrite namespace setup 2022-12-01 10:45:15 +00:00
Neil Alexander 1be0afa181
Expose /_dendrite and /_synapse on the P2P demo HTTP muxes 2022-12-01 10:24:17 +00:00
Neil Alexander 6f000e9801
Make create-account more verbose 2022-12-01 10:14:26 +00:00
devonh a8e7ffc7ab
Add p2p wakeup broadcast handling to pinecone demos (#2841)
Adds wakeup broadcast handling to the pinecone demos.
This will reset their blacklist status and interrupt any ongoing
federation queue backoffs currently in progress for this peer.
The end result is that any queued events will quickly be sent to the
peer if they had disconnected while attempting to send events to them.
2022-11-18 00:29:23 +00:00
Neil Alexander 6650712a1c
Federation fixes for virtual hosting 2022-11-15 15:05:23 +00:00
Neil Alexander 72ce6acf71
Run upgrade tests for SQLite too (#2875)
This should hopefully catch problems with database migrations in SQLite
as well as PostgreSQL.
2022-11-11 11:21:16 +00:00
Neil Alexander efa50253f6
Fix lint error 2022-11-10 10:16:56 +00:00
Neil Alexander 503d9c7586
Improve logging in upgrade tests 2022-11-10 10:07:19 +00:00
devonh 4afadebd99
Add readme for dendrite-demo-pinecone (#2851) 2022-11-02 20:29:30 +00:00
Neil Alexander ca8bc87380
Multi-stage Docker builds (#2850)
This builds on @S7evinK's work to make multi-stage Docker builds. Now
that we can build SQLite without Cgo this should be much simpler and
should make Docker builds in CI significantly faster.

Co-authored-by: Till Faelligen <tfaelligen@gmail.com>
Co-authored-by: Till Faelligen <davidf@element.io>
Co-authored-by: Till Faelligen <2353100+S7evinK@users.noreply.github.com>
2022-11-02 14:04:08 +00:00
0x1a8510f2 51ab0a8ccf
Fix moderncsqlite errors and rebase onto main (#2832)
This is #2819 but rebased on latest `main`. This PR is against main too
as opposed to the `moderncsqlite` branch.

The main change here is simply:

```go
// add query parameters to the dsn
if strings.Contains(dsn, "?") {
	dsn += "&"
} else {
	dsn += "?"
}

// wait some time before erroring if the db is locked
// https://gitlab.com/cznic/sqlite/-/issues/106#note_1058094993
dsn += "_pragma=busy_timeout%3d10000"
```

### Pull Request Checklist

<!-- Please read
https://matrix-org.github.io/dendrite/development/contributing before
submitting your pull request -->

* [x] I have added tests for PR _or_ I have justified why this PR
doesn't need tests.
* [x] Pull request includes a [sign off below using a legally
identifiable
name](https://matrix-org.github.io/dendrite/development/contributing#sign-off)
_or_ I have already signed off privately

Signed off privately.

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-11-02 13:20:10 +00:00
Neil Alexander 238b6ef2cd
Update Yggdrasil demo 2022-10-26 18:37:01 +01:00
Till Faelligen 83c9dde219
Return error if we fail to read the response body 2022-10-17 07:27:11 +02:00
Neil Alexander 085bf5e28b
Revert Docker changes 2022-10-04 11:33:05 +01:00
Till e6c992ba8b
Update Dockerfile (#2342)
Updates/adds a new multistage (build-kit) Dockerfile. (if accepted,
could make `Dockerfile.monolith` and `Dockerfile.polylith` in
`build/docker` obsolete)
There's no huge difference between the dockerfiles, except this uses a
non-root user when running the container, also doesn't copy the working
directory to the image when building.
Also adds vulnerabilities scans using
[Trivy](https://github.com/aquasecurity/trivy) for the created docker
images. (untested)

Building images is done using 
```
docker build . --target image-monolith -t dendrite-monolith
docker build . --target image-polylith -t dendrite-polylith
```

As noted in the comments, only adds `dendrite-polylith-multi` to the
polylith image and all required binaries to the monolith image.
Probably needs some docs updating, if this is accepted.

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-10-04 10:41:06 +01:00
Neil Alexander 50fa50a343
Update P2P base directories 2022-10-03 13:05:58 +01:00
Neil Alexander b7f4bab358
Hopefully fix P2P --config error (re. #2756) 2022-10-03 11:38:31 +01:00
Till Faelligen d4710217f8
Use non-HTTPS as default URL, as most people will be running behind a
reverse proxy
2022-10-02 11:31:40 +02:00
Neil Alexander 34993717fd
Update search docs 2022-09-27 17:10:47 +01:00
Till 87be32ca26
Fulltext implementation using Bleve (#2675)
Based on #2480

This actually indexes events based on their event type. They are removed
from the index if we receive a `m.room.redaction` event on the
`OutputRoomEvent` stream.
An admin endpoint is added to reindex all existing events.


Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-09-27 18:06:49 +02:00
Neil Alexander 14b7322003
Add -dir option to dendrite-demo-pinecone and dendrite-demo-yggdrasil 2022-09-23 15:44:21 +01:00
Till a5f8c07184
Hopefully fix upgrade-tests (#2717)
Wait for events to come down `/sync` before ending the test.
2022-09-15 07:26:26 +02:00