Commit graph

111 commits

Author SHA1 Message Date
devonh 8245b24100
Update gmsl to use new validated RoomID on PDUs ()
GMSL returns a `spec.RoomID` when calling `PDU.RoomID()`
2023-09-15 14:39:06 +00:00
maxberger 294eff8a7f
Add ID in error messages for ApplicationServices ()
This is to easier identify which service caused the error.

Feature is just improving logging, thus no tests added.

### Pull Request Checklist

<!-- Please read
https://matrix-org.github.io/dendrite/development/contributing before
submitting your pull request -->

* [X]  I have justified why this PR doesn't need tests
* [X] Pull request includes a [sign off below using a legally
identifiable
name](https://matrix-org.github.io/dendrite/development/contributing#sign-off)
_or_ I have already signed off privately

Signed-off-by: `Maximilian Berger <max@berger.name>`

Co-authored-by: Till <2353100+S7evinK@users.noreply.github.com>
2023-08-03 09:26:42 +02:00
Sam Wedgwood 9582827493
de-MSC-ifying space summaries (MSC2946) ()
- This PR moves and refactors the
[code](https://github.com/matrix-org/dendrite/blob/main/setup/mscs/msc2946/msc2946.go)
for
[MSC2946](https://github.com/matrix-org/matrix-spec-proposals/pull/2946)
('Space Summaries') to integrate it into the rest of the codebase.
- Means space summaries are no longer hidden behind an MSC flag
- Solves 

Signed-off-by: Sam Wedgwood <sam@wedgwood.dev>
2023-07-20 15:06:05 +01:00
devonh e4665979bf
Merge SenderID & Per Room User Key work () 2023-06-14 14:23:46 +00:00
devonh 8ea1a11105
Use SenderID Type () 2023-06-07 17:14:35 +00:00
devonh 7a1fd7f512
PDU Sender split ()
Initial cut of splitting PDU Sender into SenderID & looking up UserID where required.
2023-06-06 20:55:18 +00:00
kegsay f5b3144dc3
Use PDU not *Event in HeaderedEvent ()
Requires https://github.com/matrix-org/gomatrixserverlib/pull/376

This has numerous upsides:
 - Less type casting to `*Event` is required.
- Making Dendrite work with `PDU` interfaces means we can swap out Event
impls more easily.
 - Tests which represent weird event shapes are easier to write.

Part of a series of refactors on GMSL.
2023-05-02 15:03:16 +01:00
kegsay 1432743d1a
Use PDU in more places () 2023-04-28 16:00:22 +01:00
kegsay b189edf4f4
Remove gmsl.HeaderedEvent ()
Replaced with types.HeaderedEvent _for now_. In reality we want to move
them all to gmsl.Event and only use HeaderedEvent when we _need_ to
bundle the version/event ID with the event (seriailsation boundaries,
and even then only when we don't have the room version).

Requires https://github.com/matrix-org/gomatrixserverlib/pull/373
2023-04-27 12:54:20 +01:00
kegsay 72285b2659
refactor: update GMSL ()
Sister PR to https://github.com/matrix-org/gomatrixserverlib/pull/364

Read this commit by commit to avoid going insane.
2023-04-19 15:50:33 +01:00
kegsay 3691423626
Move GMSL client types to Dendrite ()
GMSL is intended for Federation only. Sister PR to
https://github.com/matrix-org/gomatrixserverlib/pull/357
2023-04-04 17:16:53 +00:00
Boris Rybalkin 985298cfc4
app service unix socket support ()
This is the last part of unix socket support to talk to app servers, go
based app services already support unix sockets:

5a68173fe3
```
appservice:
  # The address that the homeserver can use to connect to this appservice.
  address: unix:///var/snap/matrix/current/whatsapp.socket

  # The hostname and port where this appservice should listen.
  hostname: /var/snap/matrix/current/whatsapp.socket
  port: 0
```

### Pull Request Checklist

<!-- Please read
https://matrix-org.github.io/dendrite/development/contributing before
submitting your pull request -->

* [x] I have added Go unit tests or [Complement integration
tests](https://github.com/matrix-org/complement) for this PR _or_ I have
justified why this PR doesn't need tests
* [x] Pull request includes a [sign off below using a legally
identifiable
name](https://matrix-org.github.io/dendrite/development/contributing#sign-off)
_or_ I have already signed off privately

Signed-off-by: `Boris Rybalkin <ribalkin@gmail.com>`
2023-04-04 09:42:46 +02:00
Till c2db38d295
Add user profile tests, refactor user API methods ()
This adds tests for `/profile`.
Also, as a first change in this regard, refactors the methods defined on
the `UserInternalAPI` to not use structs as the request/response
parameters.
2023-04-03 20:19:26 +02:00
Till aa1bda4c58
Add AS invite test, fix issue with invitations being processed twice ()
The AS roomserver consumer would receive the events twice, one time as
type `OutputTypeNewInviteEvent` and the other time as
`OutputTypeNewRoomEvent`. 

[skip ci]
2023-03-27 11:26:26 +02:00
Till 5e85a00cb3
Remove BaseDendrite ()
Removes `BaseDendrite` to, hopefully, make testing and composing of
components easier in the future.
2023-03-22 09:21:32 +01:00
Till 5579121c6f
Preparations for removing BaseDendrite ()
Preparations to actually remove/replace `BaseDendrite`.
Quite a few changes:
- SyncAPI accepts an `fulltext.Indexer` interface (fulltext is removed
from `BaseDendrite`)
- Caches are removed from `BaseDendrite`
- Introduces a `Router` struct (likely to change)
  - also fixes 
- Introduces a `sqlutil.ConnectionManager`, which should remove
`base.DatabaseConnection` later on
- probably more
2023-03-17 11:09:45 +00:00
Till 232aef016c
Add basic runtime tracing ()
This allows us in almost all places to use regions to further trace down
long running tasks.
Also removes an unused function.
2023-03-13 16:45:14 +01:00
Till 6c20f8f742
Refactor StoreEvent, add MaybeRedactEvent, create an EventDatabase ()
This PR changes the following:
- `StoreEvent` now only stores an event (and possibly prev event),
instead of also doing redactions
- Adds a `MaybeRedactEvent` (pulled out from `StoreEvent`), which should
be called after storing events
- a few other things
2023-03-01 17:06:47 +01:00
Till 4594233f89
Merge keyserver & userapi ()
As discussed yesterday, a first draft of merging the keyserver and the
userapi.
2023-02-20 14:58:03 +01:00
Till 11d9b9db0e
Remove polylith/API mode ()
This removes most of the code used for polylith/API mode.

This removes the `/api` internal endpoints entirely. 

Binary size change roughly 5%: 
```
51437560 Feb 13 10:15 dendrite-monolith-server # old
48759008 Feb 13 10:15 dendrite-monolith-server # new
```
2023-02-14 12:47:47 +01:00
Till Faelligen b99349b18c
Use test.WithAllDatabases 2022-12-05 16:00:02 +01:00
Till Faelligen 07e8ed13f6
Fix CI and test.WithAllDatabases 2022-12-05 15:09:59 +01:00
Till e245a26f6b
Enable/Disable internal metrics ()
Basically enables us to use `test.WithAllDatabases` when testing
internal HTTP APIs, as this would otherwise result in Prometheus
complaining about already registered metric names.
2022-12-05 13:53:36 +01:00
Till b65f89e61e
Add tests for the AS internal API () 2022-12-02 16:42:23 +01:00
Neil Alexander 529df30b56
Virtual hosting schema and logic changes ()
Note that virtual users cannot federate correctly yet.
2022-11-11 16:41:37 +00:00
Till b367cfeddf
Implement /thirdparty endpoints ()
Implements the following endpoints 
```
GET /_matrix/client/v3/thirdparty/protocols
GET /_matrix/client/v3/thirdparty/protocols/{protocol}
GET /_matrix/client/v3/thirdparty/location
GET /_matrix/client/v3/thirdparty/location/{protocol}
GET /_matrix/client/v3/thirdparty/user
GET /_matrix/client/v3/thirdparty/user/{protocol}
```
2022-11-02 10:17:53 +00:00
Till 8c0c3441d8
Add RoomEventType nats.Header to avoid unneeded unmarshalling () 2022-10-05 12:12:42 +02:00
Tak Wai Wong 99f6b6a952
Bug fix appservice txnid should be different for each batch of events ()
See issue: [](https://github.com/matrix-org/dendrite/issues/2718)
for more details.

The fix assumes that if the number of transaction items are different,
then the txnid should be different.

txnid := OriginalServerTS()_len(transactions) 

The case that it doesn't address is if the txnid generated this way is
the same for 2 different batches of events which have the same
OriginalServerTS and the same array length.

Another option:

txnid := OriginalServerTS()_hash(transactions)

Would love to hear other ideas and ways to fix this.

### Pull Request Checklist

* [x ] I have added added tests for PR _or_ I have justified why this PR
doesn't need tests.
* [x ] Pull request includes a [sign
off](https://github.com/matrix-org/dendrite/blob/main/docs/CONTRIBUTING.md#sign-off)

Signed-off-by: `Tak Wai Wong <tak@hntlabs.com>`

Co-authored-by: Tak Wai Wong <tak@hntlabs.com>
2022-09-19 18:39:06 +02:00
Till 350a5e5393
Only send (invite) events to the AS if it's interested in it ()
This should hopefully fix  (and maybe also )
2022-09-02 16:03:33 +02:00
Neil Alexander ad6b902b84
Refactor appservices component ()
This PR refactors the app services component. It makes the following changes:

* Each appservice now gets its own NATS JetStream consumer
* The appservice database is now removed entirely, since we just use JetStream as a data source instead
* The entire component is now much simpler and we deleted lots of lines of code 💅

The result is that it should be much lighter and hopefully much more performant.
2022-09-01 09:20:40 +01:00
Neil Alexander 175f65407a
Allow batching in JetStreamConsumer ()
This allows us to receive more than one message from NATS at a time if we want.
2022-08-31 12:21:56 +01:00
Till Faelligen 21ef487ff8
Log roomserver error 2022-08-29 14:18:33 +02:00
Neil Alexander c45d0936b5
Generic-based internal HTTP API ()
* Generic-based internal HTTP API (tested out on a few endpoints in the federation API)

* Add `PerformInvite`

* More tweaks

* Fix metric name

* Fix LookupStateIDs

* Lots of changes to clients

* Some serverside stuff

* Some error handling

* Use paths as metric names

* Revert "Use paths as metric names"

This reverts commit a9323a6a34.

* Namespace metric names

* Remove duplicate entry

* Remove another duplicate entry

* Tweak error handling

* Some more tweaks

* Update error behaviour

* Some more error tweaking

* Fix API path for `PerformDeleteKeys`

* Fix another path

* Tweak federation client proxying

* Fix another path

* Don't return typed nils

* Some more tweaks, not that it makes any difference

* Tweak federation client proxying

* Maybe fix the key backup test
2022-08-11 15:29:33 +01:00
Aaron Ogle 20844942a8
Handle invite in appservice roomserver consumer ()
Signed-off-by: Aaron Ogle <aaron@geekgonecrazy.com>
2022-05-24 09:07:33 +01:00
Neil Alexander 77722c5a4f
Back out by restoring http.Clients
This creates problems with non-HTTPS endpoints and should fix .
2022-05-10 11:08:10 +01:00
Neil Alexander 09d754cfbf
One NATS instance per BaseDendrite ()
* One NATS instance per `BaseDendrite`

* Fix roomserver
2022-05-09 14:15:24 +01:00
kegsay 85704eff20
Clean up interface definitions ()
* tidy up interfaces

* remove unused GetCreatorIDForAlias

* Add RoomserverUserAPI interface

* Define more interfaces

* Use AppServiceInternalAPI for consistent naming

* clean up federationapi constructor a bit

* Fix monolith in -http mode
2022-05-06 12:39:26 +01:00
kegsay 506de4bb3d
Define component interfaces based on consumers (1/2) ()
* Specify interfaces used by appservice, do half of clientapi

* convert more deps of clientapi to finer-grained interfaces

* Convert mediaapi and rest of clientapi

* Somehow this got missed
2022-05-05 13:17:38 +01:00
Neil Alexander d9e71b93b6
Use gomatrixserverlib.Client instead of http.Client ()
* Update to 

* Use `gomatrixserverlib.Client` for phone-home stats

* Use `gomatrixserverlib.Client` for push notifications

* Use `gomatrixserverlib.Client` for appservices

* Use `gomatrixserverlib.Client` for three-PID invites
2022-05-05 11:33:16 +01:00
Neil Alexander 4ad5f9c982
Global database connection pool (for monolith mode) ()
* Allow monolith components to share a single database pool

* Don't yell about missing connection strings

* Rename field

* Setup tweaks

* Fix panic

* Improve configuration checks

* Update config

* Fix lint errors

* Update comments
2022-05-03 16:35:06 +01:00
Neil Alexander 99f2007eb6
Fix bug where state events could be requested twice in appservices ()
* Fix bug where state events could be requested twice (should fix )

* Avoid a pointless HTTP request if possible
2022-04-06 16:10:31 +01:00
S7evinK f2e550efd8
Refactor appservice & client API to use userapi internal ()
* Refactor user api internal

* Refactor clientapi to use internal userapi

* Use internal userapi instead of user DB directly

* Remove AccountDB dependency

* Fix linter issues

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-03-24 22:45:44 +01:00
Neil Alexander 98a5e410d7
Per-room consumers ()
* Roomserver input refactoring — again!

* Ensure the actor runs again

* Preserve consumer after unsubscribe

* Another sprinkling of magic

* Rename `TopicFor` to `Prefixed`

* Recreate the stream if the config is bad

* Check streams too

* Prefix subjects, preserve inboxes

* Recreate if subjects wrong

* Remove stream subject

* Reconstruct properly

* Fix mutex unlock

* Comments

* Fix tests

* Don't drop events

* Review comments

* Separate `queueInputRoomEvents` function

* Re-jig control flow a bit
2022-03-23 10:20:18 +00:00
Neil Alexander 9572f5ed19
Wait for safe shutdown of NATS Server () 2022-03-21 10:32:34 +00:00
S7evinK 8336ce972e
Remove unused partition_offset_table () 2022-03-21 10:47:41 +01:00
Neil Alexander e30aa38fb0
Stream tweaks, use same codepath for sync vs async input room events, wait for error response via NATS messages () 2022-03-16 14:21:11 +00:00
Neil Alexander 67de4dbd0c
Don't send adds_state_events in roomserver output events anymore ()
* Don't send `adds_state_events` in roomserver output events anymore

* Set `omitempty` on some output fields that aren't always set

* Add `AddsState` helper function

* No-op if no added state event IDs

* Revert "No-op if no added state event IDs"

This reverts commit 71a0ef3df1.

* Revert "Add `AddsState` helper function"

This reverts commit c9fbe45475.
2022-03-07 17:17:16 +00:00
Neil Alexander 153bfbbea5
Merge both user API databases into one ()
* Merge user API databases into one

* Remove DeviceDatabase from config

* Fix tests

* Try that again

* Clean up keyserver device keys when the devices no longer exist in the user API

* Tweak ordering

* Fix UserExists flag, device check

* Allow including empty entries so we can clean them up

* Remove logging
2022-02-18 11:31:05 +00:00
S7evinK 5a39512f5f
Add account type ()
* Add account_type for sqlite3

* Add account_type for postgres

* Remove CreateGuestAccount from interface

* Add new AccountTypes & update test

* Use newly added AccountType for account creation

* Add migrations

* Reuse type

* Add AccounnType to Device, so it can be verified on requests

* Rename migration, add missing update for appservices

* Rename sqlite3 migration

* Add missing AccountType to return value

* Update sqlite migration
Change allowance check on /admin/whois

* Fix migration, add IS NULL

* Move accountType to completeRegistration

* Fix migrations

* Add passing test
2022-02-16 18:55:38 +01:00
S7evinK 9de7efa0b0
Remove sarama/saramajetstream dependencies ()
* Remove dependency on saramajetstream & sarama

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Remove internal.ContinualConsumer from federationapi

* Remove internal.ContinualConsumer from syncapi

* Remove internal.ContinualConsumer from keyserver

* Move to new Prepare function

* Remove saramajetstream & sarama dependency

* Delete unneeded file

* Remove duplicate import

* Log error instead of silently irgnoring it

* Move `OffsetNewest` and `OffsetOldest` into keyserver types, change them to be more sane values

* Fix comments

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-02-04 13:08:13 +00:00