- Update the start-local-dendrite.sh to be able to start local dendrite
with / without authorization checks
- Update dendrite config to support command line arg
Closes HNT-244.
The following PR implements Space,Channel soft deletion using on-chain
`disabled` flag scope to space, channel respectively. On message sync,
dendrite will now gate disabled rooms by performing a leave on the user
attempting to sync unless the user is the owner (more on this later). To
re-join, given rooms (spaces,channels) are created by default using
`invite` membership state, the owner will need to undo the on-chain
`disabled` flag, setting it false then re-invite users that left the
room as a side effect of it becoming disabled previously.
The owner does not leave the space, channel because if they did then
there would be no one left to invite users let alone themselves back in
if the action is ever undone.
What is not implemented in this PR:
1. **Transitive leaves on channels in a space** - If a space is
disabled, users will leave the space but not the channels within the
space. To allow for fully disabling a space and all its' channels, the
client can offer a view to the owner that iterates over the channels and
space to disable all on-chain. Furthermore, we could implement a batch
on-chain method that fully disables all channels within a space (plus
the space) in one on-chain call to save the owner gas.
2. **Data deletion** - No data is remove from the DAGs or on-chain.
Therefore deletion is soft and reversible.
3. **New hook to check if a room is disabled** - the client can leverage
existing on-chain public read only methods `getSpaceInfoBySpaceId`,
`getChannelInfoByChannelId` to read the state of each in order to remove
spaces, channels from a member's view that are disabled.
Pulls in upstream latest changes from [dendrite-fork
](https://github.com/HereNotThere/dendrite)to subtree at
servers/dendrite here.
Co-authored-by: Tak Wai Wong <64229756+tak-hntlabs@users.noreply.github.com>
Co-authored-by: Tak Wai Wong <tak@hntlabs.com>
Co-authored-by: John Terzis <john@hntlabs.com>
Based on #2480
This actually indexes events based on their event type. They are removed
from the index if we receive a `m.room.redaction` event on the
`OutputRoomEvent` stream.
An admin endpoint is added to reindex all existing events.
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
This PR changes the handling of notifications
- removes the `StreamEvent` and `ReadUpdate` stream
- listens on the `OutputRoomEvent` stream in the UserAPI to inform the
SyncAPI about unread notifications
- listens on the `OutputReceiptEvent` stream in the UserAPI to set
receipts/update notifications
- sets the `read_markers` directly from within the internal UserAPI
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* add config yaml for enable_auth
* zion_space_manager_localhost.go
* Placeholders for authorization
* rename func and type
* re-run go mod tidy
Co-authored-by: Tak Wai Wong <tak@hntlabs.com>
Beforehand we disabled HTTP keepalives to prevent ambient system
resources from being used by excess idle connections. Now that we've
fixed some bugs in the federation API and device list updater, this
situation is now much better and we don't open so many remote
connections anyway.
Keepalives allow us to not have to handshake TLS so often (which is
quite expensive) and reusing an idle connection is much faster than
having to open a new one. This can help with response times when talking
to remote federated servers.
This PR also adds a new option to disable keepalives if needed:
```
# Disable HTTP keepalives, which also prevents connection reuse. Dendrite will typically
# keep HTTP connections open to remote hosts for 5 minutes as they can be reused much
# more quickly than opening new connections each time. Disabling keepalives will close
# HTTP connections immediately after a successful request but may result in more CPU and
# memory being used on TLS handshakes for each new connection instead.
disable_http_keepalives: false
```
This adds the main component of the fulltext search.
This PR doesn't do anything yet, besides creating an empty fulltextindex
folder if enabled. Indexing events is done in a separate PR.
This makes the following changes:
* The various `Defaults` functions are now responsible for setting sane defaults if `generate` is specified, rather than hiding them in `generate-config`
* Some configuration options have been marked as `omitempty` so that they don't appear in generated configs unnecessarily (monolith-specific vs. polylith-specific options)
* A new option `-polylith` has been added to `generate-config` to create a config that makes sense for polylith deployments (i.e. including the internal/external API listeners and per-component database sections)
* A new option `-normalise` has been added to `generate-config` to take an existing file and add any missing options and/or defaults
This PR refactors the app services component. It makes the following changes:
* Each appservice now gets its own NATS JetStream consumer
* The appservice database is now removed entirely, since we just use JetStream as a data source instead
* The entire component is now much simpler and we deleted lots of lines of code 💅
The result is that it should be much lighter and hopefully much more performant.
This refactors the `dendrite-demo-pinecone` executable so that it:
1. Converts the old `.key` file into a standard `.pem` file
2. Allows passing in the `--config` option to supply a normal Dendrite configuration file, so that you can configure PostgreSQL instead of SQLite, appservices and all the other usual stuff
* Blacklist some sytest tests that are failing in our environment
* Commenting out test that isn't reliably passing or failing, probably a race
* refresh latest dendrite main
* use rand.Seed to seed the random function
Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
* Blacklist some sytest tests that are failing in our environment
* Commenting out test that isn't reliably passing or failing, probably a race
* refresh latest dendrite main
* refresh latest dendrite main
* dendrite implementation of eip-4361
* simplify nonce generation
Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
* Generic-based internal HTTP API (tested out on a few endpoints in the federation API)
* Add `PerformInvite`
* More tweaks
* Fix metric name
* Fix LookupStateIDs
* Lots of changes to clients
* Some serverside stuff
* Some error handling
* Use paths as metric names
* Revert "Use paths as metric names"
This reverts commit a9323a6a34.
* Namespace metric names
* Remove duplicate entry
* Remove another duplicate entry
* Tweak error handling
* Some more tweaks
* Update error behaviour
* Some more error tweaking
* Fix API path for `PerformDeleteKeys`
* Fix another path
* Tweak federation client proxying
* Fix another path
* Don't return typed nils
* Some more tweaks, not that it makes any difference
* Tweak federation client proxying
* Maybe fix the key backup test
* Add race testing to tests, and fix a few small race conditions in the tests
* Enable run-sytest on MacOS
* Remove deadlock detecting mutex, per code review feedback
* Remove autoformatting related changes and a closure that is not needed
* Adjust to importing nats client as 'natsclient'
Signed-off-by: Brian Meek <brian@hntlabs.com>
* Clarify the use of gooseMutex to proect goose internal state
Signed-off-by: Brian Meek <brian@hntlabs.com>
* Remove no longer needed mutex for guarding goose
Signed-off-by: Brian Meek <brian@hntlabs.com>
* Set CORS headers for the 404s
* Use custom handlers, plus one for HTTP 405 too
* Tweak setup
* Add to muxes too
* Tidy up some more
* Use built-in HTTP 404 handler
* Don't bother setting it for federation-facing
* Try Ristretto cache
* Tweak
* It's beautiful
* Update GMSL
* More strict keyable interface
* Fix that some more
* Make less panicky
* Don't enforce mutability checks for now
* Determine mutability using deep equality
* Tweaks
* Namespace keys
* Make federation caches mutable
* Update cost estimation, add metric
* Update GMSL
* Estimate cost for metrics better
* Reduce counters a bit
* Try caching events
* Some guards
* Try again
* Try this
* Use separate caches for hopefully better hash distribution
* Fix bug with admitting events into cache
* Try to fix bugs
* Check nil
* Try that again
* Preserve order jeezo this is messy
* thanks VS Code for doing exactly the wrong thing
* Try this again
* Be more specific
* aaaaargh
* One more time
* That might be better
* Stronger sorting
* Cache expiries, async publishing of EDUs
* Put it back
* Use a shared cache again
* Cost estimation fixes
* Update ristretto
* Reduce counters a bit
* Clean up a bit
* Update GMSL
* 1GB
* Configurable cache sizees
* Tweaks
* Add `config.DataUnit` for specifying friendly cache sizes
* Various tweaks
* Update GMSL
* Add back some lazy loading caching
* Include key in cost
* Include key in cost
* Tweak max age handling, config key name
* Only register prometheus metrics if requested
* Review comments @S7evinK
* Don't return errors when creating caches (it is better just to crash since otherwise we'll `nil`-pointer exception everywhere)
* Review comments
* Update sample configs
* Update GHA Workflow
* Update Complement images to Go 1.18
* Remove the cache test from the federation API as we no longer guarantee immediate cache admission
* Don't check the caches in the renewal test
* Possibly fix the upgrade tests
* Update to matrix-org/gomatrixserverlib#322
* Update documentation to refer to Go 1.18
* Rate limiting changes
This makes the following changes:
* For logged in users, the rate limiting now applies to the device session rather than the remote IP address;
* For non-logged in users, the rate limiting continues to apply to remote address as it does today;
* It is now possible to add user IDs to the `exempt_user_ids` option under `rate_limiting` to exclude bots from rate limiting;
* Admin and appservice users are now exempt from rate limiting by default.
* Fix build with media API
* Add very basic syncapi tests
* Add a way to inject jetstream messages
* implement add_state_ids
* bugfixes
* Unbreak tests
* Remove now un-needed API call
* Linting
