Merge branch 'main' of github.com:matrix-org/dendrite into s7evink/consent-tracking

This commit is contained in:
Till Faelligen 2022-03-07 09:46:28 +01:00
commit 710007d600
12 changed files with 115 additions and 62 deletions

View file

@ -1,5 +1,31 @@
# Changelog
## Dendrite 0.6.5 (2022-03-04)
### Features
* Early support for push notifications has been added, with support for push rules, pushers, HTTP push gateways and the `/notifications` endpoint (contributions by [danpe](https://github.com/danpe), [PiotrKozimor](https://github.com/PiotrKozimor) and [tommie](https://github.com/tommie))
* Spaces Summary (MSC2946) is now correctly supported (when `msc2946` is enabled in the config)
* All media API endpoints are now available under the `/v3` namespace
* Profile updates (display name and avatar) are now sent asynchronously so they shouldn't block the client for a very long time
* State resolution v2 has been optimised further to considerably reduce the number of memory allocations
* State resolution v2 will no longer duplicate events unnecessarily when calculating the auth difference
* The `create-account` tool now has a `-reset-password` option for resetting the passwords of existing accounts
* The `/sync` endpoint now calculates device list changes much more quickly with less RAM used
* The `/messages` endpoint now lazy-loads members correctly
### Fixes
* Read receipts now work correctly by correcting bugs in the stream positions and receipt coalescing
* Topological sorting of state and join responses has been corrected, which should help to reduce the number of auth problems when joining new federated rooms
* Media thumbnails should now work properly after having unnecessarily strict rate limiting removed
* The roomserver no longer holds transactions for as long when processing input events
* Uploading device keys and cross-signing keys will now correctly no-op if there were no changes
* Parameters are now remembered correctly during registration
* Devices can now only be deleted within the appropriate UIA flow
* The `/context` endpoint now returns 404 instead of 500 if the event was not found
* SQLite mode will no longer leak memory as a result of not closing prepared statements
## Dendrite 0.6.4 (2022-02-21)
### Features
@ -210,9 +236,9 @@
### Fixes
- **SECURITY:** A bug in SQLite mode which could cause the registration flow to complete unexpectedly for existing accounts has been fixed (PostgreSQL deployments are not affected)
- A panic in the federation sender has been fixed when shutting down destination queues
- The `/keys/upload` endpoint now correctly returns the number of one-time keys in response to an empty upload request
* **SECURITY:** A bug in SQLite mode which could cause the registration flow to complete unexpectedly for existing accounts has been fixed (PostgreSQL deployments are not affected)
* A panic in the federation sender has been fixed when shutting down destination queues
* The `/keys/upload` endpoint now correctly returns the number of one-time keys in response to an empty upload request
## Dendrite 0.3.10 (2021-02-17)

View file

@ -2,6 +2,7 @@
Dendrite is a second-generation Matrix homeserver written in Go.
It intends to provide an **efficient**, **reliable** and **scalable** alternative to [Synapse](https://github.com/matrix-org/synapse):
- Efficient: A small memory footprint with better baseline performance than an out-of-the-box Synapse.
- Reliable: Implements the Matrix specification as written, using the
[same test suite](https://github.com/matrix-org/sytest) as Synapse as well as
@ -9,12 +10,14 @@ It intends to provide an **efficient**, **reliable** and **scalable** alternativ
- Scalable: can run on multiple machines and eventually scale to massive homeserver deployments.
As of October 2020, Dendrite has now entered **beta** which means:
- Dendrite is ready for early adopters. We recommend running in Monolith mode with a PostgreSQL database.
- Dendrite has periodic semver releases. We intend to release new versions as we land significant features.
- Dendrite supports database schema upgrades between releases. This means you should never lose your messages when upgrading Dendrite.
- Breaking changes will not occur on minor releases. This means you can safely upgrade Dendrite without modifying your database or config file.
This does not mean:
- Dendrite is bug-free. It has not yet been battle-tested in the real world and so will be error prone initially.
- All of the CS/Federation APIs are implemented. We are tracking progress via a script called 'Are We Synapse Yet?'. In particular,
presence and push notifications are entirely missing from Dendrite. See [CHANGES.md](CHANGES.md) for updates.
@ -34,11 +37,13 @@ If you have further questions, please take a look at [our FAQ](docs/FAQ.md) or j
To build Dendrite, you will need Go 1.16 or later.
For a usable federating Dendrite deployment, you will also need:
- A domain name (or subdomain)
- A valid TLS certificate issued by a trusted authority for that domain
- SRV records or a well-known file pointing to your deployment
Also recommended are:
- A PostgreSQL database engine, which will perform better than SQLite with many users and/or larger rooms
- A reverse proxy server, such as nginx, configured [like this sample](https://github.com/matrix-org/dendrite/blob/master/docs/nginx/monolith-sample.conf)
@ -76,18 +81,18 @@ Then point your favourite Matrix client at `http://localhost:8008` or `https://l
We use a script called Are We Synapse Yet which checks Sytest compliance rates. Sytest is a black-box homeserver
test rig with around 900 tests. The script works out how many of these tests are passing on Dendrite and it
updates with CI. As of January 2022 we're at around 65% CS API coverage and 92% Federation coverage, though check
updates with CI. As of March 2022 we're at around 76% CS API coverage and 95% Federation coverage, though check
CI for the latest numbers. In practice, this means you can communicate locally and via federation with Synapse
servers such as matrix.org reasonably well. There's a long list of features that are not implemented, notably:
- Push
- Search and Context
- Search
- User Directory
- Presence
- Guests
We are prioritising features that will benefit single-user homeservers first (e.g Receipts, E2E) rather
than features that massive deployments may be interested in (User Directory, OpenID, Guests, Admin APIs, AS API).
This means Dendrite supports amongst others:
- Core room functionality (creating rooms, invites, auth rules)
- Federation in rooms v1-v7
- Backfilling locally and via federation
@ -97,9 +102,11 @@ This means Dendrite supports amongst others:
- Media APIs
- Redaction
- Tagging
- Context
- E2E keys and device lists
- Receipts
- Push
- Guests
## Contributing
@ -112,6 +119,7 @@ For example, if the test `Local device key changes get to remote servers` was ma
test file (e.g via `grep` or via the
[CI log output](https://buildkite.com/matrix-dot-org/dendrite/builds/2826#39cff5de-e032-4ad0-ad26-f819e6919c42)
it's `tests/50federation/40devicelists.pl` ) then to run Sytest:
```
docker run --rm --name sytest
-v "/Users/kegan/github/sytest:/sytest"
@ -121,10 +129,12 @@ docker run --rm --name sytest
-e "POSTGRES=1" -e "DENDRITE_TRACE_HTTP=1"
matrixdotorg/sytest-dendrite:latest tests/50federation/40devicelists.pl
```
See [sytest.md](docs/sytest.md) for the full description of these flags.
You can try running sytest outside of docker for faster runs, but the dependencies can be temperamental
and we recommend using docker where possible.
```
cd sytest
export PERL5LIB=$HOME/lib/perl5
@ -149,6 +159,7 @@ Dendrite in Monolith + SQLite works in a range of environments including iOS and
For small homeserver installations joined on ~10s rooms on matrix.org with ~100s of users in those rooms, including some
encrypted rooms:
- Memory: uses around 100MB of RAM, with peaks at around 200MB.
- Disk space: After a few months of usage, the database grew to around 2GB (in Monolith mode).
- CPU: Brief spikes when processing events, typically idles at 1% CPU.

2
go.mod
View file

@ -1,6 +1,6 @@
module github.com/matrix-org/dendrite
replace github.com/nats-io/nats-server/v2 => github.com/neilalexander/nats-server/v2 v2.7.4-0.20220302103432-6b04b9f12740
replace github.com/nats-io/nats-server/v2 => github.com/neilalexander/nats-server/v2 v2.7.2-0.20220217100407-087330ed46ad
replace github.com/nats-io/nats.go => github.com/neilalexander/nats.go v1.11.1-0.20220104162523-f4ddebe1061c

14
go.sum
View file

@ -480,6 +480,8 @@ github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaS
github.com/golang/protobuf v1.5.2 h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw=
github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/golang/snappy v0.0.3 h1:fHPg5GQYlCeLIPB9BZqMVR5nR9A+IM5zcgeTdjMYmLA=
github.com/golang/snappy v0.0.3/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/gologme/log v1.2.0/go.mod h1:gq31gQ8wEHkR+WekdWsqDuf8pXTUZA9BnnzTuPz1Y9U=
github.com/gologme/log v1.3.0 h1:l781G4dE+pbigClDSDzSaaYKtiueHCILUa/qSDsmHAo=
github.com/gologme/log v1.3.0/go.mod h1:yKT+DvIPdDdDoPtqFrFxheooyVmoqi0BAsw+erN3wA4=
@ -711,8 +713,8 @@ github.com/klauspost/compress v1.9.7/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0
github.com/klauspost/compress v1.10.3/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs=
github.com/klauspost/compress v1.11.3/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs=
github.com/klauspost/compress v1.11.13/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs=
github.com/klauspost/compress v1.14.4 h1:eijASRJcobkVtSt81Olfh7JX43osYLwy5krOJo6YEu4=
github.com/klauspost/compress v1.14.4/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk=
github.com/klauspost/compress v1.13.4 h1:0zhec2I8zGnjWcKyLl6i3gPqKANCCn5e9xmviEEeX6s=
github.com/klauspost/compress v1.13.4/go.mod h1:8dP1Hq4DHOhN9w426knH3Rhby4rFm6D8eO+e+Dq5Gzg=
github.com/klauspost/cpuid v1.2.1/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek=
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/konsorten/go-windows-terminal-sequences v1.0.2/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
@ -1027,8 +1029,8 @@ github.com/miekg/dns v1.1.31/go.mod h1:KNUDUusw/aVsxyTYZM1oqvCicbwhgbNgztCETuNZ7
github.com/miekg/pkcs11 v1.0.3/go.mod h1:XsNlhZGX73bx86s2hdc/FuaLm2CPZJemRLMA+WTFxgs=
github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1 h1:lYpkrQH5ajf0OXOcUbGjvZxxijuBwbbmlSxLiuofa+g=
github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1/go.mod h1:pD8RvIylQ358TN4wwqatJ8rNavkEINozVn9DtGI3dfQ=
github.com/minio/highwayhash v1.0.2 h1:Aak5U0nElisjDCfPSG79Tgzkn2gl66NxOMspRrKnA/g=
github.com/minio/highwayhash v1.0.2/go.mod h1:BQskDq+xkJ12lmlUUi7U0M5Swg3EWR+dLTk+kldvVxY=
github.com/minio/highwayhash v1.0.1 h1:dZ6IIu8Z14VlC0VpfKofAhCy74wu/Qb5gcn52yWoz/0=
github.com/minio/highwayhash v1.0.1/go.mod h1:BQskDq+xkJ12lmlUUi7U0M5Swg3EWR+dLTk+kldvVxY=
github.com/minio/sha256-simd v0.0.0-20190131020904-2d45a736cd16/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U=
github.com/minio/sha256-simd v0.0.0-20190328051042-05b4dd3047e5/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U=
github.com/minio/sha256-simd v0.1.0/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U=
@ -1130,8 +1132,8 @@ github.com/nbio/st v0.0.0-20140626010706-e9e8d9816f32/go.mod h1:9wM+0iRr9ahx58uY
github.com/ncw/swift v1.0.47/go.mod h1:23YIA4yWVnGwv2dQlN4bB7egfYX6YLn0Yo/S6zZO/ZM=
github.com/neelance/astrewrite v0.0.0-20160511093645-99348263ae86/go.mod h1:kHJEU3ofeGjhHklVoIGuVj85JJwZ6kWPaJwCIxgnFmo=
github.com/neelance/sourcemap v0.0.0-20151028013722-8c68805598ab/go.mod h1:Qr6/a/Q4r9LP1IltGz7tA7iOK1WonHEYhu1HRBA7ZiM=
github.com/neilalexander/nats-server/v2 v2.7.4-0.20220302103432-6b04b9f12740 h1:RJrc+z35RHZlrjR6UBt9UmVRAlFh4SgYyEA0YpQdPHM=
github.com/neilalexander/nats-server/v2 v2.7.4-0.20220302103432-6b04b9f12740/go.mod h1:eJUrA5gm0ch6sJTEv85xmXIgQWsB0OyjkTsKXvlHbYc=
github.com/neilalexander/nats-server/v2 v2.7.2-0.20220217100407-087330ed46ad h1:Z2nWMQsXWWqzj89nW6OaLJSdkFknqhaR5whEOz4++Y8=
github.com/neilalexander/nats-server/v2 v2.7.2-0.20220217100407-087330ed46ad/go.mod h1:tckmrt0M6bVaDT3kmh9UrIq/CBOBBse+TpXQi5ldaa8=
github.com/neilalexander/nats.go v1.11.1-0.20220104162523-f4ddebe1061c h1:G2qsv7D0rY94HAu8pXmElMluuMHQ85waxIDQBhIzV2Q=
github.com/neilalexander/nats.go v1.11.1-0.20220104162523-f4ddebe1061c/go.mod h1:BPko4oXsySz4aSWeFgOHLZs3G4Jq4ZAyE6/zMCxRT6w=
github.com/neilalexander/utp v0.1.1-0.20210622132614-ee9a34a30488/go.mod h1:NPHGhPc0/wudcaCqL/H5AOddkRf8GPRhzOujuUKGQu8=

View file

@ -17,7 +17,7 @@ var build string
const (
VersionMajor = 0
VersionMinor = 6
VersionPatch = 4
VersionPatch = 5
VersionTag = "" // example: "rc1"
)

View file

@ -294,7 +294,7 @@ func (r *Inputer) processRoomEvent(
}
// Store the event.
_, _, stateAtEvent, redactionEvent, redactedEventID, err := r.DB.StoreEvent(ctx, event, authEventNIDs, isRejected)
_, _, stateAtEvent, redactionEvent, redactedEventID, err := r.DB.StoreEvent(ctx, event, authEventNIDs, isRejected || softfail)
if err != nil {
return fmt.Errorf("updater.StoreEvent: %w", err)
}

View file

@ -154,6 +154,7 @@ func (s *eventStateKeyStatements) BulkSelectEventStateKey(
if err != nil {
return nil, err
}
defer selectPrep.Close()
stmt := sqlutil.TxStmt(txn, selectPrep)
rows, err := stmt.QueryContext(ctx, iEventStateKeyNIDs...)
if err != nil {

View file

@ -140,6 +140,7 @@ func (s *eventTypeStatements) BulkSelectEventTypeNID(
if err != nil {
return nil, err
}
defer selectPrep.Close()
stmt := sqlutil.TxStmt(txn, selectPrep)
///////////////

View file

@ -198,11 +198,12 @@ func (s *eventStatements) BulkSelectStateEventByID(
iEventIDs[k] = v
}
selectOrig := strings.Replace(bulkSelectStateEventByIDSQL, "($1)", sqlutil.QueryVariadic(len(iEventIDs)), 1)
selectStmt, err := s.db.Prepare(selectOrig)
selectPrep, err := s.db.Prepare(selectOrig)
if err != nil {
return nil, err
}
selectStmt = sqlutil.TxStmt(txn, selectStmt)
defer selectPrep.Close() // nolint:errcheck
selectStmt := sqlutil.TxStmt(txn, selectPrep)
///////////////
rows, err := selectStmt.QueryContext(ctx, iEventIDs...)
@ -266,11 +267,12 @@ func (s *eventStatements) BulkSelectStateEventByNID(
}
}
selectOrig += " ORDER BY event_type_nid, event_state_key_nid ASC"
selectStmt, err := s.db.Prepare(selectOrig)
selectPrep, err := s.db.Prepare(selectOrig)
if err != nil {
return nil, fmt.Errorf("s.db.Prepare: %w", err)
}
selectStmt = sqlutil.TxStmt(txn, selectStmt)
defer selectPrep.Close() // nolint:errcheck
selectStmt := sqlutil.TxStmt(txn, selectPrep)
rows, err := selectStmt.QueryContext(ctx, params...)
if err != nil {
return nil, fmt.Errorf("selectStmt.QueryContext: %w", err)
@ -307,11 +309,12 @@ func (s *eventStatements) BulkSelectStateAtEventByID(
iEventIDs[k] = v
}
selectOrig := strings.Replace(bulkSelectStateAtEventByIDSQL, "($1)", sqlutil.QueryVariadic(len(iEventIDs)), 1)
selectStmt, err := s.db.Prepare(selectOrig)
selectPrep, err := s.db.Prepare(selectOrig)
if err != nil {
return nil, err
}
selectStmt = sqlutil.TxStmt(txn, selectStmt)
defer selectPrep.Close() // nolint:errcheck
selectStmt := sqlutil.TxStmt(txn, selectPrep)
///////////////
rows, err := selectStmt.QueryContext(ctx, iEventIDs...)
if err != nil {
@ -390,10 +393,11 @@ func (s *eventStatements) BulkSelectStateAtEventAndReference(
if err != nil {
return nil, err
}
selectPrep = sqlutil.TxStmt(txn, selectPrep)
defer selectPrep.Close() // nolint:errcheck
selectStmt := sqlutil.TxStmt(txn, selectPrep)
//////////////
rows, err := sqlutil.TxStmt(txn, selectPrep).QueryContext(ctx, iEventNIDs...)
rows, err := sqlutil.TxStmt(txn, selectStmt).QueryContext(ctx, iEventNIDs...)
if err != nil {
return nil, fmt.Errorf("sqlutil.TxStmt.QueryContext: %w", err)
}
@ -441,6 +445,7 @@ func (s *eventStatements) BulkSelectEventReference(
if err != nil {
return nil, err
}
defer selectPrep.Close() // nolint:errcheck
///////////////
selectStmt := sqlutil.TxStmt(txn, selectPrep)
@ -471,11 +476,12 @@ func (s *eventStatements) BulkSelectEventID(ctx context.Context, txn *sql.Tx, ev
iEventNIDs[k] = v
}
selectOrig := strings.Replace(bulkSelectEventIDSQL, "($1)", sqlutil.QueryVariadic(len(iEventNIDs)), 1)
selectStmt, err := s.db.Prepare(selectOrig)
selectPrep, err := s.db.Prepare(selectOrig)
if err != nil {
return nil, err
}
selectStmt = sqlutil.TxStmt(txn, selectStmt)
defer selectPrep.Close() // nolint:errcheck
selectStmt := sqlutil.TxStmt(txn, selectPrep)
///////////////
rows, err := selectStmt.QueryContext(ctx, iEventNIDs...)
@ -526,11 +532,12 @@ func (s *eventStatements) bulkSelectEventNID(ctx context.Context, txn *sql.Tx, e
} else {
selectOrig = strings.Replace(bulkSelectEventNIDSQL, "($1)", sqlutil.QueryVariadic(len(iEventIDs)), 1)
}
selectStmt, err := s.db.Prepare(selectOrig)
selectPrep, err := s.db.Prepare(selectOrig)
if err != nil {
return nil, err
}
selectStmt = sqlutil.TxStmt(txn, selectStmt)
defer selectPrep.Close() // nolint:errcheck
selectStmt := sqlutil.TxStmt(txn, selectPrep)
///////////////
rows, err := selectStmt.QueryContext(ctx, iEventIDs...)
if err != nil {
@ -560,6 +567,7 @@ func (s *eventStatements) SelectMaxEventDepth(ctx context.Context, txn *sql.Tx,
if err != nil {
return 0, err
}
defer sqlPrep.Close()
err = sqlutil.TxStmt(txn, sqlPrep).QueryRowContext(ctx, iEventIDs...).Scan(&result)
if err != nil {
return 0, fmt.Errorf("sqlutil.TxStmt.QueryRowContext: %w", err)
@ -575,12 +583,13 @@ func (s *eventStatements) SelectRoomNIDsForEventNIDs(
if err != nil {
return nil, err
}
sqlPrep = sqlutil.TxStmt(txn, sqlPrep)
defer sqlPrep.Close()
sqlStmt := sqlutil.TxStmt(txn, sqlPrep)
iEventNIDs := make([]interface{}, len(eventNIDs))
for i, v := range eventNIDs {
iEventNIDs[i] = v
}
rows, err := sqlPrep.QueryContext(ctx, iEventNIDs...)
rows, err := sqlStmt.QueryContext(ctx, iEventNIDs...)
if err != nil {
return nil, err
}

View file

@ -233,12 +233,13 @@ func (s *roomStatements) SelectRoomVersionsForRoomNIDs(
if err != nil {
return nil, err
}
sqlPrep = sqlutil.TxStmt(txn, sqlPrep)
defer sqlPrep.Close() // nolint:errcheck
sqlStmt := sqlutil.TxStmt(txn, sqlPrep)
iRoomNIDs := make([]interface{}, len(roomNIDs))
for i, v := range roomNIDs {
iRoomNIDs[i] = v
}
rows, err := sqlPrep.QueryContext(ctx, iRoomNIDs...)
rows, err := sqlStmt.QueryContext(ctx, iRoomNIDs...)
if err != nil {
return nil, err
}

View file

@ -108,11 +108,12 @@ func (s *stateBlockStatements) BulkSelectStateBlockEntries(
intfs[i] = int64(stateBlockNIDs[i])
}
selectOrig := strings.Replace(bulkSelectStateBlockEntriesSQL, "($1)", sqlutil.QueryVariadic(len(intfs)), 1)
selectStmt, err := s.db.Prepare(selectOrig)
selectPrep, err := s.db.Prepare(selectOrig)
if err != nil {
return nil, err
}
selectStmt = sqlutil.TxStmt(txn, selectStmt)
defer selectPrep.Close() // nolint:errcheck
selectStmt := sqlutil.TxStmt(txn, selectPrep)
rows, err := selectStmt.QueryContext(ctx, intfs...)
if err != nil {
return nil, err

View file

@ -113,11 +113,12 @@ func (s *stateSnapshotStatements) BulkSelectStateBlockNIDs(
nids[k] = v
}
selectOrig := strings.Replace(bulkSelectStateBlockNIDsSQL, "($1)", sqlutil.QueryVariadic(len(nids)), 1)
selectStmt, err := s.db.Prepare(selectOrig)
selectPrep, err := s.db.Prepare(selectOrig)
if err != nil {
return nil, err
}
selectStmt = sqlutil.TxStmt(txn, selectStmt)
defer selectPrep.Close() // nolint:errcheck
selectStmt := sqlutil.TxStmt(txn, selectPrep)
rows, err := selectStmt.QueryContext(ctx, nids...)
if err != nil {