Commit graph

2863 commits

Author SHA1 Message Date
devonh bb2ab62cbf
Handle event_format federation in /sync responses (#3192) 2023-08-31 15:33:38 +00:00
Till Faelligen 11fd2f019b
Fix Complement scheduled CI
[skip CI]
2023-08-30 07:37:14 +02:00
Omar Pakker b538f237df
[helm] Update Ingress hosts to account for IPv6 (server+client) and scheme (client) (#3182)
This updates the matchers for deriving the host values from the dendrite
config. The original version turned out to have 2 complications:
- It did not support IPv6 addresses as host value
- It failed for `well_known_client_host` which is a (base) URL instead
of a hostname+port.

I've verified `well_known_server_name` with
```
dendrite.example.net:443
dendrite.example.net
192.168.1.1
192.168.1.1:1324
[dead::beef]:1234
[dead::beef]
[ffff:dead::beef]
```
and `well_known_client_name` with:
```
https://dendrite.example.net:443
https://dendrite.example.net
https://dendrite.example.net/
http://dendrite.example.net:8080/
http://192.168.1.1
http://192.168.1.1:8080/
http://[dead::beef]:1234
http://[dead::beef]/
http://[ffff:dead::beef]
```

Fixes #3175

### Pull Request Checklist

<!-- Please read
https://matrix-org.github.io/dendrite/development/contributing before
submitting your pull request -->

* [x] I have added Go unit tests or [Complement integration
tests](https://github.com/matrix-org/complement) for this PR _or_ I have
justified why this PR doesn't need tests
* [x] Pull request includes a [sign off below using a legally
identifiable
name](https://matrix-org.github.io/dendrite/development/contributing#sign-off)
_or_ I have already signed off privately

Signed-off-by: `Omar Pakker <Omar007@users.noreply.github.com>`

---------

Signed-off-by: Omar Pakker <Omar007@users.noreply.github.com>

[skip CI]
2023-08-29 08:20:37 +02:00
Till e3a7039c81
Fix CI, upgrade image used for upgrade tests (#3151) 2023-08-28 13:28:22 +02:00
dependabot[bot] 43b1ddb89b
Bump commonmarker from 0.23.9 to 0.23.10 in /docs (#3172)
Bumps [commonmarker](https://github.com/gjtorikian/commonmarker) from
0.23.9 to 0.23.10.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/gjtorikian/commonmarker/releases">commonmarker's
releases</a>.</em></p>
<blockquote>
<h2>v0.23.10</h2>
<h2>What's Changed</h2>
<ul>
<li>Update to 0.29.0.gfm.13 by <a
href="https://github.com/anticomputer"><code>@​anticomputer</code></a>
in <a
href="https://redirect.github.com/gjtorikian/commonmarker/pull/247">gjtorikian/commonmarker#247</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/gjtorikian/commonmarker/compare/v0.23.9...v0.23.10">https://github.com/gjtorikian/commonmarker/compare/v0.23.9...v0.23.10</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/gjtorikian/commonmarker/blob/v0.23.10/CHANGELOG.md">commonmarker's
changelog</a>.</em></p>
<blockquote>
<h2>[v0.23.10] (2023-07-31)</h2>
<ul>
<li>Update GFM release to <a
href="https://github.com/github/cmark-gfm/releases/tag/0.29.0.gfm.12"><code>0.29.0.gfm.12</code></a>
and <a
href="https://github.com/github/cmark-gfm/releases/tag/0.29.0.gfm.13"><code>0.29.0.gfm.13</code></a>,
thereby <a
href="https://github.com/github/cmark-gfm/security/advisories/GHSA-w4qg-3vf7-m9x5">fixing
a polynomial time complexity security vulnerability</a>.</li>
<li>Of note to users of this library, GFM releases
<code>0.29.0.gfm.12</code> and <code>0.29.0.gfm.13</code> also:
<ul>
<li>Normalized marker row vs. delimiter row nomenclature (<a
href="https://redirect.github.com/github/cmark-gfm/pull/273">#273</a>)</li>
<li>Exposed CMARK_NODE_FOOTNOTE_DEFINITION literal value (<a
href="https://redirect.github.com/github/cmark-gfm/pull/336">#336</a>)</li>
</ul>
</li>
</ul>
<h2><a
href="https://github.com/gjtorikian/commonmarker/tree/v0.23.4">v0.23.4</a>
(2022-03-03)</h2>
<p><a
href="https://github.com/gjtorikian/commonmarker/compare/v0.23.2...v0.23.4">Full
Changelog</a></p>
<p><strong>Fixed bugs:</strong></p>
<ul>
<li><code>#render_html</code> way slower than
<code>#render_doc.to_html</code> <a
href="https://redirect.github.com/gjtorikian/commonmarker/issues/141">#141</a></li>
</ul>
<p><strong>Closed issues:</strong></p>
<ul>
<li>allow keeping text content of unknown tags <a
href="https://redirect.github.com/gjtorikian/commonmarker/issues/169">#169</a></li>
<li>STRIKETHROUGH_DOUBLE_TILDE not working <a
href="https://redirect.github.com/gjtorikian/commonmarker/issues/168">#168</a></li>
<li>Allow disabling 4-space code blocks <a
href="https://redirect.github.com/gjtorikian/commonmarker/issues/167">#167</a></li>
<li>tables with escaped pipes are not recognized <a
href="https://redirect.github.com/gjtorikian/commonmarker/issues/166">#166</a></li>
</ul>
<p><strong>Merged pull requests:</strong></p>
<ul>
<li>CI: Drop a duplicate 'bundle install' <a
href="https://redirect.github.com/gjtorikian/commonmarker/pull/173">#173</a>
(<a href="https://github.com/olleolleolle">olleolleolle</a>)</li>
<li>CI: Drop duplicate bundle install <a
href="https://redirect.github.com/gjtorikian/commonmarker/pull/172">#172</a>
(<a href="https://github.com/olleolleolle">olleolleolle</a>)</li>
<li>Fixup benchmark and speedup a little, fixes <a
href="https://redirect.github.com/gjtorikian/commonmarker/issues/141">#141</a>
<a
href="https://redirect.github.com/gjtorikian/commonmarker/pull/171">#171</a>
(<a href="https://github.com/ojab">ojab</a>)</li>
</ul>
<h2><a
href="https://github.com/gjtorikian/commonmarker/tree/v0.23.2">v0.23.2</a>
(2021-09-17)</h2>
<p><a
href="https://github.com/gjtorikian/commonmarker/compare/v0.23.1...v0.23.2">Full
Changelog</a></p>
<p><strong>Merged pull requests:</strong></p>
<ul>
<li>Update GFM release to <code>0.29.0.gfm.2</code> <a
href="https://redirect.github.com/gjtorikian/commonmarker/pull/148">#148</a>
(<a href="https://github.com/phillmv">phillmv</a>)</li>
</ul>
<h2><a
href="https://github.com/gjtorikian/commonmarker/tree/v0.23.1">v0.23.1</a>
(2021-09-03)</h2>
<p><a
href="https://github.com/gjtorikian/commonmarker/compare/v0.23.0...v0.23.1">Full
Changelog</a></p>
<p><strong>Closed issues:</strong></p>
<ul>
<li>Incorrect processing of list and next block of code <a
href="https://redirect.github.com/gjtorikian/commonmarker/issues/146">#146</a></li>
</ul>
<p><strong>Merged pull requests:</strong></p>
<ul>
<li>Normalize parse and render options <a
href="https://redirect.github.com/gjtorikian/commonmarker/pull/145">#145</a>
(<a href="https://github.com/phillmv">phillmv</a>)</li>
</ul>
<h2><a
href="https://github.com/gjtorikian/commonmarker/tree/v0.23.0">v0.23.0</a>
(2021-08-30)</h2>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="db8cd377b5"><code>db8cd37</code></a>
Merge pull request <a
href="https://redirect.github.com/gjtorikian/commonmarker/issues/247">#247</a>
from anticomputer/update-to-0.29.0.gfm.13</li>
<li><a
href="e1e450c381"><code>e1e450c</code></a>
💎 release 0.23.10</li>
<li><a
href="08b7c4b96c"><code>08b7c4b</code></a>
Update cmark-upstream to <a
href="https://github.com/github/cmark-gfm/commit/587a12bb5">https://github.com/github/cmark-gfm/commit/587a12bb5</a>...</li>
<li><a
href="d0e81e2392"><code>d0e81e2</code></a>
I've used this version of the update_submodules script for several
releases, ...</li>
<li>See full diff in <a
href="https://github.com/gjtorikian/commonmarker/compare/v0.23.9...v0.23.10">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=commonmarker&package-manager=bundler&previous-version=0.23.9&new-version=0.23.10)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/matrix-org/dendrite/network/alerts).

</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

[skip ci]
2023-08-28 12:51:47 +02:00
devonh 1c4ec67bb6
Add configuration option for sliding sync when hosting /.well-known/matrix/client (#3189)
Adds the `org.matrix.msc3575.proxy` field (used for configuring sliding
sync) to /.well-known/matrix/client when Dendrite is serving that
endpoint and `well_known_sliding_sync_proxy` has been configured.

ie. Config values of:
``` yaml
global:
    well_known_client_name: https://example.com
    well_known_sliding_sync_proxy: https://syncv3.example.com
```
results in a /.well-known/matrix/client of:
``` json
{
    "m.homeserver": {
        "base_url": "https://example.com"
    },
    "org.matrix.msc3575.proxy": {
        "url": "https://syncv3.example.com"
    }
}
```

If `well_known_sliding_sync_proxy` is not provided, the json provided by
/.well-known/matrix/client does not include the proxy field.
ie.
``` json
{
    "m.homeserver": {
        "base_url": "https://example.com"
    }
}
```
2023-08-24 21:08:40 +00:00
Sam Wedgwood 9b5be6b9c5
[pseudoIDs] More pseudo ID fixes - Part 2 (#3181)
Fixes include:
- Translating state keys that contain user IDs to their respective room
keys for both querying and sending state events
- **NOTE**: there may be design discussion needed on what should happen
when sender keys cannot be found for users
- A simple fix for kicking guests from rooms properly
- Logic for boundary history visibilities was slightly off (I'm
surprised this only manifested in pseudo ID room versions)

Signed-off-by: `Sam Wedgwood <sam@wedgwood.dev>`
2023-08-24 16:43:51 +01:00
Devon Hudson a721294e2b
Bump pinecone docker go version 2023-08-23 08:56:44 -06:00
Till Faelligen 845800abfa
Bump helm chart version 2023-08-23 16:44:52 +02:00
Till 57ddbe015d
Version 0.13.2 (#3187) 2023-08-23 16:24:16 +02:00
Sam Wedgwood 9a12420428
[pseudoID] More pseudo ID fixes (#3167)
Signed-off-by: `Sam Wedgwood <sam@wedgwood.dev>`
2023-08-15 12:37:04 +01:00
devonh fa6c7ba456
Update pinecone to use new quic version (#3174) 2023-08-11 14:29:48 +00:00
Sam Wedgwood 35804f8493
Add config key for default room version (#3171)
This PR adds a config key `room_server.default_config_key` to set the
default room version for the room server.

Signed-off-by: `Sam Wedgwood <sam@wedgwood.dev>`
2023-08-08 14:20:05 +01:00
maxberger 294eff8a7f
Add ID in error messages for ApplicationServices (#3162)
This is to easier identify which service caused the error.

Feature is just improving logging, thus no tests added.

### Pull Request Checklist

<!-- Please read
https://matrix-org.github.io/dendrite/development/contributing before
submitting your pull request -->

* [X]  I have justified why this PR doesn't need tests
* [X] Pull request includes a [sign off below using a legally
identifiable
name](https://matrix-org.github.io/dendrite/development/contributing#sign-off)
_or_ I have already signed off privately

Signed-off-by: `Maximilian Berger <max@berger.name>`

Co-authored-by: Till <2353100+S7evinK@users.noreply.github.com>
2023-08-03 09:26:42 +02:00
Sam Wedgwood c7193e24d0
Use *spec.SenderID for QuerySenderIDForUser (#3164)
There are cases where a dendrite instance is unaware of a pseudo ID for
a user, the user is not a member of that room. To represent this case,
we currently use the 'zero' value, which is often not checked and so
causes errors later down the line. To make this case more explict, and
to be consistent with `QueryUserIDForSender`, this PR changes this to
use a pointer (and `nil` to mean no sender ID).

Signed-off-by: `Sam Wedgwood <sam@wedgwood.dev>`
2023-08-02 11:12:14 +01:00
Sam Wedgwood af13fa1c75
[pseudoIDs] Fixes for room alias tests (#3159)
Some (deceptively) simple fixes for some bugs that caused room alias
tests to fail (sytext `tests/30rooms/05aliases.pl`). Each commit has
details about what it fixes.

Sytest results:

- Sytest before (79d4a0e):
https://gist.github.com/swedgwood/972ac4ef93edd130d3db0930703d6c82
- Sytest after (4b09bed):
https://gist.github.com/swedgwood/504b00ac4ee892acb757b7fac55fa28a

Room aliases go from `8/15` to `15/15`, but looks like these fixes also
managed to fix about `4` other tests, which is a nice bonus :)

Signed-off-by: `Sam Wedgwood <sam@wedgwood.dev>`
2023-07-31 14:39:41 +01:00
Till 3f727485d6
Send a more generic error message to clients if the file can't be found (#3161)
Fixes #3160
2023-07-28 08:40:05 +02:00
Till Faelligen 79d4a0e399
Restore old behaviour of PurgeRoom 2023-07-26 09:09:04 +02:00
George Antoniadis 7899f47e71
add deployment strategy option to helm chart (re #3021) (#3155)
@S7evinK sorry for the spam but any chance we get get this merged into
main at some point? It was previously merged in
https://github.com/matrix-org/dendrite/pull/3021 into a temp branch that
never made it into main. If there is an issue with this being merged let
me know.

---

Minor update to the helm chart to allow setting the update strategy as
the default `RollingUpdate` one is a bit annoying if using
`ReadWriteOnce` volumes for media. Hope this makes sense.

---

### Pull Request Checklist

<!-- Please read
https://matrix-org.github.io/dendrite/development/contributing before
submitting your pull request -->

* [x] ~~I have added Go unit tests or [Complement integration
tests](https://github.com/matrix-org/complement) for this PR _or_ I have
justified why this PR doesn't need tests~~ Haven't touched any go files.
* [x] Pull request includes a [sign off below using a legally
identifiable
name](https://matrix-org.github.io/dendrite/development/contributing#sign-off)
_or_ I have already signed off privately

Signed-off-by: `George Antoniadis <george@noodles.gr>` [skip ci]
2023-07-26 08:16:43 +02:00
Devon Hudson a48c7d33a5
Don't quit if unknown msc in config, log it and keep going 2023-07-21 13:08:28 -06:00
devonh c809e95335
Fix event federation with pseudoID rooms (#3156) 2023-07-21 16:08:40 +00:00
Till e216c2fbf0
Update ConnectionManager to still allow component defined connections (#3154) 2023-07-21 08:34:01 +02:00
Sam Wedgwood 9582827493
de-MSC-ifying space summaries (MSC2946) (#3134)
- This PR moves and refactors the
[code](https://github.com/matrix-org/dendrite/blob/main/setup/mscs/msc2946/msc2946.go)
for
[MSC2946](https://github.com/matrix-org/matrix-spec-proposals/pull/2946)
('Space Summaries') to integrate it into the rest of the codebase.
- Means space summaries are no longer hidden behind an MSC flag
- Solves #3096

Signed-off-by: Sam Wedgwood <sam@wedgwood.dev>
2023-07-20 15:06:05 +01:00
Till 297479ea49
Use pointer when passing the connection manager around (#3152)
As otherwise existing connections aren't reused.
2023-07-19 13:37:04 +02:00
devonh a01faee17c
Extend context timeout on send_join to allow for joining complex rooms (#3153)
Background federated joins are currently broken since they timeout after
30s. This timeout didn't exist before the refactor. It should still exist but it needs to be extended to allow for the additional time it can take a server to generate the /send_join response when joining a complex room.
2023-07-18 18:48:05 +00:00
Till Faelligen 33ff309572
Don't HTTP500 if a profile does't exist 2023-07-14 14:24:31 +02:00
Till Faelligen 6011ddc0a8
Discard "illegal base64 data at input byte 0" errors in the SyncAPI 2023-07-14 08:28:30 +02:00
Till Faelligen 3e314e028e
Avoid panic due to being unable to query the userID 2023-07-14 08:04:25 +02:00
Till 5267cc0f54
Optimise getting local members and membership counts (#3150)
The previous version was getting **ALL** membership events (as
`ClientEvents`, so going through `NewEventFromTrustedJSONWithID`) for a
given room.
Now we are querying only locally joined users as `ClientEvents`, which
should **significantly** reduce allocations.

Take for example a large room with 2k membership events, but only 1
local user - avoiding 1999 `NewEventFromTrustedJSONWithID` calls just to
calculate the `roomSize` which we can also query by other means.

This is also getting called for every `OutputRoomEvent` in the userAPI.

Benchmark with 1 local user and 100 remote users.
```
pkg: github.com/matrix-org/dendrite/userapi/consumers
cpu: 12th Gen Intel(R) Core(TM) i5-12500H
                    │   old.txt   │               new.txt               │
                    │   sec/op    │   sec/op     vs base                │
LocalRoomMembers-16   375.9µ ± 7%   327.6µ ± 6%  -12.85% (p=0.000 n=10)

                    │    old.txt    │               new.txt                │
                    │     B/op      │     B/op      vs base                │
LocalRoomMembers-16   79.426Ki ± 0%   8.507Ki ± 0%  -89.29% (p=0.000 n=10)

                    │   old.txt   │              new.txt               │
                    │  allocs/op  │ allocs/op   vs base                │
LocalRoomMembers-16   1015.0 ± 0%   277.0 ± 0%  -72.71% (p=0.000 n=10)
```
2023-07-13 14:19:08 +02:00
Till f12982472c
Tweaks around /messages (#3149)
Try to mitigate some issues with `/messages`
2023-07-13 14:18:37 +02:00
Till Faelligen 0df982a2e5
Update NATS again [skip ci] 2023-07-13 14:17:48 +02:00
Till 99f94fc735
Add revision to version string (#3147)
Since the removal of `build.sh`, we don't include any information about
the revision Dendrite was build from. Since go1.18, the revision a
binary was build from is automatically included, so we can try to get
that instead.

This also adds a `dendrite_up` metric showing the current version
(`dendrite_up{version="0.13.1+c796f20"} 1`)

Closes #2993
2023-07-11 13:56:25 +02:00
Till 69b2069dea
Avoid loops by setting end to an empty string if start == end (#3146) 2023-07-08 11:45:44 +02:00
Till Faelligen b965a08faa
Unknown issue 2023-07-07 22:52:23 +02:00
Till Faelligen ef32de928d
[NATS] Issue identified and fixed applied, workaround known. 2023-07-07 22:10:52 +02:00
Till 74a5ab6c24
Fix issues reported by Sentry (#3143)
This should fix a few issues reported by Sentry
2023-07-07 22:00:10 +02:00
Till eb9e90379d
Add event size checks similar to Synapse (#3140)
Companion to https://github.com/matrix-org/gomatrixserverlib/pull/400
This tries to mimic the logic found in Synapse, as dropping events can
break rooms (and we may end up in endless loops..)
2023-07-07 20:37:23 +02:00
Neil e93bdd56fd
Set max age for roomserver input stream to avoid excessive interior deletes (#3145)
If old messages build up in the input stream and do not get processed
successfully, this can create a significant drift between the stream
first sequence and the consumer ack floors, which results in a slow and
expensive start-up when interest-based retention is in use.

If a message is sat in the stream for 24 hours, it's probably not going
to get processed successfully, so let NATS drop them instead. Dendrite
can reconcile by fetching missing events later if it needs to.

---------

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2023-07-07 19:59:34 +02:00
Till c08c7405db
Prepare statement on an existing transaction (#3144)
This should fix an issue with the database being locked for SQLite.
2023-07-07 13:09:39 +02:00
devonh cc9b695c1e
Populate syncapi state event prev_sender with userID (#3142) 2023-07-06 23:54:35 +00:00
devonh 3a125fd8fa
Fix prev event lookup in syncapi (#3141)
The syncapi operates using userID's so when querying for the previous
state event we need to lookup the userID from the given senderID before
the state query.
2023-07-06 19:50:28 +00:00
devonh d507c5fc95
Add pseudoID compatibility to Invites (#3126) 2023-07-06 15:15:24 +00:00
Till Faelligen fea946d914
Don't spam the logs - downgrade sentry 2023-07-06 10:55:21 +02:00
Till Faelligen 9f7e14e4d0
Back to the original version for now 2023-07-06 10:44:11 +02:00
Till Faelligen 4a666932f5
[debug] Downgrade NATS 2023-07-06 10:31:32 +02:00
Till Faelligen e1d76de6c6
Increase NATS server startup timeout 2023-07-06 10:04:46 +02:00
Till 49d75d3cf6
Version 0.13.1 (#3136) 2023-07-06 09:28:39 +02:00
Till Faelligen 5a87c703fa
Fix metrics.. 2023-07-05 12:34:53 +02:00
Till 4c3a526e1b
Fix adding state events to the database (#3133)
When we're adding state to the database, we check which eventNIDs are
already in a block, if we already have that eventNID, we remove it from
the list. In its current form we would skip over eventNIDs in the case
we already found a match (we're decrementing `i` twice)
My theory is, that when we later get the state blocks, we are receiving
"too many" eventNIDs (well, yea, we stored too many), which may or may
not can result in state resets when comparing different state snapshots.
(e.g. when adding state we stored a eventNID by accident because we
skipped it, later we add more state and are not adding it because we
don't skip it)
2023-07-04 17:15:44 +02:00
Till 2ee03fd657
Version 0.13.0 (#3127) 2023-06-30 08:49:37 +02:00