Commit graph

147 commits

Author SHA1 Message Date
Neil Alexander ee35c15467
Update sytest-whitelist 2021-11-08 09:25:01 +00:00
kegsay 42973e7d65
Add part of #1322 tests (#2034)
* Add part of #1322 tests

* Fix #1322
2021-11-01 15:05:20 +00:00
kegsay 92026a73ce
Remove tests from the blacklist (#2031)
* Remove tests from the blacklist

They seem to be more reliable now

* Update whitelist
2021-10-29 15:01:22 +01:00
Andrew Morgan a22a6d360f
Add 'Federation publicRoom Name/topic keys are correct' to the sytest whitelist (#2029)
Sytest had two tests with the name `Name/topic keys are correct`. https://github.com/matrix-org/sytest/pull/1098 edited one so that the names were unique.

It seems that this test name was in the dendrite Sytest whitelist. This PR adds the additional, new name to the whitelist as well. Time to see if CI passes!
2021-10-29 12:06:38 +01:00
kegsay 3b7a4ad640
sytest: update blacklist and whitelist (#2025)
Should make CI happy again
2021-10-21 15:35:36 +01:00
Neil Alexander d751e97e03
Update sytest blacklist/whitelist for known flaky tests (closes matrix-org/sytest#1155) 2021-10-13 09:19:00 +01:00
Neil Alexander ff21675c5b
Cross-signing fixes, notifications via sync, federation (#1974)
* Initial work on signing key update EDUs

* Fix build

* Produce/consume EDUs

* Producer logging

* Only produce key change notifications for local users

* Better naming

* Try to notify sync

* Enable feature

* Use key change topic

* Don't bother verifying signatures, validate key lengths if we can, notifier fixes

* Copyright notices

* Remove tests from whitelist until matrix-org/sytest#1117

* Some review comment fixes

* Update to matrix-org/gomatrixserverlib@f9416ac

* Remove unneeded parameter
2021-08-17 13:44:30 +01:00
Neil Alexander b1377d991a
Cross-signing signature handling (#1965)
* Handle other signatures

* Decorate key ID properly

* Match by key IDs

* Tweaks

* Fixes

* Fix /user/keys/query bug, review comments, update sytest-whitelist

* Various wtweaks

* Fix wiring for keyserver in API mode

* Additional fixes
2021-08-09 14:35:24 +01:00
Neil Alexander eb0efa4636
Cross-signing groundwork (#1953)
* Cross-signing groundwork

* Update to matrix-org/gomatrixserverlib#274

* Fix gobind builds, which stops unit tests in CI from yelling

* Some changes from review comments

* Fix build by passing in UIA

* Update to matrix-org/gomatrixserverlib@bec8d22

* Process master/self-signing keys from devices call

* nolint

* Enum-ify the key type in the database

* Process self-signing key too

* Fix sanity check in device list updater

* Fix check

* Fix sytest, hopefully

* Fix build
2021-08-04 17:56:29 +01:00
kegsay 32bf14a37c
Key Backups (3/3) : Implement querying keys and various bugfixes (#1946)
* Add querying device keys

Makes a bunch of sytests pass

* Apparently only the current version supports uploading keys

* Linting
2021-07-27 19:29:32 +01:00
kegsay b3754d68fc
Key Backups (2/3) : Add E2E backup key tables (#1945)
* Add PUT key backup endpoints and glue them to PerformKeyBackup

* Add tables for storing backup keys and glue them into the user API

* Don't create tables whilst still WIPing

* writer on sqlite please

* Linting
2021-07-27 17:08:53 +01:00
kegsay 32538640db
Key backups (1/2) : Add E2E session backup metadata tables (#1943)
* Initial key backup paths and userapi API

* Fix unit tests

* Add key backup table

* Glue REST API to database

* Linting

* use writer on sqlite
2021-07-27 12:47:32 +01:00
Neil Alexander c1447a58e5
Various alias fixes (#1934)
* Generate m.room.canonical_alias instead of legacy m.room.aliases

* Add omitempty tags

* Add aliases endpoint to client API

* Check power levels when setting aliases

* Don't return null on /aliases

* Don't return error if the state event fails

* Update sytest-whitelist

* Don't send updated m.room.canonical_alias events

* Don't check PLs after all because for local aliases they are apparently irrelevant

* Fix some bugs

* Allow deleting a local alias with enough PL

* Fix some more bugs

* Update sytest-whitelist

* Fix copyright notices

* Review comments
2021-07-21 16:53:50 +01:00
David Spenler 8d8fe485b4
Fix failing ban tests (#1884)
* Add room membership and powerlevel checks for func SendBan

* Added non-error return to func GetStateEvent when no state events with the specified state key are found

* Add passing tests to whitelist

* Fixed formatting

* Update roomserver/storage/shared/storage.go

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: kegsay <kegan@matrix.org>
Co-authored-by: kegsay <kegsay@gmail.com>
2021-07-19 18:33:05 +01:00
kegsay 728061db03
fedsender: try to satisfy all notary key requests from the cache first (#1925)
* fedsender: try to satisfy all notary key requests from the cache first

* Linting
2021-07-16 11:35:42 +01:00
kegsay c102adaf43
fedsender: add cache tables for notary keys (#1923)
* Add notary server tables for postgres

* Add sqlite tables

* fedsender: GetServerKeys -> QueryServerKeys

As it now checks a cache and can return multiple responses
2021-07-15 17:45:37 +01:00
Kegan Dougal c211fdf581 Add missing sytest to whitelist 2021-07-14 18:13:00 +01:00
kegsay e8de96b78c
sytests: fix failing PL tests by updating GMSL (#1920) 2021-07-14 15:48:14 +01:00
kegsay e80098e186
bugfix: retire invites even when we cannot talk to the remote server to make/send_leave (#1918)
* bugfix: retire invites even when we cannot talk to the remote server to make/send_leave

Also modify the leave response in /sync to include a fake event as this is ultimately
what clients (and sytest) will use to determine leave-ness.

* hash the event ID

* Base64 not hex
2021-07-14 10:39:17 +01:00
kegsay f698142073
update whitelist (#1914)
* update whitelist

* newline
2021-07-12 19:42:05 +01:00
Kegan Dougal 0530302cd6 Add shared secret sytests to whitelist 2021-07-12 11:48:08 +01:00
Bruce MacDonald d27607af78
Implement OpenID module (#599) (#1812)
* Implement OpenID module (#599)

- Unrelated: change Riot references to Element in client API routing

Signed-off-by: Bruce MacDonald <contact@bruce-macdonald.com>

* OpenID module tweaks (#599)

- specify expiry is ms rather than vague ts
- add OpenID token lifetime to configuration
- use Go naming conventions for the path params
- store plaintext token rather than hash
- remove openid table sqllite mutex

* Add default OpenID token lifetime (#599)

* Update dendrite-config.yaml

Co-authored-by: Kegsay <kegsay@gmail.com>
Co-authored-by: Kegsay <kegan@matrix.org>
2021-04-07 13:26:20 +01:00
Neil Alexander 5acf30cd3c
Update sytest-whitelist 2021-03-08 13:32:21 +00:00
Neil Alexander 1ad96e2e2d
Tweak AS registration check and AS component HTTP clients (#1785)
* Tweak AS registration check

* Check appservice usernames using correct function

* Update sytest-whitelist

* Use gomatrixserverlib.Client since that allows us to disable TLS validation using the config

* Add appservice-specific client and ability to control TLS validation for appservices only

* Set timeout on appservice client

* Review comments

* Remove dead code

* Enforce LoginTypeApplicationService after all

* Check correct auth type field
2021-03-05 10:40:27 +00:00
Neil Alexander 81312b8a78
Return the current OTK count on an empty upload request (#1774)
* Always return OTK counts

* Fix parameter ordering

* Send IDs over to keyserver internal API

* Review comments

* Fix syntax error

* Fix panic, hopefully

* Require user ID to be set

* Fix user API call
2021-03-02 11:40:20 +00:00
Neil Alexander 745ee20b90
Update sytest-whitelist after matrix-org/sytest#991 2021-01-22 12:56:20 +00:00
Neil Alexander b70238f2d5
Basic sync filtering (#1721)
* Add some filtering (postgres only for now)

* Fix build error

* Try to use request filter

* Use default filter as a template when retrieving from the database

* Remove unused strut

* Update sytest-whitelist

* Add filtering to SelectEarlyEvents

* Fix Postgres selectEarlyEvents query

* Attempt filtering on SQLite

* Test limit, set field for limit/order in prepareWithFilters

* Remove debug logging, add comments

* Tweaks, debug logging

* Separate SQLite stream IDs

* Fix filtering in current state table

* Fix lock issues

* More tweaks

* Current state requires room ID

* Review comments
2021-01-19 18:00:42 +00:00
Neil Alexander 55cfe391f7
Update /messages pagination token behaviour (#1708)
* Tweak pagination tokens

* start should be the specified from

* Don't reverse start and end

* Tweak getStartEnd again

* Update sytest-whitelist

* NOTSPEC: Re-add iOS end of topology
2021-01-13 12:59:29 +00:00
Neil Alexander 4fe4c180e6
Ensure we wake for our own device list updates (#1661)
* Make sure we wake up for our own key changes

* Whitelist 'Users receive device_list updates for their own devices'
2020-12-18 14:04:17 +00:00
Neil Alexander 50963b724b
More sane next batch handling, typing notification tweaks, give invites their own stream position, device list fix (#1641)
* Update sync responses

* Fix positions, add ApplyUpdates

* Fix MarshalText as non-pointer, PrevBatch is optional

* Increment by number of read receipts

* Merge branch 'master' into neilalexander/devicelist

* Tweak typing

* Include keyserver position tweak

* Fix typing next position in all cases

* Tweaks

* Fix typo

* Tweaks, restore StreamingToken.MarshalText which somehow went missing?

* Rely on positions from notifier rather than manually advancing them

* Revert "Rely on positions from notifier rather than manually advancing them"

This reverts commit 53112a62cc.

* Give invites their own position, fix other things

* Fix test

* Fix invites maybe

* Un-whitelist tests that look to be genuinely wrong

* Use real receipt positions

* Ensure send-to-device uses real positions too
2020-12-18 11:11:21 +00:00
Neil Alexander 9c03b0a4fa
Refactor sync tokens (#1628)
* Refactor sync tokens

* Comment out broken notifier test

* Update types, sytest-whitelist

* More robust token checking

* Remove New functions for streaming tokens

* Export Logs in StreamingToken

* Fix tests
2020-12-10 18:57:10 +00:00
Neil Alexander 68d92fd3b4
Update sytest-whitelist/blacklist 2020-12-04 14:09:21 +00:00
alexkursell 2b03d24358
Fix /joined_members API response (#1606)
* Fix /joined_members API response

* Fix golint issue
2020-12-03 11:01:49 +00:00
bodqhrohro 1cf9f20d2f
Allow = in user identifiers (#1590)
While I was breaking through all the TDD bureaucracy to finally push this fix,
it turned out that it already got fixed in #1578. Still I push the rest of the
changes (basically, everything except of actually fixing the bug ;)).

`=` is crucial for matrix-bifrost which turns XMPP @'s into =40.

Signed-off-by: Bohdan Horbeshko <bodqhrohro@gmail.com>
2020-11-18 22:56:06 +00:00
David Spenler 35ea55e70b
Implemented whois endpoint (#1573)
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-11-17 10:07:03 +00:00
Neil Alexander 3034d8e805
Update sytest lists to blacklist flaky 'Forgotten room messages cannot be paginated' test 2020-11-16 10:51:55 +00:00
S7evinK bcb89ada5e
Implement read receipts (#1528)
* fix conversion from int to string yields a string of one rune, not a string of digits

* Add receipts table to syncapi

* Use StreamingToken as the since value

* Add required method to testEDUProducer

* Make receipt json creation "easier" to read

* Add receipts api to the eduserver

* Add receipts endpoint

* Add eduserver kafka consumer

* Add missing kafka config

* Add passing tests to whitelist

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Fix copy & paste error

* Fix column count error

* Make outbound federation receipts pass

* Make "Inbound federation rejects receipts from wrong remote" pass

* Don't use errors package

* - Add TODO for batching requests
- Rename variable

* Return a better error message

* - Use OutputReceiptEvent instead of InputReceiptEvent as result
- Don't use the errors package for errors
- Defer CloseAndLogIfError to close rows
- Fix Copyright

* Better creation/usage of JoinResponse

* Query all joined rooms instead of just one

* Update gomatrixserverlib

* Add sqlite3 migration

* Add postgres migration

* Ensure required sequence exists before running migrations

* Clarification on comment

* - Fix a bug when creating client receipts
- Use concrete types instead of interface{}

* Remove dead code
Use key for timestamp

* Fix postgres query...

* Remove single purpose struct

* Use key/value directly

* Only apply receipts on initial sync or if edu positions differ,
otherwise we'll be sending the same receipts over and over again.

* Actually update the id, so it is correctly send in syncs

* Set receipt on request to /read_markers

* Fix issue with receipts getting overwritten

* Use fmt.Errorf instead of pkg/errors

* Revert "Add postgres migration"

This reverts commit 722fe5a046.

* Revert "Add sqlite3 migration"

This reverts commit d113b03f64.

* Fix selectRoomReceipts query

* Make golangci-lint happy

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-11-09 18:46:11 +00:00
S7evinK eccd0d2c1b
Implement forgetting about rooms (#1572)
* Add basic storage methods

* Add internal api handler

* Add check for forgotten room

* Add /rooms/{roomID}/forget endpoint

* Add missing rsAPI method

* Remove unused parameters

* Add passing tests

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Add missing file

* Add postgres migration

* Add sqlite migration

* Use Forgetter to forget room

* Remove empty line

* Update HTTP status codes

It looks like the spec calls for these to be 400, rather than 403: https://matrix.org/docs/spec/client_server/r0.6.1#post-matrix-client-r0-rooms-roomid-forget

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-11-05 10:19:23 +00:00
Neil Alexander 7ca89ef511
Update gomatrixserverlib 2020-10-20 17:10:37 +01:00
Neil Alexander c7bf122a26
Update sytest lists 2020-10-19 15:38:42 +01:00
Neil Alexander 5d80ff11a0
Update sytest-whitelist 2020-10-19 15:17:28 +01:00
Neil Alexander 6e63df1d9a
KindOld (#1531)
* Add KindOld

* Don't process latest events/memberships for old events

* Allow federationsender to ignore duplicate key entries when LatestEventIDs is duplicated by RS output events

* Signal to downstream components if an event has become a forward extremity

* Don't exclude from sync

* Soft-fail checks on KindNew

* Don't run the latest events updater at all for KindOld

* Don't make federation sender change after all

* Kind in federation sender join

* Don't send isForwardExtremity

* Fix syncapi

* Update comments

* Fix SendEventWithState

* Update sytest-whitelist

* Generate old output events

* Sync API consumes old room events

* Update comments
2020-10-19 14:59:13 +01:00
Neil Alexander 6f12b8f85c
Ignore typing events where sender doesn't match origin (#1523)
* Ignore typing notifications where the sender doesn't match the origin

* Update sytest-whitelist

* Fix formatting directives
2020-10-14 16:49:25 +01:00
Neil Alexander e3a3908654
Update sytest-whitelist (closes #1315) 2020-10-14 16:07:15 +01:00
Kegsay 9096bfcee8
Validate m.room.create events in send_join responses (#1505)
* Validate m.room.create events in send_join responses

For sytest compliance, refs #1315 and #1317

Fixes #1317

* Linting
2020-10-10 00:21:15 +01:00
Pestdoktor c4c8bfd027
reject invalid UTF-8 (#1472)
* reject invalid UTF-8

Signed-off-by: Jonas Fentker <jonas@fentker.eu>

* update sytest-whitelist

Signed-off-by: Jonas Fentker <jonas@fentker.eu>

Co-authored-by: Kegsay <kegan@matrix.org>
2020-10-09 09:15:51 +01:00
Loïck Bonniot f3e8ae01ef
Implement fully read markers (#1475)
See #653

Signed-off-by: Loïck Bonniot <git@lesterpig.com>

Co-authored-by: Kegsay <kegan@matrix.org>
2020-10-09 09:15:35 +01:00
Neil Alexander d821f9d3c9
Deep checking of forward extremities (#1491)
* Deep forward extremity calculation

* Use updater txn

* Update error

* Update error

* Create previous event references in StoreEvent

* Use latest events updater to row-lock prev events

* Fix unexpected fallthrough

* Fix deadlock

* Don't roll back

* Update comments in calculateLatest

* Don't include events that we can't find references for in the forward extremities

* Add another passing test
2020-10-07 14:05:33 +01:00
Neil Alexander 2e71d2708f
Resolve state after event against current room state when determining latest state changes (#1479)
* Resolve state after event against current room state when determining latest state changes

* Update sytest-whitelist

* Update sytest-whitelist, blacklist
2020-10-05 17:47:08 +01:00
Loïck Bonniot 4e8c484618
Implement account deactivation (#1455)
* Implement account deactivation

See #610
Signed-off-by: Loïck Bonniot <git@lesterpig.com>

* Rename 'is_active' to 'is_deactivated'

Signed-off-by: Loïck Bonniot <git@lesterpig.com>

Co-authored-by: Kegsay <kegan@matrix.org>
2020-10-02 17:18:20 +01:00