Upgrade gomatrixserverlib dependency (#808)

* Upgrade gomatrixserverlib dependency

Signed-off-by: Thibaut CHARLES cromfr@gmail.com

* Added missing passing sytest

Signed-off-by: Thibaut CHARLES cromfr@gmail.com

* Fix login using identifier key

Not a full fix, it only really supports logging in with
the localpart of an mxid.

Signed-off-by: Serra Allgood <serra@allgood.dev>

* Replace deprecated prometheus.InstrumentHandler and unsafe time.Ticker

* goimports

* re-add temporarily missing deps?

* Refactor InstrumentHandlerCounter definition

* URL decode args

* Return server names  (#833)

* Remove unnecessary map->array processing

* Return server names in room federation directory query

* Knock off a TODO

* Fix /send_join and /send_leave (#821)

Fix the /send_join and /send_leave endpoints, so that they use the v2 endpoints as mandated by MSC1802. Also comment out the SyTest tests that are failing because of lack of support for the v1 endpoints.

* Refuse /send_join without m.room.create (#824)

Signed-off-by: Abhishek Kumar <abhishekkumar2718@gmail.com>

* AS should use the v1 endpoint, rather than r0 (#827)

* docker: Passthrough parameters to dendrite-monolith-server

* Fix copy & paste error (#812)

* Use gomatrixserverlib.Transaction instead of local type (#590) (#811)

* Move files back if linting fails (#810)

* replaced gometalinter description with golangci-lint (#837)

* Amend syncapi SQL queries to return missing columns (#840)

* This commit updates a couple of the syncapi SQL queries to return additional columns that are required/expected by rowsToStreamEvents in output_room_events_table.go.

It's not exactly clear to me yet what transaction_id and session_id do, but these being added n #367 results in state events breaking the /sync endpoint.

This is a temporary fix. We need to come up with a better solution.

* gomatrix to gomatrixserverlib on some weird line change

* Tweaks from @babolivier review comments

* Implement storage interfaces (#841)

* Implement interfaces for federationsender storage

* Implement interfaces for mediaapi storage

* Implement interfaces for publicroomsapi storage

* Implement interfaces for roomserver storage

* Implement interfaces for syncapi storage

* Implement interfaces for keydb storage

* common.PartitionStorer in publicroomsapi interface

* Update copyright notices

* make cmd directory path absolute in build.sh (#830)

* Resync testfile with current sytest pass/fail (#832)

* Resync testfile with current sytest pass/fail

* Add displayname test

* Fall back to postgres when database connection string parsing fails (#842)

* Fall back to postgres when parsing the database connection string for a URI schema fails

* Fix behaviour so that it really tries postgres when URL parsing fails and it complains about unknown schema if it succeeds

* Fix #842

* Fix #842 - again...

* Federation fixes (#845)

* Update gomatrixserverlib to p2p commit 92c0338, other tweaks

* Update gomatrixserverlib to p2p commit e5dcc65

* Rewrite getAuthChain

* Update gomatrixserverlib in go.mod/go.sum

* Correct a couple of package refs for updated gmsl/gomatrix

* Update gomatrixserverlib ref in go.mod/go.sum

* Update getAuthChain comments following @babolivier review

* Add a Sytest blacklist file (#849)

* Add more passing tests to the testfile, add test blacklist file (#848)

* CS API: Support for /messages, fixes for /sync (#847)

* Merge forward

* Tidy up a bit

* TODO: What to do with NextBatch here?

* Replace SyncPosition with PaginationToken throughout syncapi

* Fix PaginationTokens

* Fix lint errors

* Add a couple of missing functions into the syncapi external storage interface

* Some updates based on review comments from @babolivier

* Some updates based on review comments from @babolivier

* argh whitespacing

* Fix opentracing span

* Remove dead code

* Don't overshadow err (fix lint issue)

* Handle extremities after inserting event into topology

* Try insert event topology as ON CONFLICT DO NOTHING

* Prevent OOB error in addRoomDeltaToResponse

* Thwarted by gocyclo again

* Fix NewPaginationTokenFromString, define unit test for it

* Update pagination token test

* Update sytest-whitelist

* Hopefully fix some of the sync batch tokens

* Remove extraneous sync position func

* Revert to topology tokens in addRoomDeltaToResponse etc

* Fix typo

* Remove prevPDUPos as dead now that backwardTopologyPos is used instead

* Fix selectEventsWithEventIDsSQL

* Update sytest-blacklist

* Update sytest-whitelist

* Some fixes for #847 (#850)

* Fix a couple of cases where backfilling events we already had causes panics, hopefully fix ordering of events, update GMSL dependency for backfill URL fixes

* Remove commented out lines from output_room_events_table schema

* Wire up publicroomsapi for roomserver events (#851)

* Wire up publicroomsapi to roomserver events

* Remove parameter that was incorrectly brought over from p2p work

* nolint containsBackwardExtremity for now

* Store our own keys in the keydb (#853)

* Store our own keys in the keydb

The DirectKeyFetcher makes the assumption that you can always reach the key/v2/server endpoint of any server, including our own. We previously haven't bothered to store our own keys in the keydb so this would mean we end up making key requests to ourselves.

In the libp2p world as an example, self-dialling is not possible, therefore this would render it impossible to get our own keys.

This commit adds our own keys into the keydb so that we don't create unnecessarily (and maybe impossible) requests.

* Use golang.org/x/crypto/ed25519 instead of crypto/ed25519 for pre-Go 1.13

* More sync fixes (#854)

* Further sync tweaks

* Remove unnecessary blank line

* getBackwardTopologyPos always returns a usable value

* Revert order fixing

* Implement GET endpoints for account_data in clientapi (#861)

* Implement GET endpoints for account_data in clientapi

* Fix accountDB parameter

* Remove fmt.Println

* Add empty push rules into account data on account creation (#862)

* Handle kind=guest query parameter on /register (#860)

* Handle kind=guest query parameter on /register

* Reorganized imports

* Pass device_id as nil

* Added tests to systest-whitelist

* Update sytest-whitelist

* Blacklist 'displayname updates affect room member events' (#859)

* Room version abstractions (#865)

* Rough first pass at adding room version abstractions

* Define newer room versions

* Update room version metadata

* Fix roomserver/versions

* Try to fix whitespace in roomsSchema

* Implement room version capabilities in CS API (#866)

* Add wiring for querying the roomserver for the default room version

* Try to implement /capabilities for room versions

* Update copyright notices

* Update sytests, add /capabilities endpoint into CS API

* Update sytest-whitelist

* Add GetDefaultRoomVersion

* Fix cases where state package was shadowed

* Fix version formatting

* Update Dockerfile to Go 1.13.6

* oh yes types I remember

* And fix the default too

* Update documentation for Go 1.13 (#867)

* Pass cfg by reference around the codebase (#819)

* Pass cfg by reference around the codebase

* Merge branch 'master' into pass-cfg-by-ref

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>

* Implement missing device management features (#835)

* Implement missing device management features

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Add a little more documentation

* Undo changes

* Use non-anonymous struct to decode devices list

* Update sytest-whitelist

* Update sytest-whitelist

* Update sytest-blacklist

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>

* Adding sslmode: disable to sytest server config (#813)

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>

* Fix AppService bind addrs in test (#805)

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>

* Always defer *sql.Rows.Close and consult with Err (#844)

* Always defer *sql.Rows.Close and consult with Err

database/sql.Rows.Next() makes sure to call Close only after exhausting
result rows which would NOT happen when returning early from a bad Scan.
Close being idempotent makes it a great candidate to get always deferred
regardless of what happens later on the result set.

This change also makes sure call Err() after exhausting Next() and
propagate non-nil results from it as the documentation advises.

Closes #764

Signed-off-by: Kiril Vladimiroff <kiril@vladimiroff.org>

* Override named result parameters in last returns

Signed-off-by: Kiril Vladimiroff <kiril@vladimiroff.org>

* Do the same over new changes that got merged

Signed-off-by: Kiril Vladimiroff <kiril@vladimiroff.org>

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>

* Clean up

Co-authored-by: Serra Allgood <serra@allgood.dev>
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>
Co-authored-by: Abhishek Kumar <31231064+abhishekkumar2718@users.noreply.github.com>
Co-authored-by: Will Hunt <will@half-shot.uk>
Co-authored-by: S7evinK <tfaelligen@gmail.com>
Co-authored-by: Arshpreet <30545756+arsh-7@users.noreply.github.com>
Co-authored-by: Prateek Sachan <42961174+prateek2211@users.noreply.github.com>
Co-authored-by: Behouba Manassé <behouba@gmail.com>
Co-authored-by: aditsachde <23707194+aditsachde@users.noreply.github.com>
Co-authored-by: Kiril Vladimiroff <kiril@vladimiroff.org>

go.mod

@@ -18,8 +18,6 @@ require (
 	github.com/matrix-org/naffka v0.0.0-20171115094957-662bfd0841d0
 	github.com/matrix-org/util v0.0.0-20171127121716-2e2df66af2f5
 	github.com/miekg/dns v1.1.12 // indirect
-	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
-	github.com/modern-go/reflect2 v1.0.1 // indirect
 	github.com/nfnt/resize v0.0.0-20160724205520-891127d8d1b5
 	github.com/opentracing/opentracing-go v1.0.2
 	github.com/pierrec/lz4 v0.0.0-20161206202305-5c9560bfa9ac // indirect

go.sum

@@ -183,6 +183,7 @@ gopkg.in/h2non/bimg.v1 v1.0.18 h1:qn6/RpBHt+7WQqoBcK+aF2puc6nC78eZj5LexxoalT4=
 gopkg.in/h2non/bimg.v1 v1.0.18/go.mod h1:PgsZL7dLwUbsGm1NYps320GxGgvQNTnecMCZqxV11So=
 gopkg.in/h2non/gock.v1 v1.0.14 h1:fTeu9fcUvSnLNacYvYI54h+1/XEteDyHvrVCZEEEYNM=
 gopkg.in/h2non/gock.v1 v1.0.14/go.mod h1:sX4zAkdYX1TRGJ2JY156cFspQn4yRWn6p9EMdODlynE=
+gopkg.in/macaroon.v2 v2.1.0 h1:HZcsjBCzq9t0eBPMKqTN/uSN6JOm78ZJ2INbqcBQOUI=
 gopkg.in/macaroon.v2 v2.1.0/go.mod h1:OUb+TQP/OP0WOerC2Jp/3CwhIKyIa9kQjuc7H24e6/o=
 gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=

syncapi/api/query.go

@@ -0,0 +1,123 @@
+// Copyright 2020 The Matrix.org Foundation C.I.C.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package api
+
+import (
+	"context"
+	"net/http"
+
+	commonHTTP "github.com/matrix-org/dendrite/common/http"
+	"github.com/matrix-org/util"
+	opentracing "github.com/opentracing/opentracing-go"
+)
+
+const (
+	SyncAPIQuerySyncPath      = "/api/syncapi/querySync"
+	SyncAPIQueryStatePath     = "/api/syncapi/queryState"
+	SyncAPIQueryStateTypePath = "/api/syncapi/queryStateType"
+	SyncAPIQueryMessagesPath  = "/api/syncapi/queryMessages"
+)
+
+func NewSyncQueryAPIHTTP(syncapiURL string, httpClient *http.Client) SyncQueryAPI {
+	if httpClient == nil {
+		httpClient = http.DefaultClient
+	}
+	return &httpSyncQueryAPI{syncapiURL, httpClient}
+}
+
+type httpSyncQueryAPI struct {
+	syncapiURL string
+	httpClient *http.Client
+}
+
+type SyncQueryAPI interface {
+	QuerySync(ctx context.Context, request *QuerySyncRequest, response *QuerySyncResponse) error
+	QueryState(ctx context.Context, request *QueryStateRequest, response *QueryStateResponse) error
+	QueryStateType(ctx context.Context, request *QueryStateTypeRequest, response *QueryStateTypeResponse) error
+	QueryMessages(ctx context.Context, request *QueryMessagesRequest, response *QueryMessagesResponse) error
+}
+
+type QuerySyncRequest struct{}
+
+type QueryStateRequest struct {
+	RoomID string
+}
+
+type QueryStateTypeRequest struct {
+	RoomID    string
+	EventType string
+	StateKey  string
+}
+
+type QueryMessagesRequest struct {
+	RoomID string
+}
+
+type QuerySyncResponse util.JSONResponse
+type QueryStateResponse util.JSONResponse
+type QueryStateTypeResponse util.JSONResponse
+type QueryMessagesResponse util.JSONResponse
+
+// QueryLatestEventsAndState implements SyncQueryAPI
+func (h *httpSyncQueryAPI) QuerySync(
+	ctx context.Context,
+	request *QuerySyncRequest,
+	response *QuerySyncResponse,
+) error {
+	span, ctx := opentracing.StartSpanFromContext(ctx, "QuerySync")
+	defer span.Finish()
+
+	apiURL := h.syncapiURL + SyncAPIQuerySyncPath
+	return commonHTTP.PostJSON(ctx, span, h.httpClient, apiURL, request, response)
+}
+
+// QueryStateAfterEvents implements SyncQueryAPI
+func (h *httpSyncQueryAPI) QueryState(
+	ctx context.Context,
+	request *QueryStateRequest,
+	response *QueryStateResponse,
+) error {
+	span, ctx := opentracing.StartSpanFromContext(ctx, "QueryState")
+	defer span.Finish()
+
+	apiURL := h.syncapiURL + SyncAPIQueryStatePath
+	return commonHTTP.PostJSON(ctx, span, h.httpClient, apiURL, request, response)
+}
+
+// QueryEventsByID implements SyncQueryAPI
+func (h *httpSyncQueryAPI) QueryStateType(
+	ctx context.Context,
+	request *QueryStateTypeRequest,
+	response *QueryStateTypeResponse,
+) error {
+	span, ctx := opentracing.StartSpanFromContext(ctx, "QueryStateType")
+	defer span.Finish()
+
+	apiURL := h.syncapiURL + SyncAPIQueryStateTypePath
+	return commonHTTP.PostJSON(ctx, span, h.httpClient, apiURL, request, response)
+}
+
+// QueryMembershipForUser implements SyncQueryAPI
+func (h *httpSyncQueryAPI) QueryMessages(
+	ctx context.Context,
+	request *QueryMessagesRequest,
+	response *QueryMessagesResponse,
+) error {
+	span, ctx := opentracing.StartSpanFromContext(ctx, "QueryMessages")
+	defer span.Finish()
+
+	apiURL := h.syncapiURL + SyncAPIQueryMessagesPath
+	return commonHTTP.PostJSON(ctx, span, h.httpClient, apiURL, request, response)
+}

syncapi/routing/state.go

@@ -22,7 +22,6 @@ import (
 	"github.com/matrix-org/dendrite/clientapi/jsonerror"
 	"github.com/matrix-org/dendrite/syncapi/storage"
 	"github.com/matrix-org/dendrite/syncapi/types"
-	"github.com/matrix-org/gomatrix"
 	"github.com/matrix-org/gomatrixserverlib"
 	"github.com/matrix-org/util"
 	log "github.com/sirupsen/logrus"
@@ -45,10 +44,10 @@ func OnIncomingStateRequest(req *http.Request, db storage.Database, roomID strin
 	// TODO(#287): Auth request and handle the case where the user has left (where
 	// we should return the state at the poin they left)
 
-	stateFilterPart := gomatrix.DefaultFilterPart()
+	stateFilter := gomatrixserverlib.DefaultStateFilter()
-	// TODO: stateFilterPart should not limit the number of state events (or only limits abusive number of events)
+	// TODO: stateFilter should not limit the number of state events (or only limits abusive number of events)
 
-	stateEvents, err := db.GetStateEventsForRoom(req.Context(), roomID, &stateFilterPart)
+	stateEvents, err := db.GetStateEventsForRoom(req.Context(), roomID, &stateFilter)
 	if err != nil {
 		return httputil.LogThenError(req, err)
 	}

syncapi/storage/postgres/account_data_table.go

@@ -22,7 +22,7 @@ import (
 	"github.com/lib/pq"
 	"github.com/matrix-org/dendrite/common"
 	"github.com/matrix-org/dendrite/syncapi/types"
-	"github.com/matrix-org/gomatrix"
+	"github.com/matrix-org/gomatrixserverlib"
 )
 
 const accountDataSchema = `
@@ -99,7 +99,7 @@ func (s *accountDataStatements) selectAccountDataInRange(
 	ctx context.Context,
 	userID string,
 	oldPos, newPos types.StreamPosition,
-	accountDataFilterPart *gomatrix.FilterPart,
+	accountDataEventFilter *gomatrixserverlib.EventFilter,
 ) (data map[string][]string, err error) {
 	data = make(map[string][]string)
 
@@ -111,9 +111,9 @@ func (s *accountDataStatements) selectAccountDataInRange(
 	}
 
 	rows, err := s.selectAccountDataInRangeStmt.QueryContext(ctx, userID, oldPos, newPos,
-		pq.StringArray(filterConvertTypeWildcardToSQL(accountDataFilterPart.Types)),
+		pq.StringArray(filterConvertTypeWildcardToSQL(accountDataEventFilter.Types)),
-		pq.StringArray(filterConvertTypeWildcardToSQL(accountDataFilterPart.NotTypes)),
+		pq.StringArray(filterConvertTypeWildcardToSQL(accountDataEventFilter.NotTypes)),
-		accountDataFilterPart.Limit,
+		accountDataEventFilter.Limit,
 	)
 	if err != nil {
 		return

syncapi/storage/postgres/current_room_state_table.go

@@ -23,7 +23,6 @@ import (
 	"github.com/lib/pq"
 	"github.com/matrix-org/dendrite/common"
 	"github.com/matrix-org/dendrite/syncapi/types"
-	"github.com/matrix-org/gomatrix"
 	"github.com/matrix-org/gomatrixserverlib"
 )
 
@@ -185,16 +184,16 @@ func (s *currentRoomStateStatements) selectRoomIDsWithMembership(
 // CurrentState returns all the current state events for the given room.
 func (s *currentRoomStateStatements) selectCurrentState(
 	ctx context.Context, txn *sql.Tx, roomID string,
-	stateFilterPart *gomatrix.FilterPart,
+	stateFilter *gomatrixserverlib.StateFilter,
 ) ([]gomatrixserverlib.Event, error) {
 	stmt := common.TxStmt(txn, s.selectCurrentStateStmt)
 	rows, err := stmt.QueryContext(ctx, roomID,
-		pq.StringArray(stateFilterPart.Senders),
+		pq.StringArray(stateFilter.Senders),
-		pq.StringArray(stateFilterPart.NotSenders),
+		pq.StringArray(stateFilter.NotSenders),
-		pq.StringArray(filterConvertTypeWildcardToSQL(stateFilterPart.Types)),
+		pq.StringArray(filterConvertTypeWildcardToSQL(stateFilter.Types)),
-		pq.StringArray(filterConvertTypeWildcardToSQL(stateFilterPart.NotTypes)),
+		pq.StringArray(filterConvertTypeWildcardToSQL(stateFilter.NotTypes)),
-		stateFilterPart.ContainsURL,
+		stateFilter.ContainsURL,
-		stateFilterPart.Limit,
+		stateFilter.Limit,
 	)
 	if err != nil {
 		return nil, err

syncapi/storage/postgres/output_room_events_table.go

@@ -23,7 +23,6 @@ import (
 
 	"github.com/matrix-org/dendrite/roomserver/api"
 	"github.com/matrix-org/dendrite/syncapi/types"
-	"github.com/matrix-org/gomatrix"
 
 	"github.com/lib/pq"
 	"github.com/matrix-org/dendrite/common"
@@ -154,18 +153,18 @@ func (s *outputRoomEventsStatements) prepare(db *sql.DB) (err error) {
 // two positions, only the most recent state is returned.
 func (s *outputRoomEventsStatements) selectStateInRange(
 	ctx context.Context, txn *sql.Tx, oldPos, newPos types.StreamPosition,
-	stateFilterPart *gomatrix.FilterPart,
+	stateFilter *gomatrixserverlib.StateFilter,
 ) (map[string]map[string]bool, map[string]types.StreamEvent, error) {
 	stmt := common.TxStmt(txn, s.selectStateInRangeStmt)
 
 	rows, err := stmt.QueryContext(
 		ctx, oldPos, newPos,
-		pq.StringArray(stateFilterPart.Senders),
+		pq.StringArray(stateFilter.Senders),
-		pq.StringArray(stateFilterPart.NotSenders),
+		pq.StringArray(stateFilter.NotSenders),
-		pq.StringArray(filterConvertTypeWildcardToSQL(stateFilterPart.Types)),
+		pq.StringArray(filterConvertTypeWildcardToSQL(stateFilter.Types)),
-		pq.StringArray(filterConvertTypeWildcardToSQL(stateFilterPart.NotTypes)),
+		pq.StringArray(filterConvertTypeWildcardToSQL(stateFilter.NotTypes)),
-		stateFilterPart.ContainsURL,
+		stateFilter.ContainsURL,
-		stateFilterPart.Limit,
+		stateFilter.Limit,
 	)
 	if err != nil {
 		return nil, nil, err

syncapi/storage/postgres/syncserver.go

@@ -26,7 +26,6 @@ import (
 
 	"github.com/matrix-org/dendrite/clientapi/auth/authtypes"
 	"github.com/matrix-org/dendrite/roomserver/api"
-	"github.com/matrix-org/gomatrix"
 
 	// Import the postgres database driver.
 	_ "github.com/lib/pq"
@@ -237,10 +236,10 @@ func (d *SyncServerDatasource) GetStateEvent(
 // Returns an empty slice if no state events could be found for this room.
 // Returns an error if there was an issue with the retrieval.
 func (d *SyncServerDatasource) GetStateEventsForRoom(
-	ctx context.Context, roomID string, stateFilterPart *gomatrix.FilterPart,
+	ctx context.Context, roomID string, stateFilter *gomatrixserverlib.StateFilter,
 ) (stateEvents []gomatrixserverlib.Event, err error) {
 	err = common.WithTransaction(d.db, func(txn *sql.Tx) error {
-		stateEvents, err = d.roomstate.selectCurrentState(ctx, txn, roomID, stateFilterPart)
+		stateEvents, err = d.roomstate.selectCurrentState(ctx, txn, roomID, stateFilter)
 		return err
 	})
 	return
@@ -422,7 +421,7 @@ func (d *SyncServerDatasource) addPDUDeltaToResponse(
 	var succeeded bool
 	defer common.EndTransaction(txn, &succeeded)
 
-	stateFilterPart := gomatrix.DefaultFilterPart() // TODO: use filter provided in request
+	stateFilter := gomatrixserverlib.DefaultStateFilter() // TODO: use filter provided in request
 
 	// Work out which rooms to return in the response. This is done by getting not only the currently
 	// joined rooms, but also which rooms have membership transitions for this user between the 2 PDU stream positions.
@@ -432,11 +431,11 @@ func (d *SyncServerDatasource) addPDUDeltaToResponse(
 	var joinedRoomIDs []string
 	if !wantFullState {
 		deltas, joinedRoomIDs, err = d.getStateDeltas(
-			ctx, &device, txn, fromPos, toPos, device.UserID, &stateFilterPart,
+			ctx, &device, txn, fromPos, toPos, device.UserID, &stateFilter,
 		)
 	} else {
 		deltas, joinedRoomIDs, err = d.getStateDeltasForFullStateSync(
-			ctx, &device, txn, fromPos, toPos, device.UserID, &stateFilterPart,
+			ctx, &device, txn, fromPos, toPos, device.UserID, &stateFilter,
 		)
 	}
 	if err != nil {
@@ -587,12 +586,12 @@ func (d *SyncServerDatasource) getResponseWithPDUsForCompleteSync(
 		return
 	}
 
-	stateFilterPart := gomatrix.DefaultFilterPart() // TODO: use filter provided in request
+	stateFilter := gomatrixserverlib.DefaultStateFilter() // TODO: use filter provided in request
 
 	// Build up a /sync response. Add joined rooms.
 	for _, roomID := range joinedRoomIDs {
 		var stateEvents []gomatrixserverlib.Event
-		stateEvents, err = d.roomstate.selectCurrentState(ctx, txn, roomID, &stateFilterPart)
+		stateEvents, err = d.roomstate.selectCurrentState(ctx, txn, roomID, &stateFilter)
 		if err != nil {
 			return
 		}
@@ -681,7 +680,7 @@ var txReadOnlySnapshot = sql.TxOptions{
 // If there was an issue with the retrieval, returns an error
 func (d *SyncServerDatasource) GetAccountDataInRange(
 	ctx context.Context, userID string, oldPos, newPos types.StreamPosition,
-	accountDataFilterPart *gomatrix.FilterPart,
+	accountDataFilterPart *gomatrixserverlib.EventFilter,
 ) (map[string][]string, error) {
 	return d.accountData.selectAccountDataInRange(ctx, userID, oldPos, newPos, accountDataFilterPart)
 }
@@ -931,7 +930,7 @@ func (d *SyncServerDatasource) fetchMissingStateEvents(
 func (d *SyncServerDatasource) getStateDeltas(
 	ctx context.Context, device *authtypes.Device, txn *sql.Tx,
 	fromPos, toPos types.StreamPosition, userID string,
-	stateFilterPart *gomatrix.FilterPart,
+	stateFilter *gomatrixserverlib.StateFilter,
 ) ([]stateDelta, []string, error) {
 	// Implement membership change algorithm: https://github.com/matrix-org/synapse/blob/v0.19.3/synapse/handlers/sync.py#L821
 	// - Get membership list changes for this user in this sync response
@@ -944,7 +943,7 @@ func (d *SyncServerDatasource) getStateDeltas(
 	var deltas []stateDelta
 
 	// get all the state events ever between these two positions
-	stateNeeded, eventMap, err := d.events.selectStateInRange(ctx, txn, fromPos, toPos, stateFilterPart)
+	stateNeeded, eventMap, err := d.events.selectStateInRange(ctx, txn, fromPos, toPos, stateFilter)
 	if err != nil {
 		return nil, nil, err
 	}
@@ -964,7 +963,7 @@ func (d *SyncServerDatasource) getStateDeltas(
 				if membership == gomatrixserverlib.Join {
 					// send full room state down instead of a delta
 					var s []types.StreamEvent
-					s, err = d.currentStateStreamEventsForRoom(ctx, txn, roomID, stateFilterPart)
+					s, err = d.currentStateStreamEventsForRoom(ctx, txn, roomID, stateFilter)
 					if err != nil {
 						return nil, nil, err
 					}
@@ -1006,7 +1005,7 @@ func (d *SyncServerDatasource) getStateDeltas(
 func (d *SyncServerDatasource) getStateDeltasForFullStateSync(
 	ctx context.Context, device *authtypes.Device, txn *sql.Tx,
 	fromPos, toPos types.StreamPosition, userID string,
-	stateFilterPart *gomatrix.FilterPart,
+	stateFilter *gomatrixserverlib.StateFilter,
 ) ([]stateDelta, []string, error) {
 	joinedRoomIDs, err := d.roomstate.selectRoomIDsWithMembership(ctx, txn, userID, gomatrixserverlib.Join)
 	if err != nil {
@@ -1018,7 +1017,7 @@ func (d *SyncServerDatasource) getStateDeltasForFullStateSync(
 
 	// Add full states for all joined rooms
 	for _, joinedRoomID := range joinedRoomIDs {
-		s, stateErr := d.currentStateStreamEventsForRoom(ctx, txn, joinedRoomID, stateFilterPart)
+		s, stateErr := d.currentStateStreamEventsForRoom(ctx, txn, joinedRoomID, stateFilter)
 		if stateErr != nil {
 			return nil, nil, stateErr
 		}
@@ -1030,7 +1029,7 @@ func (d *SyncServerDatasource) getStateDeltasForFullStateSync(
 	}
 
 	// Get all the state events ever between these two positions
-	stateNeeded, eventMap, err := d.events.selectStateInRange(ctx, txn, fromPos, toPos, stateFilterPart)
+	stateNeeded, eventMap, err := d.events.selectStateInRange(ctx, txn, fromPos, toPos, stateFilter)
 	if err != nil {
 		return nil, nil, err
 	}
@@ -1061,9 +1060,9 @@ func (d *SyncServerDatasource) getStateDeltasForFullStateSync(
 
 func (d *SyncServerDatasource) currentStateStreamEventsForRoom(
 	ctx context.Context, txn *sql.Tx, roomID string,
-	stateFilterPart *gomatrix.FilterPart,
+	stateFilter *gomatrixserverlib.StateFilter,
 ) ([]types.StreamEvent, error) {
-	allState, err := d.roomstate.selectCurrentState(ctx, txn, roomID, stateFilterPart)
+	allState, err := d.roomstate.selectCurrentState(ctx, txn, roomID, stateFilter)
 	if err != nil {
 		return nil, err
 	}

syncapi/storage/storage.go

@@ -25,7 +25,6 @@ import (
 	"github.com/matrix-org/dendrite/syncapi/storage/postgres"
 	"github.com/matrix-org/dendrite/syncapi/types"
 	"github.com/matrix-org/dendrite/typingserver/cache"
-	"github.com/matrix-org/gomatrix"
 	"github.com/matrix-org/gomatrixserverlib"
 )
 
@@ -35,11 +34,11 @@ type Database interface {
 	Events(ctx context.Context, eventIDs []string) ([]gomatrixserverlib.Event, error)
 	WriteEvent(context.Context, *gomatrixserverlib.Event, []gomatrixserverlib.Event, []string, []string, *api.TransactionID, bool) (types.StreamPosition, error)
 	GetStateEvent(ctx context.Context, roomID, evType, stateKey string) (*gomatrixserverlib.Event, error)
-	GetStateEventsForRoom(ctx context.Context, roomID string, stateFilterPart *gomatrix.FilterPart) (stateEvents []gomatrixserverlib.Event, err error)
+	GetStateEventsForRoom(ctx context.Context, roomID string, stateFilterPart *gomatrixserverlib.StateFilter) (stateEvents []gomatrixserverlib.Event, err error)
 	SyncPosition(ctx context.Context) (types.PaginationToken, error)
 	IncrementalSync(ctx context.Context, device authtypes.Device, fromPos, toPos types.PaginationToken, numRecentEventsPerRoom int, wantFullState bool) (*types.Response, error)
 	CompleteSync(ctx context.Context, userID string, numRecentEventsPerRoom int) (*types.Response, error)
-	GetAccountDataInRange(ctx context.Context, userID string, oldPos, newPos types.StreamPosition, accountDataFilterPart *gomatrix.FilterPart) (map[string][]string, error)
+	GetAccountDataInRange(ctx context.Context, userID string, oldPos, newPos types.StreamPosition, accountDataFilterPart *gomatrixserverlib.EventFilter) (map[string][]string, error)
 	UpsertAccountData(ctx context.Context, userID, roomID, dataType string) (types.StreamPosition, error)
 	AddInviteEvent(ctx context.Context, inviteEvent gomatrixserverlib.Event) (types.StreamPosition, error)
 	RetireInviteEvent(ctx context.Context, inviteEventID string) error

syncapi/sync/requestpool.go

@@ -24,7 +24,6 @@ import (
 	"github.com/matrix-org/dendrite/clientapi/jsonerror"
 	"github.com/matrix-org/dendrite/syncapi/storage"
 	"github.com/matrix-org/dendrite/syncapi/types"
-	"github.com/matrix-org/gomatrix"
 	"github.com/matrix-org/gomatrixserverlib"
 	"github.com/matrix-org/util"
 	log "github.com/sirupsen/logrus"
@@ -142,14 +141,14 @@ func (rp *RequestPool) currentSyncForUser(req syncRequest, latestPos types.Pagin
 		return
 	}
 
-	accountDataFilter := gomatrix.DefaultFilterPart() // TODO: use filter provided in req instead
+	accountDataFilter := gomatrixserverlib.DefaultEventFilter() // TODO: use filter provided in req instead
-	res, err = rp.appendAccountData(res, req.device.UserID, req, int64(latestPos.PDUPosition), &accountDataFilter)
+	res, err = rp.appendAccountData(res, req.device.UserID, req, latestPos.PDUPosition, &accountDataFilter)
 	return
 }
 
 func (rp *RequestPool) appendAccountData(
-	data *types.Response, userID string, req syncRequest, currentPos int64,
+	data *types.Response, userID string, req syncRequest, currentPos types.StreamPosition,
-	accountDataFilter *gomatrix.FilterPart,
+	accountDataFilter *gomatrixserverlib.EventFilter,
 ) (*types.Response, error) {
 	// TODO: Account data doesn't have a sync position of its own, meaning that
 	// account data might be sent multiple time to the client if multiple account

sytest-whitelist

@@ -81,6 +81,7 @@ Can't forget room you're still in
 Can get rooms/{roomId}/members
 Can create filter
 Can download filter
+Lazy loading parameters in the filter are strictly boolean
 Can sync
 Can sync a joined room
 Newly joined room is included in an incremental sync