* - Make all PartitionKeys include the tablename
- Update specific PKs to be item specific
- Add validation to the PerformQueryXX methods
- Fix queries that fail validation
* - Revert the PK back to CollectionName as it already includes the TableName
Co-authored-by: alexf@example.com <alexf@example.com>
* - Create CosmosDocument as a base class
- Add CT and UT
- Refactor all tables to use the CosmosDocument
* - Add UpsertDocument method to perform updates in a generic way
- Add SetUpdateTime() to update the UT for updates
- Refactor it all
* - Add Performquery method
- Refactor appservice_events_table
* - Update naffka Topics and Messages to use the common pattern
* - Update keyserver to use the common pattern for collection and PK
* - Update mediaapi to use the common pattern for collection and pk
* - Update roomserver to use the common pattern for collectionname and pk
* - Update signingkeyserver to use the common pattern for collectionname and pk
* - Update userapi touse the common pattern for collectionname and pk
* - Update partitionOffset to use the common collectionname and pk
- Remove generic GetPartitionKey() method
Co-authored-by: alexf@example.com <alexf@example.com>
* - Create CosmosDocument as a base class
- Add CT and UT
- Refactor all tables to use the CosmosDocument
* - Add UpsertDocument method to perform updates in a generic way
- Add SetUpdateTime() to update the UT for updates
- Refactor it all
Co-authored-by: alexf@example.com <alexf@example.com>
* - Add CosmosDB back
- Add missing methods to blacklist_table.go
- Add missing methods to device_keys_table.go
- Add missing methods to events_table.go
- Add missing methods to membership_table.go
- Update state_block_table.go (due to reafctor SQL)
- Update state_snapshot_table.go (due to reafctor SQL)
- Add new key_backup_table.go
- Add new key_backup_version_table.go
- Code compiles but has runtime errors
* Message sending + receiving working
Rooms and DMs working
- Add CrossSigningKeys table
- Add CrossSigningSigs table
- Refactor DeviceKeys yable
- Fix OneTimeKeys
- Update the KeyServer storage.go to use a PartitionStorer instead of a specific SQL PartitionOffsetStatements
- Fix small issues from the previous commit
- Implement DeleteSendToDeviceMessages
Co-authored-by: alexf@example.com <alexf@example.com>
* Delete device keys/signatures from key server when deleting device from user API
* Move loop to within database transaction
* Don't fall over deleting no rows
* Handle other signatures
* Decorate key ID properly
* Match by key IDs
* Tweaks
* Fixes
* Fix /user/keys/query bug, review comments, update sytest-whitelist
* Various wtweaks
* Fix wiring for keyserver in API mode
* Additional fixes
* Enable unstable feature again
* Try to verify when a device signs a key
* Try to verify when a key signs a device
* It's the self-signing key, not the master key
* Fix error
* Try to verify master key uploads
* Actually we can't guarantee we can do that so nevermind
* Add signatures into /devices/list request
* Fix nil pointer
* Reprioritise map creation
* Don't skip devices that don't have signatures
* Add some debug logging
* Fix logic error in QuerySignatures
* Fix bugs
* Expose master and self-signing keys on /devices/list hopefully
* maps are tedious
* Expose signatures via /keys/query
* Upload signatures when uploading keys
* Fixes
* Disable the feature again
* Cross-signing groundwork
* Update to matrix-org/gomatrixserverlib#274
* Fix gobind builds, which stops unit tests in CI from yelling
* Some changes from review comments
* Fix build by passing in UIA
* Update to matrix-org/gomatrixserverlib@bec8d22
* Process master/self-signing keys from devices call
* nolint
* Enum-ify the key type in the database
* Process self-signing key too
* Fix sanity check in device list updater
* Fix check
* Fix sytest, hopefully
* Fix build
* Add tenant-id to all docs and cross-tenant queries
* Use TenantName in doc ids too
* Space
* - Rename variables and ensure code compiles
* - Fix error handling for GroupBy query
* - Fix typo
Co-authored-by: Alex Flatow <alexf@criticalarc.com>
* Try to optimize SelectOneTimeKeys
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
* Use pg.Array when using ANY...
Co-authored-by: Kegsay <kegan@matrix.org>
- Update the Config to use Cosmos for the sync API
- Ensure Cosmos DocId does not contain escape chars
- Create a shared Cosmos PartitionOffet table and refactor to use it
- Hardcode the "nafka" Connstring to use the "file:naffka.db"
- Create seq documents for each of the nextXXXID methods
* - Implement Cosmos for the devices_table
- Use the ConnectionString in the YAML to include the Tenant
- Revert all other non implemented tables back to use SQLLite3
* - Change the Config to use "test.criticicalarc.com" Container
- Add generic function GetDocumentOrNil to standardize GetDocument
- Add func to return CrossPartition queries for Aggregates
- Add func GetNextSequence() as generic seq generator for AutoIncrement
- Add cosmosdbutil.ErrNoRows to return (emulate) sql.ErrNoRows
- Add a "fake" ExclusiveWriterFake
- Add standard "getXX", "setXX" and "queryXX" to all TABLE class files
- Add specific Table SEQ for the Events table
- Add specific Table SEQ for the Rooms table
- Add specific Table SEQ for the StateSnapshot table
* - Use CosmosDB for the KeyServer
- Replace the ConnString in the YAML to Cosmos
- Update the 4 tables to use Cosmos
- Use the SQLLite as a base for the CosmosDB package(s)
- Update the ConnString to use file: from cosmosdb: so it still works
- Add a yaml file for the config to use CosmosDB
* Always return OTK counts
* Fix parameter ordering
* Send IDs over to keyserver internal API
* Review comments
* Fix syntax error
* Fix panic, hopefully
* Require user ID to be set
* Fix user API call
Fix#1511
On 32-bits systems, int(hash.Sum32()) can be negative.
This makes the computation of array indices using modulo invalid, crashing dendrite.
Signed-off-by: Loïck Bonniot <git@lesterpig.com>
* Update sync responses
* Fix positions, add ApplyUpdates
* Fix MarshalText as non-pointer, PrevBatch is optional
* Increment by number of read receipts
* Merge branch 'master' into neilalexander/devicelist
* Tweak typing
* Include keyserver position tweak
* Fix typing next position in all cases
* Tweaks
* Fix typo
* Tweaks, restore StreamingToken.MarshalText which somehow went missing?
* Rely on positions from notifier rather than manually advancing them
* Revert "Rely on positions from notifier rather than manually advancing them"
This reverts commit 53112a62cc.
* Give invites their own position, fix other things
* Fix test
* Fix invites maybe
* Un-whitelist tests that look to be genuinely wrong
* Use real receipt positions
* Ensure send-to-device uses real positions too
* Start Kafka connection for each component that needs one
* Fix roomserver unit tests
* Rename to naffkaInstance (@Kegsay review comment)
* Fix import cycle
* Replace all usages of txn.Stmt with sqlutil.TxStmt
Signed-off-by: Sam Day <me@samcday.com>
* Fix sign off link in PR template.
Signed-off-by: Sam Day <me@samcday.com>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Offset updates take place using TransactionWriter
* Refactor TransactionWriter in current state server
* Refactor TransactionWriter in federation sender
* Refactor TransactionWriter in key server
* Refactor TransactionWriter in media API
* Refactor TransactionWriter in server key API
* Refactor TransactionWriter in sync API
* Refactor TransactionWriter in user API
* Fix deadlocking Sync API tests
* Un-deadlock device database
* Fix appservice API
* Rename TransactionWriters to Writers
* Move writers up a layer in sync API
* Document sqlutil.Writer interface
* Add note to Writer documentation
* Add FederationClient interface to federationsender
- Use a shim struct in HTTP mode to keep the same API as `FederationClient`.
- Use `federationsender` instead of `FederationClient` in `keyserver`.
* Pointers not values
* Review comments
* Fix unit tests
* Rejig backoff
* Unbreak test
* Remove debug logs
* Review comments and linting
* Updated TransactionWriters, moved locks in roomserver, various other tweaks
* Fix redaction deadlocks
* Fix lint issue
* Rename SQLiteTransactionWriter to ExclusiveTransactionWriter
* Fix us not sending transactions through in latest events updater
We did this already for local `/keys/upload` but didn't for
remote `/users/devices`. This meant any resyncs would spam produce
events, hammering disk i/o and spamming the logs.
- As a last resort, query the DB when exhausting all possible remote query
endpoints, but keep the field in `failures` so clients can detect that this
is stale data.
- Unblock `DeviceListUpdater.Update` on failures rather than timing out.
- Use a mutex when writing directly to `res`, not just for failures.
* WIP: Eagerly sync device lists on /user/keys/query requests
Also notify servers when a user's device display name changes. Few
caveats:
- sytest `Device deletion propagates over federation` fails
- `populateResponseWithDeviceKeysFromDatabase` is called from multiple
goroutines and hence is unsafe.
* Handle deleted devices correctly over federation
* Add sync mechanism to block when updating device lists
With a timeout, mainly for sytest to fix the test
"Server correctly handles incoming m.device_list_update"
which is flakey because it assumes that when `/send` 200 OKs
that the server has updated the device lists in prep for
`/keys/query` which is not always true when using workers.
* Fix UT
* Add new working test
