Commit graph

26 commits

Author SHA1 Message Date
alexfca 3088238419
Add UniqueId to PartitionKey for some Dendrite tables (where possible) (#19)
* - Make all PartitionKeys include the tablename
- Update specific PKs to be item specific
- Add validation to the PerformQueryXX methods
- Fix queries that fail validation

* - Revert the PK back to CollectionName as it already includes the TableName

Co-authored-by: alexf@example.com <alexf@example.com>
2021-09-23 14:48:32 +10:00
alexfca 927238a686
Use a common way to generate CollectionName and PartitionKey (#18)
* - Create CosmosDocument as a base class
- Add CT and UT
- Refactor all tables to use the CosmosDocument

* - Add UpsertDocument method to perform updates in a generic way
- Add SetUpdateTime() to update the UT for updates
- Refactor it all

* - Add Performquery method
- Refactor appservice_events_table

* - Update naffka Topics and Messages to use the common pattern

* - Update keyserver to use the common pattern for collection and PK

* - Update mediaapi to use the common pattern for collection and pk

* - Update roomserver to use the common pattern for collectionname and pk

* - Update signingkeyserver to use the common pattern for collectionname and pk

* - Update userapi touse the common pattern for collectionname and pk

* - Update partitionOffset to use the common collectionname and pk
- Remove generic GetPartitionKey() method

Co-authored-by: alexf@example.com <alexf@example.com>
2021-09-23 09:02:37 +10:00
alexfca acf63daf79
Add CT and UT to all documents and refactor (#17)
* - Create CosmosDocument as a base class
- Add CT and UT
- Refactor all tables to use the CosmosDocument

* - Add UpsertDocument method to perform updates in a generic way
- Add SetUpdateTime() to update the UT for updates
- Refactor it all

Co-authored-by: alexf@example.com <alexf@example.com>
2021-09-20 17:41:04 +10:00
Jared Jacobson 84c8cb052b Merge branch 'master' into develop
# Conflicts:
#	go.mod
#	go.sum
2021-09-03 13:54:05 +10:00
jahmai-ca 199c4c3cd1
Add tenant-id to all docs and cross-tenant queries (#14)
* Add tenant-id to all docs and cross-tenant queries

* Use TenantName in doc ids too

* Space

* - Rename variables and ensure code compiles

* - Fix error handling for GroupBy query

* - Fix typo

Co-authored-by: Alex Flatow <alexf@criticalarc.com>
2021-07-27 14:10:35 +10:00
Neil Alexander f63068df3b
Only include go-sqlite3 on the relevant binaries (#1900)
* Only include go-sqlite3 on the relevant binaries

* The driver name is always sqlite3 now

* Update to matrix-org/go-sqlite3-js@e537baa
2021-07-20 11:18:14 +01:00
alexfca 3ca96b13b3
- Implement the SycAPI to use CosmosDB (#8)
- Update the Config to use Cosmos for the sync API
- Ensure Cosmos DocId does not contain escape chars
- Create a shared Cosmos PartitionOffet table and refactor to use it
- Hardcode the "nafka" Connstring to use the "file:naffka.db"
- Create seq documents for each of the nextXXXID methods
2021-05-27 18:45:53 +10:00
alexfca 5d68daef80
Implement Cosmos DB for the RoomServer Service (#5)
* - Implement Cosmos for the devices_table
- Use the ConnectionString in the YAML to include the Tenant
- Revert all other non implemented tables back to use SQLLite3

* - Change the Config to use "test.criticicalarc.com" Container
- Add generic function GetDocumentOrNil to standardize GetDocument
- Add func to return CrossPartition queries for Aggregates
- Add func GetNextSequence() as generic seq generator for AutoIncrement
- Add cosmosdbutil.ErrNoRows to return (emulate) sql.ErrNoRows
- Add a "fake" ExclusiveWriterFake
- Add standard "getXX", "setXX" and "queryXX" to all TABLE class files
- Add specific Table SEQ for the Events table
- Add specific Table SEQ for the Rooms table
- Add specific Table SEQ for the StateSnapshot table
2021-05-20 14:42:33 +10:00
alexfca b696923333
- Implement Cosmos for the devices_table (#4)
- Use the ConnectionString in the YAML to include the Tenant
- Revert all other non implemented tables back to use SQLLite3
2021-05-12 16:30:49 +10:00
Alex Flatow 5ded872da9 - Add CosmosDB as a Datasource type
- Use the SQLLite as a base for the CosmosDB package(s)
- Update the ConnString to use file: from cosmosdb: so it still works
- Add a yaml file for the config to use CosmosDB
2021-05-06 15:09:44 +10:00
Alex Flatow 4b42a0addb storage.go 2021-05-06 14:41:27 +10:00
Neil Alexander b5aa7ca3ab
Top-level setup package (#1605)
* Move config, setup, mscs into "setup" top-level folder

* oops, forgot the EDU server

* Add setup

* goimports
2020-12-02 17:41:00 +00:00
Neil Alexander c636be5070
Update last seen on sync requests (#1593)
* Update last seen on sync requests

* Fix MSC2836 unit tests

* Only update once per minute

* Remove debug logging

* Configurable option

* Simplify updateLastSeen/cleanLastSeen
2020-11-20 11:29:02 +00:00
David Spenler 35ea55e70b
Implemented whois endpoint (#1573)
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-11-17 10:07:03 +00:00
Kegsay 4a7fb9c045
Automatically upgrade databases on startup (#1529)
* Support auto-upgrading accounts DB

* Auto-upgrade device DB deltas

* Support up/downgrading from cmd/goose

* Linting

* Create tables then do migrations then prepare statements

To avoid failing due to some things not existing

* Linting
2020-10-15 18:09:41 +01:00
S7evinK 2bd0449c5b
Use default value when adding column which is not null (#1501)
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
2020-10-09 12:41:53 +01:00
S7evinK 1cd525ef0d
Extend device_devices table (#1471)
* Add last_used_ts and IP to database

* Add migrations

* Rename column
Prepare statements

* Add interface method and implement it

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Rename struct fields

* Add user_agent to database

* Add userAgent to registration calls

* Add missing "IF NOT EXISTS"

* use txn writer

* Add UserAgent to Device

Co-authored-by: Kegsay <kegan@matrix.org>
2020-10-09 09:17:23 +01:00
Neil Alexander 5076925c18
Password changes (#1397)
* User API support for password changes

* Password changes in client API

* Update sytest-whitelist

* Remove debug logging

* Default logout_devices to true

* Fix deleting devices by local part
2020-09-04 15:16:13 +01:00
Kegsay 9af2f5f1f2
Remove device DB from clientapi (#1352)
* Remove device DB from clientapi

* Remove device DB from startup configuration

It's all an impl detail now in user API
2020-08-27 18:53:40 +01:00
Neil Alexander 9d53351dc2
Component-wide TransactionWriters (#1290)
* Offset updates take place using TransactionWriter

* Refactor TransactionWriter in current state server

* Refactor TransactionWriter in federation sender

* Refactor TransactionWriter in key server

* Refactor TransactionWriter in media API

* Refactor TransactionWriter in server key API

* Refactor TransactionWriter in sync API

* Refactor TransactionWriter in user API

* Fix deadlocking Sync API tests

* Un-deadlock device database

* Fix appservice API

* Rename TransactionWriters to Writers

* Move writers up a layer in sync API

* Document sqlutil.Writer interface

* Add note to Writer documentation
2020-08-21 10:42:08 +01:00
Neil Alexander b24747b305
Transaction writer changes, move roomserver writers (#1285)
* Updated TransactionWriters, moved locks in roomserver, various other tweaks

* Fix redaction deadlocks

* Fix lint issue

* Rename SQLiteTransactionWriter to ExclusiveTransactionWriter

* Fix us not sending transactions through in latest events updater
2020-08-19 15:38:27 +01:00
Neil Alexander 4b09f445c9
Configuration format v1 (#1230)
* Initial pass at refactoring config (not finished)

* Don't forget current state and EDU servers

* More shifting around

* Update server key API tests

* Fix roomserver test

* Fix more tests

* Further tweaks

* Fix current state server test (sort of)

* Maybe fix appservices

* Fix client API test

* Include database connection string in database options

* Fix sync API build

* Update config test

* Fix unit tests

* Fix federation sender build

* Fix gobind build

* Set Listen address for all services in HTTP monolith mode

* Validate config, reinstate appservice derived in directory, tweaks

* Tweak federation API test

* Set MaxOpenConnections/MaxIdleConnections to previous values

* Update generate-config
2020-08-10 14:18:04 +01:00
Kegsay b5cb1d1534
Fix edge cases around device lists (#1234)
* Fix New users appear in /keys/changes

* Create blank device keys when logging in on a new device

* Add PerformDeviceUpdate and fix a few bugs

- Correct device deletion query on sqlite
- Return no keys on /keys/query rather than an empty key

* Unbreak sqlite properly

* Use a real DB for currentstateserver integration tests

* Race fix
2020-07-31 14:40:45 +01:00
Kegsay 541a23f712
Handle inbound federation E2E key queries/claims (#1215)
* Handle inbound /keys/claim and /keys/query requests

* Add display names to device key responses

* Linting
2020-07-22 17:04:57 +01:00
Neil Alexander b6bc132485
Use TransactionWriter in other component SQLite (#1209)
* Use TransactionWriter on other component SQLites

* Fix sync API tests

* Fix panic in media API

* Fix a couple of transactions

* Fix wrong query, add some logging output

* Add debug logging into StoreEvent

* Adjust InsertRoomNID

* Update logging
2020-07-21 15:48:21 +01:00
Kegsay e09d24e732
Move account/device DBs to userapi (#1141) 2020-06-17 12:05:56 +01:00