Commit graph

1476 commits

Author SHA1 Message Date
alexfca af4219f38e
Implement Cosmos DB for the AppService (#7)
* - Implement Cosmos for the devices_table
- Use the ConnectionString in the YAML to include the Tenant
- Revert all other non implemented tables back to use SQLLite3

* - Change the Config to use "test.criticicalarc.com" Container
- Add generic function GetDocumentOrNil to standardize GetDocument
- Add func to return CrossPartition queries for Aggregates
- Add func GetNextSequence() as generic seq generator for AutoIncrement
- Add cosmosdbutil.ErrNoRows to return (emulate) sql.ErrNoRows
- Add a "fake" ExclusiveWriterFake
- Add standard "getXX", "setXX" and "queryXX" to all TABLE class files
- Add specific Table SEQ for the Events table
- Add specific Table SEQ for the Rooms table
- Add specific Table SEQ for the StateSnapshot table

* - Use CosmosDB for the KeyServer
- Replace the ConnString in the YAML to Cosmos
- Update the 4 tables to use Cosmos

* - Add SEQ for Event and Counters
- Replace SQLLite with Cosmos in Config and Code

* - Fix typo
2021-05-21 10:12:39 +10:00
alexfca b4382bd8b9
Implement Cosmos DB for the KeyServer Service (#6)
* - Implement Cosmos for the devices_table
- Use the ConnectionString in the YAML to include the Tenant
- Revert all other non implemented tables back to use SQLLite3

* - Change the Config to use "test.criticicalarc.com" Container
- Add generic function GetDocumentOrNil to standardize GetDocument
- Add func to return CrossPartition queries for Aggregates
- Add func GetNextSequence() as generic seq generator for AutoIncrement
- Add cosmosdbutil.ErrNoRows to return (emulate) sql.ErrNoRows
- Add a "fake" ExclusiveWriterFake
- Add standard "getXX", "setXX" and "queryXX" to all TABLE class files
- Add specific Table SEQ for the Events table
- Add specific Table SEQ for the Rooms table
- Add specific Table SEQ for the StateSnapshot table

* - Use CosmosDB for the KeyServer
- Replace the ConnString in the YAML to Cosmos
- Update the 4 tables to use Cosmos
2021-05-21 09:34:30 +10:00
alexfca 5d68daef80
Implement Cosmos DB for the RoomServer Service (#5)
* - Implement Cosmos for the devices_table
- Use the ConnectionString in the YAML to include the Tenant
- Revert all other non implemented tables back to use SQLLite3

* - Change the Config to use "test.criticicalarc.com" Container
- Add generic function GetDocumentOrNil to standardize GetDocument
- Add func to return CrossPartition queries for Aggregates
- Add func GetNextSequence() as generic seq generator for AutoIncrement
- Add cosmosdbutil.ErrNoRows to return (emulate) sql.ErrNoRows
- Add a "fake" ExclusiveWriterFake
- Add standard "getXX", "setXX" and "queryXX" to all TABLE class files
- Add specific Table SEQ for the Events table
- Add specific Table SEQ for the Rooms table
- Add specific Table SEQ for the StateSnapshot table
2021-05-20 14:42:33 +10:00
alexfca b696923333
- Implement Cosmos for the devices_table (#4)
- Use the ConnectionString in the YAML to include the Tenant
- Revert all other non implemented tables back to use SQLLite3
2021-05-12 16:30:49 +10:00
alexfca dfd5d445ac
Merge pull request #3 from criticalarc/af/CA-5532-AddCosmosAccounts
Add CosmosDB backend for the Accounts DB in Dendrite
2021-05-11 15:02:52 +10:00
Alex Flatow 6a154f22b3 - Add DB POCOs for all objects
- Update the names of the json to be consistent
- Move SQL into stmt lines as per the original code
- Change the _object to be a more descriptive property similar to SZ
- Reformat using the linter for the project (golangci-lint)
2021-05-11 14:31:17 +10:00
Alex Flatow a5ddb710d8 - Add GO CosmosDB library github.com/vippsas/go-cosmosdb
- Update YAML file to use file: everywhere except for Accounts
- Use the CosmosDB conn string in the YAML
- Add cosmosdbapi package to wrap the external package
- Add Tenant.go to store the tenancy settings - to be removed when tenancy is implemented
- Update the 5 tables to use the internal CosmosDBAPI package instead of SQL
- Remove sql from storage.go and other files
2021-05-11 09:11:33 +10:00
alexfca 234a89db5d
Merge pull request #2 from criticalarc/af/CA-5532-AddCosmosDB
Add CosmosDB Package and support to Dendrite
2021-05-10 13:43:17 +10:00
Alex Flatow 5ded872da9 - Add CosmosDB as a Datasource type
- Use the SQLLite as a base for the CosmosDB package(s)
- Update the ConnString to use file: from cosmosdb: so it still works
- Add a yaml file for the config to use CosmosDB
2021-05-06 15:09:44 +10:00
Alex Flatow 4b42a0addb storage.go 2021-05-06 14:41:27 +10:00
Jared Jacobson 312874a209
Merge pull request #1 from criticalarc/JJ/FromDev/SetupVsCodeDevEnvironment
Added some changes/helpers to dev locally using hydrogen-web client
2021-04-22 09:23:25 +10:00
jjacobson 192a67c362 Added some changes/helpers to dev locally using hydrogen-web client 2021-04-21 16:18:09 +10:00
Fero d6e9b7b307
Remove the 'Content-Type' request header requirement (#1834) 2021-04-19 16:29:51 +01:00
Neil Alexander a9faa1bc44
Fix registration error when disabled 2021-04-15 09:58:26 +01:00
Kegsay 656d11ec90
fedsender: tolerate dupe membership events (#1824)
* fedsender: tolerate dupe membership events

Previously if the fedsender got a duplicate membership event it would cause
the entire process to crash. Now it doesn't. This masks an issue with the
roomserver where it can emit duplicate membership events.

* Update joined_hosts_table.go
2021-04-14 11:11:54 +01:00
Fero 653e30619c
Remove the Content-Length requirement on upload request (#1831)
* Remove the Content-Length requirement

* Make sure that the file size does not exceed max content size

* Address review comment - universally check if temp file size exceeds max file size
2021-04-14 10:53:24 +01:00
Neil Alexander 080ae6a829
Move room mutex in federation API (#1830)
* Move room mutex in federation API to surround resolveStatesAndCheck

* Guard processEventWithMissingState instead

* Revert "Guard processEventWithMissingState instead"

This reverts commit 0ce88036aa.
2021-04-13 11:13:07 +01:00
Kegsay e08942fb00
Remove legacy register endpoint (#1822)
* Remove legacy register endpoint

We only support `/r0` CS API paths, not `/v1`.

* Finish removing
2021-04-09 10:21:35 +01:00
Kegsay b769d5a25e
Optimise memory usage when calling /g_m_e (#1819)
* Optimise memory usage when calling /g_m_e

* cache more events

* refactor handling of device list update pokes

* Sigh
2021-04-08 13:50:39 +01:00
Tim McCormack 5ade348d14
Document need for max-body-size change in reverse proxy as well (#1816)
Just changing the Media API's `max_file_size_bytes` isn't enough if
Dendrite is running behind a proxy; document the need for a proxy config
change in the place the admin is most likely to notice it.

Signed-off-by: Tim McCormack <cortex@brainonfire.net>
2021-04-08 12:08:38 +01:00
Bruce MacDonald d27607af78
Implement OpenID module (#599) (#1812)
* Implement OpenID module (#599)

- Unrelated: change Riot references to Element in client API routing

Signed-off-by: Bruce MacDonald <contact@bruce-macdonald.com>

* OpenID module tweaks (#599)

- specify expiry is ms rather than vague ts
- add OpenID token lifetime to configuration
- use Go naming conventions for the path params
- store plaintext token rather than hash
- remove openid table sqllite mutex

* Add default OpenID token lifetime (#599)

* Update dendrite-config.yaml

Co-authored-by: Kegsay <kegsay@gmail.com>
Co-authored-by: Kegsay <kegan@matrix.org>
2021-04-07 13:26:20 +01:00
Kegsay f8d3a762c4
Add a per-room mutex to federationapi when processing transactions (#1810)
* Add a per-room mutex to federationapi when processing transactions

This has numerous benefits:
 - Prevents us doing lots of state resolutions in busy rooms. Previously, room forks would always result
   in a state resolution being performed immediately, without checking if we were already doing this in
   a different transaction. Now they will queue up, resulting in fewer calls to `/state_ids`, `/g_m_e`, etc.
 - Prevents memory usage from growing too large as a result and potentially OOMing.

And costs:
 - High traffic rooms will be slightly slower due to head-of-line blocking from other servers,
   though this has always been an issue as roomserver has a per-room mutex already.

* Fix unit tests

* Correct mutex lock ordering
2021-03-30 10:01:32 +01:00
Eric Eastwood 0ee1c56ffd
Use log directory that we will be able to write to (#1799)
Fix https://github.com/matrix-org/dendrite/issues/1644
2021-03-30 09:53:02 +01:00
Kegsay af41f6d454
Add Sentry support (#1803)
* Add Sentry support

* Use HTTP Sentry properly maybe

* Capture panics

* Log fed Sentry stuff correctly

* British english linter
2021-03-24 10:25:24 +00:00
Kegsay 802f1c96f8
Add more metrics (#1802)
* Add more metrics

* Linting
2021-03-23 15:22:00 +00:00
Kegsay a1b7e4ef3f
log less for failed key querys, add counters for incoming pdus/edus (#1801)
* log less for failed key querys, add counters for incoming pdus/edus

* use labels

* Blacklist flakey test

* Fix metrics
2021-03-23 11:33:36 +00:00
Neil Alexander 01267a34b9
Fix nil pointer crash in QueryMembershipsForRoom 2021-03-17 13:58:04 +00:00
Kegsay 3c419be6af
roomserver: don't make_join with ourselves if clients ask us to (#1797)
* roomserver: don't make_join with ourselves if clients ask us to

* delete properly
2021-03-08 18:16:28 +00:00
Kegsay 77fb981da5
device lists: backoff for longer if the wrong error type is returned (#1796) 2021-03-08 17:45:20 +00:00
Kegan Dougal e865a1507a Make sure the component max open conns doesn't exceed 100 2021-03-08 14:50:37 +00:00
Neil Alexander 5912429d53
Return a more useful error on /register spec compliance violation (#1792) 2021-03-08 13:57:15 +00:00
Neil Alexander 5acf30cd3c
Update sytest-whitelist 2021-03-08 13:32:21 +00:00
Kegsay 850abb1dde
Make bcrypt cost configurable (#1793) 2021-03-08 13:19:02 +00:00
Neil Alexander c3ad2cca49
Fix database default connection limits for CI (#1794) 2021-03-08 13:18:29 +00:00
Neil Alexander 6aa262ead8
Use default transport for AS traffic (#1789)
* Use default transport for AS traffic

* Update gmsl and use default client

* Remove replace

* Fix go.sum

* Update gomatrixserverlib

* Go back to appservices managing their own HTTP clients because argh

* Add missing context
2021-03-05 16:40:32 +00:00
Will Hunt fe021d3742
Treat the sender_localpart as an exclusive namespace of one user (#1790) 2021-03-05 14:57:42 +00:00
Neil Alexander 1ad96e2e2d
Tweak AS registration check and AS component HTTP clients (#1785)
* Tweak AS registration check

* Check appservice usernames using correct function

* Update sytest-whitelist

* Use gomatrixserverlib.Client since that allows us to disable TLS validation using the config

* Add appservice-specific client and ability to control TLS validation for appservices only

* Set timeout on appservice client

* Review comments

* Remove dead code

* Enforce LoginTypeApplicationService after all

* Check correct auth type field
2021-03-05 10:40:27 +00:00
Will Hunt 9557ccada4
Fix appsevice alias queries part 2 (#1684)
* Check membership of room

* Use QueryStateAfterEventsResponse

* Fix complexity

* Add field ShouldHitAppservice to GetRoomIDForAlias

* Hit appservice when trying to join a non-existent alias

* remove unused

* Changes that I made a long time ago

* Rename to appserviceJoinedAtEvent

* Check membership in GetMemberships

* Update QueryMembershipsForRoom

* Tweaks in client API

* Update appserviceJoinedAtEvent

* Comments

* Try QueryMembershipForUser instead

* Undo some changes to client API that shouldn't be needed

* More /event tweaks

* Refactor /event bit

* Go back to QueryMembershipsForRoom because appservices are hard

* Fix bugs in onMessage

* Add comments

* More logical naming, clean up a bit

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-03-03 17:00:31 +00:00
Will Hunt a2773922d2
Send events to appservice based on room membership (#1680)
* Check membership of room

* Use QueryStateAfterEventsResponse

* Fix complexity

* Changes that I made a long time ago

* Rename to appserviceJoinedAtEvent

* Check membership in GetMemberships

* Update QueryMembershipsForRoom

* Tweaks in client API

* Update appserviceJoinedAtEvent

* Comments

* Try QueryMembershipForUser instead

* Undo some changes to client API that shouldn't be needed

* More /event tweaks

* Refactor /event bit

* Go back to QueryMembershipsForRoom because appservices are hard

* Fix bugs in onMessage

* Add comments

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-03-03 16:27:44 +00:00
Neil Alexander d15836e260
Increase gocyclo complexity to 25 (and remove all but 2 golint directives related to it) (#1783) 2021-03-03 14:35:57 +00:00
Neil Alexander f0139f12ca
Don't return error when account conflict is handled gracefully (#1782) 2021-03-03 14:01:34 +00:00
stoically 41fd15b9b6
Docker fix and tweak (#1781)
* Fix image building with recent docker version

* Mount media path in docker to prevent data loss
2021-03-03 11:20:40 +00:00
Neil Alexander 25dc99f1b0
Upgrade dependencies (#1779) 2021-03-03 11:08:41 +00:00
Kegsay 586cc5be5e
Use new path for MSC2946 (#1778)
* Use new path for MSC2946

* Update GMSL
2021-03-02 16:41:28 +00:00
Neil Alexander 6a35d9f1b1
Version 0.3.11 2021-03-02 11:46:35 +00:00
Neil Alexander 81312b8a78
Return the current OTK count on an empty upload request (#1774)
* Always return OTK counts

* Fix parameter ordering

* Send IDs over to keyserver internal API

* Review comments

* Fix syntax error

* Fix panic, hopefully

* Require user ID to be set

* Fix user API call
2021-03-02 11:40:20 +00:00
Neil Alexander f5cf241877
Fix user registration bug (#1777) 2021-03-02 10:43:25 +00:00
Neil Alexander 3069079e37
Revert quic-go dependency update 2021-02-18 10:18:14 +00:00
Neil Alexander db637515a5
Update libp2p dependencies 2021-02-18 10:14:24 +00:00
Neil Alexander 11830fd4af
Fix transitive dependency for P2P demo 2021-02-18 09:45:55 +00:00