Commit graph

2504 commits

Author SHA1 Message Date
Brendan Abolivier b15ce900ab Implement public rooms APIs (#185)
* Move events contents to common

* Basic database structure

* Complete database update

* Support visibility update and retrieval

* Add HTTP methods for visibility update and retrieval

* Add the database for the new component

* Add a listener for the new component

* Fix attribute update statements

* Create public rooms component

* Fix failing test

* Add roomserver consumer

* Fix a bug in aliases creation

* Add a check on type

* Implement public rooms directory

* Use auth API for visibility update

* Support filtering

* Add component to monolith

* Various fixes

* Fix computation of next public rooms batch

* Retrieve state events from the roomserver query API + avoid dupes on join

* Split update of string or boolean attribute in two separate functions

* Use event type to detect duplicate joins

* Improve the joined members counter computation

* Use event.RoomID()
2017-08-22 11:12:51 +01:00
Richard van der Hoff fc86821a90 notes on running monolith server (#193)
* notes on running monolith server

* Clarify default ports for monolith server
2017-08-22 11:01:14 +01:00
Richard van der Hoff 46877b6baa Add installation instructions to repo (#192) 2017-08-21 17:34:05 +01:00
Mark Haines 808c2e09f6 Make txn *sql.Tx arguments optional everywhere using a utility function (#191)
* Make txn *sql.Tx arguments optional everywhere using a utility function

* Clarify that if the txn is nil the stmt will run outside a transaction
2017-08-21 17:20:23 +01:00
Mark Haines 57b7097368 Add input API for adding invites to the roomserver. (#187)
* Add input API for adding invites to the roomserver.

This API handles invites received over federation that occur outside of a room.

* Add some docstring for withTransaction

* Use a nicer pattern for wrapping transactions

* Fix MembershipUpdater method to not commit the transaction before returning it

* Use the Transaction interface from common
2017-08-21 16:37:11 +01:00
Brendan Abolivier 5950293e79 Membership viewing API (#174)
* Basic memberships retrieval

* Change the way the memberships are saved in the client API database

* Retrieve single membership

* Get memberships only if the user is or has been in the room

* Check server name on room ID instead of user ID

* Save the join membership event and updates it when necessary

* Membership events retrieval + update on leave

* Implement the API on the roomserver and client API server

* Fix comments

* Remove the functions and attributes used before the new query API

* Explicitely state what we return in query

* Remove tab
2017-08-21 16:34:26 +01:00
Mark Haines 81179a0595 Fix integration test proxy certificate arguments (#190) 2017-08-21 16:23:59 +01:00
Brendan Abolivier 35df4767c6 Remove defaults for TLS so the proxy can more easily serve over HTTP (#189) 2017-08-21 14:25:47 +01:00
Mark Haines efbc14f7b9 vars["txnID"] is always empty for state events (#188) 2017-08-18 16:10:28 +01:00
Brendan Abolivier f607ef29c2 Add a route matching the trailing slash on the state event sending route (#182)
* Add a route matching the trailing slash on the state event sending route

* Use single route for both cases

* Use synapse regexp
2017-08-18 15:33:40 +01:00
Mark Haines 877ea5cb62 Remove StopProcessingAfter from the roomserver consumer as it is unused (#186) 2017-08-18 11:33:10 +01:00
Mark Haines ba8b5d8bf9 Fix kafka consumer setup in monolith. (#184)
We can't consume the same topic on a single kafka consumer more than
once. So when using kafka we have to create a new consumer for each
component in the monolith.
2017-08-16 16:55:34 +01:00
Mark Haines c27d1fdfb4 Optionally use naffka in the monolithic server (#183)
* dependency injection for the kafka consumers/producers

* Optionally use naffka in the monolithic server

* remember to call setupKafka()

* tweak imports

* fix integration tests

* Add use_naffka to the example config

* Update comment on the listen APIs
2017-08-16 13:36:41 +01:00
Mark Haines 0d894e3da5 gb vendor fetch github.com/matrix-org/naffka 2017-08-15 16:06:09 +01:00
Mark Haines 2071387f3c Add tables for tracking the state of invites to the room server. (#165)
* Storage functions for invite events

* Add table for tracking membership state

* More stuff

* More stuff

* Use utility methods from gomatrixserverlib, rather than reimplementing them

* More stuff

* Return string rather than pointer to string

* Update gomatrixserverlib

* Use HTTP API for roomserver input.

* Use synchronous HTTP API for writing events to the roomserver

* Remove unused config for kafka topic

* Add new output types to roomserver for invites

* Write membership updates

* Separate filtering from pairing up changes in membershipChanges

* Fix SQL

* Fix SQL

* Namespace the tables

* Fix SQL

* Use clearer names for some of the variables

* Rename senderID for consistency

* Restructure update membership

* Comments

* More comment

* Fix SQL

* More comments

* Assign state keys inside the transaction

* Comment on the purpose of the latestEventsUpdater

* Comment on the purpose of updateMembership

* Remove duplicate fields from stateChange

* Attempt to rewrite comment in 'english'

* More comments

* Fix comment

* Comment

* more comments
2017-08-08 16:38:03 +01:00
Brendan Abolivier c35803c9d8 Add TLS (https) support to the monolithic server (#180)
* Add HTTPS listen and serve, and move both listen and serve to goroutines

* Add some log so we know the initialisation went well
2017-08-07 13:39:53 +01:00
Mark Haines 4d1504ee93 Add prefixes to namespace the SQL tables. (#177)
* Add prefixes to namespace the SQL tables.

This means that multiple components can share a single database schema
without colliding with each other.

Once this lands it will be possible to run a single monolithic dendrite
against a single postgresql schema.

Hopefully this will make trivial deployments and development easier.

* Comment
2017-08-07 11:51:46 +01:00
Jonas Platte 21a8c24d5f Add .editorconfig (#179) 2017-08-05 02:25:40 +01:00
Jonas Platte c15e4572ff Fix typo: ed2519 -> ed25519 (#178) 2017-08-05 02:25:03 +01:00
Brendan Abolivier 8ccc5d108b Implement membership APIs (#171)
* Implement membership endpoints

* Use FillBuilder when possible

* Fix typo in membership event content

* Fix state key invite membership events not being correctly set

* Set membership content to match the profile of the user in state_key

* Move event building and rename common function

* Doc getMembershipStateKey

* Check if user is local before lookin up their profile
2017-08-04 16:32:10 +01:00
Brendan Abolivier 03dd456b47 Fix bad initialisation of sync API notifier in monolith (#176) 2017-08-04 14:54:08 +01:00
Mark Haines 1a28cf9767 Add a monolithic server that combines all the components into one (#175)
* Add a monolithic server that combines all the components into one

* Review comments

* Expose the matrix APIs directly
2017-08-04 13:12:36 +01:00
Mark Haines 4975eb9074 Move setting up the api mux to outside the routing.Setup functions. (#173)
This makes it possible to setup all the component APIs on a single http
listener which is necessary if we want to combine all the components
into a single monolith.
2017-08-03 15:10:39 +01:00
Mark Haines 3b07633326 Simplify preparing the roomserver SQL (#172) 2017-08-02 16:50:34 +01:00
Brendan Abolivier 0fbb8b7824 Make account data sync incremental (#170)
* Clean roomserver consumer

* Make account data sync incremental

* Use a different name for the sync AD table

* Improved error logging

* Created missing topic in tests

* Add client API topic to tests

* Add client API topic to common

* Move data batch retrieval

* Add database index for data retrieval

* Fix typo in table name

* Fix indentation
2017-08-02 16:21:35 +01:00
Brendan Abolivier 7d17df6f51 Add local room join (#169)
* Add local room join

* Update TODO
2017-07-28 15:29:12 +01:00
Brendan Abolivier 34165c7761 Add default config file name for media API and room servers (#168) 2017-07-28 11:32:16 +01:00
Brendan Abolivier c7e36ba2a1 Implement room aliases (#167)
* Add database functions to interact with aliases

* Save room alias

* Check if alias already exists

* Implement local alias lookup

* Implement alias removal

* Move alias API

* Handle case with no alias set for a room

* Apply requested changes
2017-07-28 11:31:43 +01:00
Brendan Abolivier 3e394e9e21 Save and retrieve account data (#166)
* Save function for account data

* Fix upsert + add empty routes and function

* Save account data

* Retrieval functions

* Implement retrieval in /sync

* Fix arrays not correctly initialised

* Merge account data retrieval functions

* Request DB only once per request

* Initialise array

* Fix comment
2017-07-26 14:53:11 +01:00
Brendan Abolivier 6d073dcf9f Propagate profile update through rooms (#163)
* Use gomatrixserverlib function to split user ID

* Propagate profile update via m.room.member events

* Send profile data on room join

* Send profile data on room creation

* Rename variable

* Move membership update to roomserver consumer

* Improve iteration

* Move event update from client API server to sync API server

* Change the way buildMembershipEvents is called

* Forbid update of someone else's profile

* Use gomatrixserverlib method

* Fix depth and previous events not being set

* Fix wrong removal in latest commit

* Update all events instead of only memberships

* Handle case where there is no state key

* Fix test
2017-07-25 16:10:59 +01:00
Mark Haines a904380e1b gb vendor update github.com/matrix-org/gomatrixserverlib 2017-07-24 13:42:55 +01:00
Brendan Abolivier ce311ce0fe Improve room creation (#164) 2017-07-20 13:06:14 +01:00
Mark Haines e6d77d6bde Use HTTP API for roomserver input. (#161)
* Use HTTP API for roomserver input.

* Use synchronous HTTP API for writing events to the roomserver

* Remove unused config for kafka topic

* Tweak comments
2017-07-18 13:40:03 +01:00
Brendan Abolivier d9b8e5de45 Keep track of membership in Client API (#159)
* Saving memberships

* Removed unused index

* Removed useless log

* Fixed membership not being saved on the right conditions + added membership removal

* Updated outdated comment

* Use server lib method + check server name + use new roomserver API

* Better handling of events from the room server

* Fixed membership removal

* Corrected indentation

* Fix tests (hopefully)

* Replace broken kafka mirror

* Apply requested changes on database management

* Remove useless check and function

* Moved memberships update to the database package

* Use new common function

* Remove useless function
2017-07-17 18:10:56 +01:00
Mark Haines b06d1124f7 Factor out runTransaction to common code (#162) 2017-07-17 17:20:57 +01:00
Mark Haines d3a29b7816 Add a HTTP API for roomserver input (#158)
* Add a HTTP API for roomserver input

* Fix integration tests
2017-07-13 11:41:30 +01:00
Brendan Abolivier f40b776aa3 Updated config file example with federation sender database (#157) 2017-07-12 14:48:12 +01:00
Mark Haines 4e7862e3b9 Make the room server RPC API json formats more consistent with our other APIs (#156) 2017-07-12 14:13:10 +01:00
Mark Haines cea9e31723 Make the roomserver output format more flexible (#155)
* Make the roomserver output format more flexible

* Fix the sync server integration testdata

* Fix roomserver testdata

* Actually fix the sync server tests

* Fix typo
2017-07-12 10:46:29 +01:00
Brendan Abolivier 7d36ca03af Implemented logout (#154) 2017-07-11 16:04:34 +01:00
Brendan Abolivier 355ab5eedf Notify profile update (#153)
* Profile retrieval

* Saving avatar (without propagating it)

* Saving display name (without propagating it)

* Getters for display name and avatar URL

* Doc'd

* Introduced new Kafka topic and producer

* Updated config with new kafka topic

* Switched to samara producer and now sending messages

* Doc'd

* Put kafka update after the database insert

* Doc'd profileUpdate structure

* Removed unused parameter

* Moved user updates producer to clientapi/producers
2017-07-11 14:14:06 +01:00
Brendan Abolivier 1efbad8119 Profile API (#151)
* Profile retrieval

* Saving avatar (without propagating it)

* Saving display name (without propagating it)

* Getters for display name and avatar URL

* Doc'd

* Remove unused import

* Applied requested changes

* Added auth on PUT /profile/{userID}/...

* Improved error handling/reporting

* Using utils log reporting

* Removed useless checks
2017-07-10 14:52:41 +01:00
Mark Haines 69c29172c3 Use utility methods from gomatrixserverlib. (#152)
* Use utility methods from gomatrixserverlib, rather than reimplementing them

* Return string rather than pointer to string

* Update gomatrixserverlib
2017-07-07 14:11:32 +01:00
Brendan Abolivier b13cbb18fb Response to createRoom is now the room's ID (#149) 2017-07-06 11:44:15 +01:00
Mark Haines b2f9b8c641 Update the example configs with the new unified config (#146)
* Update the example configs with the new unified config

* Review comments

* Use an int for the config version
2017-07-05 13:10:28 +01:00
Mark Haines 8f1dca4e74 Add a component for sending event to remote matrix servers using fede… (#148)
* Add a component for sending event to remote matrix servers using federation

* Use named constants

* Comment on why we combine the deltas

* Comments

* Copyright headers!

* Fix the copy+Pasted comment

* Add tests for combineDeltas

* Review comments

* More review comments

* More comments

* Even more comments

* Name the mutexes

* Document what the mutexes are protecting

* Simplify the SQL, and more comments
2017-06-28 16:10:17 +01:00
Mark Haines a6bb92520c Ignore hidden files and kafka directories 2017-06-28 10:28:13 +01:00
Mark Haines 54e7e3041b Set depth of events and whether they need to be federated. (#145)
* Set depth of events and whether they need to be federated.

Set the depth of each new event to one greater than the maximum depth
of it's direct ancestors.

Add a flag to each event passing through the roomserver that tells us
whether the event needs to be sent over federation.

We do this by passing the name of the server to send the event as.
We will need this capability if we want to support vhosting as it is
not possible to tell from the event alone which server to send it as.

(The reason for this is that sometimes a event needs to be sent on
behalf of a different remote matrix server)

* Update roomserver integration tests

* More comments

* Constants and comments

* Fix comments

* Document the int64 returned by LatestEventIDs
2017-06-27 15:28:44 +01:00
Mark Haines 572f6c399d Include the state before the event in roomserver output (#143)
* Include the state before the event in roomserver output

* Fix roomserver integration test

* Remove VisibilityEventIDs from the JSON

* More comments

* Remove spurious TODO
2017-06-27 13:20:04 +01:00
Mark Haines 524475f8a3 Stub APIs needed to let riot join a room (#147)
* Stub APIs needed to let riot join a room

* Fix comments
2017-06-27 12:37:25 +01:00