This has two benefits:
1. Using channels makes it easier to time out while waiting
2. Allows us to clean up goroutines that were waiting if we timeout the
request
The way we store the partition offsets for kafka streams means that when
we start after a crash we may get the last message we processed again.
This means that we have to be careful to ensure that the processing
handles consecutive duplicates correctly.
The HTTP handlers in the components are split into reader and writer directories. This was a fairly arbitrary distinction, and turns out to not be so helpful. Most read APIs have a corresponding write API, and it is more natural for them to be in the same file rather than in different directories.
* Implement filter POSTing and GETting.
Signed-off-by: Jan Christian Grünhage <jan.christian@gruenhage.xyz>
* Add missing '}' typo introduced during merge
* Still trying to fix that merge...
* Fix linting
* Add table for storing invites in the syncapi
* Use the invite table to list the active invites for a user
* Update the invites table from the roomserver stream
* Include the invites table when determining the maxInviteID
* Use a shared PostgreSQL sequence to generate ids.
Share an auto incrementing sequnce between the account data and
the room event table.
This means that account data updates can be received independantly of
room events updates.
This should give some basic support for fixing #212
* Remove redundant 'primary key'
* Re-number the SQL arguments
* Fewer lies in comments
* Specify HTTP methods for the client API
* Specify HTTP methods for the federation API
* Specify HTTP methods for the media API
* Specify HTTP methods for the sync API
* Add comment
* gb vendor update github.com/matrix-org/gomatrixserverlib
* Add handler for the exchange_third_party_invite endpoint
* Doc
* Use SendEvents to send the invite to the roomserver
* Add missing error check
* Add checks
* Add config for trusted ID servers
* Add new error
* Implement check for trusted ID server
* Complete unfinished comment
* Make comment more explicit in the config file
* Use go standard errors in membership.go
* Use standard errors instead of JSON responses in threepid
* Doc errors
* Remove unused parameter
* Use federation to auth the event if the server isn't in the room
* Use MakeAPI for 3pid onbind handler as it isn't a standard federation request
* Error check
* Temporarily disable tests
* Fix return on 3PID invite
* Re-enable tests
* Remove useless else
* gb vendor update github.com/matrix-org/gomatrixserverlib
* gb vendor update github.com/matrix-org/gomatrixserverlib
* Implement same behaviour as synapse
* Fix condition and array initialisation
* Log errors on iteration and throw one if no server could be reached
* Fix err not being initialised
* Fix lint
* Fix import path
* Fix response to /invite to match the format expected by synapse
* gb vendor update github.com/matrix-org/gomatrixserverlib
* Use gomatrixserverlib.RespInvite
* gb vendor update github.com/matrix-org/gomatrixserverlib
* Add missing file headers
* Move the ID server's signatures verification to common
* Allow verification without specifying a server name
* Add third-party structs to membership events content
* Add processing of 3PID onbind requests
* Use reference for third party invite data
* Fix return arguments order
* Revert "Move the ID server's signatures verification to common"
This reverts commit 9344201031.
* Revert "Allow verification without specifying a server name"
This reverts commit fd27afbf82.
* Remove checks that are already occurring in gomatrixserverlib
* Change return type of createInviteFrom3PIDInvite
* Add doc, add checks in fillDisplayName
* Use MakeFedAPI
* Invert condition
* Use AuthEvents to retrieve the 3PID invite
* Update comment
* Remove unused parameter
* gb vendor update github.com/matrix-org/gomatrixserverlib
* Remove unused struct field
* Ignore unused test data
* Remove unused variables
* Remove deadcode
* Fix up vetshadow warnings
* Convert to using gometalinter
* Update travis
* Use vendored versions of gometalinter
* Make gometalinter install its stuff
* Vendor misspell
* Create package for handling 3pid processes and move invite processing there
* Add database table and functions for tracking 3PIDs
* Add structures and functions to interact with an ID server
* Add handlers for 3PIDs management
* Fix 3PIDs retrieval sending null if no 3PID known for a user
* Include medium in database requests and function calls
* Publish an association if it has been validated and requested
* Add TODO markers for tursted ID server check
* Use a structure instead of a map to represent a 3PID
* Structure for 3PID invite
* Generate invite from 3PID known by ID server
* Load user profile in a separate function
* Generate m.room.third_party_invite if the ID server doesn't know the 3PID
* Fix URLs to the spec in comments
* Move third-party invites to a separate package and doc' it
* Handle non-OK status codes on lookup
* Send display name to identity server when asking to store an invite
* Remove join response structure
* Change the way some variables are declared or passed as argument
* Use gomatrixserverlib.Base64String instead of the builtin base64 package
* Don't copy the public keys array
* Implement case where user left the room
* Filter by membership event
* Move the logic from the storage to the query API
* Fix check on state entries iteration
* Remove aliases methods from query API
* Use structure for response to match with the spec
* Remove filtering on /members and implement /joined_members
* Add query API for listing active invites
This lists the invites for a user in a room that could be used to
join the room over federation.
* s/Lookup/Look up/
* Fix implements comments
* Use BuildEvent method on room join
* Fix building the list of room members in the sync notifier
* Fix building the list of room members in the sync notifier
* Rephrase comment
* Move events contents to common
* Basic database structure
* Complete database update
* Support visibility update and retrieval
* Add HTTP methods for visibility update and retrieval
* Add the database for the new component
* Add a listener for the new component
* Fix attribute update statements
* Create public rooms component
* Fix failing test
* Add roomserver consumer
* Fix a bug in aliases creation
* Add a check on type
* Implement public rooms directory
* Use auth API for visibility update
* Support filtering
* Add component to monolith
* Various fixes
* Fix computation of next public rooms batch
* Retrieve state events from the roomserver query API + avoid dupes on join
* Split update of string or boolean attribute in two separate functions
* Use event type to detect duplicate joins
* Improve the joined members counter computation
* Use event.RoomID()
* Add input API for adding invites to the roomserver.
This API handles invites received over federation that occur outside of a room.
* Add some docstring for withTransaction
* Use a nicer pattern for wrapping transactions
* Fix MembershipUpdater method to not commit the transaction before returning it
* Use the Transaction interface from common
* Basic memberships retrieval
* Change the way the memberships are saved in the client API database
* Retrieve single membership
* Get memberships only if the user is or has been in the room
* Check server name on room ID instead of user ID
* Save the join membership event and updates it when necessary
* Membership events retrieval + update on leave
* Implement the API on the roomserver and client API server
* Fix comments
* Remove the functions and attributes used before the new query API
* Explicitely state what we return in query
* Remove tab
We can't consume the same topic on a single kafka consumer more than
once. So when using kafka we have to create a new consumer for each
component in the monolith.
* dependency injection for the kafka consumers/producers
* Optionally use naffka in the monolithic server
* remember to call setupKafka()
* tweak imports
* fix integration tests
* Add use_naffka to the example config
* Update comment on the listen APIs
* Storage functions for invite events
* Add table for tracking membership state
* More stuff
* More stuff
* Use utility methods from gomatrixserverlib, rather than reimplementing them
* More stuff
* Return string rather than pointer to string
* Update gomatrixserverlib
* Use HTTP API for roomserver input.
* Use synchronous HTTP API for writing events to the roomserver
* Remove unused config for kafka topic
* Add new output types to roomserver for invites
* Write membership updates
* Separate filtering from pairing up changes in membershipChanges
* Fix SQL
* Fix SQL
* Namespace the tables
* Fix SQL
* Use clearer names for some of the variables
* Rename senderID for consistency
* Restructure update membership
* Comments
* More comment
* Fix SQL
* More comments
* Assign state keys inside the transaction
* Comment on the purpose of the latestEventsUpdater
* Comment on the purpose of updateMembership
* Remove duplicate fields from stateChange
* Attempt to rewrite comment in 'english'
* More comments
* Fix comment
* Comment
* more comments
* Add prefixes to namespace the SQL tables.
This means that multiple components can share a single database schema
without colliding with each other.
Once this lands it will be possible to run a single monolithic dendrite
against a single postgresql schema.
Hopefully this will make trivial deployments and development easier.
* Comment
* Implement membership endpoints
* Use FillBuilder when possible
* Fix typo in membership event content
* Fix state key invite membership events not being correctly set
* Set membership content to match the profile of the user in state_key
* Move event building and rename common function
* Doc getMembershipStateKey
* Check if user is local before lookin up their profile
This makes it possible to setup all the component APIs on a single http
listener which is necessary if we want to combine all the components
into a single monolith.
* Clean roomserver consumer
* Make account data sync incremental
* Use a different name for the sync AD table
* Improved error logging
* Created missing topic in tests
* Add client API topic to tests
* Add client API topic to common
* Move data batch retrieval
* Add database index for data retrieval
* Fix typo in table name
* Fix indentation
* Add database functions to interact with aliases
* Save room alias
* Check if alias already exists
* Implement local alias lookup
* Implement alias removal
* Move alias API
* Handle case with no alias set for a room
* Apply requested changes
* Save function for account data
* Fix upsert + add empty routes and function
* Save account data
* Retrieval functions
* Implement retrieval in /sync
* Fix arrays not correctly initialised
* Merge account data retrieval functions
* Request DB only once per request
* Initialise array
* Fix comment
