This makes it possible to setup all the component APIs on a single http
listener which is necessary if we want to combine all the components
into a single monolith.
* Clean roomserver consumer
* Make account data sync incremental
* Use a different name for the sync AD table
* Improved error logging
* Created missing topic in tests
* Add client API topic to tests
* Add client API topic to common
* Move data batch retrieval
* Add database index for data retrieval
* Fix typo in table name
* Fix indentation
* Add database functions to interact with aliases
* Save room alias
* Check if alias already exists
* Implement local alias lookup
* Implement alias removal
* Move alias API
* Handle case with no alias set for a room
* Apply requested changes
* Save function for account data
* Fix upsert + add empty routes and function
* Save account data
* Retrieval functions
* Implement retrieval in /sync
* Fix arrays not correctly initialised
* Merge account data retrieval functions
* Request DB only once per request
* Initialise array
* Fix comment
* Use gomatrixserverlib function to split user ID
* Propagate profile update via m.room.member events
* Send profile data on room join
* Send profile data on room creation
* Rename variable
* Move membership update to roomserver consumer
* Improve iteration
* Move event update from client API server to sync API server
* Change the way buildMembershipEvents is called
* Forbid update of someone else's profile
* Use gomatrixserverlib method
* Fix depth and previous events not being set
* Fix wrong removal in latest commit
* Update all events instead of only memberships
* Handle case where there is no state key
* Fix test
* Use HTTP API for roomserver input.
* Use synchronous HTTP API for writing events to the roomserver
* Remove unused config for kafka topic
* Tweak comments
* Saving memberships
* Removed unused index
* Removed useless log
* Fixed membership not being saved on the right conditions + added membership removal
* Updated outdated comment
* Use server lib method + check server name + use new roomserver API
* Better handling of events from the room server
* Fixed membership removal
* Corrected indentation
* Fix tests (hopefully)
* Replace broken kafka mirror
* Apply requested changes on database management
* Remove useless check and function
* Moved memberships update to the database package
* Use new common function
* Remove useless function
* Make the roomserver output format more flexible
* Fix the sync server integration testdata
* Fix roomserver testdata
* Actually fix the sync server tests
* Fix typo
* Profile retrieval
* Saving avatar (without propagating it)
* Saving display name (without propagating it)
* Getters for display name and avatar URL
* Doc'd
* Introduced new Kafka topic and producer
* Updated config with new kafka topic
* Switched to samara producer and now sending messages
* Doc'd
* Put kafka update after the database insert
* Doc'd profileUpdate structure
* Removed unused parameter
* Moved user updates producer to clientapi/producers
* Add a component for sending event to remote matrix servers using federation
* Use named constants
* Comment on why we combine the deltas
* Comments
* Copyright headers!
* Fix the copy+Pasted comment
* Add tests for combineDeltas
* Review comments
* More review comments
* More comments
* Even more comments
* Name the mutexes
* Document what the mutexes are protecting
* Simplify the SQL, and more comments
* Set depth of events and whether they need to be federated.
Set the depth of each new event to one greater than the maximum depth
of it's direct ancestors.
Add a flag to each event passing through the roomserver that tells us
whether the event needs to be sent over federation.
We do this by passing the name of the server to send the event as.
We will need this capability if we want to support vhosting as it is
not possible to tell from the event alone which server to send it as.
(The reason for this is that sometimes a event needs to be sent on
behalf of a different remote matrix server)
* Update roomserver integration tests
* More comments
* Constants and comments
* Fix comments
* Document the int64 returned by LatestEventIDs
* Include the state before the event in roomserver output
* Fix roomserver integration test
* Remove VisibilityEventIDs from the JSON
* More comments
* Remove spurious TODO
* Move all the dendrite config in to a single place
* Add tests for config parsing
* replace syncserver config with common config
* Replace client API config with common config
* Replace federation API config with common config
* Replace media api config with common config
* Replace room server config with common config
* Remove unused readKey function
* Fix the integration tests
* Comment on hardcoding roomserver to HTTP
* Add a method for getting RoomServerURL
This moves the hardcoding of HTTPs into one place.
* Add a single config file format for dendrite.
This config file combines all the configuration needed by the
dendrite components into a single file.
* Resolve relative paths against the current working directory
* Review comments
* more review comments!
* Document what the purpose of the version field is
* cmd/mediaapi-integration-tests: Add foundation for testing
* common/test: Add some server init and client request utilities
* common/test/client: Handle timed out requests for tests that passed
* cmd/syncserver-integration-tests: Port to new common/test infra
* common/test/client: Remove stray debug print
* cmd/mediaapi-integration-tests: Simplify slice initialisation
* cmd/mediaapi-integration-tests: Simplify getMediaURL argument
* cmd/mediaapi-integration-tests: Make startMediaAPI return listen address
* common/test/client: Fix uninitialised LastRequestErr
* common/test/server: Remove redundant argument
* common/test/server: Add StartProxy to create a reverse proxy
* cmd/mediaapi-integration-tests: Add proxies in front of servers
This is needed so that origins can be correctly configured and used for
remote media.
* travis: Enable media API integration tests
* travis: Build the client-api-proxy for media tests
* common/test/client: Don't panic on EOF in CanonicalJSONInput
* cmd/mediaapi-integration-tests: Add upload/download/thumbnail tests
* mediaapi/thumbnailer: Store thumbnail according to requested size
* cmd/mediaapi-integration-tests: Add totem.jpg test file
* cmd/client-api-proxy: Optionally listen for HTTPS
* common/test/client: Do not verify TLS certs for testing
We will commonly use self-signed certs.
* cmd/mediaapi-integration-tests: Make HTTPS requests
* cmd/mediaapi-integration-tests: Log size and method for thumbnails
* mediaapi/thumbnailer: Factor out isThumbnailExists
Appease gocyclo^w^w simplify
* mediaapi/thumbnailer: Check if request is larger than original
* travis: Install openssl and generate server.{crt,key}
* cmd/mediaapi-integration-tests: Add valid dynamic thumbnail test
* cmd/mediaapi-integration-tests: Document state of tests
* cmd/mediaapi-integration-tests: Test remote thumbnail before download
This ordering also exercises the cold cache immediate generation of a
size configured for pregeneration.
* travis: Explain openssl key+cert generation
* common/test/server: Clarify postgresContainerName
* Fetching missing state from the roomserver.
Whenever the syncserver receives an event from the room server that adds
state that isn't in the syncserver's local database it should fetch
those state events from the roomserver.
* Fix append
* Put comment back
* Comments
* s/addsStateEvents/lookupStateEvents/
* Fix spelling
* Include the stream position that a state event was added at in the current state tables
* Fix comment
* Review comments
* Add API for querying events by ID.
* Fix tense
* Start implementing federation ingress
* More stuff
* Hook up federation event receiving
* Handle the case where we are missing state
* Fix docstring and comments
* Fix infinite loop when printing unknownRoomError
* vendor: Add bimg image processing library
bimg is MIT licensed. It depends on the C library libvips which is LGPL
v2.1+ licensed. libvips must be installed separately.
* mediaapi: Add YAML config file support
* mediaapi: Add thumbnail support
* mediaapi: Add missing thumbnail files
* travis: Add ppa and install libvips-dev
* travis: Another ppa and install libvips-dev attempt
* travis: Add sudo: required for sudo apt* usage
* mediaapi/thumbnailer: Make comparison code more readable
* mediaapi: Simplify logging of thumbnail properties
* mediaapi/thumbnailer: Rename metrics to fitness
Metrics is used in the context of monitoring with Prometheus so renaming
to avoid confusion.
* mediaapi/thumbnailer: Use math.Inf() for max aspect and size
* mediaapi/thumbnailer: Limit number of parallel generators
Fall back to selecting from already-/pre-generated thumbnails or serving
the original.
* mediaapi/thumbnailer: Split bimg code into separate file
* vendor: Add github.com/nfnt/resize pure go image scaler
* mediaapi/thumbnailer: Add nfnt/resize thumbnailer
* travis: Don't install libvips-dev via ppa
* mediaapi: Add notes to README about resizers
* mediaapi: Elaborate on scaling libs in README
* Add API for querying events by ID.
* Fix tense
* Start implementing federation ingress
* More stuff
* Hook up federation event receiving
* Fix comments
* Comment on the order of the arrays
Avoid locking around db requests by only locking around active requests
and always creating an active request if there is none. A nice side
effect of this is that if many parallel requests for remote media come
in (a common case) then only one database query is made for the duration
of the query.
* Add API for querying the state after a list of events
* Implement state after events API
* Actually implement the API
* Reorder .travis.yml
* Comments
* Use common.MakeAPI
Content-Disposition is only used for communicating the filename. It does
not need to be stored in the database as we have upload_name anyway. It
does not need to be in types.MediaMetadata and does not need to be
logged.
Reorder functions to have public API functions in alphabetical order at
the top, internal package functions at the bottom in call order.
Use RawURLEncoding to avoid padding the hash with '='.
Use stronger types for paths in public API.
Simplify comments.
* Start implementing the join room API
* Hacks to get join room working
* Make the TLS fingerprint configurable
* Fix the client API proxy to handle '#' correctly
* Return a 200 OK response
* Write the join event along with current state to the room server
* Comment on the error handling
* Fix typos
* Fix tab
* Add TODO for moving authEventIDs to gomatrixserverlib
* Comment on why we ignore the key ID argument for local keys
* Avoid 'preceeded'
* Neaten the control flow
* Neaten the control flow even more
* Return pointers
* Rename err to lastErr for the loop
- Renamed `clientapi/auth/types` to `clientapi/auth/authtypes` for the same
horrible namespace clashing reasons as `storage`.
- Factored out `makeAPI` to `common`.
- Added in `makeAuthAPI`.
Previously, all database stuff was under the helpfully named
package 'storage'. However, this convention is used throughout all
of dendrite, which will clash the moment we want to add auth to all
the CS API endpoints. To prevent the package name clash, add
sub-directories which represent what is being stored so the final
usage ends up being:
```
func doThing(db *storage.SyncServerDatabase, authDB *accounts.Database)
{
// ...
}
```
* Start implementing the federation server keys API.
* Fix copyright
* Fix comments
* Comment on the key format
* Better explain what the ValidityPeriod is
* Return a 200 status code
* Rename "make" to "makeAPI" and factor out some more common code into it
Naming a function the same as a go builtin function seems like a bad
idea. Also move the call to `NewJSONRequestHander` inside the function
rather than calling it everywhere.
* Fix typo
- `s/Server/OutputRoomEvent/` in `consumers` to accurately reflect what is being consumed.
- `s/set/userIDSet/` in `notifier.go` for clarity.
- Removed lying comments.
The logic required to populate the right bits of `RoomData` tends towards
the complete `/sync` response struct, so just use the actual response struct
and save the hassle of mapping between the two. It may not make much difference
in its current form, but the next PR will make use of this.
This PR has no functional changes.
This is only 'mostly' correct currently, because what should be no-op dupe
joins will actually trigger the entire room state to be re-sent.
Bizarrely, it's significantly easier to just do that than work out if we should,
and there are no client-visible effects to doing so, so we just do it for now.
- Test data for the sync server is now in its own file.
- Rejig the sync server tests to support multiple /sync requests and corresponding
assertions.
- Fixed a minor bug which resulted in state events to appear twice in /sync
responses when syncing without a `since` parameter.
* Split out notifying /sync requests and calculating sync responses
The logic for notifying /sync requests is about to get really
complicated as we optimise when to wake up requests, so split
out that code into a separate struct to isolate it and make
it easier to unit test.
This has the following known shortcomings:
- It doesn't handle missing events not in the from/to range.
- It doesn't order events in the timeline correctly.
- It doesn't handle invited/left rooms at all.
- There is no intelligent wakeup logic: events for Bob will wake up Alice's stream.
Currently the HTTP handlers are attached to an API mux which is created inside
Setup(). In the future, we'll want to split this out so a single mux can have
both sync and clientapi handlers attached to it.
This distinguishes them from tools like 'kafka-producer'
which are not required to run a dendrite server.
The prefix also groups together the dendrite binaries nicely
on `ls` as well as making it nicer to tab-complete.
This involves:
- Parsing the HTTP request
- Requesting auth events from the roomserver via the Query API
- Building the event
- Doing auth checks on the event
- Sending it to the roomserver input log
Previously, the error responses:
- were not valid matrix errors (no `errcode`)
- returned the `err.Error()` message which may contain sensitive information.
- did not get logged (at all, let alone set the level correctly).
Now the error responses:
- return valid matrix errors (`M_UNKNOWN`)
- return a generic "Internal Server Error" string
- get logged at `ERROR` level.
* Return the requested portions of current state in the query API
* Use Unique from github.com/matrix-org/util
* rewrite bulkSelectFilteredStateBlockEntries to use append for clarity
* Add test for stateKeyTupleSorter
* Replace current with a new StateEntryList rather than individually setting the fields
* Add method for calculating the room state after a list of events
* Fill out AddsState and RemovesState fields in the output event
* comment on return value
* Fix whitespace
* Clarify the meaning of LastEventIDSent
* Be more explicit on what the snapshot is
* Fix comments
* Comment on why we want the state after the previous events
* Start implementing a query API for go using net/rpc
* Use a conventional JSON POST API rather than go net/rpc
net/rpc doesn't automatically handle reconnecting and we have better
logging and metrics infrastructure for monitoring HTTP apis.
* Implement the query API and add it to the integration tests
* Increase the timeout, travis seems to be a bit slow
* Clarify that state events are the things that are not returned if they are not requested
* Add utility function for converting arrays of numeric event IDs to pq Int64Arrays
* Warn people against requesting empty state keys by accident
* Add integration tests for roomserver to travis
* Try setting the dist trusty to see if that helps with postgres
* Pretty print the test input JSON
* Add comment as to why we are using trusty
* Marshal and Unmarshal methods for roomserver input api
* Comments for why we don't json.marshal the InputRoomEvent directly
* More comments
* s/m.room.create/the first event/
Terse function names are used to make usage not stutter. For example:
```go
err := error.Forbidden("you shall not pass")
```
At the moment they are all the same fundamental `MatrixError` type. This will
be bad if we ever want to `switch` based on the kind of error. I'm hoping we
won't ever need to introspect into errors like this: ideally these errors would
be created purely for immediately being returned in an HTTP response.
`MatrixError` implements the `error` interface.
* Move partition offsets sql to a separate file
* Move event types sql to a separate file
* Move event state key sql to a separate file
* Move room sql to a separate file
* Move events sql to a separate file
* Move event json sql to a separate file
* Move state snapshot sql to a separate file
* Move state block sql to a separate file
* A kafkaesque room event consumer for the roomserver.
Implement the main input loop for the roomserver.
It will receive events from a kafkaesque event source and track
where it is in the stream.
It currently does nothing with the events it consumes.
