Commit graph

82 commits

Author SHA1 Message Date
Kegan Dougal edd7feecf0 Write OutputRoomEvents into the database and remember the arrival order
The arrival order will eventually be used to generate streaming tokens to
return to the client as a sync token.
2017-03-27 14:42:20 +01:00
Kegan Dougal 12b19a712c sync-server config: Read from a YAML file rather than hard-coded variables 2017-03-27 13:20:19 +01:00
Kegan Dougal 7b5fcd1bdc Merge branch 'master' into kegan/syncserver-config 2017-03-27 13:06:25 +01:00
Kegan Dougal 7ab0436689 Add yaml dependency 2017-03-27 13:05:46 +01:00
Kegan Dougal 32ae63ee40 Log messages as they are consumed. Fix bug with remembering offsets 2017-03-27 11:58:16 +01:00
Kegan Dougal 02f1d353b9 Merge branch 'master' into kegan/sync-server-consumer 2017-03-27 10:20:31 +01:00
Kegsay a974b90ee9 Factor out Consumer from the roomserver (#50) 2017-03-27 10:00:28 +01:00
Kegan Dougal 66db5a9f55 Add SyncServer and remove needless abstractions 2017-03-24 17:38:04 +00:00
Kegan Dougal 81943b63f3 Glue up RoomserverConsumer to a database backend 2017-03-24 17:15:16 +00:00
Kegan Dougal d2ef1a6a4b Merge branch 'kegan/factor-consumer' into kegan/sync-server-consumer 2017-03-24 16:21:05 +00:00
Kegan Dougal 39f129918a Add a RoomserverConsumer 2017-03-24 16:20:39 +00:00
Kegan Dougal 5c1be7c043 Restore comments 2017-03-24 16:10:52 +00:00
Kegan Dougal 349b991178 Fix comments 2017-03-24 14:59:29 +00:00
Kegan Dougal fb20ec09da Make roomserver Consumer use the factored out Consumer 2017-03-24 11:55:38 +00:00
Kegan Dougal f04769efb3 Make roomserver use the factored out partition offsets table 2017-03-24 11:24:08 +00:00
Kegan Dougal 6094195b68 Factor out Consumer from the roomserver
This will be used in the sync server.
2017-03-24 11:13:54 +00:00
Mark Haines e3f3eb8f3d Add metrics for the various ways of calculating room state (#49) 2017-03-23 14:14:25 +00:00
Kegsay e347aa05fe Make a dendrite-sync-server binary (#48)
Currently the HTTP handlers are attached to an API mux which is created inside
Setup(). In the future, we'll want to split this out so a single mux can have
both sync and clientapi handlers attached to it.
2017-03-22 17:39:08 +00:00
Kegsay d6a03039dc Prefix 'dendrite-' onto dendrite binaries (#47)
This distinguishes them from tools like 'kafka-producer'
which are not required to run a dendrite server.

The prefix also groups together the dendrite binaries nicely
on `ls` as well as making it nicer to tab-complete.
2017-03-21 17:43:13 +00:00
Kegsay 83b9cfeb2b Move binaries to /cmd (#46) 2017-03-21 17:14:17 +00:00
Mark Haines a82ccf5b31 Add notes on what message sending will look like in dendrite (#45) 2017-03-20 16:46:07 +00:00
Kegsay 346f5d6334 Use gomatrixserverlib.StateKeyTuple and helper functions for auth (#44)
* Update gomatrixserverib dep

* Use helper functions when creating rooms

* Use gomatrixserverlib.StateKeyTuple
2017-03-17 16:28:15 +00:00
Mark Haines 44c63254a7 Tool for writing messages to a single partition in kafka. (#43) 2017-03-17 16:27:45 +00:00
Mark Haines 812fdbe22a gb vendor update github.com/matrix-org/gomatrixserverlib 2017-03-17 13:10:25 +00:00
Mark Haines ac66d07177 Add option to stop processing input after a given number of messages (#38)
* Add option to stop processing input after a given number of messages
2017-03-17 13:01:07 +00:00
Kegsay a7390953dd Implement /state event sending (#41) 2017-03-17 11:21:52 +00:00
Mark Haines f2163c31c6 Implement state conflict resolution (#37)
* Implement state conflict resolution
2017-03-15 13:47:12 +00:00
Kegsay 742e4aca85 Factor out writing events to the roomserver input log (#40) 2017-03-15 13:36:26 +00:00
Kegsay 6739f65752 Implement event sending part of /rooms/$roomid/send/$type/$txnid (#39)
This involves:
 - Parsing the HTTP request
 - Requesting auth events from the roomserver via the Query API
 - Building the event
 - Doing auth checks on the event
 - Sending it to the roomserver input log
2017-03-15 11:22:40 +00:00
Mark Haines aa664f0eb7 Tool for creating room events (#36)
* Tool for creating room events
2017-03-14 17:05:02 +00:00
Kegsay b9a4551075 Create README.md 2017-03-13 15:56:26 +00:00
Kegsay 414ea314a6 Update gomatrixserverlib and use AuthEventProvider (#35) 2017-03-10 17:54:17 +00:00
Kegsay 8ccff1e40f Log fatal errors at error level and return generic 500s (#34)
Previously, the error responses:
 - were not valid matrix errors (no `errcode`)
 - returned the `err.Error()` message which may contain sensitive information.
 - did not get logged (at all, let alone set the level correctly).

Now the error responses:
 - return valid matrix errors (`M_UNKNOWN`)
 - return a generic "Internal Server Error" string
 - get logged at `ERROR` level.
2017-03-10 16:50:41 +00:00
Kegsay 2fcf6fd6eb Send /createRoom events to kafka (#33) 2017-03-10 16:19:23 +00:00
Kegsay 49ed708ca4 Create the initial set of state events for room creation (#32) 2017-03-10 11:32:53 +00:00
Mark Haines e667f17e14 Include the requested current state alongside the latest events in the query API. (#30)
* Return the requested portions of current state in the query API

* Use Unique from github.com/matrix-org/util

* rewrite bulkSelectFilteredStateBlockEntries to use append for clarity

* Add test for stateKeyTupleSorter

* Replace current with a new StateEntryList rather than individually setting the fields
2017-03-09 15:07:18 +00:00
Kegsay e82090e277 Update gomatrixserverlib dep and add basic /createRoom validation (#31) 2017-03-09 11:47:06 +00:00
Mark Haines 1d18da1189 Move the functions for reading room state to a separate package. (#29)
This should:
    1) Make the input package a bit cleaner.
    2) Allow use to reuse the state reading code from the query package.
2017-03-08 15:10:26 +00:00
Mark Haines 84682b33c9 Use Unique from github.com/matrix-org/util (#28)
* Update github.com/matrix-org/util

* Use Unique from github.com/matrix-org/util
2017-03-08 13:27:21 +00:00
Kegsay 8ba9d4af04 Add /createRoom endpoint and unmarshal HTTP body (#27) 2017-03-07 16:11:08 +00:00
Kegsay 5552e1f3a8 Extract access tokens from HTTP requests (#15) 2017-03-07 13:43:32 +00:00
Mark Haines 8084beb6f7 Add helper method for preparing lists of statements. (#25)
* Add helper method for preparing lists of statements.
2017-03-07 10:37:41 +00:00
Mark Haines 96fc9294cc Track the current state of the room (#24)
* Add method for calculating the room state after a list of events

* Fill out AddsState and RemovesState fields in the output event

* comment on return value

* Fix whitespace

* Clarify the meaning of LastEventIDSent

* Be more explicit on what the snapshot is

* Fix comments

* Comment on why we want the state after the previous events
2017-03-07 10:25:01 +00:00
Mark Haines 9a8a8aedcb Add a query API to the roomserver for getting the latest events in a room. (#23)
* Start implementing a query API for go using net/rpc

* Use a conventional JSON POST API rather than go net/rpc

net/rpc doesn't automatically handle reconnecting and we have better
logging and metrics infrastructure for monitoring HTTP apis.

* Implement the query API and add it to the integration tests

* Increase the timeout, travis seems to be a bit slow

* Clarify that state events are the things that are not returned if they are not requested

* Add utility function for converting arrays of numeric event IDs to pq Int64Arrays

* Warn people against requesting empty state keys by accident
2017-03-06 14:29:39 +00:00
Mark Haines 37e0b6c4c6 Add integration tests for roomserver to travis (#21)
* Add integration tests for roomserver to travis

* Try setting the dist trusty to see if that helps with postgres

* Pretty print the test input JSON

* Add comment as to why we are using trusty
2017-03-01 14:55:27 +00:00
Tyler Gillies 276cc8cad6 Update README.md (#22)
Typo
2017-03-01 09:39:05 +00:00
Mark Haines 237b912cf5 Write to an output kafka log for new room events in the roomserver. (#20)
* Track which events have been sent and the last event sent for each room

* Write output events

* comment that an event could be sent twice
2017-02-27 11:25:35 +00:00
Kegsay 45d1e61a9d Update to use util.JSONResponse (#18) 2017-02-24 12:32:27 +00:00
Mark Haines a9b296c522 Get the SQL working, and make it more idempotent (#17)
* Get the SQL working, and make it more idempotent
2017-02-22 16:51:10 +00:00
Mark Haines c96d3eec0c Marshal and Unmarshal methods for roomserver input api (#16)
* Marshal and Unmarshal methods for roomserver input api

* Comments for why we don't json.marshal the InputRoomEvent directly

* More comments

* s/m.room.create/the first event/
2017-02-22 16:05:18 +00:00