Commit graph

180 commits

Author SHA1 Message Date
Kegsay 801b9246ce Add basic sync server integration test (#91) 2017-05-05 17:59:45 +01:00
Mark Haines a56f609b74 Update gomatrixserverlib (#90) 2017-05-05 17:43:42 +01:00
Mark Haines 0309035aad Update gomatrixserverlib (#89) 2017-05-05 16:19:48 +01:00
Robert Swain 9b7defd375 common/log: Always log to stderr 2017-04-21 02:11:24 +02:00
Robert Swain 27beb8a2b3 common/log: Re-colorize terminal log output 2017-04-21 02:06:06 +02:00
Robert Swain 9b7bf8cba8 common/log: Add license header 2017-04-21 01:45:18 +02:00
Robert Swain b33ed93dd6 Merge branch 'master' into rob/common-logging 2017-04-21 01:43:36 +02:00
Robert Swain d09a65abbe Add Apache Version 2.0 license and headers to all golang files 2017-04-21 00:40:52 +02:00
Robert Swain e281fc4530 syncserver: Rename to syncapi 2017-04-20 19:21:15 +02:00
Robert Swain d8b98cec44 client-api-proxy: roomserver-integration-tests: Fix renamed references 2017-04-20 19:21:15 +02:00
Robert Swain 0939ae3686 Rename dendrite-syncserver to dendrite-sync-api-server 2017-04-20 19:21:15 +02:00
Robert Swain 209ba3f6a3 Rename dendrite-roomserver to dendrite-room-server 2017-04-20 19:21:15 +02:00
Robert Swain 2278848922 Rename dendrite-clientapi to dendrite-client-api-server 2017-04-20 19:21:15 +02:00
Kegan Dougal 2a264db85e Don't tightloop if currentPos is 0 2017-04-20 18:01:45 +01:00
Kegsay cb5081b332 Yank out clientapi config options to env vars for now (#77) 2017-04-20 17:15:34 +01:00
Kegsay 2fadb9e409 Make it possible to point Riot at Dendrite (#74) 2017-04-20 17:11:53 +01:00
Mark Haines 43d1e94e75 Rename dendrite-sync-server to dendrite-syncserver for consistency (#76) 2017-04-20 16:54:22 +01:00
Robert Swain be47984c43 common/log: Always output timestamps as UTC 2017-04-20 17:16:16 +02:00
Robert Swain f65e26bc2a cmd: Configure logging before any log messages 2017-04-20 16:28:15 +02:00
Robert Swain 254e61f727 common/log: Switch to RFC3339 format with nanoseconds and trailing zeros 2017-04-20 16:28:15 +02:00
Robert Swain e8d2d61cc2 cmd: common/log: Always configure logging
When LOG_DIR or so is not specified, just configure the formatter.
2017-04-20 16:27:48 +02:00
Robert Swain 8010083026 dendrite/common: Move logrus configuration to common 2017-04-20 16:27:44 +02:00
Kegsay db428174d2 tool: Add client-api-proxy (#73) 2017-04-20 14:00:34 +01:00
Kegsay fad7e9541b /sync: Handle missing state events and return events in the correct order (#72) 2017-04-20 11:18:26 +01:00
Kegsay 92a0325266 Implement incremental sync (#71)
This has the following known shortcomings:
 - It doesn't handle missing events not in the from/to range.
 - It doesn't order events in the timeline correctly.
 - It doesn't handle invited/left rooms at all.
 - There is no intelligent wakeup logic: events for Bob will wake up Alice's stream.
2017-04-19 16:04:01 +01:00
Kegsay f18d935134 Factor out syncRequest parsing to its own file (#70)
Also ensure every sync response has a `next_batch` token by forcing
it in the constructor, as previously timeouts would not have a token.
2017-04-18 10:32:32 +01:00
Kegsay 53ec4a255b Calculate the complete /sync response (#69) 2017-04-13 16:56:46 +01:00
Kegsay 203e706b99 Move syncserver to its own directory (#66) 2017-04-12 16:06:26 +01:00
Kegsay da0fffcc19 Add /sync API response structs (#58) 2017-04-11 11:52:26 +01:00
Kegsay 9a266762b4 Return events based on 'since' value (#57) 2017-04-10 15:12:18 +01:00
Kegsay de9e3e5417 Add structs for HTTP long-polling (#56) 2017-04-07 14:32:42 +01:00
Kegsay c1c837516a Store current room state (#55) 2017-04-05 10:30:13 +01:00
Kegsay a3c66f7fa0 Write OutputRoomEvents into the database and remember the arrival order (#54) 2017-03-30 15:29:23 +01:00
Kegsay 2d2c7e7169 sync-server config: Read from a YAML file rather than hard-coded variables (#53) 2017-03-29 14:09:27 +01:00
Kegsay a423008987 Read roomserver output log and remember position across restarts (#52) 2017-03-29 14:05:43 +01:00
Kegsay a974b90ee9 Factor out Consumer from the roomserver (#50) 2017-03-27 10:00:28 +01:00
Mark Haines e3f3eb8f3d Add metrics for the various ways of calculating room state (#49) 2017-03-23 14:14:25 +00:00
Kegsay e347aa05fe Make a dendrite-sync-server binary (#48)
Currently the HTTP handlers are attached to an API mux which is created inside
Setup(). In the future, we'll want to split this out so a single mux can have
both sync and clientapi handlers attached to it.
2017-03-22 17:39:08 +00:00
Kegsay d6a03039dc Prefix 'dendrite-' onto dendrite binaries (#47)
This distinguishes them from tools like 'kafka-producer'
which are not required to run a dendrite server.

The prefix also groups together the dendrite binaries nicely
on `ls` as well as making it nicer to tab-complete.
2017-03-21 17:43:13 +00:00
Kegsay 83b9cfeb2b Move binaries to /cmd (#46) 2017-03-21 17:14:17 +00:00
Kegsay 346f5d6334 Use gomatrixserverlib.StateKeyTuple and helper functions for auth (#44)
* Update gomatrixserverib dep

* Use helper functions when creating rooms

* Use gomatrixserverlib.StateKeyTuple
2017-03-17 16:28:15 +00:00
Mark Haines 44c63254a7 Tool for writing messages to a single partition in kafka. (#43) 2017-03-17 16:27:45 +00:00
Mark Haines ac66d07177 Add option to stop processing input after a given number of messages (#38)
* Add option to stop processing input after a given number of messages
2017-03-17 13:01:07 +00:00
Kegsay a7390953dd Implement /state event sending (#41) 2017-03-17 11:21:52 +00:00
Mark Haines f2163c31c6 Implement state conflict resolution (#37)
* Implement state conflict resolution
2017-03-15 13:47:12 +00:00
Kegsay 742e4aca85 Factor out writing events to the roomserver input log (#40) 2017-03-15 13:36:26 +00:00
Kegsay 6739f65752 Implement event sending part of /rooms/$roomid/send/$type/$txnid (#39)
This involves:
 - Parsing the HTTP request
 - Requesting auth events from the roomserver via the Query API
 - Building the event
 - Doing auth checks on the event
 - Sending it to the roomserver input log
2017-03-15 11:22:40 +00:00
Mark Haines aa664f0eb7 Tool for creating room events (#36)
* Tool for creating room events
2017-03-14 17:05:02 +00:00
Kegsay b9a4551075 Create README.md 2017-03-13 15:56:26 +00:00
Kegsay 414ea314a6 Update gomatrixserverlib and use AuthEventProvider (#35) 2017-03-10 17:54:17 +00:00
Kegsay 8ccff1e40f Log fatal errors at error level and return generic 500s (#34)
Previously, the error responses:
 - were not valid matrix errors (no `errcode`)
 - returned the `err.Error()` message which may contain sensitive information.
 - did not get logged (at all, let alone set the level correctly).

Now the error responses:
 - return valid matrix errors (`M_UNKNOWN`)
 - return a generic "Internal Server Error" string
 - get logged at `ERROR` level.
2017-03-10 16:50:41 +00:00
Kegsay 2fcf6fd6eb Send /createRoom events to kafka (#33) 2017-03-10 16:19:23 +00:00
Kegsay 49ed708ca4 Create the initial set of state events for room creation (#32) 2017-03-10 11:32:53 +00:00
Mark Haines e667f17e14 Include the requested current state alongside the latest events in the query API. (#30)
* Return the requested portions of current state in the query API

* Use Unique from github.com/matrix-org/util

* rewrite bulkSelectFilteredStateBlockEntries to use append for clarity

* Add test for stateKeyTupleSorter

* Replace current with a new StateEntryList rather than individually setting the fields
2017-03-09 15:07:18 +00:00
Kegsay e82090e277 Update gomatrixserverlib dep and add basic /createRoom validation (#31) 2017-03-09 11:47:06 +00:00
Mark Haines 1d18da1189 Move the functions for reading room state to a separate package. (#29)
This should:
    1) Make the input package a bit cleaner.
    2) Allow use to reuse the state reading code from the query package.
2017-03-08 15:10:26 +00:00
Mark Haines 84682b33c9 Use Unique from github.com/matrix-org/util (#28)
* Update github.com/matrix-org/util

* Use Unique from github.com/matrix-org/util
2017-03-08 13:27:21 +00:00
Kegsay 8ba9d4af04 Add /createRoom endpoint and unmarshal HTTP body (#27) 2017-03-07 16:11:08 +00:00
Kegsay 5552e1f3a8 Extract access tokens from HTTP requests (#15) 2017-03-07 13:43:32 +00:00
Mark Haines 8084beb6f7 Add helper method for preparing lists of statements. (#25)
* Add helper method for preparing lists of statements.
2017-03-07 10:37:41 +00:00
Mark Haines 96fc9294cc Track the current state of the room (#24)
* Add method for calculating the room state after a list of events

* Fill out AddsState and RemovesState fields in the output event

* comment on return value

* Fix whitespace

* Clarify the meaning of LastEventIDSent

* Be more explicit on what the snapshot is

* Fix comments

* Comment on why we want the state after the previous events
2017-03-07 10:25:01 +00:00
Mark Haines 9a8a8aedcb Add a query API to the roomserver for getting the latest events in a room. (#23)
* Start implementing a query API for go using net/rpc

* Use a conventional JSON POST API rather than go net/rpc

net/rpc doesn't automatically handle reconnecting and we have better
logging and metrics infrastructure for monitoring HTTP apis.

* Implement the query API and add it to the integration tests

* Increase the timeout, travis seems to be a bit slow

* Clarify that state events are the things that are not returned if they are not requested

* Add utility function for converting arrays of numeric event IDs to pq Int64Arrays

* Warn people against requesting empty state keys by accident
2017-03-06 14:29:39 +00:00
Mark Haines 37e0b6c4c6 Add integration tests for roomserver to travis (#21)
* Add integration tests for roomserver to travis

* Try setting the dist trusty to see if that helps with postgres

* Pretty print the test input JSON

* Add comment as to why we are using trusty
2017-03-01 14:55:27 +00:00
Mark Haines 237b912cf5 Write to an output kafka log for new room events in the roomserver. (#20)
* Track which events have been sent and the last event sent for each room

* Write output events

* comment that an event could be sent twice
2017-02-27 11:25:35 +00:00
Kegsay 45d1e61a9d Update to use util.JSONResponse (#18) 2017-02-24 12:32:27 +00:00
Mark Haines a9b296c522 Get the SQL working, and make it more idempotent (#17)
* Get the SQL working, and make it more idempotent
2017-02-22 16:51:10 +00:00
Mark Haines c96d3eec0c Marshal and Unmarshal methods for roomserver input api (#16)
* Marshal and Unmarshal methods for roomserver input api

* Comments for why we don't json.marshal the InputRoomEvent directly

* More comments

* s/m.room.create/the first event/
2017-02-22 16:05:18 +00:00
Mark Haines 51f9741b34 Track which events are the latest events in the room. (#14)
* Track which events are the latest events in the room.
2017-02-21 14:50:30 +00:00
Kegsay 966d3b95b7 Add error package and some Matrix errors (#13)
Terse function names are used to make usage not stutter. For example:

```go
err := error.Forbidden("you shall not pass")
```

At the moment they are all the same fundamental `MatrixError` type. This will
be bad if we ever want to `switch` based on the kind of error. I'm hoping we
won't ever need to introspect into errors like this: ideally these errors would
be created purely for immediately being returned in an HTTP response.

`MatrixError` implements the `error` interface.
2017-02-20 17:20:49 +00:00
Kegsay 5c34caa1c1 Implement logging to file via dugong (#12) 2017-02-20 16:14:23 +00:00
Kegsay f1bb59d24a Use gorilla/mux to route HTTP requests (#11)
* Add basic routing based on matched paths
* Make /sync and /send use the right API paths
2017-02-20 15:41:29 +00:00
Mark Haines 450e0f1570 Move the SQL into separate files for each table. (#7)
* Move partition offsets sql to a separate file

* Move event types sql to a separate file

* Move event state key sql to a separate file

* Move room sql to a separate file

* Move events sql to a separate file

* Move event json sql to a separate file

* Move state snapshot sql to a separate file

* Move state block sql to a separate file
2017-02-15 14:43:19 +00:00
Mark Haines 39264cbf4b Calculate and store the state at each event (#6)
* Calculate and store the state at each event

* Use type aliases for numeric IDs
2017-02-15 11:05:45 +00:00
Mark Haines c72a44097f Move the auth checks to a separate file (#5) 2017-02-10 10:44:38 +00:00
Mark Haines fc4eb85379 Check that events pass authentication checks. (#4)
* Check that events pass authentication checks.

Record the list of events that the event passes authentication checks
against.
2017-02-09 16:48:14 +00:00
Mark Haines 600f56b4b8 Store outlier room events. (#3)
* Storage functions for event types
* Consume outlier room events
2017-02-07 17:20:05 +00:00
Kegan Dougal b04dfae9c5 Add comments 2017-02-03 16:32:55 +00:00
Kegan Dougal 225cce9c77 Prefix /api as per standards 2017-02-03 16:28:11 +00:00
Kegan Dougal aafaf6ede6 Add stub clientapi webserver with readers/writers packages
As per RL discussion with Mjark
2017-02-03 16:05:46 +00:00
Mark Haines 41c6a3737e A kafkaesque room event consumer for the roomserver. (#1)
* A kafkaesque room event consumer for the roomserver.

Implement the main input loop for the roomserver.

It will receive events from a kafkaesque event source and track
where it is in the stream.

It currently does nothing with the events it consumes.
2017-02-03 13:52:32 +00:00