Commit graph

355 commits

Author SHA1 Message Date
Andrew Morgan a4460dd6ee Set up proper registration flow (#354)
* Refactor registration to align with the spec
* We now keep track of sessions and their completed registration stages.
* We only complete registration if the client has completed a full flow.
* New Derived section in config for data derived from config options.
* New config options for captcha.
* Send params back to client for each registration stage.

Signed-off-by: Andrew Morgan (https://amorgan.xyz) <andrew@amorgan.xyz>
2017-11-29 09:43:03 +00:00
Erik Johnston b7a3b92a9f
Implement /make_join and /send_join (#316) 2017-11-29 09:38:56 +00:00
Richard van der Hoff fc92fdbe0a
Add OPTIONS handlers everywhere (#355)
Everything gets pre-flighted now, because of the Authorization header, so we
need to support OPTIONS everywhere.
2017-11-27 12:18:48 +00:00
Richard van der Hoff 0786318a04
Bump gomatrixserverlib (#353)
* Bump gomatrixserverlib

Mostly because I want to use Erik's go-faster jsoning.

* Update KeyDB for new KeyFetcher API

we now need to implement FetcherName.

* Attempt to fix integ tests

CanonicalJSON doesn't like the empty string, apparently, and anyway
canonicalising it is pointless.

* More integ test fix
2017-11-27 12:05:14 +00:00
Erik Johnston 9e352e7311
Implement query to get state and auth chain (#352)
* Implement query to get state and auth chain

* Add routing for queryStateAndAuthChain

* Comments

* Fix fetching wrong set of events

* Add tests

* Shuffle and comment
2017-11-27 10:20:00 +00:00
Erik Johnston 0d72e34acf
Fix panic if upload name wasn't supplied (#351) 2017-11-22 15:45:03 +00:00
Erik Johnston f42f44391f
Fix /sync when we have no events (#341)
* Fix /sync when we have no events

We used a since token of 0 to mean that no token was given. However, naffka
streams start at 0. This causes clients to get stuck spinning forever until an
event is sent.

This changes it so that we pass around pointers instead, with nil meaning a
since token wasn't given.

* Comment

* Fix unit tests

* Comments

* Fix typo
2017-11-22 09:51:12 +00:00
Ross Schulman ea53558cca Implement room_alias federation end point (#338)
* Add room alias query endpoint

* Try to fix indentation problems

* Fix linting errors and use of httpReq.FormValue

Signed-off-by: Ross Schulman <ross@rbs.io>

* Run gofmt

* Check for empty alias parameter and fix route URL

Signed-off-by: Ross Schulman <ross@rbs.io>

* Fix some linting errors

Signed-off-by: Ross Schulman <ross@rbs.io>

* Delete extra copy of directory route
2017-11-20 14:33:49 +00:00
Erik Johnston 19a716e7da
Fix create filter API (#342) 2017-11-18 15:06:51 +00:00
Michael Telatynski bb8dcb09a1 use voip turnServers struct from gomatrix for dedup and consistency (#344)
Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>
2017-11-17 10:27:28 +00:00
Erik Johnston 8599a36fa6
Use a Postgres database rather than Memory for Naffka (#337)
* Update naffka dep

* User Postgres database rather than Memory for Naffka
2017-11-16 17:35:28 +00:00
Erik Johnston bdc44c4bde Log errors when there is an error validating token (#340) 2017-11-16 15:46:52 +00:00
Erik Johnston 4b280943dc Fix SQL 2017-11-16 15:28:15 +00:00
Erik Johnston 561315e1d6
Add gofmt linter (#339)
* Check that gofmt has been run

* gofmt files

* Tabs and spaces
2017-11-16 10:12:02 +00:00
Richard van der Hoff 4124ce2ac0
Store keys rather than json in the keydatabase (#330)
* bump gomatrixserverlib

(changes to KeyFetcher and KeyDatabase interfaces)

* Store keys rather than json in the keydatabase

Rather than storing the raw JSON returned from a /keys/v1/query call in the
table, store the key itself.

This makes keydb.Database implement the updated KeyDatabase interface.
2017-11-15 17:46:16 +00:00
Erik Johnston 7f85422471
Move /state request handling out of RequestPool (#333)
We should probably move the handling out from the syncapi, but that
requires the clientapi to stream the current state which it currently
doesn't. This at least stops the sync and state handling being done in
one file.
2017-11-15 15:42:39 +00:00
Erik Johnston 139fae988e Fix linting errors in go1.9.1 (#336)
Fix linting errors in go1.9.1
2017-11-15 11:13:09 +00:00
Richard van der Hoff 8fff0e887c Update gometalinter config (#331)
* Update gometalinter config

gometalinter now uses `maligned` instead of `aligncheck`
(https://github.com/alecthomas/gometalinter/pull/367), so we need to update our
config accordingly.

* Update gometalinter

* Disable gotype linter

gotype does not seem to play nicely with the gb vendor directory. In
particular, it wants each of our dependencies to be built and installed (see
https://github.com/golang/go/issues/10969), but (empirically) it will not
accept them being installed in `pkg` but insists on them being in `vendor/pkg`.

This presents a problem because `gb build` builds the packages into `pkg`
(which doesn't seem entirely unreasonable since `.` comes before `vendor` in
`$GOPATH`). `go install github.com/x/y` does install in `vendor/pkg` but
requires us to know the name of each package.

The general conclusion of https://github.com/alecthomas/gometalinter/issues/91
seems to have been that the easiest thing to do is to disable `gotype` for now.

* Fix `unparam` lint

* Fix goshadow lint
2017-11-15 10:25:48 +00:00
Paul Tötterman bad701c703 Add device display names (#319) 2017-11-14 09:59:02 +00:00
Andrew Morgan 8720570bb0 Check for existing filter before inserting a new one (#318)
Signed-off-by: Andrew (anoa) <anoa@openmailbox.org>
2017-11-14 09:56:23 +00:00
Richard van der Hoff bf855a7e5c
Factor out keyring initialisation (#329)
Take the keyring init from the client-api-server and the monolith out to a
shared file
2017-11-13 18:39:09 +00:00
Michael Telatynski 90396b5620 implement voip/turnServer API endpoint (#323)
Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>
2017-11-09 09:58:45 +00:00
Andrew Morgan b7cfc2e057 Federation: Implement Query Profile API (#317)
* Federation: Implement Query Profile API

Implements the server portion of: `GET
/_matrix/federation/v1/query/profile?user_id=...&field=...`

Closes #278

Signed-off-by: Andrew (anoa) <anoa@openmailbox.org>

* Properly export profile-related structs and fix wording

Signed-off-by: Andrew (anoa) <anoa@openmailbox.org>

* Check provided user's domain matches our own.
2017-11-05 14:03:54 -04:00
Erik Johnston 49c040c89f Refactor Notifier to return channel (#311)
This has two benefits:

1. Using channels makes it easier to time out while waiting
2. Allows us to clean up goroutines that were waiting if we timeout the
   request
2017-10-26 11:34:54 +01:00
Erik Johnston e5944e0fdb Move BuildEvent to common package (#315)
This is in preperation for implementing various federation APIs that
need to build events.
2017-10-25 14:44:33 +01:00
Erik Johnston 3790a8da40 Move federationapi.readers package into routing (#314)
This seems to have been missed when we folded all the readers and
writers packages into routing.
2017-10-25 11:27:44 +01:00
Erik Johnston ce562d96f6 Don't add empty room entries to sync responses (#312) 2017-10-18 10:36:45 +01:00
Paul Tötterman 6d15aec8d3 Add /devices/ and /device/{deviceID} (#313)
Signed-off-by: Paul Tötterman <paul.totterman@iki.fi>
2017-10-17 19:12:54 +01:00
Erik Johnston 5a6a950ee8 Factor out addRoomDeltaToResponse from IncrementalSync (#306) 2017-10-17 17:13:41 +01:00
Erik Johnston bd07447abe Fix some edge cases with /sync (#302)
Including:
- Handle timeout=0 correctly
- Always return immediately on initial sync
- Handle spurious wake ups from the notifier
2017-10-16 13:34:08 +01:00
Erik Johnston 35b628f5bf Handle duplicate kafka messages (#301)
The way we store the partition offsets for kafka streams means that when
we start after a crash we may get the last message we processed again.
This means that we have to be careful to ensure that the processing
handles consecutive duplicates correctly.
2017-10-16 13:20:24 +01:00
Remi Reuvekamp 1a026f16d5 Implement /logout/all (#307)
Signed-off-by: Remi Reuvekamp <git@remireuvekamp.nl>
2017-10-15 11:29:47 +01:00
Erik Johnston 464fb5de1c Update vendor libraries and dendrite to s/Sirupsen/sirupsen/ (#304) 2017-10-13 14:42:57 +01:00
Erik Johnston d66244cb0a Merge readers/writers/routing packages (#295)
The HTTP handlers in the components are split into reader and writer directories. This was a fairly arbitrary distinction, and turns out to not be so helpful. Most read APIs have a corresponding write API, and it is more natural for them to be in the same file rather than in different directories.
2017-10-11 18:16:53 +01:00
Erik Johnston ba0d0672ea Rename kafka consumers to *Consumer (#294)
The prior naming was confusing, OutputRoomEvent consumed
OutputNewRoomEvents
2017-10-11 18:13:43 +01:00
Erik Johnston c0271c2462 Use returned ID from INSERT in create filter (#297) 2017-10-10 15:17:29 +01:00
Erik Johnston f6bda82366 Implement filter POSTing and GETting. (#296)
* Implement filter POSTing and GETting.

Signed-off-by: Jan Christian Grünhage <jan.christian@gruenhage.xyz>

* Add missing '}' typo introduced during merge

* Still trying to fix that merge...

* Fix linting
2017-10-10 14:28:49 +01:00
Erik Johnston 38999c54e1 Generate new devices for each new /login (#281) 2017-10-10 10:40:52 +01:00
Marcel 13107c6b2b Implement /register/available API (#291)
Signed-off-by: MTRNord <mtrnord1@gmail.com>
2017-10-09 15:24:38 +01:00
Erik Johnston 619fec6113 Add bug number 2017-10-06 14:13:53 +01:00
Erik Johnston f36269d9b1 Implement GET /rooms/{roomID}/state/{type}/{stateKey} (#276)
* Implement GET /rooms/{roomID}/state/{type}/{stateKey}

* Add comments
2017-10-06 11:23:58 +01:00
Mark Haines 847621bc5d Add config for setting up a jaeger opentracing reporter (#271)
* Add config for setting up a jaeger opentracing reporter

* Remove redundant comment
2017-09-28 17:00:23 +01:00
Mark Haines c4947c2ffb Add opentracing Spans to the HTTP APIs (#270)
* Add opentracing Spans to the HTTP APIs

* Add opentracing spans to the HTTP RPC clients

* Set the span in the request context

* More docstring
2017-09-28 14:50:40 +01:00
Erik Johnston f11af1e78f Implement setting alias during room create (#268)
* Implement setting alias during room create
2017-09-27 16:44:40 +01:00
Erik Johnston 00adbb38c6 Add POST /send/ api (#265) 2017-09-26 12:55:48 +01:00
Mark Haines 9ed609b9df Pass requests made by remote servers for media to the media API (#263) 2017-09-26 12:53:30 +01:00
Jan Christian Grünhage b2f6f89496 Implement version endpoint (#262) 2017-09-25 11:16:47 +01:00
Mark Haines 42f264119f Stub APIs for sytest (#261) 2017-09-22 17:28:29 +01:00
Erik Johnston 8a395b0262 Make login support logging in via user id (#260) 2017-09-22 17:08:16 +01:00
Erik Johnston bdfade26ed Don't validate username when fetching registration flows (#259) 2017-09-22 16:38:22 +01:00
Erik Johnston 6bec139544 Add missing GET for /login (#258) 2017-09-22 16:13:43 +01:00
Erik Johnston 8dabca0f07 Implement shared secret registration (#257)
* Implement shared secret registration

* Use HexString from gomatrixserverlib

* Correctly check username validility
2017-09-22 16:13:19 +01:00
Brendan Abolivier 0218063339 Implement room state retrieval endpoint (#254)
* Move prev event structure to component-wide types

* Renamed key for better understandability

* Implement /state endpoint

* Change rowsToEvents() to return an empty slice instead of nil in case of empty result set

* Doc

* Fix forgotten comma

* Specify HTTP method

* Update comment

* Remove debug fmt.Println
2017-09-22 11:34:54 +01:00
Mark Haines b9a933846b Allow users to join rooms they've been invited to (#253) 2017-09-21 17:00:48 +01:00
Mark Haines ce019738ff Pass a context when downloading remote media (#251) 2017-09-21 16:20:10 +01:00
Mark Haines fef290c47e Add context to the server key database (#248) 2017-09-21 16:16:02 +01:00
Mark Haines 7596c19f3a Add context to the mediaapi database (#250) 2017-09-21 15:44:00 +01:00
Mark Haines a7773d3d3d Add context to the partition offset table (#249) 2017-09-21 15:09:19 +01:00
Mark Haines 636848c350 Add invites to the sync API (#244)
* Add table for storing invites in the syncapi

* Use the invite table to list the active invites for a user

* Update the invites table from the roomserver stream

* Include the invites table when determining the maxInviteID
2017-09-20 15:36:41 +01:00
Erik Johnston 7a30f2085a Add goconst linter (#246) 2017-09-20 15:25:25 +01:00
Erik Johnston 340a84cdc0 Add staticcheck lint (#245) 2017-09-20 14:54:17 +01:00
Erik Johnston 584acbe9a9 Add 'unused' lint (#241)
* Add 'unused' lint

* Keep testdata consts
2017-09-20 14:47:29 +01:00
Erik Johnston cc2f755cb3 Add gosimple linter (#242) 2017-09-20 14:15:38 +01:00
Erik Johnston b72142ace5 Add vet linter (#240) 2017-09-20 13:40:22 +01:00
Erik Johnston 8a36a7e89b Add errcheck linter (#238)
* Add errcheck linter

* Add config for 'fast' linters

* Fix errcheck lint stuff

* Fix errcheck lint
2017-09-20 10:59:19 +01:00
Mark Haines b91b3e729a Use a shared PostgreSQL sequence to generate ids. (#237)
* Use a shared PostgreSQL sequence to generate ids.

Share an auto incrementing sequnce between the account data and
the room event table.
This means that account data updates can be received independantly of
room events updates.

This should give some basic support for fixing #212

* Remove redundant 'primary key'

* Re-number the SQL arguments

* Fewer lies in comments
2017-09-19 17:15:46 +01:00
Mark Haines fbc4477be0 Use a read-only snapshot transaction for calculating sync responses (#236)
* Use a read-only snapshot transaction for calculating sync responses

* gb vendor update github.com/lib/pq
2017-09-19 16:22:02 +01:00
Erik Johnston 08b9940dde Make media repo use error rather than jsonErrorResponse (#235)
* Make media repo use error rather than jsonErrorResponse

* Update comments

* gb vendor github.com/pkg/errors

* Fixup error formats
2017-09-19 11:40:21 +01:00
Mark Haines 856bc5b52e Add context to the syncapi database (#234) 2017-09-18 16:52:22 +01:00
Mark Haines 238646ee3c Add contexts to device database (#233)
* Add contexts to device database

* Remove spurious whitespace
2017-09-18 15:51:26 +01:00
Mark Haines e28ee27605 Add context to the account database (#232) 2017-09-18 14:15:27 +01:00
Mark Haines 5ada8872bb Add context to the federationsender database (#231) 2017-09-18 14:15:17 +01:00
Mark Haines dc5dd4c5d2 Add contexts to the public rooms database (#230) 2017-09-14 14:46:56 +01:00
Mark Haines bfcce5bd21 Add contexts to the roomserver storage layer (#229)
* Add contexts to the roomserver storage layer

* Fix rooms_table
2017-09-13 16:30:19 +01:00
Mark Haines 3133bef797 Add contexts to the internal roomserver APIs (#228) 2017-09-13 13:37:50 +01:00
Brendan Abolivier 79adba43f0 Append target profile to invite events created from a 3PID invite (#227)
* Append target profile to invite events created from a 3PID invite

* Don't redeclare err

* Add check on invited server
2017-09-13 13:31:46 +01:00
Brendan Abolivier 4fbafc35fe Specify HTTP methods for public APIs (#226)
* Specify HTTP methods for the client API

* Specify HTTP methods for the federation API

* Specify HTTP methods for the media API

* Specify HTTP methods for the sync API
2017-09-13 11:08:11 +01:00
Mark Haines 029e71828a Add context.Context to the federation client (#225)
* Add context.Context to the federation client

* gb vendor update github.com/matrix-org/gomatrixserverlib
2017-09-13 11:03:41 +01:00
Brendan Abolivier 086683459f Implement federation route PUT /exchange_third_party_invite (#224)
* Add comment

* gb vendor update github.com/matrix-org/gomatrixserverlib

* Add handler for the exchange_third_party_invite endpoint

* Doc

* Use SendEvents to send the invite to the roomserver

* Add missing error check

* Add checks
2017-09-12 17:15:13 +01:00
Mark Haines 82c82a3412 Explain when we should use SendEvents instead of SendInvite.
Because it's easy to get confused between the two methods.
2017-09-12 16:54:37 +01:00
Brendan Abolivier f1fce55697 Add config and checks for trusted ID servers (#206)
* Add config for trusted ID servers

* Add new error

* Implement check for trusted ID server

* Complete unfinished comment

* Make comment more explicit in the config file

* Use go standard errors in membership.go

* Use standard errors instead of JSON responses in threepid

* Doc errors

* Remove unused parameter
2017-09-11 19:18:19 +01:00
Brendan Abolivier 28346b39e8 3PID invite exchange over federation (#222)
* Use federation to auth the event if the server isn't in the room

* Use MakeAPI for 3pid onbind handler as it isn't a standard federation request

* Error check

* Temporarily disable tests

* Fix return on 3PID invite

* Re-enable tests

* Remove useless else

* gb vendor update github.com/matrix-org/gomatrixserverlib

* gb vendor update github.com/matrix-org/gomatrixserverlib

* Implement same behaviour as synapse

* Fix condition and array initialisation

* Log errors on iteration and throw one if no server could be reached

* Fix err not being initialised

* Fix lint

* Fix import path
2017-09-11 19:08:24 +01:00
Mark Haines 6cb9d900b9 Fix response to federation /invite to match the format expected by synapse (#221)
* Fix response to /invite to match the format expected by synapse

* gb vendor update github.com/matrix-org/gomatrixserverlib

* Use gomatrixserverlib.RespInvite

* gb vendor update github.com/matrix-org/gomatrixserverlib
2017-09-11 18:07:12 +01:00
Mark Haines 5740cb3e58 Redact the event before verifying its signature
... in the federation invite API.
2017-09-11 15:47:50 +01:00
Brendan Abolivier 4d1d503d43 Processing of pending invites on 3PID binding (#218)
* Add missing file headers

* Move the ID server's signatures verification to common

* Allow verification without specifying a server name

* Add third-party structs to membership events content

* Add processing of 3PID onbind requests

* Use reference for third party invite data

* Fix return arguments order

* Revert "Move the ID server's signatures verification to common"

This reverts commit 9344201031.

* Revert "Allow verification without specifying a server name"

This reverts commit fd27afbf82.

* Remove checks that are already occurring in gomatrixserverlib

* Change return type of createInviteFrom3PIDInvite

* Add doc, add checks in fillDisplayName

* Use MakeFedAPI

* Invert condition

* Use AuthEvents to retrieve the 3PID invite

* Update comment

* Remove unused parameter

* gb vendor update github.com/matrix-org/gomatrixserverlib
2017-09-08 15:17:12 +01:00
Erik Johnston fad997303b Turn on 'unparam' linting (#216)
* Turn on 'unparam' linting

* Increase deadline

* Fix mediaapi integeration test
2017-09-07 12:50:39 +01:00
Erik Johnston ea80ffb3af Add get event federation API (#209)
* Add get event federation API

* Handle err correctly

* Add missing readers

* Ignore gocyclo lint for routing function

* Add todo

* Fix styling a bit
2017-09-06 12:38:22 +01:00
Erik Johnston 2dcb3a11a5 Use gometalinter (#210)
* Remove unused struct field

* Ignore unused test data

* Remove unused variables

* Remove deadcode

* Fix up vetshadow warnings

* Convert to using gometalinter

* Update travis

* Use vendored versions of gometalinter

* Make gometalinter install its stuff

* Vendor misspell
2017-09-05 17:40:46 +01:00
Mark Haines 643d05b157 Add function for making authed federation APIs (#208) 2017-09-04 13:14:01 +01:00
Brendan Abolivier 960af3d628 Implement client API routes for 3PID handling (#205)
* Create package for handling 3pid processes and move invite processing there

* Add database table and functions for tracking 3PIDs

* Add structures and functions to interact with an ID server

* Add handlers for 3PIDs management

* Fix 3PIDs retrieval sending null if no 3PID known for a user

* Include medium in database requests and function calls

* Publish an association if it has been validated and requested

* Add TODO markers for tursted ID server check

* Use a structure instead of a map to represent a 3PID
2017-09-01 10:13:10 +01:00
Richard van der Hoff 8c2e6273e3 Add command to generate test keys/certs (#204)
Add `generate-keys` command, which can be used to generate TLS keys/certs, and,
more usefully, matrix signing keys.
2017-08-31 12:28:58 +01:00
Richard van der Hoff fc51f72bf9 Avoid swallowing error from kafka (#203) 2017-08-31 12:27:53 +01:00
Brendan Abolivier 2183712e04 Process requests for 3PID invites (#201)
* Structure for 3PID invite

* Generate invite from 3PID known by ID server

* Load user profile in a separate function

* Generate m.room.third_party_invite if the ID server doesn't know the 3PID

* Fix URLs to the spec in comments

* Move third-party invites to a separate package and doc' it

* Handle non-OK status codes on lookup

* Send display name to identity server when asking to store an invite

* Remove join response structure

* Change the way some variables are declared or passed as argument

* Use gomatrixserverlib.Base64String instead of the builtin base64 package

* Don't copy the public keys array
2017-08-29 15:17:26 +01:00
Brendan Abolivier 685e056ab3 Membership viewing API when user left the room (#194)
* Implement case where user left the room

* Filter by membership event

* Move the logic from the storage to the query API

* Fix check on state entries iteration

* Remove aliases methods from query API

* Use structure for response to match with the spec

* Remove filtering on /members and implement /joined_members
2017-08-24 16:00:14 +01:00
Mark Haines 9c954501a2 Implement /federation/v1/invite/{roomID}/{eventID} (#197)
* Implement /federation/v1/invite/{roomID}/{eventID}

* Use NotJSON instead of BadJSON when the JSON couldn't be decoded
2017-08-23 15:13:47 +01:00
Mark Haines 17c60759c4 Add query API for listing active invites (#196)
* Add query API for listing active invites

This lists the invites for a user in a room that could be used to
join the room over federation.

* s/Lookup/Look up/

* Fix implements comments
2017-08-23 15:08:48 +01:00
Brendan Abolivier 166ac9d092 Fix sync not returning on room join (#195)
* Use BuildEvent method on room join

* Fix building the list of room members in the sync notifier

* Fix building the list of room members in the sync notifier

* Rephrase comment
2017-08-22 14:14:37 +01:00
Brendan Abolivier b15ce900ab Implement public rooms APIs (#185)
* Move events contents to common

* Basic database structure

* Complete database update

* Support visibility update and retrieval

* Add HTTP methods for visibility update and retrieval

* Add the database for the new component

* Add a listener for the new component

* Fix attribute update statements

* Create public rooms component

* Fix failing test

* Add roomserver consumer

* Fix a bug in aliases creation

* Add a check on type

* Implement public rooms directory

* Use auth API for visibility update

* Support filtering

* Add component to monolith

* Various fixes

* Fix computation of next public rooms batch

* Retrieve state events from the roomserver query API + avoid dupes on join

* Split update of string or boolean attribute in two separate functions

* Use event type to detect duplicate joins

* Improve the joined members counter computation

* Use event.RoomID()
2017-08-22 11:12:51 +01:00
Mark Haines 808c2e09f6 Make txn *sql.Tx arguments optional everywhere using a utility function (#191)
* Make txn *sql.Tx arguments optional everywhere using a utility function

* Clarify that if the txn is nil the stmt will run outside a transaction
2017-08-21 17:20:23 +01:00
Mark Haines 57b7097368 Add input API for adding invites to the roomserver. (#187)
* Add input API for adding invites to the roomserver.

This API handles invites received over federation that occur outside of a room.

* Add some docstring for withTransaction

* Use a nicer pattern for wrapping transactions

* Fix MembershipUpdater method to not commit the transaction before returning it

* Use the Transaction interface from common
2017-08-21 16:37:11 +01:00