Commit graph

137 commits

Author SHA1 Message Date
Robert Swain 49ec095b59 mediaapi/writers/download: Try to only request remote files once
If multiple requests arrive for the same remote file, we want to
download them once and then serve to all the remaining incoming requests
from the cache.

The main thing missing from the code at this point is a mechanism to
time out database queries. They are made across a network and so we
should be robust to network connectivity issues. This is a general
problem across dendrite and not limited to just this code.
2017-05-10 16:04:39 +02:00
Robert Swain b280722591 mediaapi/writers/download: Factor out respondFromLocalFile() 2017-05-10 16:02:56 +02:00
Robert Swain 412f408ea0 mediaapi/writers/utils: Improve debug logging of moveFile() 2017-05-10 16:01:27 +02:00
Robert Swain da7feaf741 Merge branch 'master' into rob/media-upload 2017-05-10 16:00:08 +02:00
Robert Swain cdd2617f8e Merge pull request #97 from matrix-org/rob/util-logging-context-fix
Update github.com/matrix-org/util for request context fix for logging
2017-05-10 15:56:49 +02:00
Robert Swain 042d636e34 Update github.com/matrix-org/util for request context fix for logging 2017-05-10 15:43:58 +02:00
Kegsay 04f3c154b8 Split out notifying /sync requests and calculating sync responses (#96)
* Split out notifying /sync requests and calculating sync responses

The logic for notifying /sync requests is about to get really
complicated as we optimise when to wake up requests, so split
out that code into a separate struct to isolate it and make
it easier to unit test.
2017-05-10 10:42:00 +01:00
Robert Swain a405dccb4d mediaapi/routing/routing: Comment Content-Type override behaviour 2017-05-10 11:02:01 +02:00
Robert Swain c7c3a36e1b mediaapi/writers/download: Set more metadata fields for remote files 2017-05-09 20:58:44 +02:00
Robert Swain 5bb5a28366 cmd/dendrite-media-api-server: Bump maximum file size to 10MB 2017-05-09 20:57:59 +02:00
Robert Swain 619a77e187 mediaapi/writers/download: Improve logging 2017-05-09 20:57:34 +02:00
Robert Swain 391a1be69f mediaapi/writers/download: Fix infinite loop reading response body 2017-05-09 20:56:20 +02:00
Robert Swain 8f9eb13f69 mediaapi/writers/upload: Do not overwrite fields from database
If the entry does not exist, this would set all but the origin and id to
nil.
2017-05-09 20:10:59 +02:00
Robert Swain 911b5dc170 mediaapi/writers/upload: Generate Content-Disposition if not set 2017-05-09 20:10:27 +02:00
Robert Swain 9d9b23e8e4 Merge branch 'master' into rob/media-upload 2017-05-09 19:50:17 +02:00
Robert Swain 7cf34af30b WIP: Refactoring 2017-05-09 19:49:39 +02:00
Kegsay e226d564ec Fix a race with sync server integration tests (#95) 2017-05-09 15:58:31 +01:00
Kegsay 42564e8ed6 Factor out creating/deleting/writing to kafka topics (#94) 2017-05-09 09:05:05 +01:00
Kegsay 801b9246ce Add basic sync server integration test (#91) 2017-05-05 17:59:45 +01:00
Mark Haines a56f609b74 Update gomatrixserverlib (#90) 2017-05-05 17:43:42 +01:00
Mark Haines 0309035aad Update gomatrixserverlib (#89) 2017-05-05 16:19:48 +01:00
Robert Swain a24b3e7810 mediaapi/writers/download: Obtaining a file from a remote server is supported 2017-04-28 17:15:26 +02:00
Robert Swain 52b8b9b8fe mediaapi/writers/download: Remove cruft 2017-04-28 17:15:00 +02:00
Robert Swain 90bac42edb mediaapi/writers/download: Only accept GET method 2017-04-28 17:14:16 +02:00
Robert Swain 10e843da58 mediaapi/routing: Remove OPTIONS handling from GET endpoint 2017-04-28 17:13:36 +02:00
Robert Swain 42a390f8fe mediaapi/routing: Make use of refactored request logging in util package 2017-04-28 16:24:30 +02:00
Robert Swain 5d4432218b vendor: Update github.com/matrix-org/util 2017-04-28 16:24:07 +02:00
Robert Swain f2437be52b mediaapi/routing: Correct comment on origin of duplicated code 2017-04-28 14:45:18 +02:00
Robert Swain 0a24e406af mediaapi/routing: Rename Fudge to downloadRequestHandler 2017-04-28 14:44:54 +02:00
Robert Swain 81706408bd mediaapi: Hack in /download from gotest code 2017-04-27 17:40:57 +02:00
Robert Swain c1e5974872 mediaapi/storage: Remove DEFAULT from content_disposition in schema
It isn't very effective at all as we will anyway write an empty string.
We can handle what to do about that in the query code paths.
2017-04-26 12:19:47 +02:00
Robert Swain d9ee22d043 mediaapi: Hack in SQL db storage and Erik's gotest file upload code
After this, upload in a usual case now works but the code surely needs
cleanup.
2017-04-26 12:11:22 +02:00
Robert Swain 4d1bff2f61 mediaapi: Initial commit for /upload HTTP infra 2017-04-24 18:31:44 +02:00
Robert Swain a1ce351d36 Merge pull request #75 from matrix-org/rob/common-logging
Improve logging configuration
2017-04-21 17:48:17 +02:00
Robert Swain 9b7defd375 common/log: Always log to stderr 2017-04-21 02:11:24 +02:00
Robert Swain 27beb8a2b3 common/log: Re-colorize terminal log output 2017-04-21 02:06:06 +02:00
Robert Swain 9b7bf8cba8 common/log: Add license header 2017-04-21 01:45:18 +02:00
Robert Swain b33ed93dd6 Merge branch 'master' into rob/common-logging 2017-04-21 01:43:36 +02:00
Matthew Hodgson 0391a790a6 Merge pull request #80 from matrix-org/rob/license-headers
Add Apache Version 2.0 license and headers to all golang files
2017-04-20 23:54:27 +01:00
Robert Swain d09a65abbe Add Apache Version 2.0 license and headers to all golang files 2017-04-21 00:40:52 +02:00
Robert Swain d03126c546 Merge pull request #79 from matrix-org/rob/consistent-naming
Improve naming consistency
2017-04-21 00:20:27 +02:00
Robert Swain 19c9a0c7a5 travis-test.sh: Fix import path of room server 2017-04-20 19:21:15 +02:00
Robert Swain e281fc4530 syncserver: Rename to syncapi 2017-04-20 19:21:15 +02:00
Robert Swain d8b98cec44 client-api-proxy: roomserver-integration-tests: Fix renamed references 2017-04-20 19:21:15 +02:00
Robert Swain 0939ae3686 Rename dendrite-syncserver to dendrite-sync-api-server 2017-04-20 19:21:15 +02:00
Robert Swain 209ba3f6a3 Rename dendrite-roomserver to dendrite-room-server 2017-04-20 19:21:15 +02:00
Robert Swain 2278848922 Rename dendrite-clientapi to dendrite-client-api-server 2017-04-20 19:21:15 +02:00
Kegan Dougal 2a264db85e Don't tightloop if currentPos is 0 2017-04-20 18:01:45 +01:00
Kegsay cb5081b332 Yank out clientapi config options to env vars for now (#77) 2017-04-20 17:15:34 +01:00
Kegsay 2fadb9e409 Make it possible to point Riot at Dendrite (#74) 2017-04-20 17:11:53 +01:00