mirrors/dendrite
1
0
Fork 0
mirror of https://github.com/matrix-org/dendrite.git synced 2025-12-08 15:33:09 -06:00
Code Issues Packages Projects Releases Wiki Activity
172 commits 57 branches 102 tags 160 MiB
Commit graph

172 commits

This branch
This branch
All branches
Author SHA1 Message Date
Robert Swain c5cd5a93b9 mediaapi: Use ServerName type from gomatrixserverlib 2017-05-18 11:57:44 +02:00
Robert Swain bd9db7557a mediaapi/README: Add link to spec section 2017-05-18 11:50:24 +02:00
Robert Swain 7727a8c61e cmd/dendrite-media-api-server: Add MAX_FILE_SIZE_BYTES configuration 2017-05-18 11:44:48 +02:00
Robert Swain 846aece163 mediaapi: MaxFileSize -> MaxFileSizeBytes 2017-05-18 11:36:26 +02:00
Robert Swain 35a0b5d2e9 cmd/dendrite-media-api-server: Add BASE_PATH configuration 2017-05-18 11:34:01 +02:00
Robert Swain ff3009ffdd cmd/dendrite-media-api-server: Add SERVER_NAME configuration 2017-05-18 11:32:30 +02:00
Robert Swain deee6f84c7 mediaapi/writers/upload: Move file first as db is source of truth 
The database is the source of truth. If we add the metadata to the
database and it succeeds, and then the file fails to be moved, we think
we have a file when we actually don't.
 2017-05-18 11:10:41 +02:00
Robert Swain 3f904e1cdb mediaapi/writer/upload: Remove unnecessary logic 2017-05-18 11:09:09 +02:00
Robert Swain f28235c05d mediaapi/writers/upload: Factor out removeDir 
Reduces complexity of Upload. Note that we never care about the error
from os.RemoveAll() beyond logging as we are already in an error case.
 2017-05-18 11:07:03 +02:00
Robert Swain 5348b64edc mediaapi/writers/download: Reduce complexity of copyToActiveAndPassive 2017-05-18 10:17:11 +02:00
Robert Swain b80d5ab919 cmd/mediaapi-integration-tests: Test downloading same file 100 times 
Spawns a GET request for the same file in 100 parallel go routines and
prints the body (which is some error JSON) in case of not 200 OK. Also
prints the number of successful requests.

This of course should take command line arguments for the URL and number
of requests but that can be done as soon as needed.
 2017-05-18 09:12:01 +02:00
Robert Swain 8cf507f85f mediaapi/writers: Never return server errors to user but log them 2017-05-18 09:04:36 +02:00
Robert Swain eddc3e0789 mediaapi/writers/download: Log warnings about why we return 404 2017-05-18 08:57:29 +02:00
Robert Swain c43aa20adc mediaapi/writers/download: Remove duplicate log message 2017-05-18 08:56:54 +02:00
Robert Swain a8b7130745 mediaapi/writers/download: Clean up copyToActiveAndPassive 2017-05-17 16:39:01 +02:00
Robert Swain 9f8458716d mediaapi/writers/download: Pass config members to respondFromRemoteFile 2017-05-17 16:37:11 +02:00
Robert Swain 19ffa4d75a mediaapi/writers/download: respondFromLocalFile only needs basePath 2017-05-17 16:30:01 +02:00
Robert Swain 13b1051a3e mediaapi/writers/download: Add logger to downloadRequest and use it 2017-05-17 16:24:30 +02:00
Robert Swain 6e24fb86cb mediaapi/writer/download: Make functions into methods and use MediaMetadata 2017-05-17 16:13:54 +02:00
Robert Swain 937162a722 mediaapi/routing: Pass pointer to config struct instead of copying 2017-05-17 13:29:35 +02:00
Robert Swain a821b9155f mediaapi/writers/utils: Reorder createFileWriter return parameters for consistency 2017-05-11 14:40:35 +02:00
Robert Swain 074e899000 mediaapi/writers/utils: Move and reuse createTempFileWriter 2017-05-11 14:39:40 +02:00
Robert Swain abcbb57aa1 mediaapi/writers/download: A bunch more refactoring 2017-05-11 14:31:24 +02:00
Robert Swain 6c3dbb9b48 mediaapi/writers/download: Factor out createRemoteRequest 2017-05-11 10:36:32 +02:00
Robert Swain ef7dc8c8c5 mediaapi/writers/download: Clarify log message that file is remote 2017-05-11 09:31:40 +02:00
Robert Swain 5e66ad6d39 mediaapi/writers/download: Comment the flow of respondFromRemoteFile 2017-05-11 09:30:10 +02:00
Robert Swain da28e4a3e2 mediaapi/writers/download: Update TODO comment that is done 2017-05-11 09:19:56 +02:00
Robert Swain 9c29a31e7e mediaapi/writers/download: Factor out respondFromRemoteFile 2017-05-11 09:19:34 +02:00
Robert Swain b6af021858 mediaapi/writers/download: Unlock activeRemoteRequests after too many tries 2017-05-11 09:18:02 +02:00
Robert Swain f0c6ec23c9 mediaapi/writers/download: Move log messages to correct code paths 2017-05-11 09:17:37 +02:00
Robert Swain 563330b82a mediaapi/writers/download: Rename attempts to tries 
Tries is more usual language in programming.
 2017-05-11 09:16:20 +02:00
Robert Swain 304a275e1b mediaapi/writers/download: Give remote files a placeholder UserID 2017-05-10 16:09:17 +02:00
Robert Swain df52b1aef2 mediaapi/writers/download: Only try to write as much as was read 2017-05-10 16:08:51 +02:00
Robert Swain 8f68f61117 mediaapi/writers/download: Remove done TODOs 2017-05-10 16:08:32 +02:00
Robert Swain 5d67787c46 mediaapi/writers/download: Use consistent temporary file name 2017-05-10 16:07:18 +02:00
Robert Swain 49ec095b59 mediaapi/writers/download: Try to only request remote files once 
If multiple requests arrive for the same remote file, we want to
download them once and then serve to all the remaining incoming requests
from the cache.

The main thing missing from the code at this point is a mechanism to
time out database queries. They are made across a network and so we
should be robust to network connectivity issues. This is a general
problem across dendrite and not limited to just this code.
 2017-05-10 16:04:39 +02:00
Robert Swain b280722591 mediaapi/writers/download: Factor out respondFromLocalFile() 2017-05-10 16:02:56 +02:00
Robert Swain 412f408ea0 mediaapi/writers/utils: Improve debug logging of moveFile() 2017-05-10 16:01:27 +02:00
Robert Swain da7feaf741 Merge branch 'master' into rob/media-upload 2017-05-10 16:00:08 +02:00
Robert Swain cdd2617f8e Merge pull request #97 from matrix-org/rob/util-logging-context-fix 
Update github.com/matrix-org/util for request context fix for logging
 2017-05-10 15:56:49 +02:00
Robert Swain 042d636e34 Update github.com/matrix-org/util for request context fix for logging 2017-05-10 15:43:58 +02:00
Kegsay 04f3c154b8 Split out notifying /sync requests and calculating sync responses (#96) 
* Split out notifying /sync requests and calculating sync responses

The logic for notifying /sync requests is about to get really
complicated as we optimise when to wake up requests, so split
out that code into a separate struct to isolate it and make
it easier to unit test.
 2017-05-10 10:42:00 +01:00
Robert Swain a405dccb4d mediaapi/routing/routing: Comment Content-Type override behaviour 2017-05-10 11:02:01 +02:00
Robert Swain c7c3a36e1b mediaapi/writers/download: Set more metadata fields for remote files 2017-05-09 20:58:44 +02:00
Robert Swain 5bb5a28366 cmd/dendrite-media-api-server: Bump maximum file size to 10MB 2017-05-09 20:57:59 +02:00
Robert Swain 619a77e187 mediaapi/writers/download: Improve logging 2017-05-09 20:57:34 +02:00
Robert Swain 391a1be69f mediaapi/writers/download: Fix infinite loop reading response body 2017-05-09 20:56:20 +02:00
Robert Swain 8f9eb13f69 mediaapi/writers/upload: Do not overwrite fields from database 
If the entry does not exist, this would set all but the origin and id to
nil.
 2017-05-09 20:10:59 +02:00
Robert Swain 911b5dc170 mediaapi/writers/upload: Generate Content-Disposition if not set 2017-05-09 20:10:27 +02:00
Robert Swain 9d9b23e8e4 Merge branch 'master' into rob/media-upload 2017-05-09 19:50:17 +02:00