mirrors/dendrite
1
0
Fork 0
mirror of https://github.com/matrix-org/dendrite.git synced 2025-12-08 07:23:10 -06:00
Code Issues Packages Projects Releases Wiki Activity
205 commits 57 branches 102 tags 160 MiB
Commit graph

205 commits

This branch
This branch
All branches
Author SHA1 Message Date
Robert Swain 5d5f156500 mediaapi/writers/fileutils: Rework file path layout 
From experience with synapse, splitting the files into subdirectories
based on the beginnings of the filenames helps with browsability. As we
are using MediaIDs that are base64-encoded, each character has 64
possibilities, which is a nice upper limit on the number of
subdirectories in a directory in terms of browsing. We have two levels
of single character directories for added convenience, creating up to
4096 buckets.
 2017-05-19 11:34:40 +02:00
Robert Swain 12b0cdde06 mediaapi/writers/upload: Explain the use of TeeReader 2017-05-19 11:01:44 +02:00
Robert Swain f7d11f87c1 mediaapi/writers/upload: Add comment about why we hash the file data 2017-05-19 10:59:12 +02:00
Robert Swain 3e5ac85ce1 mediaapi/writers/upload: Clarify TODO comment 2017-05-19 10:53:47 +02:00
Robert Swain cdd4222e45 mediaapi/writers/fileutils: Return errors to log using request context 2017-05-19 10:46:50 +02:00
Robert Swain 5dd90fbff3 mediaapi/writers/fileutils: Make note of further file path validation todo 2017-05-18 18:00:56 +02:00
Robert Swain 7af45e4664 mediaapi/writers/upload: Refactor Upload() into three new functions 2017-05-18 17:56:19 +02:00
Robert Swain 00e8fed3a7 mediaapi/writers: Add validation and error handling to getPathFromMediaMetadata 2017-05-18 17:39:30 +02:00
Robert Swain 10a2b2f8e6 mediaapi: Also rename all basePath variables to absBasePath for clarity 2017-05-18 17:37:32 +02:00
Robert Swain 995e1f2c99 cmd/dendrite-media-api-server: Make base path absolute 2017-05-18 17:25:12 +02:00
Robert Swain f5422787a1 mediaapi/writers: Move single-value error return assignment into if 2017-05-18 16:02:43 +02:00
Robert Swain 9fc5abdb3f mediaapi/writers: Rename utils.go to fileutils.go 
Better reflects the content of the file.
 2017-05-18 16:01:06 +02:00
Robert Swain 2e795ed8aa mediaapi/storage: Improve GetMediaMetadata description 2017-05-18 15:57:07 +02:00
Robert Swain 1f2ac60bee mediaapi/routing: Sync make() to makeAPI() as in clientapi 2017-05-18 15:53:48 +02:00
Robert Swain 04c4a2d05a cmd/dendrite-media-api-server: Move os.Getenv() for consistency 2017-05-18 15:50:09 +02:00
Robert Swain 1057e2e117 Merge branch 'master' into rob/media-upload 2017-05-18 12:47:41 +02:00
Robert Swain e4a97d13b3 Merge pull request #108 from matrix-org/rob/golang-1.8 
.travis.yml: Bump golang to 1.8
 2017-05-18 12:47:08 +02:00
Robert Swain 4df470eab5 .travis.yml: Bump golang to 1.8 2017-05-18 12:38:09 +02:00
Robert Swain ec0d584fe7 cmd/dendrite-media-api-server: Log format string with Infof not Info 2017-05-18 12:23:17 +02:00
Robert Swain 8085c1f863 mediaapi/types: Clarify what is ActiveRemoteRequests.Set's key 2017-05-18 12:19:03 +02:00
Robert Swain 28ef35d36a mediaapi/storage: Rework GetMediaMetadata API to return new MediaMetadata 2017-05-18 12:09:33 +02:00
Robert Swain 2fca4bbd65 mediaapi/config: Fix max_file_size_bytes YAML tag 2017-05-18 11:58:41 +02:00
Robert Swain c5cd5a93b9 mediaapi: Use ServerName type from gomatrixserverlib 2017-05-18 11:57:44 +02:00
Robert Swain bd9db7557a mediaapi/README: Add link to spec section 2017-05-18 11:50:24 +02:00
Robert Swain 7727a8c61e cmd/dendrite-media-api-server: Add MAX_FILE_SIZE_BYTES configuration 2017-05-18 11:44:48 +02:00
Robert Swain 846aece163 mediaapi: MaxFileSize -> MaxFileSizeBytes 2017-05-18 11:36:26 +02:00
Robert Swain 35a0b5d2e9 cmd/dendrite-media-api-server: Add BASE_PATH configuration 2017-05-18 11:34:01 +02:00
Robert Swain ff3009ffdd cmd/dendrite-media-api-server: Add SERVER_NAME configuration 2017-05-18 11:32:30 +02:00
Robert Swain deee6f84c7 mediaapi/writers/upload: Move file first as db is source of truth 
The database is the source of truth. If we add the metadata to the
database and it succeeds, and then the file fails to be moved, we think
we have a file when we actually don't.
 2017-05-18 11:10:41 +02:00
Robert Swain 3f904e1cdb mediaapi/writer/upload: Remove unnecessary logic 2017-05-18 11:09:09 +02:00
Robert Swain f28235c05d mediaapi/writers/upload: Factor out removeDir 
Reduces complexity of Upload. Note that we never care about the error
from os.RemoveAll() beyond logging as we are already in an error case.
 2017-05-18 11:07:03 +02:00
Robert Swain 5348b64edc mediaapi/writers/download: Reduce complexity of copyToActiveAndPassive 2017-05-18 10:17:11 +02:00
Robert Swain b80d5ab919 cmd/mediaapi-integration-tests: Test downloading same file 100 times 
Spawns a GET request for the same file in 100 parallel go routines and
prints the body (which is some error JSON) in case of not 200 OK. Also
prints the number of successful requests.

This of course should take command line arguments for the URL and number
of requests but that can be done as soon as needed.
 2017-05-18 09:12:01 +02:00
Robert Swain 8cf507f85f mediaapi/writers: Never return server errors to user but log them 2017-05-18 09:04:36 +02:00
Robert Swain eddc3e0789 mediaapi/writers/download: Log warnings about why we return 404 2017-05-18 08:57:29 +02:00
Robert Swain c43aa20adc mediaapi/writers/download: Remove duplicate log message 2017-05-18 08:56:54 +02:00
Kegsay 917c433fd2 Minor refactoring (#106) 
- `s/Server/OutputRoomEvent/` in `consumers` to accurately reflect what is being consumed.
- `s/set/userIDSet/` in `notifier.go` for clarity.
- Removed lying comments.
 2017-05-17 17:29:26 +01:00
Kegsay c8c5a40ebd Update README.md 2017-05-17 16:37:58 +01:00
Kegsay 0457a3bb69 State of the union update 2017-05-17 16:36:17 +01:00
Kegsay ccd0eb2851 /sync bugfix: Check transitions to 'leave' do not leak events afterwards (#105) 2017-05-17 16:21:27 +01:00
Robert Swain a8b7130745 mediaapi/writers/download: Clean up copyToActiveAndPassive 2017-05-17 16:39:01 +02:00
Kegsay d5a44fd3e8 Only wake up /sync requests which the event is for (#101) 2017-05-17 15:38:24 +01:00
Robert Swain 9f8458716d mediaapi/writers/download: Pass config members to respondFromRemoteFile 2017-05-17 16:37:11 +02:00
Robert Swain 19ffa4d75a mediaapi/writers/download: respondFromLocalFile only needs basePath 2017-05-17 16:30:01 +02:00
Robert Swain 13b1051a3e mediaapi/writers/download: Add logger to downloadRequest and use it 2017-05-17 16:24:30 +02:00
Robert Swain 6e24fb86cb mediaapi/writer/download: Make functions into methods and use MediaMetadata 2017-05-17 16:13:54 +02:00
Robert Swain 937162a722 mediaapi/routing: Pass pointer to config struct instead of copying 2017-05-17 13:29:35 +02:00
Kegsay 0a3d44a80a Handle room transitions to 'leave' in incremental /sync requests (#104) 
* Add test for transition to leave

* Add test for join/leave in the same /sync response
 2017-05-17 10:25:59 +01:00
Kegsay 675759c192 Add invites to /sync responses (#103) 2017-05-15 17:41:54 +01:00
Kegsay 94e1c62745 Remove intermediary /sync response struct (#102) 
The logic required to populate the right bits of `RoomData` tends towards
the complete `/sync` response struct, so just use the actual response struct
and save the hassle of mapping between the two. It may not make much difference
in its current form, but the next PR will make use of this.

This PR has no functional changes.
 2017-05-15 15:18:08 +01:00