mirrors/dendrite
1
0
Fork 0
mirror of https://github.com/matrix-org/dendrite.git synced 2025-12-08 23:43:11 -06:00
Code Issues Packages Projects Releases Wiki Activity
233 commits 57 branches 102 tags 160 MiB
Commit graph

233 commits

This branch
This branch
All branches
Author SHA1 Message Date
Robert Swain 227fe64252 mediaapi/writers: Pass used member instead of entire struct 2017-05-22 15:38:51 +02:00
Robert Swain 0ecb645f27 mediaapi/fileutils: Move logic from function to call site 
Makes for better readability. The only reason it was inside the function
was to avoid being too complex for gocyclo checks.
 2017-05-22 15:21:56 +02:00
Robert Swain 6d000794ec mediaapi/fileutils: Fix and simplify API doc 2017-05-22 15:16:53 +02:00
Robert Swain 703685f0dd mediaapi/storage: Correct comment about Database 2017-05-22 15:13:35 +02:00
Robert Swain 2f6db5d317 mediaapi/storage: StoreMediaMetadata error if origin and id not unique in db 2017-05-22 15:11:28 +02:00
Robert Swain 2004dfcfcf mediaapi: ContentLength -> FileSizeBytes 2017-05-22 15:07:45 +02:00
Robert Swain 1e639b0868 mediaapi/config: Document that MaxFileSizeBytes 0 means unlimited 2017-05-22 14:23:46 +02:00
Robert Swain 6fe6f45093 mediaapi/writers: Remove unnecessary comparison to true 2017-05-22 14:13:57 +02:00
Robert Swain 8684f80ebd mediaapi/fileutils: Move fileutils to its own package 2017-05-22 14:12:11 +02:00
Robert Swain 7bdcece102 mediaapi/writers/fileutils: Group error variables 2017-05-22 14:02:08 +02:00
Robert Swain 73b419a126 mediaapi/writers: Factor out common moveFileWithHashCheck functionality 2017-05-22 12:20:07 +02:00
Robert Swain 5d4b5ef1bc mediaapi/writers/download: Add Base64Hash to useful log messages 2017-05-22 12:19:06 +02:00
Robert Swain 01c565ddfb mediaapi/writers/fileutils: Store files based on hash, not media ID 
This avoids having to sanitize the origin and media ID for files from
remote servers. It also allows us to deduplicate files across all files
uploaded to this homeserver or downloaded from remote homeservers.
 2017-05-22 10:34:56 +02:00
Robert Swain 979ce964ab Merge branch 'master' into rob/media-upload 2017-05-22 10:28:37 +02:00
Robert Swain 8f7ce9adc0 mediaapi/writers/upload: Add note about Content-Disposition override 2017-05-22 10:27:48 +02:00
Robert Swain 3cea54db0b mediaapi/writers/download: Simplify user error message 
They already known the origin and media ID so it is redundant.
 2017-05-22 10:27:02 +02:00
Robert Swain 5f604cc41f mediaapi/writers/upload: Infof -> Info as no formatting in string 2017-05-22 10:26:30 +02:00
Robert Swain 5536fec902 mediaapi/writers: Add base64hash to media_repository table 
A SHA-256 hash sum in golang base64 URLEncoding format (contains only
URL-safe characters) is now calculated and stored for every file
transferred to this server.

Uploads to the server use this hash as the MediaID. Downloads from
remote servers retain their MediaID from the remote server, but can use
the hash for local deduplication and integrity checking purposes.
 2017-05-22 10:24:03 +02:00
Robert Swain 370cb74d2d mediaapi/writers: Reuse same writer code for upload and download 
This now calculates a hash for downloads from remote servers as well as
uploads to this server.
 2017-05-22 10:19:52 +02:00
Robert Swain 9af66a1963 mediaapi/writers: Reuse removeDir throughout the package 2017-05-22 10:13:37 +02:00
Mark Haines 6605333f6f Start implementing the federation server keys API. (#112) 
* Start implementing the federation server keys API.

* Fix copyright

* Fix comments

* Comment on the key format

* Better explain what the ValidityPeriod is

* Return a 200 status code
 2017-05-19 16:06:41 +01:00
Robert Swain 318531d011 mediaapi/writers/upload: Make assign in-line in if 2017-05-19 12:27:55 +02:00
Robert Swain 86cb8e32f7 mediaapi/writers/upload: Clarify order of moving file and storing metadata 2017-05-19 12:26:27 +02:00
Robert Swain 1242fdba22 mediaapi: Improve logging throughout, leveraging logrus features 2017-05-19 12:21:10 +02:00
Mark Haines aa179d451c Update version of gomatrixserverlib (#111) 2017-05-19 10:46:17 +01:00
Robert Swain 5d5f156500 mediaapi/writers/fileutils: Rework file path layout 
From experience with synapse, splitting the files into subdirectories
based on the beginnings of the filenames helps with browsability. As we
are using MediaIDs that are base64-encoded, each character has 64
possibilities, which is a nice upper limit on the number of
subdirectories in a directory in terms of browsing. We have two levels
of single character directories for added convenience, creating up to
4096 buckets.
 2017-05-19 11:34:40 +02:00
Kegsay 9d4d18ae7f Add AccountDatabase for storing user accounts (#110) 
Including the ability to add new accounts with a user/password and
select accounts using a user/password. Uses bcrypt to hash passwords.
 2017-05-19 10:27:03 +01:00
Robert Swain 12b0cdde06 mediaapi/writers/upload: Explain the use of TeeReader 2017-05-19 11:01:44 +02:00
Robert Swain f7d11f87c1 mediaapi/writers/upload: Add comment about why we hash the file data 2017-05-19 10:59:12 +02:00
Robert Swain 3e5ac85ce1 mediaapi/writers/upload: Clarify TODO comment 2017-05-19 10:53:47 +02:00
Robert Swain cdd4222e45 mediaapi/writers/fileutils: Return errors to log using request context 2017-05-19 10:46:50 +02:00
Robert Swain 5dd90fbff3 mediaapi/writers/fileutils: Make note of further file path validation todo 2017-05-18 18:00:56 +02:00
Robert Swain 7af45e4664 mediaapi/writers/upload: Refactor Upload() into three new functions 2017-05-18 17:56:19 +02:00
Robert Swain 00e8fed3a7 mediaapi/writers: Add validation and error handling to getPathFromMediaMetadata 2017-05-18 17:39:30 +02:00
Robert Swain 10a2b2f8e6 mediaapi: Also rename all basePath variables to absBasePath for clarity 2017-05-18 17:37:32 +02:00
Robert Swain 995e1f2c99 cmd/dendrite-media-api-server: Make base path absolute 2017-05-18 17:25:12 +02:00
Robert Swain f5422787a1 mediaapi/writers: Move single-value error return assignment into if 2017-05-18 16:02:43 +02:00
Robert Swain 9fc5abdb3f mediaapi/writers: Rename utils.go to fileutils.go 
Better reflects the content of the file.
 2017-05-18 16:01:06 +02:00
Robert Swain 2e795ed8aa mediaapi/storage: Improve GetMediaMetadata description 2017-05-18 15:57:07 +02:00
Robert Swain 1f2ac60bee mediaapi/routing: Sync make() to makeAPI() as in clientapi 2017-05-18 15:53:48 +02:00
Robert Swain 04c4a2d05a cmd/dendrite-media-api-server: Move os.Getenv() for consistency 2017-05-18 15:50:09 +02:00
Mark Haines 426a0365cf Rename "make" to "makeAPI" and factor out some more common code into it (#109) 
* Rename "make" to "makeAPI" and factor out some more common code into it

Naming a function the same as a go builtin function seems like a bad
idea. Also move the call to `NewJSONRequestHander` inside the function
rather than calling it everywhere.

* Fix typo
 2017-05-18 13:47:23 +01:00
Kegan Dougal cf736d746d hook: Make go vet run all tests and fix warnings 2017-05-18 12:27:11 +01:00
Robert Swain 1057e2e117 Merge branch 'master' into rob/media-upload 2017-05-18 12:47:41 +02:00
Robert Swain e4a97d13b3 Merge pull request #108 from matrix-org/rob/golang-1.8 
.travis.yml: Bump golang to 1.8
 2017-05-18 12:47:08 +02:00
Robert Swain 4df470eab5 .travis.yml: Bump golang to 1.8 2017-05-18 12:38:09 +02:00
Robert Swain ec0d584fe7 cmd/dendrite-media-api-server: Log format string with Infof not Info 2017-05-18 12:23:17 +02:00
Robert Swain 8085c1f863 mediaapi/types: Clarify what is ActiveRemoteRequests.Set's key 2017-05-18 12:19:03 +02:00
Robert Swain 28ef35d36a mediaapi/storage: Rework GetMediaMetadata API to return new MediaMetadata 2017-05-18 12:09:33 +02:00
Robert Swain 2fca4bbd65 mediaapi/config: Fix max_file_size_bytes YAML tag 2017-05-18 11:58:41 +02:00