mirrors/dendrite
1
0
Fork 0
mirror of https://github.com/matrix-org/dendrite.git synced 2024-11-23 06:41:56 -06:00
Code Issues Packages Projects Releases Wiki Activity
2078 commits 57 branches 102 tags 160 MiB
Commit graph

76 commits

Author SHA1 Message Date
kegsay e0cdf64c33
Merge branch 'main' into s7evink/consent-tracking 2022-05-03 17:31:36 +01:00
Neil Alexander dd061a172e
Tidy up AddPublicRoutes (#2412) 
* Simplify federation API `AddPublicRoutes`

* Simplify client API `AddPublicRoutes`

* Simplify media API `AddPublicRoutes`

* Simplify sync API `AddPublicRoutes`

* Simplify `AddAllPublicRoutes`
 2022-05-03 17:17:02 +01:00
Neil Alexander 4ad5f9c982
Global database connection pool (for monolith mode) (#2411) 
* Allow monolith components to share a single database pool

* Don't yell about missing connection strings

* Rename field

* Setup tweaks

* Fix panic

* Improve configuration checks

* Update config

* Fix lint errors

* Update comments
 2022-05-03 16:35:06 +01:00
Till Faelligen ef62255685 Merge branch 'main' of github.com:matrix-org/dendrite into s7evink/consent-tracking 2022-05-03 09:54:25 +02:00
Till 979a551f1e
Return null if MaxFileSizeBytes is 0 (#2409) 
* Return "null" if MaxFileSizeBytes is 0

* Add comment and nil check (better save than sorry)

* Simplify config
 2022-05-02 10:47:16 +02:00
Brian Meek d28d0ee66e
Fix TestThumbnailsStorage failing when media results come back in non-deterministic order; silence expected 
error when tests are run multiple times against the same postgres database (cherry-picked from #2395)

Signed-off-by: Brian Meek <brian@hntlabs.com>
 2022-04-29 09:32:58 +01:00
Till Faelligen 2b496be2c3 Merge branch 'main' of github.com:matrix-org/dendrite into s7evink/consent-tracking 2022-04-20 17:26:27 +02:00
Till 3ddbffd59e
Refactor media storage layer, add tests (#2352) 
* Refactor mediaapi storage layer

* Verify filetype before trying to create thumbnails

* Add media api storage tests

* Fix returned values
 2022-04-14 13:32:48 +01:00
S7evinK a0f5d8e1a5
Fix unlimitedSize uploads (#2317) 2022-04-04 10:32:53 +02:00
Neil Alexander e47dfe4786
Fix panic that can occur from nil pointer exception in media API fetchRemoteFile 2022-03-18 16:21:15 +00:00
S7evinK 485367fcfa
Return error from remote download requests (#2266) 
* Return error from remote download requests

* Remove redundant err check, return better message if file does not exist
 2022-03-16 12:16:27 +01:00
Till Faelligen c7d2254698 Update templates, remove default base URL 2022-03-07 09:45:24 +01:00
Till Faelligen fa26aa9138 Merge branch 'main' of github.com:matrix-org/dendrite into s7evink/consent-tracking 2022-03-04 15:10:23 +01:00
S7evinK 5e694cd362
Un-ratelimit calls to /thumbnail (#2251) 2022-03-04 12:03:51 +01:00
Till Faelligen e6e62497c9 Merge branch 'main' of github.com:matrix-org/dendrite into s7evink/consent-tracking 2022-03-04 09:18:34 +01:00
Neil Alexander 8996cc8059
Media endpoints on /v3 (#2242) 
* Media endpoints on `/v3`

* Keep `/v1` too?
 2022-03-02 11:35:35 +00:00
Till Faelligen 5a0ec6e443 Add policy version to create-account & mediaapi 2022-02-15 14:15:18 +01:00
Till Faelligen 89340cfc52 Verify the user has given their consent, otherwise block access 2022-02-14 18:11:56 +01:00
Neil Alexander ba1a9b98b7
Tweak some logging (#2130) 
* Modify some log levels

* Update gomatrixserverlib to matrix-org/gomatrixserverlib@336334f

* Update gomatrixserverlib to matrix-org/gomatrixserverlib@cde7ac8

* Demote warning about key change producer

* Add more useful roomserver logging

* Further tweaking
 2022-01-31 10:48:28 +00:00
S7evinK 25dcf80180
Ratelimit requests to /media/r0/download|upload (#2020) 
* Add /media/r0/config handler

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Add rate limiting to media api

* Rename variable

* Add passing tests

* Don't send multiple headers

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
 2021-11-24 12:55:44 +00:00
PiotrKozimor dec05c3347
Run gofmt on dendrite - apply go 1.17 preferred build tags (#2021) 2021-11-02 16:48:48 +00:00
database64128 5b969d172b
0️⃣ Media API: Fix maxFileSizeBytes=0 handling (#1993) 2021-11-02 16:35:15 +00:00
hacktivista 1ec5b0709b
Support WebP based thumbnails (#2033) 
Co-authored-by: Felix Freeman <libsys@hacktivista.org>
 2021-11-02 10:15:13 +00:00
Ryan W a624eab309
- Removed double imports (#1989) 
- Lower cased error messages

Signed-off-by: Ryan Whittington <twentybitdev@gmail.com>

Co-authored-by: kegsay <kegan@matrix.org>
 2021-09-08 17:31:03 +01:00
Neil Alexander f63068df3b
Only include go-sqlite3 on the relevant binaries (#1900) 
* Only include go-sqlite3 on the relevant binaries

* The driver name is always sqlite3 now

* Update to matrix-org/go-sqlite3-js@e537baa
 2021-07-20 11:18:14 +01:00
S7evinK 5094bc89bf
Set MaxFileSizeBytes <= 0 to unlimited (#1876) 
* Revert "Set MaxFileSizeBytes <= 0 to "unlimited" (#1875)"

This reverts commit 9ed0440421.

* Actually allow unlimited upload

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

Co-authored-by: kegsay <kegan@matrix.org>
 2021-07-19 17:58:51 +01:00
database64128 79181f2348
✂️ Media API: Handle unlimited file size (#1881) 2021-06-22 10:01:44 +01:00
S7evinK 9ed0440421
Set MaxFileSizeBytes <= 0 to "unlimited" (#1875) 
* Set MaxFileSizeBytes < 0 to "unlimited"

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* int64 overflows later in mediaapi/routing/upload.go[doUpload]

* Prevent int overflow when uploading
 2021-06-14 14:12:03 +01:00
S7evinK 8b22c4270d
Use LimitReader to prevent DoS risk (#1843) 
* Use LimitReader to prevent DoS risk

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Check if bytesWritten is equal to the maxFileSize
Add tests

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Use oldschool defer to cleanup after the tests

* Let LimitReader read MaxFileSizeBytes + 1

Co-authored-by: Kegsay <kegan@matrix.org>
 2021-06-07 09:17:20 +01:00
Fero d6e9b7b307
Remove the 'Content-Type' request header requirement (#1834) 2021-04-19 16:29:51 +01:00
Fero 653e30619c
Remove the Content-Length requirement on upload request (#1831) 
* Remove the Content-Length requirement

* Make sure that the file size does not exceed max content size

* Address review comment - universally check if temp file size exceeds max file size
 2021-04-14 10:53:24 +01:00
darkgallium da797c7998
Retrieve remote file size without the Content-Size header (#1537) 
* Get downloaded file size regardless of the Content-Size header

Signed-off-by: Florian Le Minoux <darkgallium+git@deuxfleurs.fr>

* fixing lint issues

* Don't exhaust memory for large files, don't limit more than necessary

* Don't use errors.Wrap in download.go

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
 2021-02-17 13:54:53 +00:00
Neil Alexander b5aa7ca3ab
Top-level setup package (#1605) 
* Move config, setup, mscs into "setup" top-level folder

* oops, forgot the EDU server

* Add setup

* goimports
 2020-12-02 17:41:00 +00:00
Neil Alexander fee1c22790
Fix #1361 (#1362) 2020-08-28 09:46:32 +01:00
Kegsay 3802efe301
Port of #945 (#1343) 
- Make sure we always cleanup the temp directory on error.
- Complain about it having an error prone API shape.
 2020-08-26 15:38:34 +01:00
Neil Alexander 287700baee
Randomly generated media IDs (#1336) 
* Filenames based on hashes but media IDs randomly generated and per-upload metadata

* Lint

* Comment generateMediaID
 2020-08-25 15:08:37 +01:00
Neil Alexander 9d53351dc2
Component-wide TransactionWriters (#1290) 
* Offset updates take place using TransactionWriter

* Refactor TransactionWriter in current state server

* Refactor TransactionWriter in federation sender

* Refactor TransactionWriter in key server

* Refactor TransactionWriter in media API

* Refactor TransactionWriter in server key API

* Refactor TransactionWriter in sync API

* Refactor TransactionWriter in user API

* Fix deadlocking Sync API tests

* Un-deadlock device database

* Fix appservice API

* Rename TransactionWriters to Writers

* Move writers up a layer in sync API

* Document sqlutil.Writer interface

* Add note to Writer documentation
 2020-08-21 10:42:08 +01:00
Neil Alexander b24747b305
Transaction writer changes, move roomserver writers (#1285) 
* Updated TransactionWriters, moved locks in roomserver, various other tweaks

* Fix redaction deadlocks

* Fix lint issue

* Rename SQLiteTransactionWriter to ExclusiveTransactionWriter

* Fix us not sending transactions through in latest events updater
 2020-08-19 15:38:27 +01:00
Neil Alexander 9677a95afc
API setup refactoring (#1266) 
* Start HTTP endpoint refactoring

* Update SetupAndServeHTTP

* Fix builds

* Don't set up external listener if no address configured

* TLS HTTP setup

* Break apart client/federation/key/media muxes

* Tweaks

* Fix P2P demos

* Fix media API routing

* Review comments @Kegsay

* Update sample config

* Fix gobind build

* Fix External -> Public in federation API test
 2020-08-13 12:16:37 +01:00
Neil Alexander 4b09f445c9
Configuration format v1 (#1230) 
* Initial pass at refactoring config (not finished)

* Don't forget current state and EDU servers

* More shifting around

* Update server key API tests

* Fix roomserver test

* Fix more tests

* Further tweaks

* Fix current state server test (sort of)

* Maybe fix appservices

* Fix client API test

* Include database connection string in database options

* Fix sync API build

* Update config test

* Fix unit tests

* Fix federation sender build

* Fix gobind build

* Set Listen address for all services in HTTP monolith mode

* Validate config, reinstate appservice derived in directory, tweaks

* Tweak federation API test

* Set MaxOpenConnections/MaxIdleConnections to previous values

* Update generate-config
 2020-08-10 14:18:04 +01:00
Neil Alexander b6bc132485
Use TransactionWriter in other component SQLite (#1209) 
* Use TransactionWriter on other component SQLites

* Fix sync API tests

* Fix panic in media API

* Fix a couple of transactions

* Fix wrong query, add some logging output

* Add debug logging into StoreEvent

* Adjust InsertRoomNID

* Update logging
 2020-07-21 15:48:21 +01:00
Neil Alexander 6bee6e77d9
Yet another v1 vs r0 media endpoint (#1190) 2020-07-08 16:39:50 +01:00
Neil Alexander 23bed196e6
Fallback parsing for Content-Disposition minefield (#1144) 2020-06-17 14:26:45 +01:00
Neil Alexander 5d5aa0a31d
Media filename handling improvements (#1140) 
* Derive content ID from hash+filename but preserve dedupe, improve Content-Disposition handling and ASCII handling

* Linter fix

* Some more comments

* Update sytest-whitelist
 2020-06-17 11:53:26 +01:00
Neil Alexander 04c99092a4
Update whitelist for sytest media fix (#1137) 
* Update sytest-whitelist, are-we-synapse-yet.list

* Update gomatrixserverlib

* Update gomatrixserverlib

* Loop avoidance

* Return UTF-8 filenames

* Replace quotes only, instead of using strconv.Quote

* Update sytest-whitelist

* Update sytest-whitelist
 2020-06-16 18:31:38 +01:00
Neil Alexander fc0e74ae0f
Fix media API for demos and possibly Synapse (#1134) 
* Fix media API for demos and possibly Synapse

* User API

* goimports
 2020-06-16 14:29:11 +01:00
Kegsay 9c77022513
Make userapi responsible for checking access tokens (#1133) 
* Make userapi responsible for checking access tokens

There's still plenty of dependencies on account/device DBs, but this
is a start. This is a breaking change as it adds a required config
value `listen.user_api`.

* Cleanup

* Review comments and test fix
 2020-06-16 14:10:55 +01:00
Kegsay ecd7accbad
Rehuffle where things are in the internal package (#1122) 
renamed:    internal/eventcontent.go -> internal/eventutil/eventcontent.go
	renamed:    internal/events.go -> internal/eventutil/events.go
	renamed:    internal/types.go -> internal/eventutil/types.go
	renamed:    internal/http/http.go -> internal/httputil/http.go
	renamed:    internal/httpapi.go -> internal/httputil/httpapi.go
	renamed:    internal/httpapi_test.go -> internal/httputil/httpapi_test.go
	renamed:    internal/httpapis/paths.go -> internal/httputil/paths.go
	renamed:    internal/routing.go -> internal/httputil/routing.go
	renamed:    internal/basecomponent/base.go -> internal/setup/base.go
	renamed:    internal/basecomponent/flags.go -> internal/setup/flags.go
	renamed:    internal/partition_offset_table.go -> internal/sqlutil/partition_offset_table.go
	renamed:    internal/postgres.go -> internal/sqlutil/postgres.go
	renamed:    internal/postgres_wasm.go -> internal/sqlutil/postgres_wasm.go
	renamed:    internal/sql.go -> internal/sqlutil/sql.go
 2020-06-12 14:55:57 +01:00
Kegsay 4f171c56a8
Split out SetupFooComponent (#1106) 
* Split out adding HTTP routes from making internal APIs for clarity

* Split out more components

* Split out more things

* Finish converting

* internal mux for internal routes
 2020-06-08 15:51:07 +01:00
Kegsay e7d1ac84c3
Add ParseFileURI and use it when dealing with file URIs (#1088) 
* Add ParseFileURI and use it when dealing with file URIs

Fixes #1059

* Missing file

* Linting
 2020-06-04 11:18:08 +01:00