Add peer-to-peer support into Dendrite via libp2p and fetch (#880)
* Use a fork of pq which supports userCurrent on wasm
* Use sqlite3_js driver when running in JS
* Add cmd/dendritejs to pull in sqlite3_js driver for wasm only
* Update to latest go-sqlite-js version
* Replace prometheus with a stub. sigh
* Hard-code a config and don't use opentracing
* Latest go-sqlite3-js version
* Generate a key for now
* Listen for fetch traffic rather than HTTP
* Latest hacks for js
* libp2p support
* More libp2p
* Fork gjson to allow us to enforce auth checks as before
Previously, all events would come down redacted because the hash
checks would fail. They would fail because sjson.DeleteBytes didn't
remove keys not used for hashing. This didn't work because of a build
tag which included a file which no-oped the index returned.
See https://github.com/tidwall/gjson/issues/157
When it's resolved, let's go back to mainline.
* Use gjson@1.6.0 as it fixes https://github.com/tidwall/gjson/issues/157
* Use latest gomatrixserverlib for sig checks
* Fix a bug which could cause exclude_from_sync to not be set
Caused when sending events over federation.
* Use query variadic to make lookups actually work!
* Latest gomatrixserverlib
* Add notes on getting p2p up and running
Partly so I don't forget myself!
* refactor: Move p2p specific stuff to cmd/dendritejs
This is important or else the normal build of dendrite will fail
because the p2p libraries depend on syscall/js which doesn't work
on normal builds.
Also, clean up main.go to read a bit better.
* Update ho-http-js-libp2p to return errors from RoundTrip
* Add an LRU cache around the key DB
We actually need this for P2P because otherwise we can *segfault*
with things like: "runtime: unexpected return pc for runtime.handleEvent"
where the event is a `syscall/js` event, caused by spamming sql.js
caused by "Checking event signatures for 14 events of room state" which
hammers the key DB repeatedly in quick succession.
Using a cache fixes this, though the underlying cause is probably a bug
in the version of Go I'm on (1.13.7)
* breaking: Add Tracing.Enabled to toggle whether we do opentracing
Defaults to false, which is why this is a breaking change. We need
this flag because WASM builds cannot do opentracing.
* Start adding conditional builds for wasm to handle lib/pq
The general idea here is to have the wasm build have a `NewXXXDatabase`
that doesn't import any postgres package and hence we never import
`lib/pq`, which doesn't work under WASM (undefined `userCurrent`).
* Remove lib/pq for wasm for syncapi
* Add conditional building to remaining storage APIs
* Update build script to set env vars correctly for dendritejs
* sqlite bug fixes
* Docs
* Add a no-op main for dendritejs when not building under wasm
* Use the real prometheus, even for WASM
Instead, the dendrite-sw.js must mock out `process.pid` and
`fs.stat` - which must invoke the callback with an error (e.g `EINVAL`)
in order for it to work:
```
global.process = {
pid: 1,
};
global.fs.stat = function(path, cb) {
cb({
code: "EINVAL",
});
}
```
* Linting
2020-03-06 04:23:55 -06:00
// Copyright 2020 The Matrix.org Foundation C.I.C.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package storage
import (
"context"
2022-08-09 04:15:58 -05:00
"time"
Add peer-to-peer support into Dendrite via libp2p and fetch (#880)
* Use a fork of pq which supports userCurrent on wasm
* Use sqlite3_js driver when running in JS
* Add cmd/dendritejs to pull in sqlite3_js driver for wasm only
* Update to latest go-sqlite-js version
* Replace prometheus with a stub. sigh
* Hard-code a config and don't use opentracing
* Latest go-sqlite3-js version
* Generate a key for now
* Listen for fetch traffic rather than HTTP
* Latest hacks for js
* libp2p support
* More libp2p
* Fork gjson to allow us to enforce auth checks as before
Previously, all events would come down redacted because the hash
checks would fail. They would fail because sjson.DeleteBytes didn't
remove keys not used for hashing. This didn't work because of a build
tag which included a file which no-oped the index returned.
See https://github.com/tidwall/gjson/issues/157
When it's resolved, let's go back to mainline.
* Use gjson@1.6.0 as it fixes https://github.com/tidwall/gjson/issues/157
* Use latest gomatrixserverlib for sig checks
* Fix a bug which could cause exclude_from_sync to not be set
Caused when sending events over federation.
* Use query variadic to make lookups actually work!
* Latest gomatrixserverlib
* Add notes on getting p2p up and running
Partly so I don't forget myself!
* refactor: Move p2p specific stuff to cmd/dendritejs
This is important or else the normal build of dendrite will fail
because the p2p libraries depend on syscall/js which doesn't work
on normal builds.
Also, clean up main.go to read a bit better.
* Update ho-http-js-libp2p to return errors from RoundTrip
* Add an LRU cache around the key DB
We actually need this for P2P because otherwise we can *segfault*
with things like: "runtime: unexpected return pc for runtime.handleEvent"
where the event is a `syscall/js` event, caused by spamming sql.js
caused by "Checking event signatures for 14 events of room state" which
hammers the key DB repeatedly in quick succession.
Using a cache fixes this, though the underlying cause is probably a bug
in the version of Go I'm on (1.13.7)
* breaking: Add Tracing.Enabled to toggle whether we do opentracing
Defaults to false, which is why this is a breaking change. We need
this flag because WASM builds cannot do opentracing.
* Start adding conditional builds for wasm to handle lib/pq
The general idea here is to have the wasm build have a `NewXXXDatabase`
that doesn't import any postgres package and hence we never import
`lib/pq`, which doesn't work under WASM (undefined `userCurrent`).
* Remove lib/pq for wasm for syncapi
* Add conditional building to remaining storage APIs
* Update build script to set env vars correctly for dendritejs
* sqlite bug fixes
* Docs
* Add a no-op main for dendritejs when not building under wasm
* Use the real prometheus, even for WASM
Instead, the dendrite-sw.js must mock out `process.pid` and
`fs.stat` - which must invoke the callback with an error (e.g `EINVAL`)
in order for it to work:
```
global.process = {
pid: 1,
};
global.fs.stat = function(path, cb) {
cb({
code: "EINVAL",
});
}
```
* Linting
2020-03-06 04:23:55 -06:00
2022-10-21 05:50:51 -05:00
"github.com/matrix-org/gomatrixserverlib"
2023-04-19 09:50:33 -05:00
"github.com/matrix-org/gomatrixserverlib/spec"
2022-10-21 05:50:51 -05:00
2023-01-23 11:55:12 -06:00
"github.com/matrix-org/dendrite/federationapi/storage/shared/receipt"
2021-11-24 04:45:23 -06:00
"github.com/matrix-org/dendrite/federationapi/types"
Add peer-to-peer support into Dendrite via libp2p and fetch (#880)
* Use a fork of pq which supports userCurrent on wasm
* Use sqlite3_js driver when running in JS
* Add cmd/dendritejs to pull in sqlite3_js driver for wasm only
* Update to latest go-sqlite-js version
* Replace prometheus with a stub. sigh
* Hard-code a config and don't use opentracing
* Latest go-sqlite3-js version
* Generate a key for now
* Listen for fetch traffic rather than HTTP
* Latest hacks for js
* libp2p support
* More libp2p
* Fork gjson to allow us to enforce auth checks as before
Previously, all events would come down redacted because the hash
checks would fail. They would fail because sjson.DeleteBytes didn't
remove keys not used for hashing. This didn't work because of a build
tag which included a file which no-oped the index returned.
See https://github.com/tidwall/gjson/issues/157
When it's resolved, let's go back to mainline.
* Use gjson@1.6.0 as it fixes https://github.com/tidwall/gjson/issues/157
* Use latest gomatrixserverlib for sig checks
* Fix a bug which could cause exclude_from_sync to not be set
Caused when sending events over federation.
* Use query variadic to make lookups actually work!
* Latest gomatrixserverlib
* Add notes on getting p2p up and running
Partly so I don't forget myself!
* refactor: Move p2p specific stuff to cmd/dendritejs
This is important or else the normal build of dendrite will fail
because the p2p libraries depend on syscall/js which doesn't work
on normal builds.
Also, clean up main.go to read a bit better.
* Update ho-http-js-libp2p to return errors from RoundTrip
* Add an LRU cache around the key DB
We actually need this for P2P because otherwise we can *segfault*
with things like: "runtime: unexpected return pc for runtime.handleEvent"
where the event is a `syscall/js` event, caused by spamming sql.js
caused by "Checking event signatures for 14 events of room state" which
hammers the key DB repeatedly in quick succession.
Using a cache fixes this, though the underlying cause is probably a bug
in the version of Go I'm on (1.13.7)
* breaking: Add Tracing.Enabled to toggle whether we do opentracing
Defaults to false, which is why this is a breaking change. We need
this flag because WASM builds cannot do opentracing.
* Start adding conditional builds for wasm to handle lib/pq
The general idea here is to have the wasm build have a `NewXXXDatabase`
that doesn't import any postgres package and hence we never import
`lib/pq`, which doesn't work under WASM (undefined `userCurrent`).
* Remove lib/pq for wasm for syncapi
* Add conditional building to remaining storage APIs
* Update build script to set env vars correctly for dendritejs
* sqlite bug fixes
* Docs
* Add a no-op main for dendritejs when not building under wasm
* Use the real prometheus, even for WASM
Instead, the dendrite-sw.js must mock out `process.pid` and
`fs.stat` - which must invoke the callback with an error (e.g `EINVAL`)
in order for it to work:
```
global.process = {
pid: 1,
};
global.fs.stat = function(path, cb) {
cb({
code: "EINVAL",
});
}
```
* Linting
2020-03-06 04:23:55 -06:00
)
type Database interface {
2023-01-23 11:55:12 -06:00
P2PDatabase
2021-11-24 04:45:23 -06:00
gomatrixserverlib . KeyDatabase
2020-07-20 10:55:20 -05:00
2022-05-17 07:23:35 -05:00
UpdateRoom ( ctx context . Context , roomID string , addHosts [ ] types . JoinedHost , removeHosts [ ] string , purgeRoomFirst bool ) ( joinedHosts [ ] types . JoinedHost , err error )
2020-07-20 10:55:20 -05:00
Add peer-to-peer support into Dendrite via libp2p and fetch (#880)
* Use a fork of pq which supports userCurrent on wasm
* Use sqlite3_js driver when running in JS
* Add cmd/dendritejs to pull in sqlite3_js driver for wasm only
* Update to latest go-sqlite-js version
* Replace prometheus with a stub. sigh
* Hard-code a config and don't use opentracing
* Latest go-sqlite3-js version
* Generate a key for now
* Listen for fetch traffic rather than HTTP
* Latest hacks for js
* libp2p support
* More libp2p
* Fork gjson to allow us to enforce auth checks as before
Previously, all events would come down redacted because the hash
checks would fail. They would fail because sjson.DeleteBytes didn't
remove keys not used for hashing. This didn't work because of a build
tag which included a file which no-oped the index returned.
See https://github.com/tidwall/gjson/issues/157
When it's resolved, let's go back to mainline.
* Use gjson@1.6.0 as it fixes https://github.com/tidwall/gjson/issues/157
* Use latest gomatrixserverlib for sig checks
* Fix a bug which could cause exclude_from_sync to not be set
Caused when sending events over federation.
* Use query variadic to make lookups actually work!
* Latest gomatrixserverlib
* Add notes on getting p2p up and running
Partly so I don't forget myself!
* refactor: Move p2p specific stuff to cmd/dendritejs
This is important or else the normal build of dendrite will fail
because the p2p libraries depend on syscall/js which doesn't work
on normal builds.
Also, clean up main.go to read a bit better.
* Update ho-http-js-libp2p to return errors from RoundTrip
* Add an LRU cache around the key DB
We actually need this for P2P because otherwise we can *segfault*
with things like: "runtime: unexpected return pc for runtime.handleEvent"
where the event is a `syscall/js` event, caused by spamming sql.js
caused by "Checking event signatures for 14 events of room state" which
hammers the key DB repeatedly in quick succession.
Using a cache fixes this, though the underlying cause is probably a bug
in the version of Go I'm on (1.13.7)
* breaking: Add Tracing.Enabled to toggle whether we do opentracing
Defaults to false, which is why this is a breaking change. We need
this flag because WASM builds cannot do opentracing.
* Start adding conditional builds for wasm to handle lib/pq
The general idea here is to have the wasm build have a `NewXXXDatabase`
that doesn't import any postgres package and hence we never import
`lib/pq`, which doesn't work under WASM (undefined `userCurrent`).
* Remove lib/pq for wasm for syncapi
* Add conditional building to remaining storage APIs
* Update build script to set env vars correctly for dendritejs
* sqlite bug fixes
* Docs
* Add a no-op main for dendritejs when not building under wasm
* Use the real prometheus, even for WASM
Instead, the dendrite-sw.js must mock out `process.pid` and
`fs.stat` - which must invoke the callback with an error (e.g `EINVAL`)
in order for it to work:
```
global.process = {
pid: 1,
};
global.fs.stat = function(path, cb) {
cb({
code: "EINVAL",
});
}
```
* Linting
2020-03-06 04:23:55 -06:00
GetJoinedHosts ( ctx context . Context , roomID string ) ( [ ] types . JoinedHost , error )
2023-04-19 09:50:33 -05:00
GetAllJoinedHosts ( ctx context . Context ) ( [ ] spec . ServerName , error )
2020-08-04 05:32:14 -05:00
// GetJoinedHostsForRooms returns the complete set of servers in the rooms given.
2023-04-19 09:50:33 -05:00
GetJoinedHostsForRooms ( ctx context . Context , roomIDs [ ] string , excludeSelf , excludeBlacklisted bool ) ( [ ] spec . ServerName , error )
2020-07-20 10:55:20 -05:00
2023-01-23 11:55:12 -06:00
StoreJSON ( ctx context . Context , js string ) ( * receipt . Receipt , error )
2020-07-20 10:55:20 -05:00
2023-04-19 09:50:33 -05:00
GetPendingPDUs ( ctx context . Context , serverName spec . ServerName , limit int ) ( pdus map [ * receipt . Receipt ] * gomatrixserverlib . HeaderedEvent , err error )
GetPendingEDUs ( ctx context . Context , serverName spec . ServerName , limit int ) ( edus map [ * receipt . Receipt ] * gomatrixserverlib . EDU , err error )
2020-12-09 04:03:22 -06:00
2023-04-19 09:50:33 -05:00
AssociatePDUWithDestinations ( ctx context . Context , destinations map [ spec . ServerName ] struct { } , dbReceipt * receipt . Receipt ) error
AssociateEDUWithDestinations ( ctx context . Context , destinations map [ spec . ServerName ] struct { } , dbReceipt * receipt . Receipt , eduType string , expireEDUTypes map [ string ] time . Duration ) error
2020-07-20 10:55:20 -05:00
2023-04-19 09:50:33 -05:00
CleanPDUs ( ctx context . Context , serverName spec . ServerName , receipts [ ] * receipt . Receipt ) error
CleanEDUs ( ctx context . Context , serverName spec . ServerName , receipts [ ] * receipt . Receipt ) error
2020-07-20 10:55:20 -05:00
2023-04-19 09:50:33 -05:00
GetPendingPDUServerNames ( ctx context . Context ) ( [ ] spec . ServerName , error )
GetPendingEDUServerNames ( ctx context . Context ) ( [ ] spec . ServerName , error )
2020-07-22 11:01:29 -05:00
2021-01-22 08:55:08 -06:00
// these don't have contexts passed in as we want things to happen regardless of the request context
2023-04-19 09:50:33 -05:00
AddServerToBlacklist ( serverName spec . ServerName ) error
RemoveServerFromBlacklist ( serverName spec . ServerName ) error
2021-05-24 05:43:24 -05:00
RemoveAllServersFromBlacklist ( ) error
2023-04-19 09:50:33 -05:00
IsServerBlacklisted ( serverName spec . ServerName ) ( bool , error )
2021-01-22 08:55:08 -06:00
2023-01-23 11:55:12 -06:00
// Adds the server to the list of assumed offline servers.
// If the server already exists in the table, nothing happens and returns success.
2023-04-19 09:50:33 -05:00
SetServerAssumedOffline ( ctx context . Context , serverName spec . ServerName ) error
2023-01-23 11:55:12 -06:00
// Removes the server from the list of assumed offline servers.
// If the server doesn't exist in the table, nothing happens and returns success.
2023-04-19 09:50:33 -05:00
RemoveServerAssumedOffline ( ctx context . Context , serverName spec . ServerName ) error
2023-01-23 11:55:12 -06:00
// Purges all entries from the assumed offline table.
RemoveAllServersAssumedOffline ( ctx context . Context ) error
// Gets whether the provided server is present in the table.
// If it is present, returns true. If not, returns false.
2023-04-19 09:50:33 -05:00
IsServerAssumedOffline ( ctx context . Context , serverName spec . ServerName ) ( bool , error )
2023-01-23 11:55:12 -06:00
2023-04-19 09:50:33 -05:00
AddOutboundPeek ( ctx context . Context , serverName spec . ServerName , roomID , peekID string , renewalInterval int64 ) error
RenewOutboundPeek ( ctx context . Context , serverName spec . ServerName , roomID , peekID string , renewalInterval int64 ) error
GetOutboundPeek ( ctx context . Context , serverName spec . ServerName , roomID , peekID string ) ( * types . OutboundPeek , error )
2021-01-22 08:55:08 -06:00
GetOutboundPeeks ( ctx context . Context , roomID string ) ( [ ] types . OutboundPeek , error )
2023-04-19 09:50:33 -05:00
AddInboundPeek ( ctx context . Context , serverName spec . ServerName , roomID , peekID string , renewalInterval int64 ) error
RenewInboundPeek ( ctx context . Context , serverName spec . ServerName , roomID , peekID string , renewalInterval int64 ) error
GetInboundPeek ( ctx context . Context , serverName spec . ServerName , roomID , peekID string ) ( * types . InboundPeek , error )
2021-01-22 08:55:08 -06:00
GetInboundPeeks ( ctx context . Context , roomID string ) ( [ ] types . InboundPeek , error )
2021-07-15 11:45:37 -05:00
// Update the notary with the given server keys from the given server name.
2023-04-19 09:50:33 -05:00
UpdateNotaryKeys ( ctx context . Context , serverName spec . ServerName , serverKeys gomatrixserverlib . ServerKeys ) error
2021-07-15 11:45:37 -05:00
// Query the notary for the server keys for the given server. If `optKeyIDs` is not empty, multiple server keys may be returned (between 1 - len(optKeyIDs))
// such that the combination of all server keys will include all the `optKeyIDs`.
2023-04-19 09:50:33 -05:00
GetNotaryKeys ( ctx context . Context , serverName spec . ServerName , optKeyIDs [ ] gomatrixserverlib . KeyID ) ( [ ] gomatrixserverlib . ServerKeys , error )
2022-08-09 04:15:58 -05:00
// DeleteExpiredEDUs cleans up expired EDUs
DeleteExpiredEDUs ( ctx context . Context ) error
2023-01-19 14:02:32 -06:00
PurgeRoom ( ctx context . Context , roomID string ) error
Add peer-to-peer support into Dendrite via libp2p and fetch (#880)
* Use a fork of pq which supports userCurrent on wasm
* Use sqlite3_js driver when running in JS
* Add cmd/dendritejs to pull in sqlite3_js driver for wasm only
* Update to latest go-sqlite-js version
* Replace prometheus with a stub. sigh
* Hard-code a config and don't use opentracing
* Latest go-sqlite3-js version
* Generate a key for now
* Listen for fetch traffic rather than HTTP
* Latest hacks for js
* libp2p support
* More libp2p
* Fork gjson to allow us to enforce auth checks as before
Previously, all events would come down redacted because the hash
checks would fail. They would fail because sjson.DeleteBytes didn't
remove keys not used for hashing. This didn't work because of a build
tag which included a file which no-oped the index returned.
See https://github.com/tidwall/gjson/issues/157
When it's resolved, let's go back to mainline.
* Use gjson@1.6.0 as it fixes https://github.com/tidwall/gjson/issues/157
* Use latest gomatrixserverlib for sig checks
* Fix a bug which could cause exclude_from_sync to not be set
Caused when sending events over federation.
* Use query variadic to make lookups actually work!
* Latest gomatrixserverlib
* Add notes on getting p2p up and running
Partly so I don't forget myself!
* refactor: Move p2p specific stuff to cmd/dendritejs
This is important or else the normal build of dendrite will fail
because the p2p libraries depend on syscall/js which doesn't work
on normal builds.
Also, clean up main.go to read a bit better.
* Update ho-http-js-libp2p to return errors from RoundTrip
* Add an LRU cache around the key DB
We actually need this for P2P because otherwise we can *segfault*
with things like: "runtime: unexpected return pc for runtime.handleEvent"
where the event is a `syscall/js` event, caused by spamming sql.js
caused by "Checking event signatures for 14 events of room state" which
hammers the key DB repeatedly in quick succession.
Using a cache fixes this, though the underlying cause is probably a bug
in the version of Go I'm on (1.13.7)
* breaking: Add Tracing.Enabled to toggle whether we do opentracing
Defaults to false, which is why this is a breaking change. We need
this flag because WASM builds cannot do opentracing.
* Start adding conditional builds for wasm to handle lib/pq
The general idea here is to have the wasm build have a `NewXXXDatabase`
that doesn't import any postgres package and hence we never import
`lib/pq`, which doesn't work under WASM (undefined `userCurrent`).
* Remove lib/pq for wasm for syncapi
* Add conditional building to remaining storage APIs
* Update build script to set env vars correctly for dendritejs
* sqlite bug fixes
* Docs
* Add a no-op main for dendritejs when not building under wasm
* Use the real prometheus, even for WASM
Instead, the dendrite-sw.js must mock out `process.pid` and
`fs.stat` - which must invoke the callback with an error (e.g `EINVAL`)
in order for it to work:
```
global.process = {
pid: 1,
};
global.fs.stat = function(path, cb) {
cb({
code: "EINVAL",
});
}
```
* Linting
2020-03-06 04:23:55 -06:00
}
2023-01-23 11:55:12 -06:00
type P2PDatabase interface {
// Stores the given list of servers as relay servers for the provided destination server.
// Providing duplicates will only lead to a single entry and won't lead to an error.
2023-04-19 09:50:33 -05:00
P2PAddRelayServersForServer ( ctx context . Context , serverName spec . ServerName , relayServers [ ] spec . ServerName ) error
2023-01-23 11:55:12 -06:00
// Get the list of relay servers associated with the provided destination server.
// If no entry exists in the table, an empty list is returned and does not result in an error.
2023-04-19 09:50:33 -05:00
P2PGetRelayServersForServer ( ctx context . Context , serverName spec . ServerName ) ( [ ] spec . ServerName , error )
2023-01-23 11:55:12 -06:00
// Deletes any entries for the provided destination server that match the provided relayServers list.
// If any of the provided servers don't match an entry, nothing happens and no error is returned.
2023-04-19 09:50:33 -05:00
P2PRemoveRelayServersForServer ( ctx context . Context , serverName spec . ServerName , relayServers [ ] spec . ServerName ) error
2023-01-23 11:55:12 -06:00
// Deletes all entries for the provided destination server.
// If the destination server doesn't exist in the table, nothing happens and no error is returned.
2023-04-19 09:50:33 -05:00
P2PRemoveAllRelayServersForServer ( ctx context . Context , serverName spec . ServerName ) error
2023-01-23 11:55:12 -06:00
}