mirrors/dendrite
1
0
Fork 0
mirror of https://github.com/matrix-org/dendrite.git synced 2026-01-15 01:53:09 -06:00
Code Issues Packages Projects Releases Wiki Activity

Update coverage docs

Browse source
This commit is contained in:
Till Faelligen 2023-05-25 13:07:54 +02:00
parent d5c7ee49e2
commit 56a74e2f15
No known key found for this signature in database
GPG key ID: ACCDC9606D472758
1 changed files with 62 additions and 47 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

109
docs/development/coverage.md
View file

@ -5,64 +5,74 @@ nav_order: 3
permalink: /development/coverage permalink: /development/coverage
--- ---
## Running unit tests with coverage enabled
Running unit tests with coverage enabled can be done with the following commands, this will generate a `integrationcover.log`
```bash
go test -covermode=atomic -coverpkg=./... -coverprofile=integrationcover.log $(go list ./... | grep -v '/cmd/')
go tool cover -func=integrationcover.log
```
## Running Sytest with coverage enabled ## Running Sytest with coverage enabled
To run Sytest with coverage enabled: To run Sytest with coverage enabled:
```bash ```bash
docker run --rm --name sytest -v "/Users/kegan/github/sytest:/sytest" \ docker run --rm --name sytest -v "/Users/kegan/github/sytest:/sytest" \
-v "/Users/kegan/github/dendrite:/src" -v "/Users/kegan/logs:/logs" \ -v "/Users/kegan/github/dendrite:/src" -v "$(pwd)/sytest_logs:/logs" \
-v "/Users/kegan/go/:/gopath" -e "POSTGRES=1" -e "DENDRITE_TRACE_HTTP=1" \ -v "/Users/kegan/go/:/gopath" -e "POSTGRES=1" \
-e "COVER=1" \ -e "COVER=1" \
matrixdotorg/sytest-dendrite:latest matrixdotorg/sytest-dendrite:latest
# to get a more accurate coverage you may also need to run Sytest using SQLite as the database:
docker run --rm --name sytest -v "/Users/kegan/github/sytest:/sytest" \
-v "/Users/kegan/github/dendrite:/src" -v "$(pwd)/sytest_logs:/logs" \
-v "/Users/kegan/go/:/gopath" \
-e "COVER=1" \
matrixdotorg/sytest-dendrite:latest
``` ```
This will generate a new file `integrationcover.log` in each server's directory, e.g `server-0/integrationcover.log`. To parse it, This will generate a folder `covdatafiles` in each server's directory, e.g `server-0/covdatafiles`. To parse them,
ensure your working directory is under the Dendrite repository then run: ensure your working directory is under the Dendrite repository then run:
```bash ```bash
go tool cover -func=/path/to/server-0/integrationcover.log go tool covdata func -i="$(find -name 'covmeta*' -type f -exec dirname {} \; | uniq | paste -s -d ',' -)"
``` ```
which will produce an output like: which will produce an output like:
``` ```
... ...
github.com/matrix-org/util/json.go:83: NewJSONRequestHandler 100.0% github.com/matrix-org/util/json.go:132: MakeJSONAPI 70.0%
github.com/matrix-org/util/json.go:90: Protect 57.1% github.com/matrix-org/util/json.go:151: respond 84.6%
github.com/matrix-org/util/json.go:110: RequestWithLogging 100.0% github.com/matrix-org/util/json.go:180: WithCORSOptions 0.0%
github.com/matrix-org/util/json.go:132: MakeJSONAPI 70.0% github.com/matrix-org/util/json.go:191: SetCORSHeaders 100.0%
github.com/matrix-org/util/json.go:151: respond 61.5% github.com/matrix-org/util/json.go:202: RandomString 100.0%
github.com/matrix-org/util/json.go:180: WithCORSOptions 0.0% github.com/matrix-org/util/json.go:210: init 100.0%
github.com/matrix-org/util/json.go:191: SetCORSHeaders 100.0% github.com/matrix-org/util/unique.go:13: Unique 91.7%
github.com/matrix-org/util/json.go:202: RandomString 100.0% github.com/matrix-org/util/unique.go:48: SortAndUnique 100.0%
github.com/matrix-org/util/json.go:210: init 100.0% github.com/matrix-org/util/unique.go:55: UniqueStrings 100.0%
github.com/matrix-org/util/unique.go:13: Unique 91.7% total (statements) 64.0%
github.com/matrix-org/util/unique.go:48: SortAndUnique 100.0%
github.com/matrix-org/util/unique.go:55: UniqueStrings 100.0%
total: (statements) 53.7%
``` ```
(after running Sytest for Postgres _and_ SQLite)
The total coverage for this run is the last line at the bottom. However, this value is misleading because Dendrite can run in many different configurations, The total coverage for this run is the last line at the bottom. However, this value is misleading because Dendrite can run in different configurations,
which will never be tested in a single test run (e.g sqlite or postgres). To get a more accurate value, additional processing is required which will never be tested in a single test run (e.g sqlite or postgres). To get a more accurate value, you'll need run Sytest for Postgres and SQLite (see commands above).
to remove packages which will never be tested and extension MSCs: Additional processing is required also to remove packages which will never be tested and extension MSCs:
(The following commands use [gocovmerge](https://github.com/wadey/gocovmerge), to merge two or more coverage logs, so make sure you have that installed)
```bash ```bash
# These commands are all similar but change which package paths are _removed_ from the output. # If you executed both commands from above, you can get the total coverage using the following commands
go tool covdata textfmt -i="$(find -name 'covmeta*' -type f -exec dirname {} \; | uniq | paste -s -d ',' -)" -o sytest.cov
grep -Ev 'relayapi|setup/mscs' sytest.cov > final.cov
go tool cover -func=final.cov
# For Postgres # If you only executed the one for Postgres:
find -name 'integrationcover.log' | xargs gocovmerge | grep -Ev 'relayapi|sqlite|setup/mscs' > final.cov && go tool cover -func=final.cov go tool covdata textfmt -i="$(find -name 'covmeta*' -type f -exec dirname {} \; | uniq | paste -s -d ',' -)" -o sytest.cov
grep -Ev 'relayapi|sqlite|setup/mscs' sytest.cov > final.cov
go tool cover -func=final.cov
# For SQLite # If you only executed the one for SQLite:
find -name 'integrationcover.log' | xargs gocovmerge | grep -Ev 'relayapi|postgres|setup/mscs' > final.cov && go tool cover -func=final.cov go tool covdata textfmt -i="$(find -name 'covmeta*' -type f -exec dirname {} \; | uniq | paste -s -d ',' -)" -o sytest.cov
``` grep -Ev 'relayapi|postgres|setup/mscs' sytest.cov > final.cov
go tool cover -func=final.cov
## Running unit tests with coverage enabled
Running unit tests with coverage enabled can be done with the following command, it will also generate a `integrationcover.log`, which can later be merged
using `gocovmerge`:
```bash
go test -covermode=atomic -coverpkg=./... -coverprofile=integrationcover.log $(go list ./... | grep -v '/cmd/')
``` ```
## Getting coverage from Complement ## Getting coverage from Complement
@ -87,10 +97,10 @@ and add the following content:
#!/bin/bash #!/bin/bash
mkdir -p /tmp/Complement/logs/$2/$1/ mkdir -p /tmp/Complement/logs/$2/$1/
docker cp $1:/dendrite/integrationcover.log /tmp/Complement/logs/$2/$1/ docker cp $1:/tmp/covdatafiles/. /tmp/Complement/logs/$2/$1/
``` ```
This will copy the `integrationcover.log` files from each container to something like This will copy the `covdatafiles` files from each container to something like
`/tmp/Complement/logs/TestLogin/94f9c428de95779d2b62a3ccd8eab9d5ddcf65cc259a40ece06bdc61687ffed3/integrationcover.log`. (`$1` is the containerID, `$2` the test name) `/tmp/Complement/logs/TestLogin/94f9c428de95779d2b62a3ccd8eab9d5ddcf65cc259a40ece06bdc61687ffed3/`. (`$1` is the containerID, `$2` the test name)
Now that we have set up everything we need, we can finally execute Complement: Now that we have set up everything we need, we can finally execute Complement:
```bash ```bash
@ -101,15 +111,20 @@ COMPLEMENT_POST_TEST_SCRIPT=$(pwd)/posttest.sh \
go test -tags dendrite_blacklist ./tests/... -count=1 -v -timeout=30m -failfast=false go test -tags dendrite_blacklist ./tests/... -count=1 -v -timeout=30m -failfast=false
``` ```
Once this is done, you can copy the resulting `integrationcover.log` files to your Dendrite repository for the next step. Once this is done, you can copy the resulting `covdatafiles` files to your Dendrite repository for the next step.
```bash ```bash
cp -pr /tmp/Complement/logs PathToYourDendriteRepository cp -pr /tmp/Complement/logs PathToYourDendriteRepository
``` ```
## Combining the results of all runs You can also run the following to get the coverage for Complement runs alone:
Now that we have all our `integrationcover.log` files within the Dendrite repository, you can now execute the following command, to get the coverage
overall (Note: update `grep -Ev` accordingly, to exclude SQLite for example):
```bash ```bash
find -name 'integrationcover.log' | xargs gocovmerge | grep -Ev 'setup/mscs|api_trace' > final.cov && go tool cover -func=final.cov go tool covdata func -i="$(find /tmp/Complement -name 'covmeta*' -type f -exec dirname {} \; | uniq | paste -s -d ',' -)"
```
## Combining the results of (almost) all runs
Now that we have all our `covdatafiles` files within the Dendrite repository, you can now execute the following command, to get the coverage
overall (excluding unit tests):
```bash
go tool covdata func -i="$(find -name 'covmeta*' -type f -exec dirname {} \; | uniq | paste -s -d ',' -)"
``` ```