mirrors/dendrite
1
0
Fork 0
mirror of https://github.com/matrix-org/dendrite.git synced 2026-01-11 16:13:10 -06:00
Code Issues Packages Projects Releases Wiki Activity

Update coverage docs

Browse source
This commit is contained in:
Till Faelligen 2023-05-25 13:07:54 +02:00
parent d5c7ee49e2
commit 56a74e2f15
No known key found for this signature in database
GPG key ID: ACCDC9606D472758
1 changed files with 62 additions and 47 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

109
docs/development/coverage.md
View file

@ -5,64 +5,74 @@ nav_order: 3
permalink: /development/coverage
---
## Running unit tests with coverage enabled
Running unit tests with coverage enabled can be done with the following commands, this will generate a `integrationcover.log`
```bash
go test -covermode=atomic -coverpkg=./... -coverprofile=integrationcover.log $(go list ./... | grep -v '/cmd/')
go tool cover -func=integrationcover.log
```
## Running Sytest with coverage enabled
To run Sytest with coverage enabled:
```bash
docker run --rm --name sytest -v "/Users/kegan/github/sytest:/sytest" \
-v "/Users/kegan/github/dendrite:/src" -v "/Users/kegan/logs:/logs" \
-v "/Users/kegan/go/:/gopath" -e "POSTGRES=1" -e "DENDRITE_TRACE_HTTP=1" \
-e "COVER=1" \
matrixdotorg/sytest-dendrite:latest
-v "/Users/kegan/github/dendrite:/src" -v "$(pwd)/sytest_logs:/logs" \
-v "/Users/kegan/go/:/gopath" -e "POSTGRES=1" \
-e "COVER=1" \
matrixdotorg/sytest-dendrite:latest
# to get a more accurate coverage you may also need to run Sytest using SQLite as the database:
docker run --rm --name sytest -v "/Users/kegan/github/sytest:/sytest" \
-v "/Users/kegan/github/dendrite:/src" -v "$(pwd)/sytest_logs:/logs" \
-v "/Users/kegan/go/:/gopath" \
-e "COVER=1" \
matrixdotorg/sytest-dendrite:latest
```
This will generate a new file `integrationcover.log` in each server's directory, e.g `server-0/integrationcover.log`. To parse it,
ensure your working directory is under the Dendrite repository then run:
This will generate a folder `covdatafiles` in each server's directory, e.g `server-0/covdatafiles`. To parse them,
ensure your working directory is under the Dendrite repository then run:
```bash
go tool cover -func=/path/to/server-0/integrationcover.log
go tool covdata func -i="$(find -name 'covmeta*' -type f -exec dirname {} \; | uniq | paste -s -d ',' -)"
```
which will produce an output like:
```
...
github.com/matrix-org/util/json.go:83: NewJSONRequestHandler 100.0%
github.com/matrix-org/util/json.go:90: Protect 57.1%
github.com/matrix-org/util/json.go:110: RequestWithLogging 100.0%
github.com/matrix-org/util/json.go:132: MakeJSONAPI 70.0%
github.com/matrix-org/util/json.go:151: respond 61.5%
github.com/matrix-org/util/json.go:180: WithCORSOptions 0.0%
github.com/matrix-org/util/json.go:191: SetCORSHeaders 100.0%
github.com/matrix-org/util/json.go:202: RandomString 100.0%
github.com/matrix-org/util/json.go:210: init 100.0%
github.com/matrix-org/util/unique.go:13: Unique 91.7%
github.com/matrix-org/util/unique.go:48: SortAndUnique 100.0%
github.com/matrix-org/util/unique.go:55: UniqueStrings 100.0%
total: (statements) 53.7%
github.com/matrix-org/util/json.go:132: MakeJSONAPI 70.0%
github.com/matrix-org/util/json.go:151: respond 84.6%
github.com/matrix-org/util/json.go:180: WithCORSOptions 0.0%
github.com/matrix-org/util/json.go:191: SetCORSHeaders 100.0%
github.com/matrix-org/util/json.go:202: RandomString 100.0%
github.com/matrix-org/util/json.go:210: init 100.0%
github.com/matrix-org/util/unique.go:13: Unique 91.7%
github.com/matrix-org/util/unique.go:48: SortAndUnique 100.0%
github.com/matrix-org/util/unique.go:55: UniqueStrings 100.0%
total (statements) 64.0%
```
(after running Sytest for Postgres _and_ SQLite)
The total coverage for this run is the last line at the bottom. However, this value is misleading because Dendrite can run in many different configurations,
which will never be tested in a single test run (e.g sqlite or postgres). To get a more accurate value, additional processing is required
to remove packages which will never be tested and extension MSCs:
(The following commands use [gocovmerge](https://github.com/wadey/gocovmerge), to merge two or more coverage logs, so make sure you have that installed)
The total coverage for this run is the last line at the bottom. However, this value is misleading because Dendrite can run in different configurations,
which will never be tested in a single test run (e.g sqlite or postgres). To get a more accurate value, you'll need run Sytest for Postgres and SQLite (see commands above).
Additional processing is required also to remove packages which will never be tested and extension MSCs:
```bash
# These commands are all similar but change which package paths are _removed_ from the output.
# If you executed both commands from above, you can get the total coverage using the following commands
go tool covdata textfmt -i="$(find -name 'covmeta*' -type f -exec dirname {} \; | uniq | paste -s -d ',' -)" -o sytest.cov
grep -Ev 'relayapi|setup/mscs' sytest.cov > final.cov
go tool cover -func=final.cov
# For Postgres
find -name 'integrationcover.log' | xargs gocovmerge | grep -Ev 'relayapi|sqlite|setup/mscs' > final.cov && go tool cover -func=final.cov
# If you only executed the one for Postgres:
go tool covdata textfmt -i="$(find -name 'covmeta*' -type f -exec dirname {} \; | uniq | paste -s -d ',' -)" -o sytest.cov
grep -Ev 'relayapi|sqlite|setup/mscs' sytest.cov > final.cov
go tool cover -func=final.cov
# For SQLite
find -name 'integrationcover.log' | xargs gocovmerge | grep -Ev 'relayapi|postgres|setup/mscs' > final.cov && go tool cover -func=final.cov
```
## Running unit tests with coverage enabled
Running unit tests with coverage enabled can be done with the following command, it will also generate a `integrationcover.log`, which can later be merged
using `gocovmerge`:
```bash
go test -covermode=atomic -coverpkg=./... -coverprofile=integrationcover.log $(go list ./... | grep -v '/cmd/')
# If you only executed the one for SQLite:
go tool covdata textfmt -i="$(find -name 'covmeta*' -type f -exec dirname {} \; | uniq | paste -s -d ',' -)" -o sytest.cov
grep -Ev 'relayapi|postgres|setup/mscs' sytest.cov > final.cov
go tool cover -func=final.cov
```
## Getting coverage from Complement
@ -87,10 +97,10 @@ and add the following content:
#!/bin/bash
mkdir -p /tmp/Complement/logs/$2/$1/
docker cp $1:/dendrite/integrationcover.log /tmp/Complement/logs/$2/$1/
docker cp $1:/tmp/covdatafiles/. /tmp/Complement/logs/$2/$1/
```
This will copy the `integrationcover.log` files from each container to something like
`/tmp/Complement/logs/TestLogin/94f9c428de95779d2b62a3ccd8eab9d5ddcf65cc259a40ece06bdc61687ffed3/integrationcover.log`. (`$1` is the containerID, `$2` the test name)
This will copy the `covdatafiles` files from each container to something like
`/tmp/Complement/logs/TestLogin/94f9c428de95779d2b62a3ccd8eab9d5ddcf65cc259a40ece06bdc61687ffed3/`. (`$1` is the containerID, `$2` the test name)
Now that we have set up everything we need, we can finally execute Complement:
```bash
@ -101,15 +111,20 @@ COMPLEMENT_POST_TEST_SCRIPT=$(pwd)/posttest.sh \
go test -tags dendrite_blacklist ./tests/... -count=1 -v -timeout=30m -failfast=false
```
Once this is done, you can copy the resulting `integrationcover.log` files to your Dendrite repository for the next step.
Once this is done, you can copy the resulting `covdatafiles` files to your Dendrite repository for the next step.
```bash
cp -pr /tmp/Complement/logs PathToYourDendriteRepository
```
## Combining the results of all runs
Now that we have all our `integrationcover.log` files within the Dendrite repository, you can now execute the following command, to get the coverage
overall (Note: update `grep -Ev` accordingly, to exclude SQLite for example):
You can also run the following to get the coverage for Complement runs alone:
```bash
find -name 'integrationcover.log' | xargs gocovmerge | grep -Ev 'setup/mscs|api_trace' > final.cov && go tool cover -func=final.cov
go tool covdata func -i="$(find /tmp/Complement -name 'covmeta*' -type f -exec dirname {} \; | uniq | paste -s -d ',' -)"
```
## Combining the results of (almost) all runs
Now that we have all our `covdatafiles` files within the Dendrite repository, you can now execute the following command, to get the coverage
overall (excluding unit tests):
```bash
go tool covdata func -i="$(find -name 'covmeta*' -type f -exec dirname {} \; | uniq | paste -s -d ',' -)"
```