From d35985170884e619508572465117be41bd5d45ae Mon Sep 17 00:00:00 2001 From: Alex Chen Date: Sun, 12 Apr 2020 20:48:24 +0800 Subject: [PATCH] Update guides for running sytest (#958) * Update guides for running sytest Bring the details for running sytest against postgres up to date; prefer docker over manual setup Signed-off-by: Alex Chen * Better flags Signed-off-by: Alex Chen --- docs/sytest.md | 69 ++++++++++++++++++++++++++++++++++---------------- 1 file changed, 47 insertions(+), 22 deletions(-) diff --git a/docs/sytest.md b/docs/sytest.md index 9385ebff3..26a46c0bb 100644 --- a/docs/sytest.md +++ b/docs/sytest.md @@ -10,9 +10,9 @@ passes. ## Finding out which tests to add -We recommend you run the tests locally by manually setting up SyTest or using a -SyTest docker image. After running the tests, a script will print the tests you -need to add to `sytest-whitelist`. +We recommend you run the tests locally by using the SyTest docker image or +manually setting up SyTest. After running the tests, a script will print the +tests you need to add to `sytest-whitelist`. You should proceed after you see no build problems for dendrite after running: @@ -20,9 +20,32 @@ You should proceed after you see no build problems for dendrite after running: ./build.sh ``` +### Using the SyTest Docker image + +Use the following commands to pull the latest SyTest image and run the tests: + +```sh +docker pull matrixdotorg/sytest-dendrite +docker run --rm -v /path/to/dendrite/:/src/ -v /path/to/log/output/:/logs/ matrixdotorg/sytest-dendrite +``` + +`/path/to/dendrite/` should be replaced with the actual path to your dendrite +source code. The test results TAP file and homeserver logging output will go to +`/path/to/log/output`. The output of the command should tell you if you need to +add any tests to `sytest-whitelist`. + +When debugging, the following Docker `run` options may also be useful: +* `-v /path/to/sytest/:/sytest/`: Use your local SyTest repository at + `/path/to/sytest` instead of pulling from GitHub. This is useful when you want + to speed things up or make modifications to SyTest. +* `--entrypoint bash`: Prevent the container from automatically starting the + tests. When used, you need to manually run `/bootstrap.sh dendrite` inside + the container to start them. + ### Manually Setting up SyTest -Make sure you have Perl v5+ installed, and get SyTest with: +If you don't want to use the Docker image, you can also run SyTest by hand. Make +sure you have Perl 5 or above, and get SyTest with: (Note that this guide assumes your SyTest checkout is next to your `dendrite` checkout.) @@ -37,12 +60,23 @@ Set up the database: ```sh sudo -u postgres psql -c "CREATE USER dendrite PASSWORD 'itsasecret'" -sudo -u postgres psql -c "CREATE DATABASE sytest_template OWNER dendrite" +for i in dendrite0 dendrite1 sytest_template; do sudo -u postgres psql -c "CREATE DATABASE $i OWNER dendrite;"; done mkdir -p "server-0" cat > "server-0/database.yaml" << EOF args: user: dendrite - database: dendrite + password: itsasecret + database: dendrite0 + host: 127.0.0.1 + sslmode: disable +type: pg +EOF +mkdir -p "server-1" +cat > "server-1/database.yaml" << EOF +args: + user: dendrite + password: itsasecret + database: dendrite1 host: 127.0.0.1 sslmode: disable type: pg @@ -52,29 +86,20 @@ EOF Run the tests: ```sh -./run-tests.pl -I Dendrite::Monolith -d ../dendrite/bin -W ../dendrite/sytest-whitelist -O tap --all | tee results.tap +POSTGRES=1 ./run-tests.pl -I Dendrite::Monolith -d ../dendrite/bin -W ../dendrite/sytest-whitelist -O tap --all | tee results.tap ``` -where `tee` lets you see the results while they're being piped to the file. +where `tee` lets you see the results while they're being piped to the file, and +`POSTGRES=1` enables testing with PostgeSQL. If the `POSTGRES` environment +variable is not set or is set to 0, SyTest will fall back to SQLite 3. For more +flags and options, see https://github.com/matrix-org/sytest#running. Once the tests are complete, run the helper script to see if you need to add -any newly passing test names to `sytest-whitelist` in the project's root directory: +any newly passing test names to `sytest-whitelist` in the project's root +directory: ```sh ../dendrite/show-expected-fail-tests.sh results.tap ../dendrite/sytest-whitelist ../dendrite/sytest-blacklist ``` If the script prints nothing/exits with 0, then you're good to go. - -### Using a SyTest Docker image - -Ensure you have the latest image for SyTest, then run the tests: - -```sh -docker pull matrixdotorg/sytest-dendrite -docker run --rm -v /path/to/dendrite/:/src/ matrixdotorg/sytest-dendrite -``` - -where `/path/to/dendrite/` should be replaced with the actual path to your -dendrite source code. The output should tell you if you need to add any tests to -`sytest-whitelist`.