From 3245ede1252f746f236bb7a95a83606e3c733649 Mon Sep 17 00:00:00 2001
From: Cnly <minecnly@gmail.com>
Date: Wed, 19 Jun 2019 22:29:30 +0800
Subject: [PATCH] Refine test guides in docs/sytest.md

Signed-off-by: Alex Chen <minecnly@gmail.com>
---
 docs/sytest.md | 73 +++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 54 insertions(+), 19 deletions(-)

diff --git a/docs/sytest.md b/docs/sytest.md
index 4ff93ce63..f910dd833 100644
--- a/docs/sytest.md
+++ b/docs/sytest.md
@@ -9,33 +9,68 @@ tracked through the amount of SyTest tests it passes.
 
 ## Finding out which tests to add
 
-An easy way to do this will be let CircleCI run sytest, and then search for the
-tests that should be added in the test results file.
+We recommend you run the tests locally by manually setting up sytest or using a
+sytest docker image. After running the tests, a script will print the tests you
+need to add to `testfile` for you.
 
-You may enable [CircleCI](https://circleci.com/) for your own fork, or simply
-refer to the CircleCI build for your PR. Once the build is complete, you can
-find the test results in `Container N/logs/results.tap` under the **Artifacts**
-tab on the CircleCI build page. To get the tests that should be added, search
-for the string:
+You should proceed after you see no build problems for dendrite by running:
 
-```
-# TODO passed but expected fail
+```sh
+./build.sh
 ```
 
-Generally, tests with this label should be the tests that your PR is allowing to
-pass.
+### Manually Setting up sytest
 
-To add a test into `testfile`, extract its test ID and append it to `testfile`.
-For example, to add this test:
+Get a copy of sytest on the `develop` branch next to your dendrite
+directory, and run the setup script:
 
-```
-ok 4 (expected fail) POST /register rejects registration of usernames with '!' # TODO passed but expected fail
+```sh
+git clone -b develop https://github.com/matrix-org/sytest
+cd sytest
+./install-deps.pl
 ```
 
-Append:
+Set up the database:
 
-```
-POST /register rejects registration of usernames with '!'
+```sh
+sudo -u postgres psql -c "CREATE USER dendrite PASSWORD 'itsasecret'"
+sudo -u postgres psql -c "CREATE DATABASE sytest_template OWNER dendrite"
+mkdir -p "server-0"
+cat > "server-0/database.yaml" << EOF
+args:
+    user: $PGUSER
+    database: $PGUSER
+    host: $PGHOST
+type: pg
+EOF
 ```
 
-to `testfile`.
+Run the tests:
+
+```sh
+./run-tests.pl -I Dendrite::Monolith -d ../dendrite/bin -W ../dendrite/testfile -O tap --all | tee results.tap
+```
+
+where `tee` lets you see the results while they're being piped to the file.
+
+Once the tests are complete, run the helper script to see if you need to add
+anything, and add them as needed:
+
+```sh
+../dendrite/show-expected-fail-tests.sh results.tap
+```
+
+If the script prints nothing/exits with 0, then you're good to go.
+
+### Using a sytest Docker Image
+
+Ensure you have the latest image for sytest, then run the tests:
+
+```sh
+docker pull matrixdotorg/sytest-dendrite
+docker run --rm -v /path/to/dendrite/:/src/ matrixdotorg/sytest-dendrite
+```
+
+where `/path/to/dendrite/` should be replaced with the actual path to your
+dendrite source code. The output should tell you if you need to add any tests to
+`testfile`.