Refine config and docs for sytest (#714)

This PR adds the "passed but expected fail" tests to testfile, as well as instructions on finding which tests to add for new PRs.

docs/sytest.md

@@ -1,8 +1,78 @@
 # SyTest
 
 Dendrite uses [SyTest](https://github.com/matrix-org/sytest) for its
-integration testing. When creating a new PR, add the test IDs that your PR
+integration testing. When creating a new PR, add the test IDs (see below) that
-should allow to pass to `testfile` in dendrite's root directory. Not all PRs
+your PR should allow to pass to `testfile` in dendrite's root directory. Not all
-need to make new tests pass. If we find your PR should be making a test pass we
+PRs need to make new tests pass. If we find your PR should be making a test pass
-may ask you to add to that file, as generally Dendrite's progress can be
+we may ask you to add to that file, as generally Dendrite's progress can be
 tracked through the amount of SyTest tests it passes.
+
+## Finding out which tests to add
+
+We recommend you run the tests locally by manually setting up SyTest or using a
+SyTest docker image. After running the tests, a script will print the tests you
+need to add to `testfile` for you.
+
+You should proceed after you see no build problems for dendrite after running:
+
+```sh
+./build.sh
+```
+
+### Manually Setting up SyTest
+
+Make sure you have Perl v5+ installed, and get SyTest with:
+
+(Note that this guide assumes your SyTest checkout is next to your
+`dendrite` checkout.)
+
+```sh
+git clone -b develop https://github.com/matrix-org/sytest
+cd sytest
+./install-deps.pl
+```
+
+Set up the database:
+
+```sh
+sudo -u postgres psql -c "CREATE USER dendrite PASSWORD 'itsasecret'"
+sudo -u postgres psql -c "CREATE DATABASE sytest_template OWNER dendrite"
+mkdir -p "server-0"
+cat > "server-0/database.yaml" << EOF
+args:
+    user: dendrite
+    database: dendrite
+    host: 127.0.0.1
+type: pg
+EOF
+```
+
+Run the tests:
+
+```sh
+./run-tests.pl -I Dendrite::Monolith -d ../dendrite/bin -W ../dendrite/testfile -O tap --all | tee results.tap
+```
+
+where `tee` lets you see the results while they're being piped to the file.
+
+Once the tests are complete, run the helper script to see if you need to add
+any newly passing test names to `testfile` in the project's root directory:
+
+```sh
+../dendrite/show-expected-fail-tests.sh results.tap
+```
+
+If the script prints nothing/exits with 0, then you're good to go.
+
+### Using a SyTest Docker image
+
+Ensure you have the latest image for SyTest, then run the tests:
+
+```sh
+docker pull matrixdotorg/sytest-dendrite
+docker run --rm -v /path/to/dendrite/:/src/ matrixdotorg/sytest-dendrite
+```
+
+where `/path/to/dendrite/` should be replaced with the actual path to your
+dendrite source code. The output should tell you if you need to add any tests to
+`testfile`.

show-expected-fail-tests.sh

@@ -0,0 +1,29 @@
+#! /bin/bash
+
+results_file=$1
+passed_but_expected_fail=$(grep ' # TODO passed but expected fail' ${results_file} | sed -E 's/^ok [0-9]+ (\(expected fail\) )?//' | sed -E 's/( \([0-9]+ subtests\))? # TODO passed but expected fail$//')
+tests_to_add=""
+already_in_testfile=""
+
+fail_build=0
+while read -r test_id; do
+	grep "${test_id}" testfile > /dev/null 2>&1
+	if [ "$?" != "0" ]; then
+		tests_to_add="${tests_to_add}${test_id}\n"
+		fail_build=1
+	else
+		already_in_testfile="${already_in_testfile}${test_id}\n"
+	fi
+done <<< "${passed_but_expected_fail}"
+
+if [ -n "${tests_to_add}" ]; then
+	echo "ERROR: The following passed tests are not present in testfile. Please append them to the file:"
+	echo -e "${tests_to_add}"
+fi
+
+if [ -n "${already_in_testfile}" ]; then
+	echo "WARN: Tests in testfile still marked as expected fail:"
+	echo -e "${already_in_testfile}"
+fi
+
+exit ${fail_build}

testfile

@@ -1,4 +1,144 @@
 GET /register yields a set of flows
 POST /register can create a user
 POST /register downcases capitals in usernames
-
+POST /register rejects registration of usernames with '!'
+POST /register rejects registration of usernames with '"'
+POST /register rejects registration of usernames with ':'
+POST /register rejects registration of usernames with '?'
+POST /register rejects registration of usernames with '\'
+POST /register rejects registration of usernames with '@'
+POST /register rejects registration of usernames with '['
+POST /register rejects registration of usernames with ']'
+POST /register rejects registration of usernames with '{'
+POST /register rejects registration of usernames with '|'
+POST /register rejects registration of usernames with '}'
+POST /register rejects registration of usernames with '£'
+POST /register rejects registration of usernames with 'é'
+POST /register rejects registration of usernames with '\n'
+POST /register rejects registration of usernames with '''
+GET /login yields a set of flows
+POST /login can log in as a user
+POST /login can log in as a user with just the local part of the id
+POST /login as non-existing user is rejected
+POST /login wrong password is rejected
+GET /events initially
+GET /initialSync initially
+Version responds 200 OK with valid structure
+PUT /profile/:user_id/displayname sets my name
+GET /profile/:user_id/displayname publicly accessible
+PUT /profile/:user_id/avatar_url sets my avatar
+GET /profile/:user_id/avatar_url publicly accessible
+GET /device/{deviceId} gives a 404 for unknown devices
+PUT /device/{deviceId} gives a 404 for unknown devices
+POST /createRoom makes a public room
+POST /createRoom makes a private room
+POST /createRoom makes a private room with invites
+POST /createRoom makes a room with a name
+POST /createRoom makes a room with a topic
+Can /sync newly created room
+GET /rooms/:room_id/state/m.room.member/:user_id fetches my membership
+GET /rooms/:room_id/state/m.room.power_levels fetches powerlevels
+POST /join/:room_alias can join a room
+POST /join/:room_id can join a room
+POST /join/:room_id can join a room with custom content
+POST /join/:room_alias can join a room with custom content
+POST /rooms/:room_id/leave can leave a room
+POST /rooms/:room_id/invite can send an invite
+POST /rooms/:room_id/ban can ban a user
+POST /rooms/:room_id/send/:event_type sends a message
+PUT /rooms/:room_id/send/:event_type/:txn_id sends a message
+PUT /rooms/:room_id/send/:event_type/:txn_id deduplicates the same txn id
+GET /rooms/:room_id/state/m.room.power_levels can fetch levels
+PUT /rooms/:room_id/state/m.room.power_levels can set levels
+PUT power_levels should not explode if the old power levels were empty
+Both GET and PUT work
+POST /rooms/:room_id/read_markers can create read marker
+User signups are forbidden from starting with '_'
+Can logout all devices
+Request to logout with invalid an access token is rejected
+Request to logout without an access token is rejected
+Room creation reports m.room.create to myself
+Room creation reports m.room.member to myself
+New room members see their own join event
+Existing members see new members' join events
+setting 'm.room.power_levels' respects room powerlevel
+Unprivileged users can set m.room.topic if it only needs level 0
+Users cannot set ban powerlevel higher than their own
+Users cannot set kick powerlevel higher than their own
+Users cannot set redact powerlevel higher than their own
+Can get rooms/{roomId}/members for a departed room (SPEC-216)
+3pid invite join with wrong but valid signature are rejected
+3pid invite join valid signature but revoked keys are rejected
+3pid invite join valid signature but unreachable ID server are rejected
+Room members can override their displayname on a room-specific basis
+Room members can join a room with an overridden displayname
+displayname updates affect room member events
+avatar_url updates affect room member events
+Real non-joined user cannot call /events on shared room
+Real non-joined user cannot call /events on invited room
+Real non-joined user cannot call /events on joined room
+Real non-joined user cannot call /events on default room
+Real non-joined users can get state for world_readable rooms
+Real non-joined users can get individual state for world_readable rooms
+Real non-joined users can get individual state for world_readable rooms after leaving
+Real non-joined users cannot send messages to guest_access rooms if not joined
+Real users can sync from world_readable guest_access rooms if joined
+Real users can sync from default guest_access rooms if joined
+Can't forget room you're still in
+Can get rooms/{roomId}/members
+Can create filter
+Can download filter
+Can sync
+Can sync a joined room
+Newly joined room is included in an incremental sync
+User is offline if they set_presence=offline in their sync
+Changes to state are included in an incremental sync
+A change to displayname should appear in incremental /sync
+Current state appears in timeline in private history
+Current state appears in timeline in private history with many messages before
+Rooms a user is invited to appear in an initial sync
+Rooms a user is invited to appear in an incremental sync
+Sync can be polled for updates
+Sync is woken up for leaves
+Newly left rooms appear in the leave section of incremental sync
+We should see our own leave event, even if history_visibility is restricted (SYN-662)
+We should see our own leave event when rejecting an invite, even if history_visibility is restricted (riot-web/3462)
+Newly left rooms appear in the leave section of gapped sync
+Previously left rooms don't appear in the leave section of sync
+Left rooms appear in the leave section of full state sync
+Newly banned rooms appear in the leave section of incremental sync
+Newly banned rooms appear in the leave section of incremental sync
+local user can join room with version 1
+User can invite local user to room with version 1
+local user can join room with version 2
+User can invite local user to room with version 2
+local user can join room with version 3
+User can invite local user to room with version 3
+local user can join room with version 4
+User can invite local user to room with version 4
+Should reject keys claiming to belong to a different user
+Can add account data
+Can add account data to room
+Latest account data appears in v2 /sync
+New account data appears in incremental v2 /sync
+Checking local federation server
+Inbound federation can query profile data
+Outbound federation can send room-join requests
+Outbound federation can send events
+Inbound federation can backfill events
+Backfill checks the events requested belong to the room
+Can upload without a file name
+Can download without a file name locally
+Can upload with ASCII file name
+Can send image in room message
+AS cannot create users outside its own namespace
+Regular users cannot register within the AS namespace
+AS can't set displayname for random users
+AS user (not ghost) can join room without registering, with user_id query param
+Changing the actions of an unknown default rule fails with 404
+Changing the actions of an unknown rule fails with 404
+Enabling an unknown default rule fails with 404
+Trying to get push rules with unknown rule_id fails with 404
+Events come down the correct room
+local user can join room with version 5
+User can invite local user to room with version 5