From adaec7c8e74ca877f125c56e3a659382abbb185d Mon Sep 17 00:00:00 2001 From: Alex Chen Date: Fri, 21 Jun 2019 23:49:37 +0800 Subject: [PATCH] Refine config and docs for sytest (#714) This PR adds the "passed but expected fail" tests to testfile, as well as instructions on finding which tests to add for new PRs. --- docs/sytest.md | 78 +++++++++++++++++++- show-expected-fail-tests.sh | 29 ++++++++ testfile | 142 +++++++++++++++++++++++++++++++++++- 3 files changed, 244 insertions(+), 5 deletions(-) create mode 100755 show-expected-fail-tests.sh diff --git a/docs/sytest.md b/docs/sytest.md index a0f7d85c1..99987fea5 100644 --- a/docs/sytest.md +++ b/docs/sytest.md @@ -1,8 +1,78 @@ # SyTest Dendrite uses [SyTest](https://github.com/matrix-org/sytest) for its -integration testing. When creating a new PR, add the test IDs that your PR -should allow to pass to `testfile` in dendrite's root directory. Not all PRs -need to make new tests pass. If we find your PR should be making a test pass we -may ask you to add to that file, as generally Dendrite's progress can be +integration testing. When creating a new PR, add the test IDs (see below) that +your PR should allow to pass to `testfile` in dendrite's root directory. Not all +PRs need to make new tests pass. If we find your PR should be making a test pass +we may ask you to add to that file, as generally Dendrite's progress can be tracked through the amount of SyTest tests it passes. + +## Finding out which tests to add + +We recommend you run the tests locally by manually setting up SyTest or using a +SyTest docker image. After running the tests, a script will print the tests you +need to add to `testfile` for you. + +You should proceed after you see no build problems for dendrite after running: + +```sh +./build.sh +``` + +### Manually Setting up SyTest + +Make sure you have Perl v5+ installed, and get SyTest with: + +(Note that this guide assumes your SyTest checkout is next to your +`dendrite` checkout.) + +```sh +git clone -b develop https://github.com/matrix-org/sytest +cd sytest +./install-deps.pl +``` + +Set up the database: + +```sh +sudo -u postgres psql -c "CREATE USER dendrite PASSWORD 'itsasecret'" +sudo -u postgres psql -c "CREATE DATABASE sytest_template OWNER dendrite" +mkdir -p "server-0" +cat > "server-0/database.yaml" << EOF +args: + user: dendrite + database: dendrite + host: 127.0.0.1 +type: pg +EOF +``` + +Run the tests: + +```sh +./run-tests.pl -I Dendrite::Monolith -d ../dendrite/bin -W ../dendrite/testfile -O tap --all | tee results.tap +``` + +where `tee` lets you see the results while they're being piped to the file. + +Once the tests are complete, run the helper script to see if you need to add +any newly passing test names to `testfile` in the project's root directory: + +```sh +../dendrite/show-expected-fail-tests.sh results.tap +``` + +If the script prints nothing/exits with 0, then you're good to go. + +### Using a SyTest Docker image + +Ensure you have the latest image for SyTest, then run the tests: + +```sh +docker pull matrixdotorg/sytest-dendrite +docker run --rm -v /path/to/dendrite/:/src/ matrixdotorg/sytest-dendrite +``` + +where `/path/to/dendrite/` should be replaced with the actual path to your +dendrite source code. The output should tell you if you need to add any tests to +`testfile`. diff --git a/show-expected-fail-tests.sh b/show-expected-fail-tests.sh new file mode 100755 index 000000000..277a2c8b9 --- /dev/null +++ b/show-expected-fail-tests.sh @@ -0,0 +1,29 @@ +#! /bin/bash + +results_file=$1 +passed_but_expected_fail=$(grep ' # TODO passed but expected fail' ${results_file} | sed -E 's/^ok [0-9]+ (\(expected fail\) )?//' | sed -E 's/( \([0-9]+ subtests\))? # TODO passed but expected fail$//') +tests_to_add="" +already_in_testfile="" + +fail_build=0 +while read -r test_id; do + grep "${test_id}" testfile > /dev/null 2>&1 + if [ "$?" != "0" ]; then + tests_to_add="${tests_to_add}${test_id}\n" + fail_build=1 + else + already_in_testfile="${already_in_testfile}${test_id}\n" + fi +done <<< "${passed_but_expected_fail}" + +if [ -n "${tests_to_add}" ]; then + echo "ERROR: The following passed tests are not present in testfile. Please append them to the file:" + echo -e "${tests_to_add}" +fi + +if [ -n "${already_in_testfile}" ]; then + echo "WARN: Tests in testfile still marked as expected fail:" + echo -e "${already_in_testfile}" +fi + +exit ${fail_build} diff --git a/testfile b/testfile index d90a9458c..ea6fc9172 100644 --- a/testfile +++ b/testfile @@ -1,4 +1,144 @@ GET /register yields a set of flows POST /register can create a user POST /register downcases capitals in usernames - +POST /register rejects registration of usernames with '!' +POST /register rejects registration of usernames with '"' +POST /register rejects registration of usernames with ':' +POST /register rejects registration of usernames with '?' +POST /register rejects registration of usernames with '\' +POST /register rejects registration of usernames with '@' +POST /register rejects registration of usernames with '[' +POST /register rejects registration of usernames with ']' +POST /register rejects registration of usernames with '{' +POST /register rejects registration of usernames with '|' +POST /register rejects registration of usernames with '}' +POST /register rejects registration of usernames with '£' +POST /register rejects registration of usernames with 'é' +POST /register rejects registration of usernames with '\n' +POST /register rejects registration of usernames with ''' +GET /login yields a set of flows +POST /login can log in as a user +POST /login can log in as a user with just the local part of the id +POST /login as non-existing user is rejected +POST /login wrong password is rejected +GET /events initially +GET /initialSync initially +Version responds 200 OK with valid structure +PUT /profile/:user_id/displayname sets my name +GET /profile/:user_id/displayname publicly accessible +PUT /profile/:user_id/avatar_url sets my avatar +GET /profile/:user_id/avatar_url publicly accessible +GET /device/{deviceId} gives a 404 for unknown devices +PUT /device/{deviceId} gives a 404 for unknown devices +POST /createRoom makes a public room +POST /createRoom makes a private room +POST /createRoom makes a private room with invites +POST /createRoom makes a room with a name +POST /createRoom makes a room with a topic +Can /sync newly created room +GET /rooms/:room_id/state/m.room.member/:user_id fetches my membership +GET /rooms/:room_id/state/m.room.power_levels fetches powerlevels +POST /join/:room_alias can join a room +POST /join/:room_id can join a room +POST /join/:room_id can join a room with custom content +POST /join/:room_alias can join a room with custom content +POST /rooms/:room_id/leave can leave a room +POST /rooms/:room_id/invite can send an invite +POST /rooms/:room_id/ban can ban a user +POST /rooms/:room_id/send/:event_type sends a message +PUT /rooms/:room_id/send/:event_type/:txn_id sends a message +PUT /rooms/:room_id/send/:event_type/:txn_id deduplicates the same txn id +GET /rooms/:room_id/state/m.room.power_levels can fetch levels +PUT /rooms/:room_id/state/m.room.power_levels can set levels +PUT power_levels should not explode if the old power levels were empty +Both GET and PUT work +POST /rooms/:room_id/read_markers can create read marker +User signups are forbidden from starting with '_' +Can logout all devices +Request to logout with invalid an access token is rejected +Request to logout without an access token is rejected +Room creation reports m.room.create to myself +Room creation reports m.room.member to myself +New room members see their own join event +Existing members see new members' join events +setting 'm.room.power_levels' respects room powerlevel +Unprivileged users can set m.room.topic if it only needs level 0 +Users cannot set ban powerlevel higher than their own +Users cannot set kick powerlevel higher than their own +Users cannot set redact powerlevel higher than their own +Can get rooms/{roomId}/members for a departed room (SPEC-216) +3pid invite join with wrong but valid signature are rejected +3pid invite join valid signature but revoked keys are rejected +3pid invite join valid signature but unreachable ID server are rejected +Room members can override their displayname on a room-specific basis +Room members can join a room with an overridden displayname +displayname updates affect room member events +avatar_url updates affect room member events +Real non-joined user cannot call /events on shared room +Real non-joined user cannot call /events on invited room +Real non-joined user cannot call /events on joined room +Real non-joined user cannot call /events on default room +Real non-joined users can get state for world_readable rooms +Real non-joined users can get individual state for world_readable rooms +Real non-joined users can get individual state for world_readable rooms after leaving +Real non-joined users cannot send messages to guest_access rooms if not joined +Real users can sync from world_readable guest_access rooms if joined +Real users can sync from default guest_access rooms if joined +Can't forget room you're still in +Can get rooms/{roomId}/members +Can create filter +Can download filter +Can sync +Can sync a joined room +Newly joined room is included in an incremental sync +User is offline if they set_presence=offline in their sync +Changes to state are included in an incremental sync +A change to displayname should appear in incremental /sync +Current state appears in timeline in private history +Current state appears in timeline in private history with many messages before +Rooms a user is invited to appear in an initial sync +Rooms a user is invited to appear in an incremental sync +Sync can be polled for updates +Sync is woken up for leaves +Newly left rooms appear in the leave section of incremental sync +We should see our own leave event, even if history_visibility is restricted (SYN-662) +We should see our own leave event when rejecting an invite, even if history_visibility is restricted (riot-web/3462) +Newly left rooms appear in the leave section of gapped sync +Previously left rooms don't appear in the leave section of sync +Left rooms appear in the leave section of full state sync +Newly banned rooms appear in the leave section of incremental sync +Newly banned rooms appear in the leave section of incremental sync +local user can join room with version 1 +User can invite local user to room with version 1 +local user can join room with version 2 +User can invite local user to room with version 2 +local user can join room with version 3 +User can invite local user to room with version 3 +local user can join room with version 4 +User can invite local user to room with version 4 +Should reject keys claiming to belong to a different user +Can add account data +Can add account data to room +Latest account data appears in v2 /sync +New account data appears in incremental v2 /sync +Checking local federation server +Inbound federation can query profile data +Outbound federation can send room-join requests +Outbound federation can send events +Inbound federation can backfill events +Backfill checks the events requested belong to the room +Can upload without a file name +Can download without a file name locally +Can upload with ASCII file name +Can send image in room message +AS cannot create users outside its own namespace +Regular users cannot register within the AS namespace +AS can't set displayname for random users +AS user (not ghost) can join room without registering, with user_id query param +Changing the actions of an unknown default rule fails with 404 +Changing the actions of an unknown rule fails with 404 +Enabling an unknown default rule fails with 404 +Trying to get push rules with unknown rule_id fails with 404 +Events come down the correct room +local user can join room with version 5 +User can invite local user to room with version 5