sigb.us/dendrite
1
0
Fork 0
Code Issues Pull requests Actions Packages Projects Releases 1 Wiki Activity

Refine config and docs for sytest (#714)

Browse source 
This PR adds the "passed but expected fail" tests to testfile, as well as instructions on finding which tests to add for new PRs.
This commit is contained in:
Alex Chen 2019-06-21 23:49:37 +08:00 committed by Andrew Morgan
parent bc382bba46
commit adaec7c8e7
3 changed files with 244 additions and 5 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

78
docs/sytest.md
View file

@ -1,8 +1,78 @@
# SyTest # SyTest
Dendrite uses [SyTest](https://github.com/matrix-org/sytest) for its Dendrite uses [SyTest](https://github.com/matrix-org/sytest) for its
integration testing. When creating a new PR, add the test IDs that your PR integration testing. When creating a new PR, add the test IDs (see below) that
should allow to pass to `testfile` in dendrite's root directory. Not all PRs your PR should allow to pass to `testfile` in dendrite's root directory. Not all
need to make new tests pass. If we find your PR should be making a test pass we PRs need to make new tests pass. If we find your PR should be making a test pass
may ask you to add to that file, as generally Dendrite's progress can be we may ask you to add to that file, as generally Dendrite's progress can be
tracked through the amount of SyTest tests it passes. tracked through the amount of SyTest tests it passes.
## Finding out which tests to add
We recommend you run the tests locally by manually setting up SyTest or using a
SyTest docker image. After running the tests, a script will print the tests you
need to add to `testfile` for you.
You should proceed after you see no build problems for dendrite after running:
```sh
./build.sh
```
### Manually Setting up SyTest
Make sure you have Perl v5+ installed, and get SyTest with:
(Note that this guide assumes your SyTest checkout is next to your
`dendrite` checkout.)
```sh
git clone -b develop https://github.com/matrix-org/sytest
cd sytest
./install-deps.pl
```
Set up the database:
```sh
sudo -u postgres psql -c "CREATE USER dendrite PASSWORD 'itsasecret'"
sudo -u postgres psql -c "CREATE DATABASE sytest_template OWNER dendrite"
mkdir -p "server-0"
cat > "server-0/database.yaml" << EOF
args:
user: dendrite
database: dendrite
host: 127.0.0.1
type: pg
EOF
```
Run the tests:
```sh
./run-tests.pl -I Dendrite::Monolith -d ../dendrite/bin -W ../dendrite/testfile -O tap --all | tee results.tap
```
where `tee` lets you see the results while they're being piped to the file.
Once the tests are complete, run the helper script to see if you need to add
any newly passing test names to `testfile` in the project's root directory:
```sh
../dendrite/show-expected-fail-tests.sh results.tap
```
If the script prints nothing/exits with 0, then you're good to go.
### Using a SyTest Docker image
Ensure you have the latest image for SyTest, then run the tests:
```sh
docker pull matrixdotorg/sytest-dendrite
docker run --rm -v /path/to/dendrite/:/src/ matrixdotorg/sytest-dendrite
```
where `/path/to/dendrite/` should be replaced with the actual path to your
dendrite source code. The output should tell you if you need to add any tests to
`testfile`.

29
show-expected-fail-tests.sh Executable file
View file

@ -0,0 +1,29 @@
#! /bin/bash
results_file=$1
passed_but_expected_fail=$(grep ' # TODO passed but expected fail' ${results_file} | sed -E 's/^ok [0-9]+ (\(expected fail\) )?//' | sed -E 's/( \([0-9]+ subtests\))? # TODO passed but expected fail$//')
tests_to_add=""
already_in_testfile=""
fail_build=0
while read -r test_id; do
grep "${test_id}" testfile > /dev/null 2>&1
if [ "$?" != "0" ]; then
tests_to_add="${tests_to_add}${test_id}\n"
fail_build=1
else
already_in_testfile="${already_in_testfile}${test_id}\n"
fi
done <<< "${passed_but_expected_fail}"
if [ -n "${tests_to_add}" ]; then
echo "ERROR: The following passed tests are not present in testfile. Please append them to the file:"
echo -e "${tests_to_add}"
fi
if [ -n "${already_in_testfile}" ]; then
echo "WARN: Tests in testfile still marked as expected fail:"
echo -e "${already_in_testfile}"
fi
exit ${fail_build}

142
testfile
View file

@ -1,4 +1,144 @@
GET /register yields a set of flows GET /register yields a set of flows
POST /register can create a user POST /register can create a user
POST /register downcases capitals in usernames POST /register downcases capitals in usernames
POST /register rejects registration of usernames with '!'
POST /register rejects registration of usernames with '"'
POST /register rejects registration of usernames with ':'
POST /register rejects registration of usernames with '?'
POST /register rejects registration of usernames with '\'
POST /register rejects registration of usernames with '@'
POST /register rejects registration of usernames with '['
POST /register rejects registration of usernames with ']'
POST /register rejects registration of usernames with '{'
POST /register rejects registration of usernames with '|'
POST /register rejects registration of usernames with '}'
POST /register rejects registration of usernames with '£'
POST /register rejects registration of usernames with 'é'
POST /register rejects registration of usernames with '\n'
POST /register rejects registration of usernames with '''
GET /login yields a set of flows
POST /login can log in as a user
POST /login can log in as a user with just the local part of the id
POST /login as non-existing user is rejected
POST /login wrong password is rejected
GET /events initially
GET /initialSync initially
Version responds 200 OK with valid structure
PUT /profile/:user_id/displayname sets my name
GET /profile/:user_id/displayname publicly accessible
PUT /profile/:user_id/avatar_url sets my avatar
GET /profile/:user_id/avatar_url publicly accessible
GET /device/{deviceId} gives a 404 for unknown devices
PUT /device/{deviceId} gives a 404 for unknown devices
POST /createRoom makes a public room
POST /createRoom makes a private room
POST /createRoom makes a private room with invites
POST /createRoom makes a room with a name
POST /createRoom makes a room with a topic
Can /sync newly created room
GET /rooms/:room_id/state/m.room.member/:user_id fetches my membership
GET /rooms/:room_id/state/m.room.power_levels fetches powerlevels
POST /join/:room_alias can join a room
POST /join/:room_id can join a room
POST /join/:room_id can join a room with custom content
POST /join/:room_alias can join a room with custom content
POST /rooms/:room_id/leave can leave a room
POST /rooms/:room_id/invite can send an invite
POST /rooms/:room_id/ban can ban a user
POST /rooms/:room_id/send/:event_type sends a message
PUT /rooms/:room_id/send/:event_type/:txn_id sends a message
PUT /rooms/:room_id/send/:event_type/:txn_id deduplicates the same txn id
GET /rooms/:room_id/state/m.room.power_levels can fetch levels
PUT /rooms/:room_id/state/m.room.power_levels can set levels
PUT power_levels should not explode if the old power levels were empty
Both GET and PUT work
POST /rooms/:room_id/read_markers can create read marker
User signups are forbidden from starting with '_'
Can logout all devices
Request to logout with invalid an access token is rejected
Request to logout without an access token is rejected
Room creation reports m.room.create to myself
Room creation reports m.room.member to myself
New room members see their own join event
Existing members see new members' join events
setting 'm.room.power_levels' respects room powerlevel
Unprivileged users can set m.room.topic if it only needs level 0
Users cannot set ban powerlevel higher than their own
Users cannot set kick powerlevel higher than their own
Users cannot set redact powerlevel higher than their own
Can get rooms/{roomId}/members for a departed room (SPEC-216)
3pid invite join with wrong but valid signature are rejected
3pid invite join valid signature but revoked keys are rejected
3pid invite join valid signature but unreachable ID server are rejected
Room members can override their displayname on a room-specific basis
Room members can join a room with an overridden displayname
displayname updates affect room member events
avatar_url updates affect room member events
Real non-joined user cannot call /events on shared room
Real non-joined user cannot call /events on invited room
Real non-joined user cannot call /events on joined room
Real non-joined user cannot call /events on default room
Real non-joined users can get state for world_readable rooms
Real non-joined users can get individual state for world_readable rooms
Real non-joined users can get individual state for world_readable rooms after leaving
Real non-joined users cannot send messages to guest_access rooms if not joined
Real users can sync from world_readable guest_access rooms if joined
Real users can sync from default guest_access rooms if joined
Can't forget room you're still in
Can get rooms/{roomId}/members
Can create filter
Can download filter
Can sync
Can sync a joined room
Newly joined room is included in an incremental sync
User is offline if they set_presence=offline in their sync
Changes to state are included in an incremental sync
A change to displayname should appear in incremental /sync
Current state appears in timeline in private history
Current state appears in timeline in private history with many messages before
Rooms a user is invited to appear in an initial sync
Rooms a user is invited to appear in an incremental sync
Sync can be polled for updates
Sync is woken up for leaves
Newly left rooms appear in the leave section of incremental sync
We should see our own leave event, even if history_visibility is restricted (SYN-662)
We should see our own leave event when rejecting an invite, even if history_visibility is restricted (riot-web/3462)
Newly left rooms appear in the leave section of gapped sync
Previously left rooms don't appear in the leave section of sync
Left rooms appear in the leave section of full state sync
Newly banned rooms appear in the leave section of incremental sync
Newly banned rooms appear in the leave section of incremental sync
local user can join room with version 1
User can invite local user to room with version 1
local user can join room with version 2
User can invite local user to room with version 2
local user can join room with version 3
User can invite local user to room with version 3
local user can join room with version 4
User can invite local user to room with version 4
Should reject keys claiming to belong to a different user
Can add account data
Can add account data to room
Latest account data appears in v2 /sync
New account data appears in incremental v2 /sync
Checking local federation server
Inbound federation can query profile data
Outbound federation can send room-join requests
Outbound federation can send events
Inbound federation can backfill events
Backfill checks the events requested belong to the room
Can upload without a file name
Can download without a file name locally
Can upload with ASCII file name
Can send image in room message
AS cannot create users outside its own namespace
Regular users cannot register within the AS namespace
AS can't set displayname for random users
AS user (not ghost) can join room without registering, with user_id query param
Changing the actions of an unknown default rule fails with 404
Changing the actions of an unknown rule fails with 404
Enabling an unknown default rule fails with 404
Trying to get push rules with unknown rule_id fails with 404
Events come down the correct room
local user can join room with version 5
User can invite local user to room with version 5