Dismantle REST API #55

houkime · 2023-08-14T20:06:16+03:00

houkime commented

2023-08-14 20:06:16 +03:00

Triggered by: CI having problems with TestClient as used in REST testing.

How do we do it:

Remove tests that have GraphQL counterparts
- Auth
- System
- Users
- Services
Add tests to GraphQL so that we can remove them from REST
- Services
- System
- Users
Remove all REST tests
Remove all REST code
Migrate to new service settings storage scheme
Bump version to 3.0.0

For adding tests to be efficient, one wants #40 to merge first, ideally.

Triggered by: CI having problems with TestClient as used in REST testing. How do we do it: - [x] Remove tests that have GraphQL counterparts - [x] Auth - [x] System - [x] Users - [x] Services - [x] Add tests to GraphQL so that we can remove them from REST - [x] Services - [x] System - [x] Users - [x] Remove all REST tests - [x] Remove all REST code - [x] Migrate to new service settings storage scheme - [x] Bump version to 3.0.0 For adding tests to be efficient, one wants #40 to merge first, ideally.

houkime added 3 commits 2023-08-14 20:06:17 +03:00

3f1fe16f39 test(rest-dismantling): remove auth tests after confirming gql counterparts exist

48a8f86865 test(rest-dismantling): remove system tests with gql counterparts

continuous-integration/drone/push Build is failing Details

a8fe3474b2 test(rest-dismantling): remove user tests with gql counterparts

houkime added 53 commits 2023-10-11 21:31:26 +03:00

09d1f7d74e feature(backup): remember the reason for making a snapshot

ab4eb53a6c test(backup): test reasons

45d975e68c feature(redis): compatibility with str enums

388b3a73c9 feature(backups): caps for autobackups

55d4125d80 feature(backups): actual finegrained quotas

18e9eda6e5 test(backups): test quotas with actual backups

0bc4295c0a refactor(backups): clean up caps code

475fd88c22 flx(backups): commit forgotten time.py

51370f823b BROKEN(backups): hooking up quotas to API fails. AutobackupQuotas needs to be an input type, but if input type, it fails because it needs to be an Output type, which is not documented

2e32814aa9 fix(graphql): backup quotas field typing

786c26dba8 fix(backup): trim auto-snapshots on setting the quotas

833018d3f2 refactor(backups): use restic-like rotation policy

412a6284e2 chore:bump version

83a6101349 fix(backups): consider failing services MORE and not try to stop them

5b70609c4a fix(backups): default timeout to 5 min for service starting and stopping in backup operations

5920376939 refactor(backups): a better backup-related service timeout error

458a6c5e99 feature(backups): report the error text in a job

efc6448416 feature(backups): report status text for restore jobs

fc15d42637 feature(backups): temporarily revert restore job status to created for inplace restore to run backup

8d821b889b test(backups): more checks regarding tmpdirs and mounting

e9ff118519 feature(backups): do not rely on mounting

0c6ca7d0d3 refactor(backups): clean up unused mounting tools

cecfa8c288 style: remove unused imports

87f1299f07 style: linting

2cbd1a98ec feature(backups): a task to autorefresh cache. Redis expiry abolished

642fc52fc9 fix(backups): expose snapshot reason and set the default value

598d1b8b51 test(backup): ensure we use correct repo folder

2b1b760f56 test(backup): ensure we actually call backup fixture and related resets

d4cdd00b98 test(backup): break out dummy service fixture

7c6bdf625c test(services): break out graphql basics

ae946da0f5 test(services): add services query generator

e82929f99e test(services): disable usual services for testing

2461c4a920 test(services): check id and status

30ed958217 test(service): startstop

0ea9aeee09 test(service): enabled status get

b846854d1e test(service): startstop return values

d366bff584 test(service): breakout raw api calls with ids

e49368a49b test(service): enable-disable return values

4c4d6cb770 test(service): start nonexistent service

dde933ebd3 test(service): stop nonexistent service

9d474e7626 test(service): enable nonexistent service

b8fbbcae07 test(service): disable nonexistent service

b37304c172 test(service): start service unauthorized

ff809cf611 test(service): other unauthorized mutations

c7860d7ab6 test(service): an unauthorized query

ba33cf10ed test(service): restart

eb04888a1d test(service): moving errors

ec2674ed51 fix(blockdevices): handle nested lsblk

7a45f855d0 feature(service): error handling on moves

797e755570 test(service): somewhat support moves for dummy service

dab05460b5 test(block devices): delete an extra update

41bb2916d0 test(services): try moving to the same device

continuous-integration/drone/push Build is failing Details

7670813c32 fix(backup): do not store maybe unpicklable service on the queue

houkime added 24 commits 2023-11-24 13:30:01 +02:00

3f722ca188 test(services): test double enables and disables

71249ebf0a test(services): delete redundant rest tests from bitwarden

96d86096a9 refactor(services): make a default implementation of enable/disable

7da8f74c0a refactor(services): delete enable/disable from gitea

dda8f3ef40 refactor(services): add default implementation to get_enabled

bd8648023e refactor(services): delete is_enabled() from gitea

26fc435770 refactor(services): delete xxenablexx functions from jitsi

1147b16987 fix(services): do not randomly exit the huey immediate mode

2a3121c772 test(backups, token_repo): move non-graphql tests out of graphql

ae5c4d6ed7 better error reporting in graphql tests

69848e8f36 feature(services): better error reporting in disable and enable service

9e861484c5 fix(services): replace stray gitea reference with a generic identifier in deiable/enable

3b7186ac81 test(services): test that undisableable services are handled correctly

d30ecdc2ba test(services): use actual json enabling and disabling

5e8aaf91c5 test(services): add a test that we actually read json

4a7be85abb test(services): add a test that we actually write json

bb0a49e5d2 test(services): fix last tests being not enabled, and a field typo0

440545d9c8 test(services): test possibly undefined json fields. On writing

62b93ade55 test(services): missing info on service enabled status returns False

1f8ddf2dab test(services): move non-gql enable+json tests out of gql tests towards backend tests

5697d643fa test(services): remove bitwarden tests because redundant

4c18974c45 test(services): add unauthorized move test

069c57617e test(services): remove redundant gitea tests

continuous-integration/drone/push Build is failing Details

de94e55dbd test(services, system): untie dkim tests from rest

houkime added 1 commit 2023-11-24 13:32:15 +02:00

continuous-integration/drone/push Build is failing Details

4e25362493 test(services): remove redundant nextcloud tests

houkime added 1 commit 2023-11-24 13:34:34 +02:00

continuous-integration/drone/push Build is failing Details

c88cb14b44 test(services): remove redundant ocserv tests

houkime added 1 commit 2023-11-24 13:37:56 +02:00

continuous-integration/drone/push Build is failing Details

5733e36e0c test(services): remove redundant pleroma tests

houkime added 1 commit 2023-11-24 13:54:45 +02:00

continuous-integration/drone/push Build is failing Details

05f86184ec test(services): remove legacy restic test data

houkime added 1 commit 2023-11-24 16:17:06 +02:00

continuous-integration/drone/push Build is failing Details

81ed0aae75 test(services): untie dkim-related service tests from rest

houkime force-pushed remove-rest from 81ed0aae75 to b49343555b

2023-12-08 10:51:11 +02:00

Compare

inex requested changes 2023-12-12 06:56:18 +02:00

selfprivacy_api/backup/tasks.py Outdated

													
				@ -16,3 +18,4 @@

				from selfprivacy_api.jobs import Jobs, JobStatus, Job

				SNAPSHOT_CACHE_TTL_HOURS = 6

inex commented

2023-12-12 06:24:38 +02:00

Duplicate constant declaration

houkime commented

2023-12-25 19:15:41 +02:00

Poster

addressed

houkime marked this conversation as resolved

selfprivacy_api/graphql/mutations/services_mutations.py Outdated

													
				@ -167,3 +179,1 @@

				            service=service_to_graphql_service(service),

				            job=job_to_api_job(job),

				        )

				        if job.status == JobStatus.FINISHED:

inex commented

2023-12-12 06:29:01 +02:00

I think that it is likely that this will return 400 all the time. It should return the message like "Started moving the service".

houkime commented

2023-12-22 14:23:28 +02:00

Poster

It is not an async call, just a very long one. It returns when it is either finished or failed.

houkime commented

2024-01-08 15:19:11 +02:00

Poster

added more handlers

houkime marked this conversation as resolved

selfprivacy_api/utils/__init__.py Outdated

													
				@ -170,2 +170,4 @@

				def get_dkim_key(domain: str, parse: bool = True) -> typing.Optional[str]:

				    """Get DKIM key from /var/dkim/<domain>.selector.txt"""

				    if os.path.exists("/var/dkim/" + domain + ".selector.txt"):

				        # Is this really neccessary to use Popen here?

inex commented

2023-12-12 06:31:57 +02:00

No, looks like a leftover that can be refactored.

houkime marked this conversation as resolved

tests/test_common.py Outdated

													
				@ -8,0 +21,4 @@

				TESTFILE_2_BODY = "testissimo!"

				@pytest.fixture()

inex commented

2023-12-12 06:42:05 +02:00

These fixtures are unused in this file. Maybe they should be in conftest.py?

These fixtures are unused in this file. Maybe they should be in `conftest.py`?

houkime marked this conversation as resolved

tests/test_services_dkim.py Outdated

													
				@ -0,0 +91,4 @@

				###############################################################################

				def test_dkim_key(authorized_client, mock_subproccess_popen):

inex commented

2023-12-12 06:54:29 +02:00

...I wonder how did we end up with dkim in these tests?

These are clearly not related to DKIM anymore. File should be renamed too.

...I wonder how did we end up with `dkim` in these tests? These are clearly not related to DKIM anymore. File should be renamed too.

houkime marked this conversation as resolved

houkime added 43 commits 2023-12-22 12:18:25 +02:00

4ed63179ac feature(ssh): change ssh settings from graphql

a4c692a11c test(ssh): add graphql ssh status query test

c5a6581579 test(ssh): try disabling ssh

f1edd4aa0b test(ssh): flip flop ssh

1fcbdb6825 test(ssh): test that query is in sync

e40c116c44 test(ssh): test idempotency of enablement

a582d2827f test(ssh): cleanup

857067de45 test(ssh): add json storage writing tests

c4e4ac669b test(ssh): add json storage reading tests

4e4c815c03 test(ssh): remove rest tests for undefined ssh settings

7ff4802929 test(ssh): add test for unauthorized settings getting

042f8522d3 test(ssh): remove basic unauthorized tests from rest ssh tests

cae10c220a test(ssh): remove rest enablement tests

53c5043765 test(ssh): full ssh enablement-via-gql readwrite testing

16896c4d23 test(ssh): remove rest ssh enablement tests

5b006c90c4 test(ssh): add rootkey json tests

8aaf444482 test(ssh): add docstring with scope to tests/test_ssh

cec2c807d4 test(ssh): delete undefined root keys

a05369aabd test(ssh): delete rest test of undefined root key deletion

bbdc8db72b test(ssh): add root key when none

dc699aaba8 test(ssh): add one more root key

f6a199371e test(ssh): remove corresponding rest test

f818af712c test(ssh): adding same key

f63f475a41 test(ssh): get root key

7be5e23b6c test(ssh): get root key when none

885d650eb3 test(ssh): remove root key

bbe1770837 test(ssh): remove root key nonexistent

5d50cfb1b6 test(ssh): admin keys getting

eda99cc771 test(ssh): admin keys deleting

d2dbac4244 test(ssh): admin keys json storage tests

aa1ae1699d test(ssh): parametrized testing of ssh key addition

dc3d4dc1f6 test(ssh): parametrized testing of ssh key addition, more

f8ed06c065 test(ssh): parametrized testing of ssh key addition, existing and invalid

09111ca8c7 test(ssh): remove rest user getting tests (they are covered by users tests

799dff80a0 test(ssh): dealing with undefined users

521534ccff test(ssh): regular users

75a129c14c test(ssh): parametrized removing keys

841cf79686 test(ssh): delete redundant ssh tests

c5e2bd241b test(ssh): json storage of user keys : reading

e5fde70f98 test(ssh): user key storage test: adding

f74dedfd08 test(ssh): user key storage test: removing

742bf103ab test(ssh): user key storage test: removing user keys on undefined

continuous-integration/drone/push Build is failing Details

820f905d6a test(ssh): remove the rest of rest ssh tests bc redundant

houkime added 5 commits 2023-12-25 19:15:10 +02:00

af83042c6d refactor(backup): remove a redundant constant

aab06ebf0d refactor(dkim): do not use popen

8c5412a8fa test(service, backup): move dummy service fixtures to conftest.py

4ff4616b79 test(service): refactor systemctl calltests

continuous-integration/drone/push Build is failing Details

ee48c0971a test(service): rename service systemctl calltests

houkime added 8 commits 2023-12-27 16:47:48 +02:00

15d9719f69 test(services): test mailserver dkim quirk

4808840f89 fix(users): handle no admin case

e6d0490848 test(users): test adding users with missing (not just empty) fields

8db4422649 test(users): test invalid usernames (and delete it from rest)

1b45a091b1 fix(users): handle no admin name defined when adding a user

8c31f8322b test(users): delete a user and CHECK that it was deleted

23285c06da test(users): delete redundant rest admin deletion tests

continuous-integration/drone/push Build is failing Details

1634886b4e test(users): delete rest user tests

houkime added 3 commits 2023-12-29 15:14:42 +02:00

94f8c95c9e feature(system): better error handling for shell calls

ecdd074227 test(system): test generic shell calls

continuous-integration/drone/push Build is failing Details

603ac9384e test(system): remove rest system tests

houkime added 5 commits 2023-12-29 17:20:26 +02:00

189bd74529 feature(system): remove rest system code

4064b5b4f8 feature(users): remove rest users code

b6b904b8e0 feature(services): remove rest services code

8562c4d28c feature(auth): remove rest auth code

continuous-integration/drone/push Build is failing Details

8b661472a3 feature(rest): remove rest

houkime added 4 commits 2024-01-03 21:33:24 +02:00

264023849c fix(services): handle the async nature of moving.

9e98497e6c refactor(services): use generic code for enabling and disabling

c24dbd39f5 feature(services): introduce 'modules' field in userdata and group services settings there

continuous-integration/drone/push Build is failing Details

5e2757e38e chore(version): bump version to 3.0, no Rest API

houkime force-pushed remove-rest from 5e2757e38e to 2b21df9ad3

2024-01-08 15:16:25 +02:00

Compare

houkime changed title from ~~WIP: Dismantle REST API~~ to Dismantle REST API

2024-01-08 15:19:55 +02:00

houkime commented

2024-01-08 15:20:10 +02:00

Poster

@inex I think this one is ready.

houkime commented

2024-01-08 15:20:47 +02:00

Poster

but there is an unrelated error in CI, will work on it separately.

houkime requested review from inex 2024-01-08 15:20:58 +02:00

inex approved these changes 2024-01-09 18:13:08 +02:00

inex left a comment

LGTM. ~~Do not merge yet.~~

LGTM. ~~Do not merge yet.~~

selfprivacy_api/migrations/__init__.py

													
				@ -37,6 +38,7 @@ migrations = [

				    MigrateToSelfprivacyChannelFrom2205(),

				    MigrateToSelfprivacyChannelFrom2211(),

				    LoadTokensToRedis(),

				    CreateModulesField(),

inex commented

2024-01-09 17:53:00 +02:00

Not needed, 3.0.0 will run on the system which already has a new userdata structure. I will make an additional PR to account for changes occurred during migration.