Rewrite backups #35

Merged

inex merged 475 commits from restic-rewrite into master 2023-07-20 18:42:57 +03:00

Conversation 1 Commits 475 Files Changed 72

houkime commented 2023-02-22 21:51:50 +02:00

Collaborator

• Backend support for multiple clouds via multiple tools
• In-Memory backups for testing (partial, cannot persist even for a session)
• In-file backups for testing (works, but slow)
• Repo init
• Folder backup
• Snapshot listing
• Restoring a service from snapshot
• Measurement of snapshot size
• Encryption and simple key management
• Loading provider from json
• Storing provider in redis
• Keeping track of initted vs uninitted repos
• Huey tasks
• Integration with Jobs
• Integration with API

- [x] Backend support for multiple clouds via multiple tools - [x] In-Memory backups for testing (partial, cannot persist even for a session) - [x] In-file backups for testing (works, but slow) - [x] Repo init - [x] Folder backup - [x] Snapshot listing - [x] Restoring a service from snapshot - [x] Measurement of snapshot size - [x] Encryption and simple key management - [x] Loading provider from json - [x] Storing provider in redis - [x] Keeping track of initted vs uninitted repos - [x] Huey tasks - [x] Integration with Jobs - [x] Integration with API

houkime force-pushed restic-rewrite from 4aab5e0b57 to f6402f2394 2023-04-04 02:32:04 +03:00 Compare

houkime added 5 commits 2023-04-10 16:33:04 +03:00

644a0b96b8 test(backups): test last backup date retrieval

942c35b7e6 feature(backups): add snapshot cache sync functions

d68e9a4141 feature(backups): enable snapshot cache usage

9d772ea2e2 test(backups): test that we do use cache

a11627da7d refactor(backups): split out storage

houkime added 2 commits 2023-04-10 19:37:18 +03:00

d7316f8e79 test(backups): test autobackup timing

3d4d05ff11 feature(backups): automatic backup

houkime force-pushed restic-rewrite from 6c84359649 to 03d4632465 2023-05-17 23:31:44 +03:00 Compare

houkime added 14 commits 2023-06-05 14:55:01 +03:00

0dc6f74754 fix(backups): make sure location and credentials get properly passed around

97e4c529f6 feature(backups): init repo mutation

345e2c10c4 feature(backups): a graphql query to get provider info

c3696d3a4b feature(backups): global init instead of per-service

9c4d2a0ba5 feature(backups): drop repository call

603ed2ddf9 feature(backups): start backup graphql API

4b07d4de41 refactor(backups): global snapshots

f7c0821675 fix(backups): return correct snapshots per service

50952d688a feature(backups): graphql mutation for restore

9a94f7624e feature(backups): a graphql call to invalidate cache

9d5335f62c feature(backups): feature(backups): return a snapshot from start_backup

2168037a10 refactor(backups): introduce get_all_snapshots()

d6609b29e8 feature(backups): get all snapshots if requested by api

ff58bdccbb feature(backups): set autobackup period from gql

houkime added 1 commit 2023-06-07 17:11:50 +03:00

0c95c5913b BREAKING CHANGE(backups): support only individual service backup requests(combinable)

houkime added 1 commit 2023-06-07 17:25:38 +03:00

4b8abb6a66 fix(backups): return 400, not 300

houkime added 1 commit 2023-06-07 17:29:44 +03:00

98de85e569 fix(backups): return one job, not an array of one

houkime added 1 commit 2023-06-07 18:06:24 +03:00

7f984b678f feature(backups): integration between restore and jobs

houkime added 1 commit 2023-06-07 18:55:57 +03:00

981445d594 refactor(backups): expect one more error of restic json output parsing

houkime added 1 commit 2023-06-07 19:33:40 +03:00

549b149aaf feature(backups): check available space before restoring

inex added 1 commit 2023-06-13 23:54:09 +03:00

d02302c7b2 fix: BackupConfiguration argument order

inex added 1 commit 2023-06-14 00:03:08 +03:00

f950dd1e93 feat(backups): register backups in GraphQL schema

houkime added 3 commits 2023-06-14 13:18:07 +03:00

865e304f42 test(backups): test backup API - backing up

d9bde6930b fix(backups): register queries

c47977c100 test(backups): actual testfile

houkime added 4 commits 2023-06-14 13:29:41 +03:00

0ef6569d97 fix(backups): try to actually get backup configuration

f4df1f6a62 fix(backups): return type of encryption key

d356fad534 fix(backups): Handle orphaned snapshots

8554879dc2 Merge remote-tracking branch 'upstream/restic-rewrite-api' into restic-rewrite

houkime added 1 commit 2023-06-14 14:15:09 +03:00

928f026e7c test(backups): snapshot query

houkime added 7 commits 2023-06-14 16:01:10 +03:00

f1654c699c fix(backups): shorten snapshot query signature

aa9d082adc test(backups): display errors from api

21791f20a2 test(backups): make dummy service more compliant

92322d8fad test(backups): test dummy service compliance

39cd4b714b test(backups): check snapshots getting created

17c7dffb07 test(backups): use get_data

4862cdc72f test(backups): test restore

houkime added 1 commit 2023-06-14 16:06:51 +03:00

cbedd196e4 fix(backups): non-nullable service when backing up

houkime added 2 commits 2023-06-14 17:17:19 +03:00

9e0b0d8caa refactor(backups): make localfile repos normal

b3724e240e refactor(backups): cleanup localfile-specific logic

houkime added 2 commits 2023-06-16 17:34:03 +03:00

33c60f971d test(backups): test reinitting repository

8e1e37c766 test(backups): preliminary test of repo reset

houkime added 1 commit 2023-06-16 17:39:15 +03:00

47aee3c1f1 fix(backups): fix output API return types for configuration

houkime added 1 commit 2023-06-16 18:09:54 +03:00

79c3b9598c feature(backups): resetting json config too

houkime added 1 commit 2023-06-16 18:19:40 +03:00

f02e27cf06 test(backups): add a backend json reset test

houkime added 2 commits 2023-06-16 19:03:52 +03:00

3449837de9 refactor(backups): make a dir for backuppers

4f2f24daac refactor(backups): NoneBackupper class for those cases when we do not know

houkime added 1 commit 2023-06-19 14:09:48 +03:00

612a46ee42 feature(backups): actually dealing with situation when the provider is not configured

houkime added 1 commit 2023-06-19 14:17:30 +03:00

b5e2499a30 refactor(backups): delete legacy provider setting

houkime added 1 commit 2023-06-19 16:28:34 +03:00

25f3115c05 test(backups): setting autobackup period

houkime added 1 commit 2023-06-19 17:13:19 +03:00

e3a87f1d98 test(backups): ensure asking to reload snaps does not explode the server

inex added 1 commit 2023-06-21 06:47:03 +03:00

c6919293b6 refactor(api): Group mutations ...

I've learned that there is no problem in grouping mutations like we do with queries.
This is a big mistake from my side, now we have legacy not so conveniently placed endpoints.
I've grouped all mutations, left the copies of old ones flattened in the root for backwards compatibility.
We will migrate to mutation groups on client side, and backups now only use grouped mutations.
Tests are updated.

houkime changed title from WIP: Rewrite backups to Rewrite backups 2023-06-21 15:26:08 +03:00

houkime requested review from inex 2023-06-21 15:27:33 +03:00

houkime changed title from Rewrite backups to WIP: Rewrite backups 2023-06-21 15:52:50 +03:00

houkime commented 2023-06-21 15:53:52 +03:00

Poster

Collaborator

This PR will be ready to be merged when backblaze tests on CI pass.

This PR will be ready to be merged when backblaze tests on CI pass.

inex added 1 commit 2023-06-23 04:50:06 +03:00

c9cfb7d7bc ci: fix killing redis-server

inex added 1 commit 2023-06-23 12:40:23 +03:00

e7e0fdc4a1 refactor(backups): fix typing errors

inex added 1 commit 2023-06-23 12:57:44 +03:00

0f1d8e22f2 ci: ignore the failure when trying to kill redis

inex added 1 commit 2023-06-23 13:02:59 +03:00

1f558d6cf9 ci: only run on push event

inex added 2 commits 2023-06-23 14:36:24 +03:00

b86d0cd850 fix(redis): Do not shut down redis on ctrl c ...

see https://github.com/NixOS/nix/issues/2141

1664f857ea fix(backups): Providers were not initialized corretly

houkime added 1 commit 2023-06-23 14:58:28 +03:00

6ca68fae17 fix(tokens-repo): persistent hashing

houkime added 1 commit 2023-06-23 15:04:53 +03:00

2f71469f39 refactoring(backups): backuper -> backupper

houkime added 1 commit 2023-06-23 16:06:01 +03:00

273a1935a8 feature(dev_qol): mypy type checking and rope refactoring support

houkime added 14 commits 2023-06-26 23:06:08 +03:00

8604caa331 refactor(backups): api readability reorg

e159d2f1a7 refactor(backups): reorder imports

81d1762518 refactor(backups): make lookup_provider not public

71eeed926d refactor(backups): make construct_provider not public

1593474dc1 refactor(backups): make redis and json provider related lowlevels private

e6efd1b42d refactor(backups): move reset() to top because toplevel interface

2f2c4f14af refactor(backups): group operations together

b9be0be6a2 refactor(backups): privatize assert_restorable and restore_snapshot_from_id

4ad4c3cc67 refactor(backups): delete sync_service_snapshots

f8029d133a refactor(backups): straighten get_all_snapshots

a76834d1ba refactor(backups): merge sync_all_snapshots with force_snapshot_reload

bae81b2f69 refactor(backups): rename force_snapshot_reload to force_snapshot_cache_reload

7147f97077 refactor(backups): remove the by-service getting of cached snapshots

ae16a527a2 refactor(backups): rename service_snapshot_size to snapshot_restored_size

inex added 1 commit 2023-06-28 12:10:18 +03:00

d9b26e12e2 fix(backups): missing space in rclone args

houkime added 1 commit 2023-06-28 12:41:21 +03:00

59fef1d016 fix(backups): register the correct tasks

houkime added 1 commit 2023-06-28 12:48:49 +03:00

35258bad38 fix(services): cleanup a stray get_location

houkime added 1 commit 2023-06-28 14:45:27 +03:00

e4865aa094 fix(services): proper backup progress reporting

inex added 1 commit 2023-06-28 15:56:38 +03:00

11184a55e8 feat(jobs): return type_id of the job in graphql api

houkime added 1 commit 2023-06-28 16:06:31 +03:00

3ee90617ba test(backup): make large testfile larger

houkime added 1 commit 2023-06-28 16:23:09 +03:00

cd2e9d3ba3 fix(jobs): make finishing the job set progress to 100

inex added 2 commits 2023-06-29 13:45:05 +03:00

b480c84041 style: fix styling

15ce344bc8 feat(backups): expose if the service can be backed up

inex added 1 commit 2023-06-29 14:27:13 +03:00

64e7afe53e feat(backups): Add backup descriptions for UI

houkime added 6 commits 2023-07-03 18:34:22 +03:00

9793201ca1 feature(backups): a wrapper for rclone sync

08cc7740b3 test(backups): actually list folders

6a00d3cff9 refactor(backups): move output yielding into backup utils

953860a02c refactor(backups): move syncing (non-restic) into backup utils

c5088e0e2c test(backups): remove the 100mb file after test

b6eb27dc5e feature(backups): mounting a repo

houkime added 1 commit 2023-07-03 23:42:10 +03:00

3dc6fb91f2 refactor(backups): download a copy before replacing original

houkime added 1 commit 2023-07-03 23:53:35 +03:00

03313b739a feature(backups): check restore exit code

houkime added 1 commit 2023-07-05 16:13:50 +03:00

53bb5cc4e2 feature(backups): forgetting snapshots

houkime added 5 commits 2023-07-07 16:25:03 +03:00

c74b3df32c feature(backups): restore strategies enum

af5edb695f feature(backups): implement inplace restore strategy

9075afd38a test(backups): test out that pre-restore backup plays nice with jobs

2743441e1e fix(backups): actually mount if asked for an inplace restore

1cefaefa3b feature(backups): expose restore strategies to the API

houkime added 8 commits 2023-07-12 19:55:24 +03:00

6523105d89 feature(utils): a hopefully reusable waitloop

169e9ad57d test(backups): simulating async service start n stop

e7a6700522 test(backups): test async service start n stop simulation

713296c520 test(backups): make delay settable per dummyservice

de8ef744eb refactor(backups): make a StoppedService context manager

326e3d3b0c test(backups): do not store the status file in backupped folders

7af7600599 feature(backups): stop services before backups

e43478d437 feature(backups): stop services before restores

houkime added 1 commit 2023-07-12 20:03:50 +03:00

92cfd00f93 feature(servers): set default timeout of server operations to 10 min

houkime added 1 commit 2023-07-14 13:58:47 +03:00

4aa87edf47 fix(servers): hopefully fix moving

houkime added 1 commit 2023-07-14 14:41:24 +03:00

926d0c27c5 test(backup): test moving preventing backups

houkime added 1 commit 2023-07-14 15:35:02 +03:00

9f19c677d0 feature(backup):calculate needed space for inplace restoration

inex changed title from WIP: Rewrite backups to Rewrite backups 2023-07-17 16:04:33 +03:00

inex force-pushed restic-rewrite from 9f19c677d0 to 45011450c5 2023-07-17 16:05:04 +03:00 Compare

inex referenced this pull request from SelfPrivacy/selfprivacy-nixos-config 2023-07-17 17:14:47 +03:00

WIP: Automatic backblaze backups #9

inex added 1 commit 2023-07-17 17:39:43 +03:00

8805f73812 CI: Disable redis errors on write failures

inex added 1 commit 2023-07-17 19:45:05 +03:00

7fe802eb1d ci: Fix redis port of redis-cli command

inex added 1 commit 2023-07-17 19:55:23 +03:00

20f3e5c564 ci: Give redis some time to start?

inex added 1 commit 2023-07-18 20:15:27 +03:00

8b504993d0 fix(backups): backup strategies were unused

houkime added 231 commits 2023-07-19 13:36:39 +03:00

80f2c99ba4 refactor(backup): delete unused import

060117a355 refactor(backup): extract rclone args

4871958534 refactor(backup): extract restic repo

cf3cbd179f refactor(backup): pass key and account to exec

740b072b07 refactor(backup): do not use config file

6d73405118 feature(backups): add backup structures and queries

a98314bb60 feature(backups): placeholders for the modules of the new backup system

23e3d553e6 feature(backups): placeholders for the backupers and backup providers

24d170b3cb feature(backups): copy cli logic to new restic backuper

af6a49b0a3 test(backup): provider class selection

934afeb531 test(backup): dummy service

f5066082c8 feat(backup): allow no auth

a66a3f35fc feat(backup): add in-memory backup

e54224f15d test(backup): init an in-memory backup class

90c2524800 test(backup): make a testfile to backup

474f04f923 feat(backup): add backuping to restic backuper

6fa72dbac1 fix(backup): add memory backup class,forgot to add to git

17df21964a test(backup): try to back up!

d5881cc523 refactor(backup): add a placeholder Backups singleton class

7e243146db refactor(backup): add a backup function to Backups singleton class

bf263f68b9 test(backup): use a backup service function

3970524bce feat(backup): hooks

1568ee68a4 feature(backup): add a restore function to restic backuper

3afe196901 feature(backup): loading snapshots

e447419f96 refactor(backup): snapshot model

a043392b7b test(backup): no snapshots

c9fd656181 refactor(backups): snapshotlist and local secret groundwork

86e2b90bef refactor(backups): add repo init

bd744ba211 refactor(backups): repo init service method

865c8f5280 test(backups): test repo init

c493a49daa feature(backups): localfile repo

f1987edd99 feature(backups): register localfile backend

e944f4a267 test(backups): basic file backend init test

e091fbd4a2 feature(backups): throw an error if repo init fails

06deb83b33 test(backups): localfile repo by default in tests

01797cb922 fix(backups): singleton metaclass was screwing with tests

b3633d1a96 feat(backups): throw an error on a failed backup

8a1b3be9fa test(backups): reenable snapshot testing

c32353fe9b feat(backups): return proper snapshot structs when listing

98e1c9ebaa feat(backups): a better error on failed snapshot retrieval

dc3032eb03 feat(backups): add restore_snapshot and restore_service_from_snapshot

d62d7534d7 test(backups): test restoring a file

23ee958bfb feat(backups): sizing up snapshots

4e329299e0 feat(backups): local secret generation and storage

ea6a047478 feature(backups): load from json

a0afe63b3d refactor(backups): redis model storage utils

6524c98131 refactor(backups): add a provider model for redis storage

4b2cecac8f feature(backups): provider storage and retrieval

a2dd47130b feature(backups): repo init tracking

144e4e5e91 refactor(backups): make backups stateless

772b499b46 feature(backups): huey task to back up

2eb64db199 feature(backups): return snapshot info from backup function

ae065867b3 test(backups): test that we do return snapshot on backup

18d5cd2b83 feature(backups): cache snapshots and last backup timestamps

45f33e2d31 feature(backups): check, set and unset service autobackup status

417533fc04 fix(backups): remove self from static method

e043720289 feature(backups): methods for autobackup period setting and getting

3c42d8c413 test(backups): test setting services as enabled for autobackups

d4cad61d56 test(backups): test setting autobackup period

146b0ca02f feature(backups): add a datetime validator function for huey autobackups

7699ba0d9b test(backups): test last backup date retrieval

a2ff74244e feature(backups): add snapshot cache sync functions

5ad9f50b94 feature(backups): enable snapshot cache usage

f25e57c51a test(backups): test that we do use cache

283c8d09cc refactor(backups): split out storage

ec85f060f8 test(backups): test autobackup timing

4018dca184 feature(backups): automatic backup

0ae4192081 test(backups): register dummy service

1b1052d205 refactor(services): rename get_location() to get_drive()

919ba1ad03 refactor(backups): make a dedicated get_folders() function

08739f7ca8 refactor(backups): make api accept a list of folders

fa26379a68 refactor(backups): actually accept a list of folders

30ac990985 refactor(backups): set a list of folders for our dummy service

a68e94fad3 test(backups): actually back up 2 folders

48359ffd20 test(backups): test 2-folder restoration

378f0ebd73 test(backups): implement get_folders() for bitwarden

1c0e3f0f92 test(backups): implement get_folders() for gitea

0207b5a473 fix(services): use get_foldername() for moving around

8a4f256c12 refactor(services): add get_folders() to the rest of the services

a34b14449b refactor(services): make get_folders() a mandatory part of Service interface

789fd71e2f refactor(services): add a generic storage counter

f8edcac33f refactor(services): remove special storage counting from bitwarden

c040f0825c refactor(services): remove special storage counting from gitea

312328af95 refactor(services): remove special storage counting from jitsi

042a2e4cf2 refactor(services): remove special storage counting from mail

4475bcea45 documentation(services): move the storage count docstring to parent service class

617f7f1628 refactor(services): remove special storage counting from nextcloud

3605a71c1d refactor(services): remove special storage counting from ocserv

a664ab3dd4 refactor(services): remove special storage counting from pleroma

0b7d2d0bf4 refactor(services): add OwnedPath struct

c34eb6d447 refactor(services): add overridable get owner and get group

ad66513f27 refactor(services): add folder owner derivation

238a656cd9 refactor(services): make a foldermove from owned path

043675ce14 fix(services): folder methods typing

ba72fadb8d test(services): test that we indeed return correct folders and owned folders from real services

dde86725b9 test(services): test derivation of foldermoves

65acd3173a refactor(services): use fully generic foldermoves

fa360655fe refactor(services): make local secret setting public

72f4fc8ae7 test(backups): test local secrets

d1425561d9 feature(backups): restore task

108fca0eb3 feature(backups): simplest jobs intergration in tasks: created and finished

cb1dd1011e test(backups): assure that jobs are created and not duplicated

3eb8382d9b feature(backups): also create a job if not called from a task

c0dfbb3ca2 refactor(backups): delete unused redis import from backups ckass

ad6cc5e1bc refactor(backups): make a backup job running when the backup code itself is executed

d340b0ca67 test(backups): break out a finished job checker

396b42b69c test(backups): test jobs starting and finishing when from Backups

0bf18dcdc7 refactor(backups): cleanup unused imports in tasks

b2e231ebae feature(backups): set job status to error if backup fails

7ddfad10d4 refactor(backups): quick-expiration logs of jobs status updates

e25aa2cb33 test(backups): test that the job has run

a6b3a5e590 feature(backups): deny adding a backup job if another one is already queued

deb857bca9 refactor(backups): use single repo and multiplex by tags

ddd2176a5d refactor(backups): break out job logs status prefix

60806cd536 feature(backups): job progress logs

54a8e0b2b0 test(backups): break out obtaining finished jobs

b1d0a80963 feature(backups): realtime progress updates of backups

9fab596f91 refactor(backups): refactor realtime updating

03d4632465 refactor(backups): remove extraneous asserts from jobs

0dc6f74754 fix(backups): make sure location and credentials get properly passed around

97e4c529f6 feature(backups): init repo mutation

345e2c10c4 feature(backups): a graphql query to get provider info

c3696d3a4b feature(backups): global init instead of per-service

9c4d2a0ba5 feature(backups): drop repository call

603ed2ddf9 feature(backups): start backup graphql API

4b07d4de41 refactor(backups): global snapshots

f7c0821675 fix(backups): return correct snapshots per service

50952d688a feature(backups): graphql mutation for restore

9a94f7624e feature(backups): a graphql call to invalidate cache

9d5335f62c feature(backups): feature(backups): return a snapshot from start_backup

2168037a10 refactor(backups): introduce get_all_snapshots()

d6609b29e8 feature(backups): get all snapshots if requested by api

ff58bdccbb feature(backups): set autobackup period from gql

0c95c5913b BREAKING CHANGE(backups): support only individual service backup requests(combinable)

4b8abb6a66 fix(backups): return 400, not 300

98de85e569 fix(backups): return one job, not an array of one

7f984b678f feature(backups): integration between restore and jobs

981445d594 refactor(backups): expect one more error of restic json output parsing

549b149aaf feature(backups): check available space before restoring

d02302c7b2 fix: BackupConfiguration argument order

f950dd1e93 feat(backups): register backups in GraphQL schema

0ef6569d97 fix(backups): try to actually get backup configuration

f4df1f6a62 fix(backups): return type of encryption key

d356fad534 fix(backups): Handle orphaned snapshots

865e304f42 test(backups): test backup API - backing up

d9bde6930b fix(backups): register queries

c47977c100 test(backups): actual testfile

8554879dc2 Merge remote-tracking branch 'upstream/restic-rewrite-api' into restic-rewrite

928f026e7c test(backups): snapshot query

f1654c699c fix(backups): shorten snapshot query signature

aa9d082adc test(backups): display errors from api

21791f20a2 test(backups): make dummy service more compliant

92322d8fad test(backups): test dummy service compliance

39cd4b714b test(backups): check snapshots getting created

17c7dffb07 test(backups): use get_data

4862cdc72f test(backups): test restore

cbedd196e4 fix(backups): non-nullable service when backing up

9e0b0d8caa refactor(backups): make localfile repos normal

b3724e240e refactor(backups): cleanup localfile-specific logic

33c60f971d test(backups): test reinitting repository

8e1e37c766 test(backups): preliminary test of repo reset

47aee3c1f1 fix(backups): fix output API return types for configuration

79c3b9598c feature(backups): resetting json config too

f02e27cf06 test(backups): add a backend json reset test

3449837de9 refactor(backups): make a dir for backuppers

4f2f24daac refactor(backups): NoneBackupper class for those cases when we do not know

612a46ee42 feature(backups): actually dealing with situation when the provider is not configured

b5e2499a30 refactor(backups): delete legacy provider setting

25f3115c05 test(backups): setting autobackup period

e3a87f1d98 test(backups): ensure asking to reload snaps does not explode the server

c6919293b6 refactor(api): Group mutations ...

I've learned that there is no problem in grouping mutations like we do with queries.
This is a big mistake from my side, now we have legacy not so conveniently placed endpoints.
I've grouped all mutations, left the copies of old ones flattened in the root for backwards compatibility.
We will migrate to mutation groups on client side, and backups now only use grouped mutations.
Tests are updated.

c9cfb7d7bc ci: fix killing redis-server

e7e0fdc4a1 refactor(backups): fix typing errors

0f1d8e22f2 ci: ignore the failure when trying to kill redis

1f558d6cf9 ci: only run on push event

b86d0cd850 fix(redis): Do not shut down redis on ctrl c ...

see https://github.com/NixOS/nix/issues/2141

1664f857ea fix(backups): Providers were not initialized corretly

6ca68fae17 fix(tokens-repo): persistent hashing

2f71469f39 refactoring(backups): backuper -> backupper

273a1935a8 feature(dev_qol): mypy type checking and rope refactoring support

8604caa331 refactor(backups): api readability reorg

e159d2f1a7 refactor(backups): reorder imports

81d1762518 refactor(backups): make lookup_provider not public

71eeed926d refactor(backups): make construct_provider not public

1593474dc1 refactor(backups): make redis and json provider related lowlevels private

e6efd1b42d refactor(backups): move reset() to top because toplevel interface

2f2c4f14af refactor(backups): group operations together

b9be0be6a2 refactor(backups): privatize assert_restorable and restore_snapshot_from_id

4ad4c3cc67 refactor(backups): delete sync_service_snapshots

f8029d133a refactor(backups): straighten get_all_snapshots

a76834d1ba refactor(backups): merge sync_all_snapshots with force_snapshot_reload

bae81b2f69 refactor(backups): rename force_snapshot_reload to force_snapshot_cache_reload

7147f97077 refactor(backups): remove the by-service getting of cached snapshots

ae16a527a2 refactor(backups): rename service_snapshot_size to snapshot_restored_size

d9b26e12e2 fix(backups): missing space in rclone args

59fef1d016 fix(backups): register the correct tasks

35258bad38 fix(services): cleanup a stray get_location

e4865aa094 fix(services): proper backup progress reporting

11184a55e8 feat(jobs): return type_id of the job in graphql api

3ee90617ba test(backup): make large testfile larger

cd2e9d3ba3 fix(jobs): make finishing the job set progress to 100

b480c84041 style: fix styling

15ce344bc8 feat(backups): expose if the service can be backed up

64e7afe53e feat(backups): Add backup descriptions for UI

9793201ca1 feature(backups): a wrapper for rclone sync

08cc7740b3 test(backups): actually list folders

6a00d3cff9 refactor(backups): move output yielding into backup utils

953860a02c refactor(backups): move syncing (non-restic) into backup utils

c5088e0e2c test(backups): remove the 100mb file after test

b6eb27dc5e feature(backups): mounting a repo

3dc6fb91f2 refactor(backups): download a copy before replacing original

03313b739a feature(backups): check restore exit code

53bb5cc4e2 feature(backups): forgetting snapshots

c74b3df32c feature(backups): restore strategies enum

af5edb695f feature(backups): implement inplace restore strategy

9075afd38a test(backups): test out that pre-restore backup plays nice with jobs

2743441e1e fix(backups): actually mount if asked for an inplace restore

1cefaefa3b feature(backups): expose restore strategies to the API

6523105d89 feature(utils): a hopefully reusable waitloop

169e9ad57d test(backups): simulating async service start n stop

e7a6700522 test(backups): test async service start n stop simulation

713296c520 test(backups): make delay settable per dummyservice

de8ef744eb refactor(backups): make a StoppedService context manager

326e3d3b0c test(backups): do not store the status file in backupped folders

7af7600599 feature(backups): stop services before backups

e43478d437 feature(backups): stop services before restores

92cfd00f93 feature(servers): set default timeout of server operations to 10 min

4aa87edf47 fix(servers): hopefully fix moving

926d0c27c5 test(backup): test moving preventing backups

9f19c677d0 feature(backup):calculate needed space for inplace restoration

a303e5ce37 Merge branch 'restic-rewrite' of https://git.selfprivacy.org/SelfPrivacy/selfprivacy-rest-api into restic-rewrite

houkime added 1 commit 2023-07-19 13:52:02 +03:00

fa53264136 refactor(backup):remove unused verify arg from do_restore

inex added 3 commits 2023-07-19 16:00:16 +03:00

3067d353d8 fix(backups): Do not remove failsafe snapshot after recovery

5253780cc8 style: linting of backups module

a757dc7cc1 chore: add rclone as a nix shell dependency

inex added 1 commit 2023-07-19 16:49:43 +03:00

5bee124500 style: set 88 as a line length limit

houkime added 2 commits 2023-07-19 18:41:52 +03:00

466160dbf9 feature(backup):disable finegrain control over services to back up

f4ac3d29a9 feature(backup):remove code for finegrained autobackup control

inex added 1 commit 2023-07-20 17:11:14 +03:00

2df448a4a9 fix(backups): Do not shut down the service during backup ...

We do not want the user to expirience outages during automatic backups.
Generally, they are not even needed.
We should use hooks to service-specific tasks, such as
creating the database dump, so we don't have to shut down Postgres.

inex added 3 commits 2023-07-20 18:28:12 +03:00

0245d629fd refactor(backups): linting

4b575b6138 fix(graphql): duplicate imports

ac236569b0 chore: bump version to 2.2.0

inex approved these changes 2023-07-20 18:36:11 +03:00

inex merged commit f965546ca6 into master 2023-07-20 18:42:57 +03:00

inex deleted branch restic-rewrite 2023-07-20 18:42:57 +03:00

inex referenced this issue from a commit 2023-07-20 18:42:58 +03:00

Merge pull request 'Rewrite backups' (#35) from restic-rewrite into master

Sign in to join this conversation.

Reviewers

No reviewers

inex

Labels

Clear labels   

Bug

  

Contributions welcome

  

Did not do

  

Duplicate

  

Feature

  

Module

Backups

  

Module

GraphQL

  

Priority

High

  

Priority

Low

  

Priority

Medium

  

Refactor

  

Severity

High

  

Severity

Low

  

Severity

Medium

No Label

Bug

Contributions welcome

Did not do

Duplicate

Feature

Module

Backups

Module

GraphQL

Priority

High

Priority

Low

Priority

Medium

Refactor

Severity

High

Severity

Low

Severity

Medium

Milestone

Clear milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

No Assignees

2 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: SelfPrivacy/selfprivacy-rest-api#35

There is no content yet.