Compare commits

..

230 Commits

Author SHA1 Message Date
Houkime 9f19c677d0 feature(backup):calculate needed space for inplace restoration
continuous-integration/drone/push Build is failing Details
2023-07-14 12:34:45 +00:00
Houkime 926d0c27c5 test(backup): test moving preventing backups
continuous-integration/drone/push Build is failing Details
2023-07-14 11:41:03 +00:00
Houkime 4aa87edf47 fix(servers): hopefully fix moving
continuous-integration/drone/push Build is failing Details
2023-07-14 10:58:31 +00:00
Houkime 92cfd00f93 feature(servers): set default timeout of server operations to 10 min
continuous-integration/drone/push Build is failing Details
2023-07-12 17:03:24 +00:00
Houkime e43478d437 feature(backups): stop services before restores
continuous-integration/drone/push Build is failing Details
2023-07-12 16:53:49 +00:00
Houkime 7af7600599 feature(backups): stop services before backups 2023-07-12 16:43:26 +00:00
Houkime 326e3d3b0c test(backups): do not store the status file in backupped folders 2023-07-12 16:00:36 +00:00
Houkime de8ef744eb refactor(backups): make a StoppedService context manager 2023-07-12 15:14:54 +00:00
Houkime 713296c520 test(backups): make delay settable per dummyservice 2023-07-12 12:27:55 +00:00
Houkime e7a6700522 test(backups): test async service start n stop simulation 2023-07-12 12:10:40 +00:00
Houkime 169e9ad57d test(backups): simulating async service start n stop 2023-07-10 17:03:10 +00:00
Houkime 6523105d89 feature(utils): a hopefully reusable waitloop 2023-07-10 10:33:01 +00:00
Houkime 1cefaefa3b feature(backups): expose restore strategies to the API
continuous-integration/drone/push Build is failing Details
2023-07-07 13:24:05 +00:00
Houkime 2743441e1e fix(backups): actually mount if asked for an inplace restore 2023-07-07 13:14:30 +00:00
Houkime 9075afd38a test(backups): test out that pre-restore backup plays nice with jobs 2023-07-07 12:50:20 +00:00
Houkime af5edb695f feature(backups): implement inplace restore strategy 2023-07-07 11:54:48 +00:00
Houkime c74b3df32c feature(backups): restore strategies enum 2023-07-07 10:50:59 +00:00
Houkime 53bb5cc4e2 feature(backups): forgetting snapshots
continuous-integration/drone/push Build is failing Details
2023-07-05 13:13:30 +00:00
Houkime 03313b739a feature(backups): check restore exit code
continuous-integration/drone/push Build is failing Details
2023-07-03 20:53:21 +00:00
Houkime 3dc6fb91f2 refactor(backups): download a copy before replacing original
continuous-integration/drone/push Build is failing Details
2023-07-03 20:41:52 +00:00
Houkime b6eb27dc5e feature(backups): mounting a repo
continuous-integration/drone/push Build is failing Details
2023-07-03 15:32:00 +00:00
Houkime c5088e0e2c test(backups): remove the 100mb file after test 2023-07-03 15:32:00 +00:00
Houkime 953860a02c refactor(backups): move syncing (non-restic) into backup utils 2023-07-03 15:32:00 +00:00
Houkime 6a00d3cff9 refactor(backups): move output yielding into backup utils 2023-07-03 15:32:00 +00:00
Houkime 08cc7740b3 test(backups): actually list folders 2023-07-03 15:32:00 +00:00
Houkime 9793201ca1 feature(backups): a wrapper for rclone sync 2023-07-03 15:32:00 +00:00
Inex Code 64e7afe53e feat(backups): Add backup descriptions for UI
continuous-integration/drone/push Build is failing Details
2023-06-29 14:27:08 +03:00
Inex Code 15ce344bc8 feat(backups): expose if the service can be backed up
continuous-integration/drone/push Build is failing Details
2023-06-29 13:45:00 +03:00
Inex Code b480c84041 style: fix styling 2023-06-29 13:44:29 +03:00
Houkime cd2e9d3ba3 fix(jobs): make finishing the job set progress to 100
continuous-integration/drone/push Build is failing Details
2023-06-28 13:22:53 +00:00
Houkime 3ee90617ba test(backup): make large testfile larger
continuous-integration/drone/push Build is failing Details
2023-06-28 13:06:03 +00:00
Inex Code 11184a55e8 feat(jobs): return type_id of the job in graphql api
continuous-integration/drone/push Build is failing Details
2023-06-28 15:56:30 +03:00
Houkime e4865aa094 fix(services): proper backup progress reporting
continuous-integration/drone/push Build is failing Details
2023-06-28 11:45:07 +00:00
Houkime 35258bad38 fix(services): cleanup a stray get_location
continuous-integration/drone/push Build is failing Details
2023-06-28 09:48:32 +00:00
Houkime 59fef1d016 fix(backups): register the correct tasks
continuous-integration/drone/push Build is failing Details
2023-06-28 09:41:03 +00:00
Inex Code d9b26e12e2 fix(backups): missing space in rclone args
continuous-integration/drone/push Build is failing Details
2023-06-28 12:10:12 +03:00
Houkime ae16a527a2 refactor(backups): rename service_snapshot_size to snapshot_restored_size
continuous-integration/drone/push Build is failing Details
2023-06-26 19:41:26 +00:00
Houkime 7147f97077 refactor(backups): remove the by-service getting of cached snapshots 2023-06-26 19:29:29 +00:00
Houkime bae81b2f69 refactor(backups): rename force_snapshot_reload to force_snapshot_cache_reload 2023-06-26 19:20:49 +00:00
Houkime a76834d1ba refactor(backups): merge sync_all_snapshots with force_snapshot_reload 2023-06-26 19:16:18 +00:00
Houkime f8029d133a refactor(backups): straighten get_all_snapshots 2023-06-26 19:01:26 +00:00
Houkime 4ad4c3cc67 refactor(backups): delete sync_service_snapshots 2023-06-26 18:50:37 +00:00
Houkime b9be0be6a2 refactor(backups): privatize assert_restorable and restore_snapshot_from_id 2023-06-26 18:42:26 +00:00
Houkime 2f2c4f14af refactor(backups): group operations together 2023-06-26 18:30:31 +00:00
Houkime e6efd1b42d refactor(backups): move reset() to top because toplevel interface 2023-06-26 18:21:50 +00:00
Houkime 1593474dc1 refactor(backups): make redis and json provider related lowlevels private 2023-06-26 18:20:22 +00:00
Houkime 71eeed926d refactor(backups): make construct_provider not public 2023-06-26 18:14:15 +00:00
Houkime 81d1762518 refactor(backups): make lookup_provider not public 2023-06-26 18:11:11 +00:00
Houkime e159d2f1a7 refactor(backups): reorder imports 2023-06-26 18:07:47 +00:00
Houkime 8604caa331 refactor(backups): api readability reorg 2023-06-26 18:00:42 +00:00
Houkime 273a1935a8 feature(dev_qol): mypy type checking and rope refactoring support
continuous-integration/drone/push Build is failing Details
2023-06-23 12:17:48 +00:00
Houkime 2f71469f39 refactoring(backups): backuper -> backupper
continuous-integration/drone/push Build is failing Details
2023-06-23 12:04:33 +00:00
Houkime 6ca68fae17 fix(tokens-repo): persistent hashing
continuous-integration/drone/push Build is failing Details
2023-06-23 11:54:40 +00:00
Inex Code 1664f857ea fix(backups): Providers were not initialized corretly
continuous-integration/drone/push Build is failing Details
2023-06-23 14:36:16 +03:00
Houkime b86d0cd850 fix(redis): Do not shut down redis on ctrl c
see https://github.com/NixOS/nix/issues/2141
2023-06-23 14:35:40 +03:00
Inex Code 1f558d6cf9 ci: only run on push event
continuous-integration/drone/push Build was killed Details
2023-06-23 13:02:52 +03:00
Inex Code 0f1d8e22f2 ci: ignore the failure when trying to kill redis
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
2023-06-23 12:57:39 +03:00
Inex Code e7e0fdc4a1 refactor(backups): fix typing errors
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
2023-06-23 12:40:10 +03:00
Inex Code c9cfb7d7bc ci: fix killing redis-server 2023-06-23 04:50:00 +03:00
Inex Code c6919293b6 refactor(api): Group mutations
I've learned that there is no problem in grouping mutations like we do with queries.
This is a big mistake from my side, now we have legacy not so conveniently placed endpoints.
I've grouped all mutations, left the copies of old ones flattened in the root for backwards compatibility.
We will migrate to mutation groups on client side, and backups now only use grouped mutations.
Tests are updated.
2023-06-21 06:46:56 +03:00
Houkime e3a87f1d98 test(backups): ensure asking to reload snaps does not explode the server 2023-06-19 14:12:50 +00:00
Houkime 25f3115c05 test(backups): setting autobackup period 2023-06-19 13:28:02 +00:00
Houkime b5e2499a30 refactor(backups): delete legacy provider setting 2023-06-19 11:17:09 +00:00
Houkime 612a46ee42 feature(backups): actually dealing with situation when the provider is not configured 2023-06-19 11:09:10 +00:00
Houkime 4f2f24daac refactor(backups): NoneBackupper class for those cases when we do not know 2023-06-16 16:03:32 +00:00
Houkime 3449837de9 refactor(backups): make a dir for backuppers 2023-06-16 15:50:31 +00:00
Houkime f02e27cf06 test(backups): add a backend json reset test 2023-06-16 15:19:22 +00:00
Houkime 79c3b9598c feature(backups): resetting json config too 2023-06-16 15:09:39 +00:00
Houkime 47aee3c1f1 fix(backups): fix output API return types for configuration 2023-06-16 14:39:00 +00:00
Houkime 8e1e37c766 test(backups): preliminary test of repo reset 2023-06-16 14:33:48 +00:00
Houkime 33c60f971d test(backups): test reinitting repository 2023-06-16 13:43:41 +00:00
Houkime b3724e240e refactor(backups): cleanup localfile-specific logic 2023-06-14 14:17:04 +00:00
Houkime 9e0b0d8caa refactor(backups): make localfile repos normal 2023-06-14 14:07:51 +00:00
Houkime cbedd196e4 fix(backups): non-nullable service when backing up 2023-06-14 13:06:32 +00:00
Houkime 4862cdc72f test(backups): test restore 2023-06-14 13:00:39 +00:00
Houkime 17c7dffb07 test(backups): use get_data 2023-06-14 12:41:45 +00:00
Houkime 39cd4b714b test(backups): check snapshots getting created 2023-06-14 12:35:41 +00:00
Houkime 92322d8fad test(backups): test dummy service compliance 2023-06-14 12:08:07 +00:00
Houkime 21791f20a2 test(backups): make dummy service more compliant 2023-06-14 11:55:46 +00:00
Houkime aa9d082adc test(backups): display errors from api 2023-06-14 11:54:24 +00:00
Houkime f1654c699c fix(backups): shorten snapshot query signature 2023-06-14 11:52:57 +00:00
Houkime 928f026e7c test(backups): snapshot query 2023-06-14 11:14:52 +00:00
Houkime 8554879dc2 Merge remote-tracking branch 'upstream/restic-rewrite-api' into restic-rewrite 2023-06-14 10:27:50 +00:00
Houkime c47977c100 test(backups): actual testfile 2023-06-14 10:16:53 +00:00
Houkime d9bde6930b fix(backups): register queries 2023-06-14 10:16:53 +00:00
Houkime 865e304f42 test(backups): test backup API - backing up 2023-06-14 10:16:31 +00:00
Inex Code d356fad534 fix(backups): Handle orphaned snapshots 2023-06-14 02:52:10 +03:00
Inex Code f4df1f6a62 fix(backups): return type of encryption key 2023-06-14 01:40:53 +03:00
Inex Code 0ef6569d97 fix(backups): try to actually get backup configuration 2023-06-14 00:43:01 +03:00
Inex Code f950dd1e93 feat(backups): register backups in GraphQL schema 2023-06-14 00:00:29 +03:00
Inex Code d02302c7b2 fix: BackupConfiguration argument order 2023-06-13 23:54:02 +03:00
Houkime 549b149aaf feature(backups): check available space before restoring 2023-06-07 16:33:13 +00:00
Houkime 981445d594 refactor(backups): expect one more error of restic json output parsing 2023-06-07 15:55:38 +00:00
Houkime 7f984b678f feature(backups): integration between restore and jobs 2023-06-07 15:05:58 +00:00
Houkime 98de85e569 fix(backups): return one job, not an array of one 2023-06-07 14:27:37 +00:00
Houkime 4b8abb6a66 fix(backups): return 400, not 300 2023-06-07 14:25:17 +00:00
Houkime 0c95c5913b BREAKING CHANGE(backups): support only individual service backup requests(combinable) 2023-06-07 14:11:22 +00:00
Houkime ff58bdccbb feature(backups): set autobackup period from gql 2023-06-05 11:49:37 +00:00
Houkime d6609b29e8 feature(backups): get all snapshots if requested by api 2023-06-05 11:36:58 +00:00
Houkime 2168037a10 refactor(backups): introduce get_all_snapshots() 2023-06-05 11:28:53 +00:00
Houkime 9d5335f62c feature(backups): feature(backups): return a snapshot from start_backup 2023-06-05 11:26:58 +00:00
Houkime 9a94f7624e feature(backups): a graphql call to invalidate cache 2023-06-01 16:12:32 +00:00
Houkime 50952d688a feature(backups): graphql mutation for restore 2023-06-01 14:03:40 +00:00
Houkime f7c0821675 fix(backups): return correct snapshots per service 2023-06-01 12:44:14 +00:00
Houkime 4b07d4de41 refactor(backups): global snapshots 2023-05-31 13:16:08 +00:00
Houkime 603ed2ddf9 feature(backups): start backup graphql API 2023-05-31 12:07:47 +00:00
Houkime 9c4d2a0ba5 feature(backups): drop repository call 2023-05-29 18:24:38 +00:00
Houkime c3696d3a4b feature(backups): global init instead of per-service 2023-05-29 16:50:14 +00:00
Houkime 345e2c10c4 feature(backups): a graphql query to get provider info 2023-05-29 16:12:31 +00:00
Houkime 97e4c529f6 feature(backups): init repo mutation 2023-05-29 15:34:26 +00:00
Houkime 0dc6f74754 fix(backups): make sure location and credentials get properly passed around 2023-05-22 16:01:57 +00:00
Houkime 03d4632465 refactor(backups): remove extraneous asserts from jobs 2023-05-17 20:27:42 +00:00
Houkime 9fab596f91 refactor(backups): refactor realtime updating 2023-05-17 20:27:42 +00:00
Houkime b1d0a80963 feature(backups): realtime progress updates of backups 2023-05-17 20:27:42 +00:00
Houkime 54a8e0b2b0 test(backups): break out obtaining finished jobs 2023-05-17 20:27:42 +00:00
Houkime 60806cd536 feature(backups): job progress logs 2023-05-17 20:27:42 +00:00
Houkime ddd2176a5d refactor(backups): break out job logs status prefix 2023-05-17 20:27:42 +00:00
Houkime deb857bca9 refactor(backups): use single repo and multiplex by tags 2023-05-17 20:27:42 +00:00
Houkime a6b3a5e590 feature(backups): deny adding a backup job if another one is already queued 2023-05-17 20:27:42 +00:00
Houkime e25aa2cb33 test(backups): test that the job has run 2023-05-17 20:27:42 +00:00
Houkime 7ddfad10d4 refactor(backups): quick-expiration logs of jobs status updates 2023-05-17 20:27:42 +00:00
Houkime b2e231ebae feature(backups): set job status to error if backup fails 2023-05-17 20:27:42 +00:00
Houkime 0bf18dcdc7 refactor(backups): cleanup unused imports in tasks 2023-05-17 20:27:42 +00:00
Houkime 396b42b69c test(backups): test jobs starting and finishing when from Backups 2023-05-17 20:27:42 +00:00
Houkime d340b0ca67 test(backups): break out a finished job checker 2023-05-17 20:27:42 +00:00
Houkime ad6cc5e1bc refactor(backups): make a backup job running when the backup code itself is executed 2023-05-17 20:27:42 +00:00
Houkime c0dfbb3ca2 refactor(backups): delete unused redis import from backups ckass 2023-05-17 20:27:42 +00:00
Houkime 3eb8382d9b feature(backups): also create a job if not called from a task 2023-05-17 20:27:42 +00:00
Houkime cb1dd1011e test(backups): assure that jobs are created and not duplicated 2023-05-17 20:27:42 +00:00
Houkime 108fca0eb3 feature(backups): simplest jobs intergration in tasks: created and finished 2023-05-17 20:27:42 +00:00
Houkime d1425561d9 feature(backups): restore task 2023-05-17 20:27:42 +00:00
Houkime 72f4fc8ae7 test(backups): test local secrets 2023-05-17 20:27:42 +00:00
Houkime fa360655fe refactor(services): make local secret setting public 2023-05-17 20:27:42 +00:00
Houkime 65acd3173a refactor(services): use fully generic foldermoves 2023-05-17 20:27:42 +00:00
Houkime dde86725b9 test(services): test derivation of foldermoves 2023-05-17 20:27:42 +00:00
Houkime ba72fadb8d test(services): test that we indeed return correct folders and owned folders from real services 2023-05-17 20:27:42 +00:00
Houkime 043675ce14 fix(services): folder methods typing 2023-05-17 20:27:42 +00:00
Houkime 238a656cd9 refactor(services): make a foldermove from owned path 2023-05-17 20:27:42 +00:00
Houkime ad66513f27 refactor(services): add folder owner derivation 2023-05-17 20:27:41 +00:00
Houkime c34eb6d447 refactor(services): add overridable get owner and get group 2023-05-17 20:27:41 +00:00
Houkime 0b7d2d0bf4 refactor(services): add OwnedPath struct 2023-05-17 20:27:41 +00:00
Houkime a664ab3dd4 refactor(services): remove special storage counting from pleroma 2023-05-17 20:27:41 +00:00
Houkime 3605a71c1d refactor(services): remove special storage counting from ocserv 2023-05-17 20:27:41 +00:00
Houkime 617f7f1628 refactor(services): remove special storage counting from nextcloud 2023-05-17 20:27:41 +00:00
Houkime 4475bcea45 documentation(services): move the storage count docstring to parent service class 2023-05-17 20:27:41 +00:00
Houkime 042a2e4cf2 refactor(services): remove special storage counting from mail 2023-05-17 20:27:41 +00:00
Houkime 312328af95 refactor(services): remove special storage counting from jitsi 2023-05-17 20:27:41 +00:00
Houkime c040f0825c refactor(services): remove special storage counting from gitea 2023-05-17 20:27:41 +00:00
Houkime f8edcac33f refactor(services): remove special storage counting from bitwarden 2023-05-17 20:27:41 +00:00
Houkime 789fd71e2f refactor(services): add a generic storage counter 2023-05-17 20:27:41 +00:00
Houkime a34b14449b refactor(services): make get_folders() a mandatory part of Service interface 2023-05-17 20:27:41 +00:00
Houkime 8a4f256c12 refactor(services): add get_folders() to the rest of the services 2023-05-17 20:27:41 +00:00
Houkime 0207b5a473 fix(services): use get_foldername() for moving around 2023-05-17 20:27:41 +00:00
Houkime 1c0e3f0f92 test(backups): implement get_folders() for gitea 2023-05-17 20:27:41 +00:00
Houkime 378f0ebd73 test(backups): implement get_folders() for bitwarden 2023-05-17 20:27:41 +00:00
Houkime 48359ffd20 test(backups): test 2-folder restoration 2023-05-17 20:27:41 +00:00
Houkime a68e94fad3 test(backups): actually back up 2 folders 2023-05-17 20:27:41 +00:00
Houkime 30ac990985 refactor(backups): set a list of folders for our dummy service 2023-05-17 20:27:41 +00:00
Houkime fa26379a68 refactor(backups): actually accept a list of folders 2023-05-17 20:27:41 +00:00
Houkime 08739f7ca8 refactor(backups): make api accept a list of folders 2023-05-17 20:27:41 +00:00
Houkime 919ba1ad03 refactor(backups): make a dedicated get_folders() function 2023-05-17 20:27:41 +00:00
Houkime 1b1052d205 refactor(services): rename get_location() to get_drive() 2023-05-17 20:27:41 +00:00
Houkime 0ae4192081 test(backups): register dummy service 2023-05-17 20:27:41 +00:00
Houkime 4018dca184 feature(backups): automatic backup 2023-05-17 20:27:41 +00:00
Houkime ec85f060f8 test(backups): test autobackup timing 2023-05-17 20:27:41 +00:00
Houkime 283c8d09cc refactor(backups): split out storage 2023-05-17 20:27:41 +00:00
Houkime f25e57c51a test(backups): test that we do use cache 2023-05-17 20:27:41 +00:00
Houkime 5ad9f50b94 feature(backups): enable snapshot cache usage 2023-05-17 20:27:41 +00:00
Houkime a2ff74244e feature(backups): add snapshot cache sync functions 2023-05-17 20:27:41 +00:00
Houkime 7699ba0d9b test(backups): test last backup date retrieval 2023-05-17 20:27:41 +00:00
Houkime 146b0ca02f feature(backups): add a datetime validator function for huey autobackups 2023-05-17 20:27:41 +00:00
Houkime d4cad61d56 test(backups): test setting autobackup period 2023-05-17 20:27:41 +00:00
Houkime 3c42d8c413 test(backups): test setting services as enabled for autobackups 2023-05-17 20:27:41 +00:00
Houkime e043720289 feature(backups): methods for autobackup period setting and getting 2023-05-17 20:27:41 +00:00
Houkime 417533fc04 fix(backups): remove self from static method 2023-05-17 20:27:41 +00:00
Houkime 45f33e2d31 feature(backups): check, set and unset service autobackup status 2023-05-17 20:27:41 +00:00
Houkime 18d5cd2b83 feature(backups): cache snapshots and last backup timestamps 2023-05-17 20:27:41 +00:00
Houkime ae065867b3 test(backups): test that we do return snapshot on backup 2023-05-17 20:27:41 +00:00
Houkime 2eb64db199 feature(backups): return snapshot info from backup function 2023-05-17 20:27:41 +00:00
Houkime 772b499b46 feature(backups): huey task to back up 2023-05-17 20:27:41 +00:00
Houkime 144e4e5e91 refactor(backups): make backups stateless 2023-05-17 20:27:41 +00:00
Houkime a2dd47130b feature(backups): repo init tracking 2023-05-17 20:27:41 +00:00
Houkime 4b2cecac8f feature(backups): provider storage and retrieval 2023-05-17 20:27:41 +00:00
Houkime 6524c98131 refactor(backups): add a provider model for redis storage 2023-05-17 20:27:41 +00:00
Houkime a0afe63b3d refactor(backups): redis model storage utils 2023-05-17 20:27:41 +00:00
Houkime ea6a047478 feature(backups): load from json 2023-05-17 20:27:41 +00:00
Houkime 4e329299e0 feat(backups): local secret generation and storage 2023-05-17 20:27:40 +00:00
Houkime 23ee958bfb feat(backups): sizing up snapshots 2023-05-17 20:27:40 +00:00
Houkime d62d7534d7 test(backups): test restoring a file 2023-05-17 20:27:40 +00:00
Houkime dc3032eb03 feat(backups): add restore_snapshot and restore_service_from_snapshot 2023-05-17 20:27:40 +00:00
Houkime 98e1c9ebaa feat(backups): a better error on failed snapshot retrieval 2023-05-17 20:27:40 +00:00
Houkime c32353fe9b feat(backups): return proper snapshot structs when listing 2023-05-17 20:27:40 +00:00
Houkime 8a1b3be9fa test(backups): reenable snapshot testing 2023-05-17 20:27:40 +00:00
Houkime b3633d1a96 feat(backups): throw an error on a failed backup 2023-05-17 20:27:40 +00:00
Houkime 01797cb922 fix(backups): singleton metaclass was screwing with tests 2023-05-17 20:27:40 +00:00
Houkime 06deb83b33 test(backups): localfile repo by default in tests 2023-05-17 20:27:40 +00:00
Houkime e091fbd4a2 feature(backups): throw an error if repo init fails 2023-05-17 20:27:40 +00:00
Houkime e944f4a267 test(backups): basic file backend init test 2023-05-17 20:27:40 +00:00
Houkime f1987edd99 feature(backups): register localfile backend 2023-05-17 20:27:40 +00:00
Houkime c493a49daa feature(backups): localfile repo 2023-05-17 20:27:40 +00:00
Houkime 865c8f5280 test(backups): test repo init 2023-05-17 20:27:40 +00:00
Houkime bd744ba211 refactor(backups): repo init service method 2023-05-17 20:27:40 +00:00
Houkime 86e2b90bef refactor(backups): add repo init 2023-05-17 20:27:40 +00:00
Houkime c9fd656181 refactor(backups): snapshotlist and local secret groundwork 2023-05-17 20:27:40 +00:00
Houkime a043392b7b test(backup): no snapshots 2023-05-17 20:27:40 +00:00
Houkime e447419f96 refactor(backup): snapshot model 2023-05-17 20:27:40 +00:00
Houkime 3afe196901 feature(backup): loading snapshots 2023-05-17 20:27:40 +00:00
Houkime 1568ee68a4 feature(backup): add a restore function to restic backuper 2023-05-17 20:27:40 +00:00
Houkime 3970524bce feat(backup): hooks 2023-05-17 20:27:40 +00:00
Houkime bf263f68b9 test(backup): use a backup service function 2023-05-17 20:27:40 +00:00
Houkime 7e243146db refactor(backup): add a backup function to Backups singleton class 2023-05-17 20:27:40 +00:00
Houkime d5881cc523 refactor(backup): add a placeholder Backups singleton class 2023-05-17 20:27:40 +00:00
Houkime 17df21964a test(backup): try to back up! 2023-05-17 20:27:40 +00:00
Houkime 6fa72dbac1 fix(backup): add memory backup class,forgot to add to git 2023-05-17 20:27:40 +00:00
Houkime 474f04f923 feat(backup): add backuping to restic backuper 2023-05-17 20:27:40 +00:00
Houkime 90c2524800 test(backup): make a testfile to backup 2023-05-17 20:27:40 +00:00
Houkime e54224f15d test(backup): init an in-memory backup class 2023-05-17 20:27:40 +00:00
Houkime a66a3f35fc feat(backup): add in-memory backup 2023-05-17 20:27:40 +00:00
Houkime f5066082c8 feat(backup): allow no auth 2023-05-17 20:27:40 +00:00
Houkime 934afeb531 test(backup): dummy service 2023-05-17 20:27:40 +00:00
Houkime af6a49b0a3 test(backup): provider class selection 2023-05-17 20:27:40 +00:00
Houkime 24d170b3cb feature(backups): copy cli logic to new restic backuper 2023-05-17 20:27:40 +00:00
Houkime 23e3d553e6 feature(backups): placeholders for the backupers and backup providers 2023-05-17 20:27:40 +00:00
Houkime a98314bb60 feature(backups): placeholders for the modules of the new backup system 2023-05-17 20:27:40 +00:00
Houkime 6d73405118 feature(backups): add backup structures and queries 2023-05-17 20:27:40 +00:00
Houkime 740b072b07 refactor(backup): do not use config file 2023-05-17 20:27:40 +00:00
Houkime cf3cbd179f refactor(backup): pass key and account to exec 2023-05-17 20:27:40 +00:00
Houkime 4871958534 refactor(backup): extract restic repo 2023-05-17 20:27:40 +00:00
Houkime 060117a355 refactor(backup): extract rclone args 2023-05-17 20:27:40 +00:00
Houkime 80f2c99ba4 refactor(backup): delete unused import 2023-05-17 20:27:40 +00:00
7 changed files with 68 additions and 65 deletions

1
.gitignore vendored
View File

@ -147,4 +147,3 @@ cython_debug/
# End of https://www.toptal.com/developers/gitignore/api/flask
*.db
*.rdb

64
api.nix Normal file
View File

@ -0,0 +1,64 @@
{ lib, python39Packages }:
with python39Packages;
buildPythonApplication {
pname = "selfprivacy-api";
version = "2.0.0";
propagatedBuildInputs = [
setuptools
portalocker
pytz
pytest
pytest-mock
pytest-datadir
huey
gevent
mnemonic
pydantic
typing-extensions
psutil
fastapi
uvicorn
(buildPythonPackage rec {
pname = "strawberry-graphql";
version = "0.123.0";
format = "pyproject";
patches = [
./strawberry-graphql.patch
];
propagatedBuildInputs = [
typing-extensions
python-multipart
python-dateutil
# flask
pydantic
pygments
poetry
# flask-cors
(buildPythonPackage rec {
pname = "graphql-core";
version = "3.2.0";
format = "setuptools";
src = fetchPypi {
inherit pname version;
sha256 = "sha256-huKgvgCL/eGe94OI3opyWh2UKpGQykMcJKYIN5c4A84=";
};
checkInputs = [
pytest-asyncio
pytest-benchmark
pytestCheckHook
];
pythonImportsCheck = [
"graphql"
];
})
];
src = fetchPypi {
inherit pname version;
sha256 = "KsmZ5Xv8tUg6yBxieAEtvoKoRG60VS+iVGV0X6oCExo=";
};
})
];
src = ./.;
}

2
default.nix Normal file
View File

@ -0,0 +1,2 @@
{ pkgs ? import <nixpkgs> {} }:
pkgs.callPackage ./api.nix {}

View File

@ -27,4 +27,4 @@ async def get_token_header(
def get_api_version() -> str:
"""Get API version"""
return "2.1.3"
return "2.1.2"

View File

@ -22,9 +22,6 @@ from selfprivacy_api.migrations.providers import CreateProviderFields
from selfprivacy_api.migrations.prepare_for_nixos_2211 import (
MigrateToSelfprivacyChannelFrom2205,
)
from selfprivacy_api.migrations.prepare_for_nixos_2305 import (
MigrateToSelfprivacyChannelFrom2211,
)
migrations = [
FixNixosConfigBranch(),
@ -34,7 +31,6 @@ migrations = [
CheckForFailedBindsMigration(),
CreateProviderFields(),
MigrateToSelfprivacyChannelFrom2205(),
MigrateToSelfprivacyChannelFrom2211(),
]

View File

@ -1,58 +0,0 @@
import os
import subprocess
from selfprivacy_api.migrations.migration import Migration
class MigrateToSelfprivacyChannelFrom2211(Migration):
"""Migrate to selfprivacy Nix channel.
For some reason NixOS 22.11 servers initialized with the nixos channel instead of selfprivacy.
This stops us from upgrading to NixOS 23.05
"""
def get_migration_name(self):
return "migrate_to_selfprivacy_channel_from_2211"
def get_migration_description(self):
return "Migrate to selfprivacy Nix channel from NixOS 22.11."
def is_migration_needed(self):
try:
output = subprocess.check_output(
["nix-channel", "--list"], start_new_session=True
)
output = output.decode("utf-8")
first_line = output.split("\n", maxsplit=1)[0]
return first_line.startswith("nixos") and (
first_line.endswith("nixos-22.11")
)
except subprocess.CalledProcessError:
return False
def migrate(self):
# Change the channel and update them.
# Also, go to /etc/nixos directory and make a git pull
current_working_directory = os.getcwd()
try:
print("Changing channel")
os.chdir("/etc/nixos")
subprocess.check_output(
[
"nix-channel",
"--add",
"https://channel.selfprivacy.org/nixos-selfpricacy",
"nixos",
]
)
subprocess.check_output(["nix-channel", "--update"])
nixos_config_branch = subprocess.check_output(
["git", "rev-parse", "--abbrev-ref", "HEAD"], start_new_session=True
)
if nixos_config_branch.decode("utf-8").strip() == "api-redis":
print("Also changing nixos-config branch from api-redis to master")
subprocess.check_output(["git", "checkout", "master"])
subprocess.check_output(["git", "pull"])
os.chdir(current_working_directory)
except subprocess.CalledProcessError:
os.chdir(current_working_directory)
print("Error")

View File

@ -2,7 +2,7 @@ from setuptools import setup, find_packages
setup(
name="selfprivacy_api",
version="2.1.3",
version="2.1.2",
packages=find_packages(),
scripts=[
"selfprivacy_api/app.py",