SelfPrivacy
/
selfprivacy-nixos-config
6
3
Fork 6
Code Issues 17 Pull Requests 1 Projects Releases Wiki Activity

Develop the manifest format for packaging services #40

New Issue
Open
opened 2023-09-30 21:21:43 +03:00 by inex · 3 comments
inex commented 2023-09-30 21:21:43 +03:00
Owner

TBD.

The format should provide the API with at least the following:

  • Service name
  • Service description
    • Should be available in different languages
  • Service icon
  • Which systemd units have to be managed
  • Which folders to back up
  • Options to be provided as flake inputs
    • These options should have enough metadata to build the end-user UI
    • We should declare all the possible basic types beforehand
    • We may have advanced types which extend the basic type. This can be used to use more useful UI elements, while still allowing the setting to be changed on the old app version
    • Of course, the name, the icon, and the description for the option
  • Actions that can be made
    • Basically, we allow users to call the CLI commands from the user interface
    • Name, description, icon, which command it calls
    • There may be arguments?..
    • How do we protect from the potentially dangerous actions, when working with untrusted flakes?
  • Additional metadata for the "App store" listing
    • Screenshots?
    • Possible risks?
    • Anti-features? (Like, in F-Droid)
    • Support status? We have deprecated services.
  • Installer metadata
    • Basically, it's the options, but the ones we have to ask the user before installing the server
    • There also should be a domain
    • Speaking of domains, some services (federated ones) does not allow changing domain after installation. We must tell the user about it.
  • Possible relations to other SP modules, when the relation is optional?
    • Matrix may use the Jitsi server
    • Mastodon may use the LibreTranslate

Speaking of icons for options and actions, we may use the string IDs of the Material Icons.

Format should at least be readable by our Python API. JSON Schema? We could use existing work on JSON schemas to write UI generators and validators. We should have schema versioning from the beginning.

TBD. The format should provide the API with at least the following: - Service name - Service description - Should be available in different languages - Service icon - Which systemd units have to be managed - Which folders to back up - Options to be provided as flake inputs - These options should have enough metadata to build the end-user UI - We should declare all the possible basic types beforehand - We may have advanced types which extend the basic type. This can be used to use more useful UI elements, while still allowing the setting to be changed on the old app version - Of course, the name, the icon, and the description for the option - Actions that can be made - Basically, we allow users to call the CLI commands from the user interface - Name, description, icon, which command it calls - There may be arguments?.. - How do we protect from the potentially dangerous actions, when working with untrusted flakes? - Additional metadata for the "App store" listing - Screenshots? - Possible risks? - Anti-features? (Like, in F-Droid) - Support status? We have deprecated services. - Installer metadata - Basically, it's the options, but the ones we **have** to ask the user before installing the server - There also should be a domain - Speaking of domains, some services (federated ones) does not allow changing domain after installation. We must tell the user about it. - Possible relations to other SP modules, when the relation is optional? - Matrix may use the Jitsi server - Mastodon may use the LibreTranslate Speaking of icons *for options and actions*, we may use the string IDs of the Material Icons. Format should at least be readable by our Python API. JSON Schema? We could use existing work on JSON schemas to write UI generators and validators. We should have schema versioning from the beginning.
inex added this to the SelfPrivacy service packaging format project 2023-10-05 17:50:08 +03:00
inex added a new dependency 2023-10-05 17:50:46 +03:00
#41 Move E-mail server into SP module
inex added a new dependency 2023-10-05 17:51:19 +03:00
#42 Move Bitwarden into SP module
inex added a new dependency 2023-10-05 17:51:29 +03:00
#43 Change Gitea to Frogejo and move to SP module
inex added a new dependency 2023-10-05 17:51:39 +03:00
#44 Move Nextcloud into SP module
inex added a new dependency 2023-10-05 17:51:48 +03:00
#45 Move Pleroma into SP module
inex added a new dependency 2023-10-05 17:52:01 +03:00
#46 Move JItsi into SP module
alexoundos commented 2023-11-14 00:23:03 +02:00
Collaborator

Based on the current development progress, much of the manifest information can be retrieved from a SP module flake, such as:

  • SP module name

  • SP module description

  • systemd services (units?) to be added, configured, etc

  • configurable options (from NixOS module generated JSON documentation):

    • option path
    • option type
    • option description
    • option default value
    • option example

    Basically, such information is enough to generate search.nixos.org GUI. But it lacks icons and maybe some additional information for better options displaying in GUI.

  • a list of NixOS config paths, which a SP module needs (it adds a layer of security to forbid access to other parts of a NixOS configuration for untrusted SP modules), example: 4419a1323a/sp-modules/simple-nixos-mailserver/config-paths-needed.json

  • possible relations to other SP modules (and maybe other stock NixOS services) can be potentially derived by automatic analysis of common options paths

I referred to the manifest subject as "SP module", but the naming has not been settled and can be "SP service" or "SP extension", etc.

Based on the current development progress, much of the manifest information can be retrieved from a SP module flake, such as: - SP module name - SP module description - systemd services (units?) to be added, configured, etc - configurable options (from NixOS module [generated](https://gist.github.com/furrycatherder/a32e58c97c6cc4faaef7e6a1674864ac) JSON documentation): - option path - option type - option description - option default value - option example Basically, such information is enough to generate [search.nixos.org](https://search.nixos.org/options?channel=unstable&from=0&size=50&sort=relevance&type=packages&query=privacy) GUI. But it lacks icons and maybe some additional information for better options displaying in GUI. - a list of NixOS `config` paths, which a SP module needs (it adds a layer of security to forbid access to other parts of a NixOS configuration for untrusted SP modules), example: https://git.selfprivacy.org/SelfPrivacy/selfprivacy-nixos-config/src/commit/4419a1323adb05d542da1cd54c5dbfecb8aa2a77/sp-modules/simple-nixos-mailserver/config-paths-needed.json - possible relations to other SP modules (and maybe other stock NixOS services) can be potentially derived by automatic analysis of common options paths ---- I referred to the manifest subject as "SP module", but the naming has not been settled and can be "SP service" or "SP extension", etc.
alexoundos commented 2023-11-15 14:52:54 +02:00
Collaborator

Since much of the manifest data can be evaluated from Nix code of a SP module, it is not clear whether

  • the final manifest document should be part of the SP module or
  • generated from SP module (and stored somewhere, e.g. in a SP module hub)

It becomes more complex if we want to add more information for options, except those, which NixOS module has. It needs to be added either in:

  • NixOS module code
  • additional JSON file, which mimics NixOS module structure but with more fields
Since much of the manifest data can be evaluated from Nix code of a SP module, it is not clear whether - the final manifest document should be part of the SP module or - generated from SP module (and stored somewhere, e.g. in a SP module hub) It becomes more complex if we want to add more information for options, except those, which NixOS module has. It needs to be added either in: - NixOS module code - additional JSON file, which mimics NixOS module structure but with more fields
alexoundos commented 2024-02-01 14:38:19 +02:00
Collaborator

Example of NixOS options documentation generation in JSON format:

$ nix build --impure --expr '
let
  pkgs = (builtins.getFlake "nixpkgs").legacyPackages.x86_64-linux;
  sp-module = builtins.getFlake "git+https://git.selfprivacy.org/SelfPrivacy/selfprivacy-nixos-config?ref=flakes&dir=sp-modules/jitsi-meet";
in (pkgs.nixosOptionsDoc { inherit (pkgs.lib.evalModules { modules = [ { _module.check = false; } sp-module.nixosModules.default ]; }) options; }).optionsJSON'

It uses nixpkgs registry flake for nixosOptionsDoc and lib.evalModules functions. Alternatively exact nixpkgs commit can be specified as github:nixos/nixpkgs?rev=c1be43e8e837b8dbee2b3665a007e761680f0c3d.

sp-module variable is the flake where SP module is contained.

In case of success result/share/doc/nixos/options.json file will be produced like this:

{
  "selfprivacy.modules.jitsi-meet.enable": {
    "declarations": [
      "/nix/store/xh0a7sfim17hj2zxwlkprzm2b7n0izyy-source/module.nix"
    ],
    "default": {
      "_type": "literalExpression",
      "text": "false"
    },
    "description": "Enable jitsi-meet SP module",
    "loc": [
      "selfprivacy",
      "modules",
      "jitsi-meet",
      "enable"
    ],
    "readOnly": false,
    "type": "boolean"
  }
}
Example of NixOS options documentation generation in JSON format: ```console $ nix build --impure --expr ' let pkgs = (builtins.getFlake "nixpkgs").legacyPackages.x86_64-linux; sp-module = builtins.getFlake "git+https://git.selfprivacy.org/SelfPrivacy/selfprivacy-nixos-config?ref=flakes&dir=sp-modules/jitsi-meet"; in (pkgs.nixosOptionsDoc { inherit (pkgs.lib.evalModules { modules = [ { _module.check = false; } sp-module.nixosModules.default ]; }) options; }).optionsJSON' ``` It uses `nixpkgs` registry flake for `nixosOptionsDoc` and `lib.evalModules` functions. Alternatively exact nixpkgs commit can be specified as `github:nixos/nixpkgs?rev=c1be43e8e837b8dbee2b3665a007e761680f0c3d`. `sp-module` variable is the flake where SP module is contained. In case of success `result/share/doc/nixos/options.json` file will be produced like this: ```json { "selfprivacy.modules.jitsi-meet.enable": { "declarations": [ "/nix/store/xh0a7sfim17hj2zxwlkprzm2b7n0izyy-source/module.nix" ], "default": { "_type": "literalExpression", "text": "false" }, "description": "Enable jitsi-meet SP module", "loc": [ "selfprivacy", "modules", "jitsi-meet", "enable" ], "readOnly": false, "type": "boolean" } } ```
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
inex/test-collect-garbage
flakes
inex/test-autobackup
inex/test-systemd-rebuild
master
pre-release
flake-to-override-folder
flake-to-override
systemd-limits
ldap
api-redis
pleroma-fixes
development
rolling-testing
Labels
Clear labels   
Contributions welcome
   
Service packaging
   
bug

Something is not working   
duplicate

This issue or pull request already exists   
enhancement

New feature   
help wanted

Need some help   
invalid

Something is wrong   
question

More information is needed   
wontfix

This won't be fixed
No Label
Contributions welcome
Service packaging
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
SelfPrivacy service packaging format
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Blocks
#65 Support for the SelfPrivacy flake format
SelfPrivacy/selfprivacy-rest-api
#46 Move JItsi into SP module
SelfPrivacy/selfprivacy-nixos-config
Reference: SelfPrivacy/selfprivacy-nixos-config#40
There is no content yet.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?