better change file handling

README.md

@@ -19,25 +19,42 @@ This script helps you manage and transfer specific Drupal configuration files by
 ## Installation and Setup
 
 The script operates in the current working directory where you run it. It will create the following subdirectories if they don't exist:
-- `original_config/` - Place your source configuration files here
+- `ingest/` - Drop your exported Drupal config archive here (named `config*.tar.gz`)
+- `original_config/` - YAML files are extracted here automatically from the ingest archive
 - `new_recipe_config/` - Matched files will be copied here (with UUID and _core removed)
-- `old_recipe_config/` - Optional: Place previous recipe version here for comparison
 - `changed_files/` - Files that changed between versions will be placed here
 - `ignored_files.yml` - Optional: List files to exclude from change tracking
 - `deleted_files.csv` - Tracks files that should be deleted
 
 ## Usage
 
-1. Copy or move the script to your working directory (or use the full path to run it)
-2. Place your source configuration files in the `original_config` directory
-3. Run the script:
+### Ingest: export from Drupal into `ingest/`
+
+1. In the Drupal admin UI, open the **full configuration export** page: **Configuration → Development → Configuration synchronization → Export**, then use the **Full archive** export (direct URL path: `admin/config/development/configuration/full/export`).
+2. Download the generated archive (a `.tar.gz` of all site configuration).
+3. Place that file in the `ingest/` directory under this tool’s working directory. The filename must match `config*.tar.gz` (for example `config.tar.gz` as downloaded, or a descriptive name like `config-localhost_3001-2026-04-09.tar.gz`).
+4. Run `python3 config_selector.py` (or `python3 config_selector.py --force` if `original_config/` already has files and you want to replace them from the new archive).
+
+### Standard run (skip extraction if original_config/ already has files)
 
 ```bash
 python3 config_selector.py
 ```
 
-4. The script will create a default `config_prefixes.json` in the current directory if it doesn't exist
-5. Use the TUI to manage your file selections and copy files
+### Force re-extraction from the ingest archive
+
+Use `--force` when you have placed a new archive in `ingest/` and want to replace the current `original_config/` contents:
+
+```bash
+python3 config_selector.py --force
+```
+
+### Workflow
+
+1. Follow **Ingest** above to place a full export `config*.tar.gz` in `ingest/`.
+2. Run the script. It extracts the archive into `original_config/` automatically (skipped on subsequent runs unless `--force` is passed).
+3. The script will create a default `config_prefixes.json` in the current directory if it doesn't exist.
+4. Use the TUI to manage your file selections and copy files.
 
 ## Hierarchical Navigation
 
@@ -63,7 +80,7 @@ The Text User Interface provides the following controls:
 - **O**: Choose a prefix from the list of unmatched prefixes
 - **D**: Delete the selected prefix (in matched view) or add the selected prefix (in unmatched view)
 - **C**: Copy all matched files to the new_recipe_config directory (with UUID and _core removal)
-- **M**: Compare old and new recipe configs, track changes and deletions
+- **M**: Compare committed recipe vs `new_recipe_config`, copy new/modified to `changed_files/`, list deletions
 - **S**: Save the current list of prefixes to the JSON file
 - **Q**: Quit the application
 
@@ -98,16 +115,21 @@ You can modify this file directly or use the TUI to manage the prefixes.
 
 The script can compare configuration files between recipe versions:
 
-- **M**: Compare old and new recipe configurations
-  - Compares files in `old_recipe_config/` with `new_recipe_config/`
-  - Copies changed files to `changed_files/` directory
-  - Tracks deleted files in `deleted_files.csv`
+- **M**: Compare committed recipe to `new_recipe_config/`
+  - Baseline: `recipes/wisski_default_data_model/config/` (committed recipe)
+  - **New** files: present in `new_recipe_config/` but not in the committed recipe → copied to `changed_files/` (same layout, including `language/<lang>/…`)
+  - **Modified** files: same relative path in both, different content → copied to `changed_files/`
+  - **Deleted** files: present in the committed recipe but missing from `new_recipe_config/` → listed in `deleted_files.csv`
+  - Each compare run **clears** `changed_files/` first, then repopulates it
+  - `changed_files_manifest.csv` lists every copied path with `kind` = `new` or `modified`
 
-This feature helps you track what has changed between recipe versions and what files need to be removed.
+Files matched in `ignored_files.yml` are not copied to `changed_files/` (whether new or modified).
+
+This feature helps you track additions, updates, and removals relative to the committed recipe.
 
 ## Ignoring Files from Change Tracking
 
-You can create an `ignored_files.yml` file to prevent specific configuration files from being copied to the `changed_files/` folder during comparison. This is useful when certain files have intentional differences that should not be tracked as changes.
+You can create an `ignored_files.yml` file to prevent specific configuration files from being copied to the `changed_files/` folder during comparison (for both **new** and **modified** paths). This is useful when certain files have intentional differences or should not be promoted from the export.
 
 Example `ignored_files.yml`:
 
@@ -120,18 +142,21 @@ another_category:
   reason: custom modification required
 ```
 
-Files listed in this YAML will be excluded from the `changed_files/` directory even if they have changed between versions.
+Files listed in this YAML will be excluded from the `changed_files/` directory even if they are new or changed relative to the committed recipe.
 
 ## Directory Structure
 
 ```
 /your/working/directory/
-├── config_selector.py      # Main script
-├── config_prefixes.json    # Configuration prefixes list
-├── ignored_files.yml       # Optional: List of files to ignore from change tracking
-├── original_config/        # Source directory containing all configuration files
-├── new_recipe_config/      # Target directory where matched files will be copied (cleaned)
-├── old_recipe_config/      # Previous version of recipe config (for comparison)
-├── changed_files/          # Files that have changed between old and new versions
-└── deleted_files.csv       # List of files that should be deleted from the recipe
+├── config_selector.py                          # Main script
+├── config_prefixes.json                        # Configuration prefixes list
+├── ignored_files.yml                           # Optional: List of files to ignore from change tracking
+├── ingest/                                     # Drop config*.tar.gz archives here
+├── original_config/                            # Extracted source YAML files (auto-populated from ingest/)
+├── new_recipe_config/                          # Target directory where matched files will be copied (cleaned)
+├── recipes/wisski_default_data_model/config/   # Committed recipe config (used as baseline for comparison)
+├── changed_files/                              # New + modified YAML vs committed recipe (cleared each compare)
+├── changed_files_manifest.csv                  # relative_path + kind (new|modified) for changed_files/
+├── new_recipe_not_in_original.csv              # Paths in new_recipe_config missing from original export
+└── deleted_files.csv                           # Paths in committed recipe missing from new_recipe_config
 ```