drupal_config_extract/README.md

# Drupal Configuration Selector

A Python utility to selectively copy configuration files from a source directory to a target directory based on file prefixes.

## Overview

This script helps you manage and transfer specific Drupal configuration files by:
- Reading a list of configuration file prefixes from a JSON file
- Identifying matching files in a source directory
- Copying selected files to a target directory
- Providing a TUI (Text User Interface) to manage the selection process

## Requirements

- Python 3.6+
- curses library (built into most Python installations on Linux/Mac)
- PyYAML (`pyyaml`)

## Installation and Setup

From the `configure_man/` directory:

```bash
cd configure_man

# Create a virtual environment
python3 -m venv .venv

# Activate it (Linux/macOS)
source .venv/bin/activate

# On Windows (PowerShell)
# .venv\Scripts\Activate.ps1

# Install dependencies
pip install --upgrade pip
pip install -r requirements.txt
```

To run the tool later, activate the venv again (`source .venv/bin/activate`) before calling `python3 config_selector.py`.

The script operates in the current working directory where you run it. It will create the following subdirectories if they don't exist:
- `ingest/` - Drop your exported Drupal config archive here (named `config*.tar.gz`)
- `original_config/` - YAML files are extracted here automatically from the ingest archive
- `new_recipe_config/` - Matched files will be copied here (with UUID and _core removed)
- `changed_files/` - Files that changed between versions will be placed here
- `ignored_files.yml` - Optional: List files to exclude from change tracking
- `deleted_files.csv` - Tracks files that should be deleted

## Usage

### Ingest: export from Drupal into `ingest/`

1. In the Drupal admin UI, open the **full configuration export** page: **Configuration → Development → Configuration synchronization → Export**, then use the **Full archive** export (direct URL path: `admin/config/development/configuration/full/export`).
2. Download the generated archive (a `.tar.gz` of all site configuration).
3. Place that file in the `ingest/` directory under this tool’s working directory. The filename must match `config*.tar.gz` (for example `config.tar.gz` as downloaded, or a descriptive name like `config-localhost_3001-2026-04-09.tar.gz`).
4. Run `python3 config_selector.py` (or `python3 config_selector.py --force` if `original_config/` already has files and you want to replace them from the new archive).

### Standard run (skip extraction if original_config/ already has files)

```bash
python3 config_selector.py
```

### Force re-extraction from the ingest archive

Use `--force` when you have placed a new archive in `ingest/` and want to replace the current `original_config/` contents:

```bash
python3 config_selector.py --force
```

### Workflow

1. Follow **Ingest** above to place a full export `config*.tar.gz` in `ingest/`.
2. Run the script. It extracts the archive into `original_config/` automatically (skipped on subsequent runs unless `--force` is passed).
3. The script will create a default `config_prefixes.json` in the current directory if it doesn't exist.
4. Use the TUI to manage your file selections and copy files.

## Hierarchical Navigation

The script organizes configuration files in a tree-like structure based on dot-separated prefixes:

- Files like `field.storage.node.comment.yml` are split into segments: `field` > `field.storage` > `field.storage.node` > etc.
- Use **RIGHT ARROW** to navigate deeper into a selected prefix
- Use **LEFT ARROW** to go back up one level in the hierarchy
- At each level, you see only the unique segments available at that level
- Each segment shows the total count of files under it
- Segments with children have a "▶" indicator
- You can perform actions (add, delete, select files) at any level in the hierarchy

## TUI Navigation

The Text User Interface provides the following controls:

- **UP/DOWN**: Navigate through the list of prefixes
- **LEFT/RIGHT**: Navigate through the hierarchy tree (drill down/go back)
- **SPACE**: Toggle between matched and unmatched prefixes view
- **ENTER**: Select individual files from the selected prefix
- **A**: Add a new prefix (press ESC or Enter with empty input to cancel)
- **O**: Choose a prefix from the list of unmatched prefixes
- **D**: Delete the selected prefix (in matched view) or add the selected prefix (in unmatched view)
- **C**: Copy all matched files to the new_recipe_config directory (with UUID and _core removal)
- **M**: Compare committed recipe vs `new_recipe_config`, copy new/modified to `changed_files/`, list deletions
- **S**: Save the current list of prefixes to the JSON file
- **Q**: Quit the application

## Individual File Selection

When you press **ENTER** on a prefix, a dialog opens that allows you to:

- In matched view: Select individual files to remove from the matching set
- In unmatched view: Select individual files to add to a matching prefix

Within the file selection dialog:
- Use **UP/DOWN** to navigate between files
- Press **SPACE** to toggle selection of a file
- Press **ENTER** to confirm your selection
- Press **ESC** to cancel

## Configuration File

The script uses a JSON file to store prefixes. The default file is created at first run:

```json
{
  "prefixes": [
    "put.your.prefixes.here"
  ]
}
```

You can modify this file directly or use the TUI to manage the prefixes.

## File Comparison and Change Tracking

The script can compare configuration files between recipe versions:

- **M**: Compare committed recipe to `new_recipe_config/`
  - Baseline: `recipes/wisski_default_data_model/config/` (committed recipe)
  - **New** files: present in `new_recipe_config/` but not in the committed recipe → copied to `changed_files/` (same layout, including `language/<lang>/…`)
  - **Modified** files: same relative path in both, different content → copied to `changed_files/`
  - **Deleted** files: present in the committed recipe but missing from `new_recipe_config/` → listed in `deleted_files.csv`
  - Each compare run **clears** `changed_files/` first, then repopulates it
  - `changed_files_manifest.csv` lists every copied path with `kind` = `new` or `modified`

Files matched in `ignored_files.yml` are not copied to `changed_files/` (whether new or modified).

This feature helps you track additions, updates, and removals relative to the committed recipe.

## Ignoring Files from Change Tracking

You can create an `ignored_files.yml` file to prevent specific configuration files from being copied to the `changed_files/` folder during comparison (for both **new** and **modified** paths). This is useful when certain files have intentional differences or should not be promoted from the export.

Example `ignored_files.yml`:

```yaml
field:
  name: field.field.wisski_individual.bb48a22d36f1c15cd16b143d23466812.field__vf__title
  reason: because we need target id instead of uuid
another_category:
  name: some.other.config.file.yml
  reason: custom modification required
```

Files listed in this YAML will be excluded from the `changed_files/` directory even if they are new or changed relative to the committed recipe.

## Directory Structure

```
/your/working/directory/
├── config_selector.py                          # Main script
├── config_prefixes.json                        # Configuration prefixes list
├── ignored_files.yml                           # Optional: List of files to ignore from change tracking
├── ingest/                                     # Drop config*.tar.gz archives here
├── original_config/                            # Extracted source YAML files (auto-populated from ingest/)
├── new_recipe_config/                          # Target directory where matched files will be copied (cleaned)
├── recipes/wisski_default_data_model/config/   # Committed recipe config (used as baseline for comparison)
├── changed_files/                              # New + modified YAML vs committed recipe (cleared each compare)
├── changed_files_manifest.csv                  # relative_path + kind (new|modified) for changed_files/
├── new_recipe_not_in_original.csv              # Paths in new_recipe_config missing from original export
└── deleted_files.csv                           # Paths in committed recipe missing from new_recipe_config
```

## Update WissKI Model

1. **Export config from WissKI** — download a full export from `/admin/config/development/configuration/full/export` and place the archive in `ingest/`.

2. **Run the selector (force)** — activate the virtualenv and start the script:

   ```bash
   source .venv/bin/activate
   python3 config_selector.py --force
   ```

3. **Copy and compare** — choose `copy`, then `compare`.

4. **Review changes** — inspect `changed_files_manifest.csv` and `deleted_files.csv`; control scrap and failures before continuing.

5. **Update the recipe** — run the selector again (without `--force`) and choose `update`:

   ```bash
   python3 config_selector.py
   ```

6. **Commit and push** — stage changes, then push with the helper script:

   ```bash
   git add /opt/drupal/private-files/drupal_config_extract/update
   bin/wisski-push.sh <your-name> "your commit message"
   ```

7. **Pull the recipe** — update the local recipe checkout:

   ```bash
   cd /opt/drupal/recipes/wisski_default_data_model
   git pull
   ```

8. **Reset the WissKI instance** — delete bundle, fields, and pathbuilder on the instance.

9. **Re-apply the recipe** — from the Drupal docroot:

   ```bash
   drush cr
   drush recipe ../recipes/wisski_default_data_model
   drush wisski-core:recreate-menus
   ```