Drupal Configuration Selector

A Python utility to selectively copy configuration files from a source directory to a target directory based on file prefixes.

Overview

This script helps you manage and transfer specific Drupal configuration files by:

• Reading a list of configuration file prefixes from a JSON file
• Identifying matching files in a source directory
• Copying selected files to a target directory
• Providing a TUI (Text User Interface) to manage the selection process

Requirements

• Python 3.6+
• curses library (built into most Python installations on Linux/Mac)
• PyYAML (pyyaml)

Installation and Setup

From the configure_man/ directory:

cd configure_man

# Create a virtual environment
python3 -m venv .venv

# Activate it (Linux/macOS)
source .venv/bin/activate

# On Windows (PowerShell)
# .venv\Scripts\Activate.ps1

# Install dependencies
pip install --upgrade pip
pip install -r requirements.txt

To run the tool later, activate the venv again (source .venv/bin/activate) before calling python3 config_selector.py.

The script operates in the current working directory where you run it. It will create the following subdirectories if they don't exist:

• ingest/ - Drop your exported Drupal config archive here (named config*.tar.gz)
• original_config/ - YAML files are extracted here automatically from the ingest archive
• new_recipe_config/ - Matched files will be copied here (with UUID and _core removed)
• changed_files/ - Files that changed between versions will be placed here
• ignored_files.yml - Optional: List files to exclude from change tracking
• deleted_files.csv - Tracks files that should be deleted

Usage

Ingest: export from Drupal into ingest/

1. In the Drupal admin UI, open the full configuration export page: Configuration → Development → Configuration synchronization → Export, then use the Full archive export (direct URL path: admin/config/development/configuration/full/export).
2. Download the generated archive (a .tar.gz of all site configuration).
3. Place that file in the ingest/ directory under this tool’s working directory. The filename must match config*.tar.gz (for example config.tar.gz as downloaded, or a descriptive name like config-localhost_3001-2026-04-09.tar.gz).
4. Run python3 config_selector.py (or python3 config_selector.py --force if original_config/ already has files and you want to replace them from the new archive).

Standard run (skip extraction if original_config/ already has files)

python3 config_selector.py

Force re-extraction from the ingest archive

Use --force when you have placed a new archive in ingest/ and want to replace the current original_config/ contents:

python3 config_selector.py --force

Workflow

1. Follow Ingest above to place a full export config*.tar.gz in ingest/.
2. Run the script. It extracts the archive into original_config/ automatically (skipped on subsequent runs unless --force is passed).
3. The script will create a default config_prefixes.json in the current directory if it doesn't exist.
4. Use the TUI to manage your file selections and copy files.

Hierarchical Navigation

The script organizes configuration files in a tree-like structure based on dot-separated prefixes:

• Files like field.storage.node.comment.yml are split into segments: field > field.storage > field.storage.node > etc.
• Use RIGHT ARROW to navigate deeper into a selected prefix
• Use LEFT ARROW to go back up one level in the hierarchy
• At each level, you see only the unique segments available at that level
• Each segment shows the total count of files under it
• Segments with children have a "▶" indicator
• You can perform actions (add, delete, select files) at any level in the hierarchy

TUI Navigation

The Text User Interface provides the following controls:

• UP/DOWN: Navigate through the list of prefixes
• LEFT/RIGHT: Navigate through the hierarchy tree (drill down/go back)
• SPACE: Toggle between matched and unmatched prefixes view
• ENTER: Select individual files from the selected prefix
• A: Add a new prefix (press ESC or Enter with empty input to cancel)
• O: Choose a prefix from the list of unmatched prefixes
• D: Delete the selected prefix (in matched view) or add the selected prefix (in unmatched view)
• C: Copy all matched files to the new_recipe_config directory (with UUID and _core removal)
• M: Compare committed recipe vs new_recipe_config, copy new/modified to changed_files/, list deletions
• S: Save the current list of prefixes to the JSON file
• Q: Quit the application

Individual File Selection

When you press ENTER on a prefix, a dialog opens that allows you to:

• In matched view: Select individual files to remove from the matching set
• In unmatched view: Select individual files to add to a matching prefix

Within the file selection dialog:

• Use UP/DOWN to navigate between files
• Press SPACE to toggle selection of a file
• Press ENTER to confirm your selection
• Press ESC to cancel

Configuration File

The script uses a JSON file to store prefixes. The default file is created at first run:

{
  "prefixes": [
    "put.your.prefixes.here"
  ]
}

You can modify this file directly or use the TUI to manage the prefixes.

File Comparison and Change Tracking

The script can compare configuration files between recipe versions:

• M: Compare committed recipe to new_recipe_config/
  • Baseline: recipes/wisski_default_data_model/config/ (committed recipe)
  • New files: present in new_recipe_config/ but not in the committed recipe → copied to changed_files/ (same layout, including language/<lang>/…)
  • Modified files: same relative path in both, different content → copied to changed_files/
  • Deleted files: present in the committed recipe but missing from new_recipe_config/ → listed in deleted_files.csv
  • Each compare run clears changed_files/ first, then repopulates it
  • changed_files_manifest.csv lists every copied path with kind = new or modified

Files matched in ignored_files.yml are not copied to changed_files/ (whether new or modified).

This feature helps you track additions, updates, and removals relative to the committed recipe.

Ignoring Files from Change Tracking

You can create an ignored_files.yml file to prevent specific configuration files from being copied to the changed_files/ folder during comparison (for both new and modified paths). This is useful when certain files have intentional differences or should not be promoted from the export.

Example ignored_files.yml:

field:
  name: field.field.wisski_individual.bb48a22d36f1c15cd16b143d23466812.field__vf__title
  reason: because we need target id instead of uuid
another_category:
  name: some.other.config.file.yml
  reason: custom modification required

Files listed in this YAML will be excluded from the changed_files/ directory even if they are new or changed relative to the committed recipe.

Directory Structure

/your/working/directory/
├── config_selector.py                          # Main script
├── config_prefixes.json                        # Configuration prefixes list
├── ignored_files.yml                           # Optional: List of files to ignore from change tracking
├── ingest/                                     # Drop config*.tar.gz archives here
├── original_config/                            # Extracted source YAML files (auto-populated from ingest/)
├── new_recipe_config/                          # Target directory where matched files will be copied (cleaned)
├── recipes/wisski_default_data_model/config/   # Committed recipe config (used as baseline for comparison)
├── changed_files/                              # New + modified YAML vs committed recipe (cleared each compare)
├── changed_files_manifest.csv                  # relative_path + kind (new|modified) for changed_files/
├── new_recipe_not_in_original.csv              # Paths in new_recipe_config missing from original export
└── deleted_files.csv                           # Paths in committed recipe missing from new_recipe_config

Update WissKI Model

1. Export config from WissKI — download a full export from /admin/config/development/configuration/full/export and place the archive in ingest/.

2. Run the selector (force) — activate the virtualenv and start the script:

   source .venv/bin/activate
   python3 config_selector.py --force
   

3. Copy and compare — choose copy, then compare.

4. Review changes — inspect changed_files_manifest.csv and deleted_files.csv; control scrap and failures before continuing.

5. Update the recipe — run the selector again (without --force) and choose update:

   python3 config_selector.py
   

6. Commit and push — stage changes, then push with the helper script:

   git add /opt/drupal/private-files/drupal_config_extract/update
   bin/wisski-push.sh <your-name> "your commit message"
   

7. Pull the recipe — update the local recipe checkout:

   cd /opt/drupal/recipes/wisski_default_data_model
   git pull
   

8. Reset the WissKI instance — delete bundle, fields, and pathbuilder on the instance.

9. Re-apply the recipe — from the Drupal docroot:

   drush cr
   drush recipe ../recipes/wisski_default_data_model
   drush wisski-core:recreate-menus