Data Importer (WissKI pipeline UI)

Web application for importing tabular data into WissKI: upload CSV, map columns to a live Pathbuilder model, load staging tables, and import entities via the WissKI API. PostgreSQL is mandatory — it stores app state, staging tables, and Settings credentials. WissKI API and SPARQL are configured in Settings after login.

This repository is deployment-agnostic: it ships no site mappings, CSV dumps, or host credentials. Configure connections and mappings per project in the UI (or import your own mapping JSON).

Project layout

Directory	Role
core/wisski_data_importer/	Pure Python library: Pathbuilder model, staging DDL, RAW→staging SQL compiler, WissKI import, SPARQL helpers
backend/	Django API and orchestration
frontend/	Next.js SPA
docs/	MkDocs documentation
tests/	Pytest suite

Install (Docker, standalone)

From this directory:

cp env-example .env
# Edit: WISSKI_DATA_IMPORTER_DOMAIN, WISSKI_DATA_IMPORTER_DJANGO_SECRET_KEY,
#       WISSKI_DATA_IMPORTER_AUTH_USER, WISSKI_DATA_IMPORTER_AUTH_PASSWORD

docker compose up -d --build

Open http://localhost:8000/ (or your WISSKI_DATA_IMPORTER_DOMAIN behind your own TLS reverse proxy). Configure WissKI API and SPARQL in Settings after logging in. PostgreSQL starts automatically with docker compose up (see docker-compose.yml).

The image installs wisski_py via Compose additional_contexts (default: GitHub main). Point WISSKI_PY_CONTEXT or override build.additional_contexts.wisski_py at a local fork when needed.

Custom integration: Compose auto-merges docker-compose.override.yml if present. That file is gitignored — use it to attach the app to an existing stack (e.g. nginx sub-path on a shared Docker network). See docs/deployment.md.

Develop locally (no image rebuild)

docker compose up -d postgres   # required before dev script
pip install -e ".[wisski-data-importer]"
cd frontend && pnpm install
cd .. && ./scripts/dev-wisski-data-importer.sh

Open http://127.0.0.1:5173/wisski-data-importer/ (Next proxies the API). Details: docs/development.md.

Optional Docker API with bind mounts:

docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile dev up -d wisski-data-importer-web-dev

Documentation

pip install -e ".[docs]"
mkdocs serve   # http://127.0.0.1:8000

• User guide — stations and Settings in the GUI
• Architecture — core / backend / frontend boundaries
• Deployment — standalone vs reverse-proxy override
• Development
• When to rebuild

Contribute

• Match existing style in the touched area; keep diffs focused.
• Run tests from this directory: pytest (requires pip install -e ".[wisski-data-importer]").
• Do not commit .env, docker-compose.override.yml, uploads, CSV dumps, or credentials.