Rediflow
English ▾

How to populate fixtures

Supported fixtures: Example TKI projects (JSON), example non-working days (JSON under scripts/data/), EU projects from org-details (JSON from eu_playwright), export-from-DB (example-compatible JSON), and comprehensive testing data (WPs, tasks, 120 persons, assignments).
Deprecated: Example project from docs/example.ods and ODS-based seed (seed_from_ods.py, load_example_project). Deprecated fixtures remain for tests and legacy use; prefer example fixtures where applicable.

For a short “push test data to DB” guide with a recommended flow and one-shot commands, see push-test-data-to-db.

Projects list with fixtures

Prerequisites

  • Database: Containerised (Podman) PostgreSQL 18+ running; DATABASE_URL set (e.g. from .env.dev).
  • Migrations: Applied. Run uv run python main.py migrate first.

To start from a clean database (only project states, no orgs or lookups), see Clean database start.

Example TKI projects (supported)

To seed example TKI projects from the downloaded JSON (schools/institutes, funding programmes, strength areas, and one project per entry):

  1. Download the data (if not already present) from the fixture repo. The download script must accept --org (default: EUINI); e.g. zz_download_example_rdi_data.py --org EUINI or place active_projects_by_school.json in tmp/downloaded_data/ as needed.
  2. Seed example projects:
    uv run python scripts/seed_example_projects.py --top-org EXAMPLE
    Optional: --json / -j path to a different active_projects_by_school.json.

The seed analyses the import file to infer the organisation structure: keys like Technology School (IT Institute) are parsed as school (parent) + institute (leaf). It then creates orgs under the given top org (e.g. EXAMPLE) in dependency order: level-1 (schools and standalone units) first, then level-2 (institutes under schools). Projects are linked to the leaf org (e.g. IT Institute, not Technology School). It also creates funding programme and strength area groups and one project per JSON entry with planned dates, project owner org, funding programme, and strength area. Idempotent: safe to run multiple times; skips existing orgs by short_name and existing projects by name.

Using an existing top org: If you already have an org with the given short_name (e.g. EXAMPLE from another fixture or manual setup), the seed uses it as the top-level organisation and only creates the child orgs if missing. To require the top org to exist and never create it, pass --no-create-top-org. Pass --top-org YOUR_CODE to use a different top org.

Example non-working days (HOME org calendar)

Example (HOME org) non-working days live under scripts/data/example_non_working_days.json (list of "YYYY-MM-DD" strings). To insert them into the DB for the org and all descendant orgs (or, with inheritance, set on the top org only; child orgs inherit if they have none of their own):

uv run python scripts/seed_example_non_working_days.py --org EXAMPLE

Optional: --json path to a different JSON file; --ods docs/example.ods to parse from ODS instead of JSON. To (re)build the JSON from an ODS (e.g. after updating the calendar sheet):

uv run python scripts/seed_example_non_working_days.py --org EXAMPLE --ods docs/example.ods --export-json

Requires the org to exist (e.g. from scripts/seed_example_projects.py or scripts/seed_from_ods.py). Idempotent: skips dates already present per org.

EU projects (begin/end dates, acronyms, awarded PMs/EUR)

To populate EU projects (from EU Funding & Tenders org-details) with long name, acronym, begin/end dates, and awarded person-months or EUR:

  1. Fetch EU data (e.g. via scripts/eu_portal/dig_eu_projects_for_org.py --org EUINI or eu_playwright CLI with --output-slug euini) → tmp/downloaded_data/eu_playwright/org_details_<slug>_results.json.
  2. Optional: Enrich dates from saved HTML: uv run python scripts/eu_portal/update_org_details_dates_from_saved_pages.py.
  3. Import into DB: uv run python scripts/eu_portal/seed_eu_projects_from_org_details.py (options: --json, --top-org, --name-style title|acronym|acronym_title, --dry-run).

See populate-eu-projects-from-org-details for full steps and options.

Export example fixture from database

To make a fixture from what is in the database (Option B: example-compatible JSON):

  1. Export org tree and projects under a root org (e.g. EXAMPLE) to the same JSON shape as active_projects_by_school.json:

    uv run python scripts/export_fixture_from_db.py --root-org EXAMPLE
uv run python scripts/export_fixture_from_db.py --root-org EXAMPLE -o tmp/downloaded_data/example/active_projects_by_school_from_db.json

    Default output: tmp/downloaded_data/example/active_projects_by_school_from_db.json.

  2. Reload the fixture (e.g. in another DB or after a reset) with the example seed script:

    uv run python scripts/seed_example_projects.py --top-org EXAMPLE --json tmp/downloaded_data/example/active_projects_by_school_from_db.json --no-create-top-org

    Use --no-create-top-org when the top org already exists; omit it to create the top org if missing.

The export script writes by_school keys from the org tree: direct children of the root use the org’s display name; deeper orgs use "Parent (Leaf)". Projects are grouped by their project owner org’s key, with name, school_label, toteutusaika, rahoitusohjelma, and strength_area so the example seed remains idempotent.

Export and re-import settings data

You can export all system-wide settings and lookup data to a single JSON file and load it later (e.g. on a clean database or another environment). This includes: role type groups and role types, deliverable type groups and types, dissemination level groups and levels, WP theme groups and themes, task theme groups and themes, task type of action groups and types, country holidays, leave types, allocation groups, home org roles, worker groups and their rates, and orgs' country (calendar) codes.

  1. Export (run anytime):

    uv run python scripts/export_settings_data.py
uv run python scripts/export_settings_data.py -o tmp/settings_export.json
uv run python scripts/export_settings_data.py --country-holidays-grouped  # grouped format for many countries

    Default output: tmp/settings_export.json. Use --country-holidays-grouped for smaller output when you have many countries (exports as {"FI": ["2025-12-23", ...], "SE": [...]}).

  2. Import (run after migrations; idempotent):

    uv run python scripts/seed_settings_data.py --json tmp/settings_export.json
uv run python scripts/seed_settings_data.py -j path/to/export.json --dry-run  # preview

    Creates or updates by natural key (name, code, group). Safe to run multiple times. Use --dry-run to see what would be added without committing. Country holidays: accepts both flat array [{"country_code": "FI", "holiday_date": "..."}] and grouped format {"FI": ["2025-12-23", ...], "SE": [...]}; use grouped for many countries (smaller, easier to maintain).

Export project data (full structure)

You can export one or all projects to a single JSON file containing settings, WPs, WBS, tasks, milestones, deliverables, and project role type codes (short names). All org and lookup references are exported by stable identifier (org short_name, type/lifecycle codes) so the data can be re-imported on another DB that has the same orgs and settings.

  1. Export (run from project root; requires DATABASE_URL and migrations):

    uv run python scripts/export_projects_data.py
uv run python scripts/export_projects_data.py --format ods
uv run python scripts/export_projects_data.py --project-id 1 -o out.ods
uv run python scripts/export_projects_data.py --project-uuid <uuid> -o tmp/my_projects.json

    With no --project-id or --project-uuid, all projects are exported. Use --format json (default) or --format ods for LibreOffice Calc. Default output: tmp/projects_export.json or tmp/projects_export.ods.

  2. Output shape (JSON): meta (source, scope, exported_at_utc, project_count) and projects array. Each project has settings, wps, wbs, tasks, milestones, deliverables, and role_type_codes (role type short names used for the project). WBS rows use participant_org_short_name and wp1_pms..wp8_pms; tasks/milestones/deliverables reference WPs by wp_number.

  3. Output shape (ODS): One sheet per entity: Projects (one row per project, settings as columns; project_name first), WPs, WBS, Tasks, Milestones, Deliverables, Role_codes. Each data sheet has a project_name column so you can filter or sort by project in Calc.

An import script to load this JSON (create/update projects and children by resolving org and lookup codes) is a possible follow-up; the export is intended for backup, migration, or feeding into such a loader.

Deprecated: Example project from ODS

Deprecated. Prefer example fixture or export-from-DB (see above).

Populate (default example project)

From the project root:

export $(grep -v '^#' .env.dev | xargs)   # or source your env
uv run python main.py migrate
uv run python scripts/seed_from_ods.py

This uses docs/example.ods (or docs/content.xml if the ODS is missing), ensures lookup data exists, and creates or updates the Example project and its orgs, WPs, WBS, tasks, milestones, deliverables, and EST Capa. Idempotent: safe to run multiple times. The default app startup (e.g. fishy.fish) does not run this; run it manually when you want example data.

Populate from another ODS file

To import a specific ODS (e.g. a per-org export):

uv run python scripts/seed_from_ods.py --ods tmp/ods_per_org/example_ACR.ods

All per-org projects

uv run python scripts/build_ods_per_org.py   # optional: build ODS files
uv run python scripts/seed_all_ods_per_org.py

See import-per-org-ods.

Tests

In tests, use the db_with_example pytest fixture so the DB is migrated and seeded before the test (uses deprecated load_example_project). No need to run the seed script manually for tests.