User Guide

This guide provides a step-by-step walkthrough for using the ExpEngine within the ExtremeXP framework.
It uses a simple demo experiment to illustrate all steps.

Prerequisites

Python (>= 3.10)
Java version 11 or 13 (required for Proactive as executionware)

Note

This guide has been tested on Unix-based systems. Windows usage is possible with minimal adjustments (e.g., virtual environment activation and path separators).

Step 1 — Set up your development environment

Open a terminal and navigate to your project directory.
```
cd /root/directory/of/your/project
```
Create a Python virtual environment:
```
python -m venv <environment_name>
```
Replace <environment_name> with something like env.

Activate the environment:

Unix/macOSWindows

source <environment_name>/bin/activate

.\<environment_name>\Scripts\activate

Install the eexp_engine package:
```
pip install eexp_engine
```
To upgrade to the latest version later:
```
pip install eexp_engine --upgrade
```

Step 2 — Configure your project

Next, you'll create a configuration file and define:

Main directories
Helper modules
Execution settings
Data management
Logging

Create the configuration file in the root folder:
```
touch eexp_config.py
```

Create a directory structure following this layout:

.
├── experiments/
├── datasets/
├── tasks/
└── dependencies/

Edit eexp_config.py:
- Check out the configuration example
- For detailed information, see the Configuration page

Step 3 — Create your experiment

From the root directory navigate to the experiments directory:
```
cd experiments
```
Create a demo experiment:
```
touch demo-experiment.xxp
```
All DSL files should have .xxp format.

Edit demo-experiment.xxp file with the following content:

example

workflow DemoWorkflow {
  START -> Task1 -> Task2 -> END;

  task Task1 {
    implementation "demo/task1";
  }

  task Task2;

  define input data InputFile;
  define output data OutputFile;

  configure data InputFile {
    path "demo/test.json";
  }

  configure data OutputFile {
    path "output/test_local/test_output.json";
  }

  InputFile --> Task1.Task1InputFile;
  Task1.Task1OutputFile --> Task2.Task2InputFile;
  Task2.Task2OutputFile --> OutputFile;
}

workflow AssembledWorkflow1 from DemoWorkflow {
  task Task2 {
    implementation "demo/Task1V1";
  }
}

workflow AssembledWorkflow2 from DemoWorkflow {
  task Task2 {
    implementation "demo_tasks/Task1V2";
  }
}

experiment DemoWP5Experiment {
  control {
    START -> S1 -> S2 -> END;
  }

  space S1 of AssembledWorkflow1 {
    strategy gridsearch;
    param_values demo_param_value = range(4, 6);
    task Task1 {
      param demo_param = demo_param_values;
    }
  }

  space S2 of AssembledWorkflow2 {
    strategy randomsearch;
    runs = 1;
    param_values demo_param_value = enum(6);
    task Task1 {
      param demo_param = demo_param_values;
    }
  }
}

For detailed information about the DSL syntax and available options, see the Experiment & Workflow DSL page

Step 4 — Create your dataset

From the root directory navigate to the datasets directory:
```
cd datasets
```
Create a directory for the demo experiment:
```
mkdir demo
```
Create the test dataset file:
```
cd demo
touch test.json
```
Edit test.json with your dataset content:
```
{
  "test_data": "your_data_here"
}
```

Step 5 - Create Task Specifications

In the demo-experiment.xxp we defined two Tasks inside the workflow block (Task1, Task2). Next up we are going to create and edit the corresponding <task>.xxp files for these two Tasks.

From the root directory navigate to the tasks directory:
```
cd tasks
```

Create .xxp files following this layout:

tasks
│──── demo
      ├── task1/
      │   ├── task.xxp
      ├── task2V1/
      │   ├── task.xxp
      └── task2V2/
            ├── task.xxp

Edit the files with the following examples:

task1 xxp file

 task Task1 {

 define input data Task1InputFile;
 define output data Task1OutputFile;

 define param demo_param {
     type Integer;
     default 10;
     range (4, 20, 2);
 }

 define metric ParamIncreasedBy5 {
     kind single-value;
     type Integer;
 }

 implementation "demo/Task1/task1.py";
 python_version "3.8";
 }

task2V1 xxp file

 task Task2V1 {

 define input data Task2InputFile;
 define output data Task2OutputFile;

 implementation "demo/task2V1/task2V1.py";

 }

task2V2 xxp file

 task Task2V2 {

 define input data Task2InputFile;
 define output data Task2OutputFile;

 implementation "demo/task2V2/task2V2.py";

 }

For comprehensive documentation on task DSL syntax and configuration options, refer to the Task DSL page.

Step 6 — Provide Task Implementations

From the root directory navigate to the tasks directory:
```
cd tasks
```

Create .py files following this layout:

tasks
│──── demo
      ├── task1/
      │   ├── task.xxp
      │   └── task1.py
      ├── task2V1/
      │   ├── task.xxp
      │   └── task2V1.py
      └── task2V2/
            ├── task.xxp
            └── task2V2.py

Edit the task.py files for each task we created with the following examples.

task1.py file

[sys.path.append(os.path.join(os.getcwd(), folder)) for folder in variables.get("dependent_modules_folders").split(",")]
import proactive_helper as ph

print("Running DemoWP5Task1")

dataset = ph.load_dataset(variables, resultMap, "DemoWP5Task1InputFile")

demo_param = variables.get("demo_param")
print(f"with value of demo_param: {demo_param}")

increment = 5
metric_name = "ParamIncreasedBy5"

print(f"Increasing this parameter by {increment} and adding the result to the metric {metric_name}")
resultMap.put(metric_name, int(demo_param) + increment)

ph.save_dataset(variables, resultMap, "DemoWP5Task1OutputFile", dataset).

task2V1.py file

[sys.path.append(os.path.join(os.getcwd(), folder)) for folder in variables.get("dependent_modules_folders").split(",")]
import proactive_helper as ph

print("Running DemoWP5Task2V1")

dataset = ph.load_dataset(variables, resultMap, "DemoWP5Task2InputFile")

ph.save_dataset(variables, resultMap, "DemoWP5Task2OutputFile", dataset).

task2V2.py file

[sys.path.append(os.path.join(os.getcwd(), folder)) for folder in variables.get("dependent_modules_folders").split(",")]
import proactive_helper as ph

print("Running DemoWP5Task2V2")

dataset = ph.load_dataset(variables, resultMap, "DemoWP5Task2InputFile")

ph.save_dataset(variables, resultMap, "DemoWP5Task2OutputFile", dataset)

For comprehensive documentation on task implementation depending on your ExecutionWare option, refer to the ExecutionWare section.

Note

Each task directory should include a .xxp file (DSL configuration) and a .py file (Task Code)

Step 7 - Run the experiment

Create run_experiment.py in your project root.

from eexp_engine import client
import eexp_config

exp_name = 'example_exp'

client.run(__file__, exp_name, eexp_config)

Run the experiment using:
```
python3 run_experiment.py
```