API reference

Experiment

class experitur.Experiment(name=None, parameters=None, configurator=None, parent=None, meta=None, active=True, volatile=None, minimize=None, maximize=None, depends_on=None, doc=None, defaults=None)

Define an experiment.

Parameters:

name (str, optional) – Name of the experiment (Default: None).
parameter_grid (dict, optional) – Parameter grid (Default: None).
parent (Experiment, optional) – Parent experiment (Default: None).
meta (dict, optional) – Dict with experiment metadata that should be recorded.
active (bool, optional) – Is the experiment active? (Default: True). When False, the experiment will not be executed.
volatile (bool, optional) – If True, the results of a successful run will not be saved (Default: False).
minimize (str or list of str, optional) – Metric or list of metrics to minimize.
maximize (str or list of str, optional) – Metric or list of metrics to maximize.
doc (str, optional) – Docstring for this experiment.
defaults (dict, optional) – Default values for parameters.

This can be used as a constructor or a decorator:

# When using as a decorator, the name of the experiment is automatically inferred.
@Experiment(...)
def exp1(trial):
    ...

# Here, the name must be supplied.
exp2 = Experiment("exp2", parent=exp1)

When the experiment is run, trial will be a Trial instance. As such, it has the following characteristics:

dict-like interface (trial[<name>]): Get the value of the parameter named name.
Attribute interface (trial.<attr>): Get meta-data for this trial.
call(): Run a function and automatically assign parameters.

See Trial for more details.

class Optimize(value): An enumeration.

child(name=None, configurator=None)

Create a child experiment.

Parameters:: name (str, optional) – Name for the child experiment. Defaults to the name of the parent.
Return type:: Experiment

command(name=None, *, target='trial')

Attach a command to an experiment.

@experiment()
def experiment1(trial):
    ...

@experiment1.command()
def frobnicate(trial):
    ...

get_matching_trials(exclude=None)

Return all trials that match the configuration of this experiment.

This includes all trials with matching configuration, regardless whether they belong to this or to another experiment.

property invariant_parameters: List[str]: Names of invariant parameters, i.e. parameters that assume only one single value.

on_pre_run(func)

Register a callback that is called before a trial runs.

Example

experiment = Experiment(…) @experiment.on_pre_run def on_experiment_pre_run(trial: Trial):

…

on_success(func)

Register a callback that is called after a trial finished successfully.

Example

experiment = Experiment(…) @experiment.on_success def on_experiment_success(trial: Trial):

…

on_update(func)

Register a callback that is called when a trial is updated.

Example

experiment = Experiment(…) @experiment.on_update def on_experiment_update(trial: Trial):

…

property parameters: List[str]: Names of the configured parameters.

pre_trial(func)

Update the pre-trial hook.

The pre-trial hook is called after the parameters for a trial are calculated and before its ID is calculated and it is run. This hook can be used to alter the parameters.

Use pre_trial(None) to reset the hook.

This can be used as a decorator:

@experiment()
def exp(trial):
    ...

@exp.pre_trial
def pre_trial_handler(ctx, trial_parameters):
    ...

Parameters:: func – A function with the signature (ctx, trial_parameters).

prepend_configurator(configurator)

Prepend a configurator.

Used by BaseConfigurator.__call__.

Return type:: None

run()

Run this experiment.

Create trials for every combination in the parameter grid and run them.

run_trial(trial): Run the current trial and save the results.

update(): Run on_update hook for all existing trials.

property varying_parameters: List[str]: Names of varying parameters, i.e. parameters that can assume more than one value.

Parameter generators

Parameter generators can be supplied using the parameters parameter of Experiment or as decorators. Simple parameter grids can be passed as a dict to the parameters parameter.

Stacking parameter generators

Multiple parameter generators can be stacked.

from experitur import Experiment
from experitur.parameters import Grid, Random
from scipy.stats.distributions import expon

@Grid({"a": [1,2,3]}
@Random({"b": expon()}, 4)
@Experiment()
def example(parameters: Trial):
    print(parameters["a"], parameters["b"])

For every value of a, four values will be drawn for b.

Conditions

Parameters can be generated differently depending on Conditions.

Optimization

experitur.parameters.SKOpt is a wrapper for skopt.Optimizer.

Note

You have to install the skopt dependency using pip:

pip install scikit-optimize

skopt allows Real, Integer and Categorical variables.

Combining SKOpt with other generators allows a sophisticated exploration of the parameter space:

from experitur import Experiment, Trial
from experitur.configurators import Grid, SKOpt


def rosenbrock_parametric(a, b, x, y):
    return (a - x) ** 2 + b * (y - x ** 2) ** 2


@Grid({"a": [1, 2], "b": [100, 101]})
@SKOpt({"x": (-10.0, 10.0), "y": (-10.0, 10.0)}, "z", 10)
@Grid({"repetition": [1, 2, 3]})
@Experiment(active=False)
def exp(trial: Trial):
    z = trial.apply("", rosenbrock_parametric)

    return {"z": z}

For every combination of a and b, ten value combinations for x and y are produced in order to minimize z, but z is averaged across three runs.

Trial

class experitur.Trial(data, root, prefix='', record_used_parameters=False)

Data related to a trial.

This is automatically instanciated by experitur and provided to the experiment function:

@Experiment(configurator={"a": [1,2,3], "prefix_a": [10]})
def exp1(trial: Trial):
    # Access current value of parameter `a` (item access)
    trial["a"]

    # Access extra data (attribute access)
    trial.id # Trial ID
    trial.wdir # Trial working directory

    def func(a=1, b=2):
        ...

    # Record default trial of `func`
    trial.record_defaults(func)

    # Call `func` with current value of parameter `a` and `b`=5
    trial.call(func, b=5)

    # Access only trial starting with a certain prefix
    trial_prefix = trial.prefix("prefix_")

    # All the above works as expected:
    # trial_prefix.<attr>, trial_prefix[<key>], trial_prefix.record_defaults, trial_prefix.call, ...
    # In our case, trial_prefix["a"] == 10.

__getitem__(name): Get the value of a parameter.

__getattr__(name): Access extra attributes.

aggregate_log(include=None, exclude=None)

Aggregate (min/max/mean/final) each field in the log.

Parameters:

include (Collection, optional) – If not None, include only these fields.
exclude (Collection, optional) – If not None, exclude these fields.

Return type:

Dict

call(func, *args, **kwargs)

Call the function applying the configured parameters.

Parameters:

func (callable) – Function to be called.
*args – Positional arguments to the function.
**kwargs – Named defaults for the function.

Return type:

TypeVar(T)

Returns:

The return value of the function.

The default values of the function are determined using inspect.signature(). Additional defaults can be given using **kwargs. These defaults are recorded into the trial.

As all default values are recorded, make sure that these have simple YAML-serializable types.

If the called function throws an exception, an exception of the same type is thrown with additional information about the parameters.

Use functools.partial to pass hidden keyword parameters that should not be recorded.

choice(parameter_name, choices, default=None)

Chose a value from an iterable whose name matches the value stored in parameter_name.

If parameter_name is not configured, the first entry is returned and recorded as default.

Parameters:

parameter_name (str) – Name of the parameter.
choices (Mapping or Iterable) – Mapping of names -> values or Iterable of values with a name (e.g. classes or functions).
default – Default key in choices.

Return type:

TypeVar(T)

Returns:

The configured value from the iterable.

copy()

Create a copy of the Trial instance.

While the data is deep-copied, the ID is the same, so saving overwrites the original data.

file(filename, make_parents=False)

Return filename relative to the trial’s working directory.

Parameters:: make_parents (bool, optional) – Make intermediary directories.

find_file(pattern, recursive=False)

Find a file of the trial.

Parameters:

pattern (str) – Filename pattern.
recursive (boolean, optional) – Search recursively. If True, the pattern ‘**’ will match any files and zero or more directories and subdirectories.

Returns:

filename

Raises:

ValueError – If not exactly one file was found.

find_files(pattern, recursive=False)

Find files of the trial.

Parameters:

pattern (str) – Filename pattern.
recursive (boolean, optional) – Search recursively. If True, the pattern ‘**’ will match any files and zero or more directories and subdirectories.

Return type:

List

Returns:

List of filenames.

get(key, default=None, setdefault=True)

Get a parameter value.

If key is not present, it is initialized with the provided default, just like Trial.setdefault().

items() → a set-like object providing a view on D's items

log(values=None, **kwargs)

Record metrics.

Parameters:

values (Mapping, optional) – Values to log.
**kwargs – Further values.

log_msg(msg): Log a message.

prefixed(prefix)

Return new Trial instance with prefix applied.

Prefixes allow you to organize parameters and save keystrokes.

Return type:: Trial

Example

trial_prefix = trial.prefix("prefix_")
trial_prefix["a"] == trial["prefix_a"] # True

record_defaults(func, **defaults)

Record default parameters from a function and additional parameters.

Parameters:

func (callable) – The keyword arguments of this function will be recorded if not already present.
**kwargs – Additional arguments that will be recorded if not already present.

Use functools.partial to pass keyword parameters to func that should not be recorded.

remove(): Remove this trial from the store.

save_checkpoint(*args, **kwargs)

Save a checkpoint that allows the experiment to be resumed.

Parameters:

*args – Arguments supplied to the experiment function upon resumption.
**kwargs –
Arguments supplied to the experiment function upon resumption.

setdefaults(defaults=None, **kwargs)

Set multiple default values for parameters that do not yet exist.

Existing keys are not overwritten. If keyword arguments are given, the keyword arguments and their values are added.

Parameters:

Iterable (defaults (Mapping or) –
optional) (Default values.) –
**kwargs (Additional default values.) –

todict(with_prefix=False)

Convert trial data to dictionary.

Parameters:: with_prefix (bool) – Use full key, even when using under Trial.prefixed.

update([E, ]**F) → None. Update D from mapping/iterable E and F.: If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v