Validation strategies

Hyperparameter selection in train() minimizes a validation objective by Bayesian optimization. The objective is pluggable: any function in echostatenetwork.validation can be passed as train(validation_strategy=...). They differ in how the wash-train-validation series is folded into training and validation intervals — everything else (closed-loop scoring, the Tikhonov grid, the optimizer bookkeeping) is shared.

How each strategy partitions the data

Fold geometry on a 12-Lyapunov-time series, redrawn after Fig. 2 of Racca & Magri (2021). Rows 1 and 2 are successive folds of the regular version; row 2c is the chaotic variant, which advances folds by about one Lyapunov time instead of the interval length so the intervals overlap and the fold count multiplies (set val_fold_step). Hatched = validation intervals recycled from inside the training data.

Choosing a strategy

RVC_Noise (the default) — chaotic recycle validation. Wout is trained once on all the data; each fold only re-washes the reservoir and scores a closed-loop run on an interval recycled from the training series. Racca & Magri (2021) find it matches K-fold's accuracy at a fraction of the cost, since there is no per-fold retraining.
SSV — single shot validation: one training/validation split. The cheapest and, for chaotic series, the least reliable — its single interval correlates weakly with test error. Provided for comparison.
WFV — walk forward validation: a fixed training window slides forward, validating on the interval just after it; each fold retrains Wout on its own window (pure arithmetic on prefix ridge sums — the teacher-forced reservoir pass is shared).
KFV — K-fold validation: leave-one-interval-out; each fold retrains on everything outside its validation interval, exactly (prefix-sum assembly), and validates on it.

All strategies select the Tikhonov parameter from tikh_range per evaluation and score their probes through a shared validation metric: any callable metric(case, Y_true, Y_pred, norm) -> float (divergence penalty included). Set validation_metric on the ESN to swap the scoring for every strategy at once; built-ins are log_nMAE (log10 range-normalized MAE, the recycle-family default) and nMSE (raw variance-normalized MSE). The qlESN-specific segment strategies (SegmentRVC_Noise, RecycledSegmentRVC_Noise) live in qlrom's qlroms.data_driven_qlroms.validation and share the same contract.

Ensembles of reservoir seeds

The reservoir matrices are random draws, so validation quality is an ensemble statement. train(n_seeds=m) trains m realizations in parallel, keeps the one with the best validation score, and stores all scores in seed_scores. scripts/compare_validation_strategies.py reproduces the paper's Table-1/Figure-8 protocol at reduced scale and evaluates that selection rule: the kept member consistently lands in the ensemble's best half on test error, usually the best quarter.

See the validation strategies tutorial for a live comparison on Lorenz 63.

Reference

Racca & Magri (2021). Robust optimization and validation of echo state networks for learning chaotic dynamics. Neural Networks, 142, 252-268 (arXiv:2103.03174).

API

`echostatenetwork.validation`

Validation strategies for EchoStateNetwork's Bayesian hyperparameter search.

Plain functions with the shared signature (x, case, U_wtv, Y_wtv, tikh_opt, hp_names, print_convergence): case is the EchoStateNetwork being validated, x the hyperparameter values under evaluation. Pass one as train(validation_strategy=...); the class aliases (EchoStateNetwork._RVC_Noise etc.) keep the old spelling working.

RVC_Noise: chaotic recycle validation, within-segment folds (the default).
SSV / WFV / KFV: the single-series strategies of Racca & Magri (2021), sharing the single_series_validation engine (one teacher-forced pass, prefix-sum ridge).

The qlESN-specific segment strategies (SegmentRVC_Noise, RecycledSegmentRVC_Noise) live in qlroms.data_driven_qlroms.validation -- they exist for ragged dwell-segment corpora, not for this package's single/regular series.

VALIDATION METRICS are shared across strategies: a metric is any callable metric(case, Y_true, Y_pred, norm) -> float scoring one closed-loop probe (divergence penalty included). Every strategy uses case.validation_metric when set, else its own default. Built-ins: log_nMAE (log10 range-normalised MAE, the recycle-family default) and nMSE (raw variance-normalised MSE).

`log_nMAE(case, Y_true, Y_pred, norm=1.0)`

Probe metric: log10 of the range-normalised MAE; a diverged probe scores a fixed +10 instead of poisoning the accumulated sum. The default of the recycle-validation family.

Source code in echostatenetwork/validation.py

def log_nMAE(case, Y_true, Y_pred, norm=1.0):
    """Probe metric: log10 of the range-normalised MAE; a diverged probe scores a
    fixed +10 instead of poisoning the accumulated sum. The default of the
    recycle-validation family."""
    err = np.log10(case.compute_nMAE(Y_true, Y_pred, norm=norm))
    return float(err) if np.isfinite(err) else 10.0

`nMSE(case, Y_true, Y_pred, norm=None)`

Probe metric: raw MSE normalised by the truth's own mean square (norm is ignored); 1e6 for a diverged probe. A common open-loop one-step objective, usable in any strategy via case.validation_metric.

Source code in echostatenetwork/validation.py

def nMSE(case, Y_true, Y_pred, norm=None):
    """Probe metric: raw MSE normalised by the truth's own mean square (`norm` is
    ignored); 1e6 for a diverged probe. A common open-loop one-step objective,
    usable in any strategy via ``case.validation_metric``."""
    m = float(np.mean((Y_pred - Y_true) ** 2)) / (float(np.mean(Y_true**2)) + 1e-14)
    return m if np.isfinite(m) else 1e6

`RVC_Noise(x, case, U_wtv, Y_wtv, tikh_opt, hp_names, print_convergence=True)`

Implements Chaotic Recycle Validation for hyperparameter optimization.

Parameters:

Name	Type	Description	Default
`x`	`list`	Hyperparameter values to evaluate.	required
`case`	`EchoStateNetwork`	Instance of the ESN being validated.	required
`U_wtv`	`ndarray`	Wash-train-validation input data.	required
`Y_wtv`	`ndarray`	Corresponding labels for train/validation data.	required
`tikh_opt`	`ndarray`	Array to store optimal Tikhonov regularization values.	required
`hp_names`	`list`	Names of the hyperparameters being optimized.	required

Returns:

Type	Description
`float: Mean (over folds) log10 closed-loop normalized MAE of the best Tikhonov candidate.`

`single_series_validation(x, case, U_wtv, Y_wtv, tikh_opt, hp_names, print_convergence, folds_of, strategy_name)`

Shared engine for the single-series validation strategies of Racca & Magri (2021): SSV, WFV and KFV differ only in fold geometry, which each supplies via folds_of; everything else -- the teacher-forced open-loop pass, the per-fold ridge solves, the closed-loop probes, the Tikhonov grid and the BHO bookkeeping -- lives here.

Noise handling is identical to RVC_Noise: U_wtv is already the noisy copy built by _split_and_format_data (inputs only; targets are clean), and no extra noise is added here.

The open-loop reservoir trajectory does not depend on Wout, so ONE teacher-forced pass over the series per objective call serves every fold and every Tikhonov candidate. Each fold's ridge system is then assembled from prefix sums of per-interval Gram terms -- per-fold retraining is pure arithmetic, with no reservoir recomputation.

Parameters:

Name	Type	Description	Default
`x`		As in `RVC_Noise`, except U_wtv/Y_wtv must be regular ndarrays with a single segment (`L == 1`); a segmented or ragged corpus raises a ValueError.	required
`case`		As in `RVC_Noise`, except U_wtv/Y_wtv must be regular ndarrays with a single segment (`L == 1`); a segmented or ragged corpus raises a ValueError.	required
`U_wtv`		As in `RVC_Noise`, except U_wtv/Y_wtv must be regular ndarrays with a single segment (`L == 1`); a segmented or ragged corpus raises a ValueError.	required
`Y_wtv`		As in `RVC_Noise`, except U_wtv/Y_wtv must be regular ndarrays with a single segment (`L == 1`); a segmented or ragged corpus raises a ValueError.	required
`tikh_opt`		As in `RVC_Noise`, except U_wtv/Y_wtv must be regular ndarrays with a single segment (`L == 1`); a segmented or ragged corpus raises a ValueError.	required
`hp_names`		As in `RVC_Noise`, except U_wtv/Y_wtv must be regular ndarrays with a single segment (`L == 1`); a segmented or ragged corpus raises a ValueError.	required
`print_convergence`		As in `RVC_Noise`, except U_wtv/Y_wtv must be regular ndarrays with a single segment (`L == 1`); a segmented or ragged corpus raises a ValueError.	required
`folds_of`	`callable`	`folds_of(case, n_post) -> list of (train_ranges, i0, n_val_k)` with `n_post` the number of post-washout rows; `train_ranges` a list of half-open row ranges `(r0, r1)` the fold trains on, `i0` the first validation row, `n_val_k` the validation length.	required
`strategy_name`	`str`	Name used in error messages ('SSV', 'WFV', 'KFV').	required

Returns:

Type	Description
`float`	Mean (over folds) log10 closed-loop normalized error of the best Tikhonov candidate -- the scalar the Bayesian optimization minimizes.

Source code in echostatenetwork/validation.py

def single_series_validation(x, case, U_wtv, Y_wtv, tikh_opt, hp_names,
                              print_convergence, folds_of, strategy_name):
    """Shared engine for the single-series validation strategies of
    Racca & Magri (2021): `SSV`, `WFV` and `KFV` differ only in fold
    geometry, which each supplies via `folds_of`; everything else -- the
    teacher-forced open-loop pass, the per-fold ridge solves, the closed-loop
    probes, the Tikhonov grid and the BHO bookkeeping -- lives here.

    Noise handling is identical to `RVC_Noise`: U_wtv is already the noisy
    copy built by `_split_and_format_data` (inputs only; targets are clean),
    and no extra noise is added here.

    The open-loop reservoir trajectory does not depend on `Wout`, so ONE
    teacher-forced pass over the series per objective call serves every fold
    and every Tikhonov candidate. Each fold's ridge system is then assembled
    from prefix sums of per-interval Gram terms -- per-fold retraining is
    pure arithmetic, with no reservoir recomputation.

    Parameters
    ----------
    x, case, U_wtv, Y_wtv, tikh_opt, hp_names, print_convergence
        As in `RVC_Noise`, except U_wtv/Y_wtv must be regular ndarrays with
        a single segment (``L == 1``); a segmented or ragged corpus raises a
        ValueError.
    folds_of : callable
        ``folds_of(case, n_post) -> list of (train_ranges, i0, n_val_k)``
        with ``n_post`` the number of post-washout rows; ``train_ranges`` a
        list of half-open row ranges ``(r0, r1)`` the fold trains on, ``i0``
        the first validation row, ``n_val_k`` the validation length.
    strategy_name : str
        Name used in error messages ('SSV', 'WFV', 'KFV').

    Returns
    -------
    float
        Mean (over folds) log10 closed-loop normalized error of the best
        Tikhonov candidate -- the scalar the Bayesian optimization minimizes.
    """
    # Re-set hyperparams as the optimization goes on
    if hp_names:
        case._reset_hyperparams(x, hp_names)

    if not isinstance(U_wtv, np.ndarray) or U_wtv.ndim != 3 or U_wtv.shape[0] != 1:
        raise ValueError(
            f'{strategy_name} requires a single contiguous training series '
            '(regular ndarray with L==1); got a segmented/ragged corpus. '
            'Use RVC_Noise instead (or the segment strategies in '
            'qlroms.data_driven_qlroms.validation).')

    metric = getattr(case, 'validation_metric', None) or log_nMAE
    U_l, Y_l = U_wtv[0], Y_wtv[0]
    norm_l = np.max(Y_l, axis=0) - np.min(Y_l, axis=0)

    if case.input_parameters is not None:
        # Hold the series' own leading parameter vector fixed for the
        # closed-loop probes (same slice trick as RVC_Noise; train()
        # restores the original input_parameters afterwards).
        N_param = case._n_param(case.input_parameters)
        case.input_parameters = U_l[0, -N_param:].reshape(N_param, 1)

    # ONE teacher-forced open-loop pass (Wout-independent). The returned
    # open-loop readouts (U_RR) are computed with the scratch Wout train()
    # zeroed before BHO -- stale, so they are discarded; the total LHS/RHS
    # are rebuilt per fold from interval sums below.
    _, _, _, R_RR = case._compute_RR_terms(U_wtv, Y_wtv)
    R = R_RR[0]  # (Nt - N_wash, N_units)
    r_aug = np.hstack([R, np.ones((R.shape[0], 1)) * case.bias_out])
    Y_t = Y_l[case.N_wash:]  # row i <-> input U_l[N_wash + i]
    n_post = r_aug.shape[0]

    if n_post < case.N_val + 1:
        raise ValueError(
            f'{strategy_name}: the series holds {n_post} post-washout steps, but '
            f'one validation interval (N_val={case.N_val}) plus at least one '
            'training step is required; reduce t_val or N_wash.')

    folds = folds_of(case, n_post)
    case.n_folds_realized = len(folds)   # surfaced by training_summary()

    # Prefix Gram sums at every training-range boundary: each fold's ridge
    # system is a difference of prefixes, so every teacher-forced row enters
    # exactly one Gram product regardless of the number of folds.
    bounds = sorted({0, *(b for tr, _, _ in folds for rng_ in tr for b in rng_)})
    A = np.zeros((case.N_units + 1, case.N_units + 1))
    B = np.zeros((case.N_units + 1, case.N_dim))
    prefix, prev = {}, 0
    for b in bounds:
        if b > prev:
            block = r_aug[prev:b]
            A = A + block.T @ block  # new arrays: stored references stay valid
            B = B + block.T @ Y_t[prev:b]
            prev = b
        prefix[b] = (A, B)

    N_tikh = len(case.tikh_range)
    nMAE = np.zeros(N_tikh)
    r_washed = None  # lazily-computed genuine washout state (i0 == 0 folds)

    for train_ranges, i0, n_val_k in folds:
        LHS = np.zeros_like(A)
        RHS = np.zeros_like(B)
        for r0, r1 in train_ranges:
            LHS += prefix[r1][0] - prefix[r0][0]
            RHS += prefix[r1][1] - prefix[r0][1]

        Y_val = Y_t[i0:i0 + n_val_k]

        # Reservoir seed for the closed-loop probe, free from the
        # teacher-forced pass: R[i0-1] is the open-loop state after
        # consuming the ENTIRE history up to input U_l[N_wash+i0-1]
        # (equivalent to, and longer than, a washout on the N_wash steps
        # preceding the interval). For i0 == 0 (interval flush with the training
        # washout) step open-loop through the genuine washout window
        # U_l[:N_wash] once; the reservoir path is Wout-independent, so one
        # pass serves every fold and Tikhonov candidate.
        if i0 >= 1:
            r_seed = R[i0 - 1][:, np.newaxis]
        else:
            if r_washed is None:
                r_washed = np.zeros((case.N_units, 1))
                for u_in in U_l[:case.N_wash]:
                    _, r_washed = case.step(u_in, r_washed)
            r_seed = r_washed

        for tik_j in range(N_tikh):
            LHS_reg = LHS.copy()
            LHS_reg.ravel()[::LHS_reg.shape[1] + 1] += case.tikh_range[tik_j]
            case.Wout = np.linalg.solve(LHS_reg, RHS)

            # Closed-loop seed: THIS candidate's readout of the shared
            # washed state -- equivalent to RVC_Noise's washout inside the
            # Tikhonov loop, since the open-loop reservoir path is
            # Wout-independent and only the seed readout depends on Wout.
            r_out = r_seed.copy()
            u_out = case.reservoir_to_physical(r_out)

            Y_closed = np.zeros_like(Y_val)
            for i in range(Y_closed.shape[0]):
                u_input = case.outputs_to_inputs(full_state=u_out)
                u_out, r_out = case.step(u_input, r_out)
                Y_closed[i] = u_out[:, 0].copy()

            # probe metric (shared across strategies; see module docstring)
            nMAE[tik_j] += metric(case, Y_val, Y_closed, norm_l)

    # select and save the optimal tikhonov (same bookkeeping as RVC_Noise:
    # _optimize_hyperparameters reads tikh_opt[best_idx] after BHO)
    a = nMAE.argmin()
    tikh_opt[case.val_k] = case.tikh_range[a]
    case.tikh = case.tikh_range[a]
    normalized_best_MAE = nMAE[a] / len(folds)

    case.val_k += 1
    if print_convergence:
        print(case.val_k, end="")
        for hp in case.hyperparameters_to_optimize:
            print(f'\t {case._get_hyperparam(hp):.3e}', end="")
        print(f'\t {normalized_best_MAE:.4f}')

    return normalized_best_MAE

`SSV(x, case, U_wtv, Y_wtv, tikh_opt, hp_names, print_convergence=True)`

Single shot validation (SSV) of Racca & Magri (2021).

The series is split once: Wout is trained (per Tikhonov candidate) on everything before the last validation interval, and the closed-loop error is computed on that single interval of N_val steps at the end of the series, the reservoir washed out open-loop on the data immediately preceding it. Racca & Magri (2021) show SSV must not be relied on for chaotic time series (a single validation interval correlates weakly with test error); it is provided for comparison with the multi-interval strategies (WFV, KFV, RVC_Noise).

Fold geometry: with n post-washout steps, one fold -- training rows [0, n - N_val), validation rows [n - N_val, n).

Parameters:

Name	Type	Description	Default
`x`	`list`	Hyperparameter values to evaluate (aligned with `hp_names`).	required
`case`	`EchoStateNetwork`	Instance of the ESN being validated.	required
`U_wtv`	`ndarray`	Wash-train-validation input data, shape `(1, Nt, N_dim_in)` -- a single contiguous series. A segmented/ragged corpus raises a ValueError (use `RVC_Noise`, or the qlroms segment strategies, for those).	required
`Y_wtv`	`ndarray`	Corresponding labels, shape `(1, Nt, N_dim)`.	required
`tikh_opt`	`ndarray`	Array to store optimal Tikhonov regularization values.	required
`hp_names`	`list`	Names of the hyperparameters being optimized.	required
`print_convergence`	`bool`	Print one convergence row per evaluation.	`True`

Returns:

Type	Description
`float`	log10 closed-loop normalized error on the single validation interval (best Tikhonov candidate).

References

Racca & Magri (2021). Robust optimization and validation of echo state networks for learning chaotic dynamics. Neural Networks, 142, 252-268 (arXiv:2103.03174).

Source code in echostatenetwork/validation.py

def SSV(x, case, U_wtv, Y_wtv, tikh_opt, hp_names, print_convergence=True):
    """Single shot validation (SSV) of Racca & Magri (2021).

    The series is split once: Wout is trained (per Tikhonov candidate) on
    everything before the last validation interval, and the closed-loop
    error is computed on that single interval of ``N_val`` steps at the end
    of the series, the reservoir washed out open-loop on the data
    immediately preceding it. Racca & Magri (2021) show SSV must not be
    relied on for chaotic time series (a single validation interval
    correlates weakly with test error); it is provided for comparison with
    the multi-interval strategies (`WFV`, `KFV`, `RVC_Noise`).

    Fold geometry: with ``n`` post-washout steps, one fold -- training rows
    ``[0, n - N_val)``, validation rows ``[n - N_val, n)``.

    Parameters
    ----------
    x : list
        Hyperparameter values to evaluate (aligned with `hp_names`).
    case : EchoStateNetwork
        Instance of the ESN being validated.
    U_wtv : np.ndarray
        Wash-train-validation input data, shape ``(1, Nt, N_dim_in)`` -- a
        single contiguous series. A segmented/ragged corpus raises a
        ValueError (use `RVC_Noise`, or the qlroms segment strategies, for those).
    Y_wtv : np.ndarray
        Corresponding labels, shape ``(1, Nt, N_dim)``.
    tikh_opt : np.ndarray
        Array to store optimal Tikhonov regularization values.
    hp_names : list
        Names of the hyperparameters being optimized.
    print_convergence : bool
        Print one convergence row per evaluation.

    Returns
    -------
    float
        log10 closed-loop normalized error on the single validation
        interval (best Tikhonov candidate).

    References
    ----------
    Racca & Magri (2021). Robust optimization and validation of echo state
    networks for learning chaotic dynamics. Neural Networks, 142, 252-268
    (arXiv:2103.03174).
    """
    def folds_of(case_, n_post):
        i0 = n_post - case_.N_val
        return [([(0, i0)], i0, case_.N_val)]

    return single_series_validation(
        x, case, U_wtv, Y_wtv, tikh_opt, hp_names, print_convergence,
        folds_of, 'SSV')

`WFV(x, case, U_wtv, Y_wtv, tikh_opt, hp_names, print_convergence=True)`

Walk forward validation (WFV) of Racca & Magri (2021).

A fixed-length training window slides forward by step per fold; each fold retrains Wout on its own window (pure arithmetic on per-interval ridge sums -- the teacher-forced reservoir pass is shared) and validates closed-loop on the N_val steps immediately after it, the reservoir washed out open-loop on the data immediately preceding the interval. Hyperparameters minimize the mean closed-loop error over the folds, which is far more robust for chaotic series than SSV.

Fold geometry: the advance between consecutive folds is step = val_fold_step or N_val; the default val_fold_step = None gives the regular WFV whose validation intervals tile the series without overlap, while val_fold_step of ~one Lyapunov time in ESN steps (< N_val) gives the paper's chaotic version with overlapping intervals and correspondingly more folds. With n post-washout steps and K = min(N_folds, 1 + (n - 1 - N_val) // step) folds (reduced with a printed note when the requested N_folds do not fit), the training-window length is m = n - (K - 1) * step - N_val; fold k (k = 0..K-1) trains on rows [k * step, k * step + m) and validates on [k * step + m, k * step + m + N_val) -- the last validation interval always ends at row n.

Parameters:

Name	Type	Description	Default
`x`	`list`	Hyperparameter values to evaluate (aligned with `hp_names`).	required
`case`	`EchoStateNetwork`	Instance of the ESN being validated.	required
`U_wtv`	`ndarray`	Wash-train-validation input data, shape `(1, Nt, N_dim_in)` -- a single contiguous series. A segmented/ragged corpus raises a ValueError (use `RVC_Noise`, or the qlroms segment strategies, for those).	required
`Y_wtv`	`ndarray`	Corresponding labels, shape `(1, Nt, N_dim)`.	required
`tikh_opt`	`ndarray`	Array to store optimal Tikhonov regularization values.	required
`hp_names`	`list`	Names of the hyperparameters being optimized.	required
`print_convergence`	`bool`	Print one convergence row per evaluation.	`True`

Returns:

Type	Description
`float`	Mean (over folds) log10 closed-loop normalized error of the best Tikhonov candidate.

References

Racca & Magri (2021). Robust optimization and validation of echo state networks for learning chaotic dynamics. Neural Networks, 142, 252-268 (arXiv:2103.03174). The chaotic versions (subscript c) advance the folds by one Lyapunov time instead of the validation-interval length, so consecutive validation intervals overlap; see val_fold_step.

Source code in echostatenetwork/validation.py

def WFV(x, case, U_wtv, Y_wtv, tikh_opt, hp_names, print_convergence=True):
    """Walk forward validation (WFV) of Racca & Magri (2021).

    A fixed-length training window slides forward by ``step`` per fold;
    each fold retrains Wout on its own window (pure arithmetic on
    per-interval ridge sums -- the teacher-forced reservoir pass is shared)
    and validates closed-loop on the ``N_val`` steps immediately after it,
    the reservoir washed out open-loop on the data immediately preceding
    the interval. Hyperparameters minimize the mean closed-loop error over
    the folds, which is far more robust for chaotic series than `SSV`.

    Fold geometry: the advance between consecutive folds is
    ``step = val_fold_step or N_val``; the default ``val_fold_step = None``
    gives the regular WFV whose validation intervals tile the series
    without overlap, while ``val_fold_step`` of ~one Lyapunov time in ESN
    steps (< ``N_val``) gives the paper's chaotic version with overlapping
    intervals and correspondingly more folds. With ``n`` post-washout
    steps and ``K = min(N_folds, 1 + (n - 1 - N_val) // step)`` folds
    (reduced with a printed note when the requested ``N_folds`` do not
    fit), the training-window length is
    ``m = n - (K - 1) * step - N_val``; fold ``k`` (``k = 0..K-1``) trains
    on rows ``[k * step, k * step + m)`` and validates on
    ``[k * step + m, k * step + m + N_val)`` -- the last validation
    interval always ends at row ``n``.

    Parameters
    ----------
    x : list
        Hyperparameter values to evaluate (aligned with `hp_names`).
    case : EchoStateNetwork
        Instance of the ESN being validated.
    U_wtv : np.ndarray
        Wash-train-validation input data, shape ``(1, Nt, N_dim_in)`` -- a
        single contiguous series. A segmented/ragged corpus raises a
        ValueError (use `RVC_Noise`, or the qlroms segment strategies, for those).
    Y_wtv : np.ndarray
        Corresponding labels, shape ``(1, Nt, N_dim)``.
    tikh_opt : np.ndarray
        Array to store optimal Tikhonov regularization values.
    hp_names : list
        Names of the hyperparameters being optimized.
    print_convergence : bool
        Print one convergence row per evaluation.

    Returns
    -------
    float
        Mean (over folds) log10 closed-loop normalized error of the best
        Tikhonov candidate.

    References
    ----------
    Racca & Magri (2021). Robust optimization and validation of echo state
    networks for learning chaotic dynamics. Neural Networks, 142, 252-268
    (arXiv:2103.03174). The chaotic versions (subscript c) advance the
    folds by one Lyapunov time instead of the validation-interval length,
    so consecutive validation intervals overlap; see `val_fold_step`.
    """
    def folds_of(case_, n_post):
        step = case_.val_fold_step or case_.N_val
        K = min(case_.N_folds, 1 + (n_post - 1 - case_.N_val) // step)
        if K < case_.N_folds and case_.val_k == 0:
            print(f'WFV: only {K} of the requested N_folds={case_.N_folds} '
                  f'walk-forward folds fit in {n_post} post-washout steps '
                  f'(N_val={case_.N_val}, step={step}); using {K} folds.')
        m = n_post - (K - 1) * step - case_.N_val  # fixed training-window length
        folds = []
        for k in range(K):
            s = k * step
            folds.append(([(s, s + m)], s + m, case_.N_val))
        return folds

    return single_series_validation(
        x, case, U_wtv, Y_wtv, tikh_opt, hp_names, print_convergence,
        folds_of, 'WFV')

`KFV(x, case, U_wtv, Y_wtv, tikh_opt, hp_names, print_convergence=True)`

K-fold validation (KFV) of Racca & Magri (2021).

Leave-one-interval-out: N_folds validation intervals of length N_val cover the post-washout data (after an initial offset absorbing the remainder, cf. the b*v offset in the paper); each fold retrains Wout on ALL rows outside its own interval (pure arithmetic on per-interval ridge sums -- the teacher-forced reservoir pass is shared) and validates closed-loop on it, the reservoir washed out open-loop on the data immediately preceding the interval.

Note: the shared teacher-forced pass drives the reservoir open-loop through the held-out interval too -- the same recycling of training data that RVC_Noise embraces for its washout windows. RVC_Noise (recycle validation) matches KFV's accuracy at lower cost by also training Wout once on all the data.

Fold geometry: the advance between consecutive validation intervals is step = val_fold_step or N_val; the default val_fold_step = None gives the regular KFV whose intervals tile the data without overlap, while val_fold_step of ~one Lyapunov time in ESN steps (< N_val) gives the paper's chaotic version with overlapping intervals and correspondingly more folds. With n post-washout steps and K = min(N_folds, 1 + (n - N_val) // step) intervals (reduced with a printed note when fewer fit), the initial offset is n - (K - 1) * step - N_val; fold k (k = 0..K-1) validates on rows [offset + k * step, offset + k * step + N_val) and trains on all rows outside its own interval.

Parameters:

Name	Type	Description	Default
`x`	`list`	Hyperparameter values to evaluate (aligned with `hp_names`).	required
`case`	`EchoStateNetwork`	Instance of the ESN being validated.	required
`U_wtv`	`ndarray`	Wash-train-validation input data, shape `(1, Nt, N_dim_in)` -- a single contiguous series. A segmented/ragged corpus raises a ValueError (use `RVC_Noise`, or the qlroms segment strategies, for those).	required
`Y_wtv`	`ndarray`	Corresponding labels, shape `(1, Nt, N_dim)`.	required
`tikh_opt`	`ndarray`	Array to store optimal Tikhonov regularization values.	required
`hp_names`	`list`	Names of the hyperparameters being optimized.	required
`print_convergence`	`bool`	Print one convergence row per evaluation.	`True`

Returns:

Type	Description
`float`	Mean (over folds) log10 closed-loop normalized error of the best Tikhonov candidate.

References

Racca & Magri (2021). Robust optimization and validation of echo state networks for learning chaotic dynamics. Neural Networks, 142, 252-268 (arXiv:2103.03174). The chaotic versions (subscript c) advance the folds by one Lyapunov time instead of the validation-interval length, so consecutive validation intervals overlap; see val_fold_step.

Source code in echostatenetwork/validation.py

def KFV(x, case, U_wtv, Y_wtv, tikh_opt, hp_names, print_convergence=True):
    """K-fold validation (KFV) of Racca & Magri (2021).

    Leave-one-interval-out: ``N_folds`` validation intervals of length
    ``N_val`` cover the post-washout data (after an initial offset
    absorbing the remainder, cf. the ``b*v`` offset in the paper); each
    fold retrains Wout on ALL rows outside its own interval (pure
    arithmetic on per-interval ridge sums -- the teacher-forced reservoir
    pass is shared) and validates closed-loop on it, the reservoir washed
    out open-loop on the data immediately preceding the interval.

    Note: the shared teacher-forced pass drives the reservoir open-loop
    through the held-out interval too -- the same recycling of training
    data that `RVC_Noise` embraces for its washout windows. `RVC_Noise`
    (recycle validation) matches KFV's accuracy at lower cost by also
    training Wout once on all the data.

    Fold geometry: the advance between consecutive validation intervals
    is ``step = val_fold_step or N_val``; the default
    ``val_fold_step = None`` gives the regular KFV whose intervals tile
    the data without overlap, while ``val_fold_step`` of ~one Lyapunov
    time in ESN steps (< ``N_val``) gives the paper's chaotic version with
    overlapping intervals and correspondingly more folds. With ``n``
    post-washout steps and ``K = min(N_folds, 1 + (n - N_val) // step)``
    intervals (reduced with a printed note when fewer fit), the initial
    offset is ``n - (K - 1) * step - N_val``; fold ``k`` (``k = 0..K-1``)
    validates on rows ``[offset + k * step, offset + k * step + N_val)``
    and trains on all rows outside its own interval.

    Parameters
    ----------
    x : list
        Hyperparameter values to evaluate (aligned with `hp_names`).
    case : EchoStateNetwork
        Instance of the ESN being validated.
    U_wtv : np.ndarray
        Wash-train-validation input data, shape ``(1, Nt, N_dim_in)`` -- a
        single contiguous series. A segmented/ragged corpus raises a
        ValueError (use `RVC_Noise`, or the qlroms segment strategies, for those).
    Y_wtv : np.ndarray
        Corresponding labels, shape ``(1, Nt, N_dim)``.
    tikh_opt : np.ndarray
        Array to store optimal Tikhonov regularization values.
    hp_names : list
        Names of the hyperparameters being optimized.
    print_convergence : bool
        Print one convergence row per evaluation.

    Returns
    -------
    float
        Mean (over folds) log10 closed-loop normalized error of the best
        Tikhonov candidate.

    References
    ----------
    Racca & Magri (2021). Robust optimization and validation of echo state
    networks for learning chaotic dynamics. Neural Networks, 142, 252-268
    (arXiv:2103.03174). The chaotic versions (subscript c) advance the
    folds by one Lyapunov time instead of the validation-interval length,
    so consecutive validation intervals overlap; see `val_fold_step`.
    """
    def folds_of(case_, n_post):
        step = case_.val_fold_step or case_.N_val
        K = min(case_.N_folds, 1 + (n_post - case_.N_val) // step)
        if K < case_.N_folds and case_.val_k == 0:
            print(f'KFV: only {K} of the requested N_folds={case_.N_folds} '
                  f'validation intervals fit in {n_post} post-washout steps '
                  f'(N_val={case_.N_val}, step={step}); using {K} folds.')
        offset = n_post - (K - 1) * step - case_.N_val
        folds = []
        for k in range(K):
            i0 = offset + k * step
            train_ranges = [(r0, r1) for r0, r1 in
                            [(0, i0), (i0 + case_.N_val, n_post)] if r1 > r0]
            folds.append((train_ranges, i0, case_.N_val))
        return folds

    return single_series_validation(
        x, case, U_wtv, Y_wtv, tikh_opt, hp_names, print_convergence,
        folds_of, 'KFV')