Benchmark

For some example on how to use the benchmark module you could check the Benchmark page on github.

Module for benchmarking the DTW loss functions implemented in this package. It includes functions to compare the performance of different DTW loss, save and visualize the results.

Authors

Alberto Zancanaro <alberto.zancanaro@uni.lu>

dtw_loss_functions.benchmark.compute_benchmark(B_list: list, T_list: list, C_list: list, loss_functions_to_use: dict, device: str, n_repetitions: int = 100, path_save: str = 'benchmark', print_progress: bool = False)[source]

For each combination of batch size, signal length and number of channels specified in the input lists, compute the loss for each loss function specified in loss_functions_to_use.

The loss_functions_to_use is a dictionary where the keys are the names of the loss functions and the values are the instances of the loss functions to test. The loss functions must be instances of the classes implemented in this package.

The results are saved in npy files at the path {path_save}/{loss_function_name}/. The loss_function_name is the key of the loss function in the loss_functions_to_use dictionary. Each file will be name as B_{batch_size}_T_{signal_length}_C_{num_channels}.npy and will contain a numpy array of shape (n_repetitions,) with the computed loss values for each repetition.

Parameters:

  • B_list (list) – List of batch sizes to test.

  • T_list (list) – List of signal lengths to test.

  • C_list (list) – List of number of channels to test.

  • loss_functions_to_use (dict) – Dictionary of loss functions to test. The keys are the names of the loss functions and the values are the instances of the loss functions to test.

  • device (str) – Device to use for the computations. Must be ‘cpu’ or ‘cuda’.

  • n_repetitions (int, optional) – Number of repetitions to perform for each combination of batch size, signal length and number of channels. The default value is 100.

  • path_save (str, optional) – Path to save the results. The default value is ‘benchmark’. If the folder does not exist, it will be created. If results are already present in the folder, they will be overwritten.

  • print_progress (bool, optional) – Whether to print the progress of the benchmark. The default value is False.

dtw_loss_functions.benchmark.get_array_to_plot(results_dict: dict, loss_function_name: str, x_axis_variable: str, B_list: list, T_list: list, C_list: list) → tuple[source]

Given the results dictionary returned by the load_benchmark_results() function, extract the array of computation times for the specified loss function name. The data will be organized as a list of arrays, where each array corresponds to a specific combination of batch size, signal length and number of channels. The x-axis variable indicates which variable should be used as x-axis in the plot. E.g. if x_axis_variable is ‘T’, the function will return a list of arrays where each array corresponds to a specific combination of batch size and number of channels, and the values in the array correspond to the computation times for different signal lengths.

This function will return 3 lists :

  • data_list_mean : list of arrays containing the mean computation times for each combination of batch size, signal length and number of channels.

  • data_list_std : list of arrays containing the standard deviation of the computation times for each combination of batch size, signal length and number of channels.

  • data_list_label : list of strings containing, where each element corresponds to the label of the data in the same position in the data_list_mean and data_list_std lists.

Remember that for each combination of batch size, signal length and number of channels, there are n_repetitions computation times stored in the results dictionary. For this reason, this function return both the mean and std for each combination.

Parameters:

  • results_dict (dict) – Dictionary containing the loaded results as returned by the load_benchmark_results() function.

  • loss_function_name (str) – Name of the loss function for which to extract the data. It must be a key of the input dictionary.

  • x_axis_variable (str) – Variable to use as x-axis in the plot. It must be one of ‘B’, ‘T’ or ‘C’, corresponding to batch size, signal length and number of channels, respectively.

dtw_loss_functions.benchmark.get_info_from_filename(file_name: str) → tuple[source]

Given a file name in the format B_{batch_size}_T_{signal_length}_C_{num_channels}.npy, extract the batch size, signal length and number of channels from the filename and return them.

Parameters:

file_name (str) – File name in the format B_{batch_size}_T_{signal_length}_C_{num_channels}.npy.

Returns:

  • batch_size (int) – Batch size extracted from the file name.

  • signal_length (int) – Signal length extracted from the file name.

  • n_channels (int) – Number of channels extracted from the file name.

dtw_loss_functions.benchmark.load_benchmark_results(path_results: str, loss_list: list = None) → dict[source]

Load all the results stored in the specified path. This function expects that the results are stored in npy files organized as described in the docstring of the compute_benchmark() function. The function will return a dictionary where the keys are the loss function names and the values are dictionaries containing the results for each combination of batch size, signal length and number of channels.

Note that by default, the function will load all the results present in the specified path. The name of the loss function will be inferred from the filename. If you want to load only a subset of the results, you can specify the list of loss function names to load in the loss_list parameter. In this case, only the results corresponding to the specified loss function names will be loaded.

Parameters:

  • path_results (str) – Path to the folder containing the results. The results must be stored in npy files with the name format B_{batch_size}_T_{signal_length}_C_{num_channels}.npy.

  • loss_list (list, optional) – List of loss function names to load. If specified, only the results corresponding to the specified loss function names will be loaded. The default value is None, which means that all the results present in the specified path will be loaded.

Returns:

results_dict – Dictionary containing the loaded results. The keys are the loss function names and the values are dictionaries containing the results for each combination of batch size, signal length and number of channels.

Return type:

dict[str, dict[str, np.ndarray]]

dtw_loss_functions.benchmark.plot_benchmark(plot_config: dict, x_axis_variable_list: list, x_axis_variable_name: str, data_to_plot_mean: list, labels_to_plot: list, data_to_plot_std: list = None, color_dict: dict = None) → tuple[source]

Plot the results of the benchmark. The data should be in the format returned by the get_array_to_plot() function, i.e. a list of arrays containing the mean computation times for each combination of batch size, signal length and number of channels, a list of strings containing the labels for each array. The x-axis variable indicates which variable should be used as x-axis in the plot. E.g. if x_axis_variable is ‘T’, the function will plot the computation times for different signal lengths. If the standard deviation of the computation times is provided, it will be plotted as shaded area around the mean values.

Note that this function is more an example of how you can plot the results than the “definitive” way to plot the results. You can customize the plot as you want, using the data returned by the get_array_to_plot() function.

Parameters:

  • plot_config (dict) –

    Dictionary containing the plot settings. The following settings are available (with their default values) : - figsize : tuple, default (10, 6)

    Size of the figure in inches.

    • fontsizeint, default 12

      Font size for the labels and legend.

    • markerstr, default ‘o’

      Marker style for the points in the plot.

    • markersizeint, default 8

      Size of the markers in the plot.

    • linewidthint, default 2

      Line width for the lines in the plot.

    • y_scale_logbool, default False

      Whether to use a logarithmic scale for the y-axis.

    • use_millisecondsbool, default False

      Whether to use milliseconds instead of seconds for the computation times. If True, the computation times will be multiplied by 1000 and the y-axis label will be updated accordingly.

    • path_savestr, default ‘benchmark/plot.png’

      If specified, the plot will be saved at the given path. If None, or not specified, the plot will not be saved.

  • x_axis_variable_list (list) – List of values for the x-axis variable.

  • x_axis_variable_name (str) – Name of the x-axis variable to use as label for the x-axis.

  • data_to_plot_mean (list) – List of arrays containing the mean computation times, as returned by the get_array_to_plot() function.

  • labels_to_plot (list) – List of strings containing the labels for each array in the data_to_plot_mean` list, as returned by the :func:`get_array_to_plot function.

  • data_to_plot_std (list, optional) – List of arrays containing the standard deviation of the computation times, as returned by the get_array_to_plot() function. If provided, the standard deviation will be plotted as shaded area around the mean values. The default value is None, which means that the standard deviation will not be plotted.

  • color_dict (list, optional) – Dictionary containing the colors to use for each label. The keys of the dictionary should be the same as the labels in the labels_to_plot list. The values should be valid color specifications for matplotlib (e.g. ‘red’, ‘#FF0000’, etc.). If not provided, the default color cycle of matplotlib will be used. The default value is None.

Returns:

  • fig (matplotlib.figure.Figure) – Figure object containing the plot.

  • ax (matplotlib.axes.Axes) – Axes object containing the plot.

dtw_loss_functions.benchmark.repeat_inference(x, x_r, loss_function: Module, n_repetitions: int = 100) → list[source]

Compute the loss for the input tensors x and x_r for a number of repetitions specified in n_repetitions and save the computation times in a list.

Parameters:

  • x (torch.tensor) – First input tensor of shape B x T x C.

  • x_r (torch.tensor) – Second input tensor of shape B x T x C.

  • loss_function (torch.nn.Module) – Loss function to use for the computation. The loss function must be an instance of a class implemented in this package.

  • n_repetitions (int) – Number of repetitions to perform. The default value is 100.

Returns:

time_list_loss – List of length n_repetitions containing the computation times for each repetition.

Return type:

list