Statistics / Writing results
=================================================
.. highlight:: c++
Statistics are functors that are called at the end of each iteration of the Bayesian optimizer. Their job is to:
- write the results to files;
- write the current state of the optimization;
- write the data that are useful for your own analyses.
All the statistics are written in a directory called ``hostname_date_pid-number``. For instance: ``wallepro-perso.loria.fr_2016-05-13_16_16_09_72226``
Limbo provides a few classes for common uses (see :ref:`statistics-stats` for details):
- ``ConsoleSummary``: writes a summary to ``std::cout`` at each iteration of the algorithm
- ``AggregatedObservations``: records the value of each evaluation of the function (after aggregation) [filename ``aggregated_observations.dat``]
- ``BestAggregatedObservations``: records the best value observed so far after each iteration [filename ``aggregated_observations.dat``]
- ``Observations``: records the value of each evaluation of the function (before aggregation) [filename: ``observations.dat``]
- ``Samples``: records the position in the search space of the evaluated points [filename: ``samples.dat``]
- ``BestObservations``: records the best observation after each iteration? [filename ``best_observations.dat``]
- ``BestSamples``: records the position in the search space of the best observation after each iteration [filename: ``best_samples.dat``]
These statistics are for "advanced users":
- ``GPAcquisitions``
- ``GPKernelHParams``
- ``GPLikelihood``
- ``GPMeanHParams``
The default statistics list is::
boost::fusion::vector, stat::AggregatedObservations,
stat::ConsoleSummary>
Writing your own statistics class
----------------------------------
Limbo only provides generic statistics classes. However, it is often useful to add user-defined statistics classes that are specific to a particular experiment.
All the statistics functors follow the same template:
.. code:: c++
template
struct Samples : public limbo::stat::StatBase {
template
void operator()(const BO& bo, const AggregatorFunction&)
{
// code
}
};
In a few words, they take a `BO` object (instance of the Bayesian optimizer) and do what they want.
For instance, we could add a statistics class that writes the worst observation at each iteration. Here is how to write this functor:
.. literalinclude:: ../../src/tutorials/statistics.cpp
:language: c++
:linenos:
:lines: 109-138
In order to configure the Bayesian optimizer to use our new statistics class, we first need to define a new statistics list which includes our new `WorstObservation`:
.. literalinclude:: ../../src/tutorials/statistics.cpp
:language: c++
:linenos:
:lines: 144-148
Then, we use it to define the optimizer:
.. literalinclude:: ../../src/tutorials/statistics.cpp
:language: c++
:linenos:
:lines: 151
The full source code is available in `src/tutorials/statistics.cpp` and reproduced here:
.. literalinclude:: ../../src/tutorials/statistics.cpp
:language: c++
:linenos:
:lines: 48-