[EN] How to Create Useful Graphs and Submit Solutions

Submission of solutions

Evolutionary algorithms are based on randomness, i.e. each run can have a slightly different outcome. Therefore, it is not possible to compare the results of only one run. Instead, we need to repeat the run a number of times and compare the performance in the average/best/worst case.

The submitted solution must contain the following:

1. A short description of what you did and tried, what was the outcome. Around 5-10 sentences are enough.
2. A plot showing the optimized criteria as a function of fitness evaluations. The plot must show the average (or median) value and first and third quartile. If you compare more algorithms, they all should be in a single plot.
3. Do not zip the plots (unless there is a lot of them}. Upload them as pictures.

Scripts to create the plots

In the repository, you can find the plotting.py file. This can be used as a baseline to create your plots. The main function there is utils.plot_experiments that takes two arguments - the path to the directory with the logs of your runs, and a list of experiment IDs of the experiments you want to plot (The experiment ID can be set in each of the scripts you will get for each of the assignments in the variable EXP_ID.) Additionally, you can also add a rename_dict where you can specify more readable names for you experiments that will be displayed in the legend. For example, if you used EXP_ID = 'pop100m0.3' for your experiment with population size of 100 individuals and mutation rate of 0.3, you can add parameter rename_dict={'pop100m0.3': 'Population size = 100, Mutation rate = 0.3'}.

You may need to adapt the script slightly, if you want to change the scaling of the plot, or limit the ranges of the axis. You can always add additional matplotlib commands later, for example, if you want to change the scaling of the y-axis to logarithmic scaling, you can add plt.yscale('log') above the line with plt.show().

Standalone Scripts

There are also standalone scripts in the directory that allow you to create the plots from the same files as those you can use with the plotting.py file. However, using the method described above is probably easier and more user friendly.

To create the plot with the fitness you can use the scripts createGraphs.ps1 (for Windows PowerShell), or createGraphs.sh(for Unix). Both scripts require to have gnuplot installed and in $PATH.

The scripts expect as an input the log of the performance of the algorithm. Each line in the file contains six numbers - the number of fitness evaluation up to this point, and 5 numbers describing the statistical properties of the fitness of the best individual in each of a number of runs - minimum, first quartile, average, third quartile, and maximum. Such files are produced by the available source codes, or you can produce them yourself (if you prefer programming in other languages and want to use the scripts to make the graphs).

Using the scripts is quite easy. They require only two parameters, which indicate where are their inputs (the outputs of the evolutionary algorithm) a how they should be named in the plot. Assume, we have outputs of two algorithms in files logs/basic.objective_stats and logs/better.objective_stats. To compare these algorithms, we have to run

createGraphs.ps1 -logFileNames basic,better -legendNames Basic,Better

This yields the comparison of the two algorithms in one plot named output.svg, the names in the legend will read “Basic” and “Better”.

This script has more useful parameters:

• output specifies the name of the output file (default output.svg)
• title sets the title of the plot (default “Objective value log”)
• logScale specifies, which axis use logarithmic scale, possible values are x, xy, y (default - both axis use linear scale)
• path specifies the directory with the outputs (default “”)
• scale sets the scaling of the horizontal axis, e.g. if -scale 1000 is used, all the numbers are divided by 1000 (default 1)
• barsEvery sets the frequency of the error bars, as the number of lines in the input file (default 20)
• limit is the upper limit of the horizontal axis (default “” - no limit)

These parameters can be used to make the resulting plot more readable. Use them and experiment with them. Try to make the plots as readable as possible.

A Python script is also available to create the graphs. It does not need gnuplot, but uses numpy and matplotlib instead. The parameters of the script are similar to those of the scripts described above. Check them in the source code.

The following hints may help you to make better graphs

• If both algorithms converge long before they are stopped, the plot contains long almost horizontal lines and most information is contained on the left side -> use a suitable value of the limit parameter to ignore the right part of the plot
• If one of the algorithms converges much faster than the other, use logScale x or logScale xy to emphasize the left part of the plot.
• If there is a large difference in the objective values in the beginning of the evolution and in the end, it can lead to almost vertical lines in the plot, use logScale y or logScale xy to make the small differences in the end more pronounced
• If error bars are too close to each other, use a larger value of the barsEvery parameter (and vice versa)