[EN] How to Create Useful Graphs and Submit Solutions

Submission of solutions

Evolutionary algorithms are based on randomness, i.e. each run can have a slightly different outcome. Therefore, it is not possible to compare the results of only one run. Instead, we need to repeat the run a number of times and compare the performance in the average/best/worst case.

The submitted solution must contain the following:

  1. A short description of what you did and tried, what was the outcome. Around 5-10 sentences are enough.
  2. A plot showing the optimized criteria as a function of fitness evaluations. The plot must show the average (or median) value and first and third quartile. If you compare more algorithms, they all should be in a single plot.
  3. Do not zip the plots (unless there is a lot of them}. Upload them as pictures.

Scripts to create the plots

To create the plot with the fitness you can use the scripts createGraphs.ps1 (for Windows PowerShell), or createGraphs.sh(for Unix). Both scripts require to have gnuplot installed and in $PATH.

The scripts expect as an input the log of the performance of the algorithm. Each line in the file contains six numbers - the number of fitness evaluation up to this point, and 5 numbers describing the statistical properties of the fitness of the best individual in each of a number of runs - minimum, first quartile, average, third quartile, and maximum. Such files are produced by the available source codes, or you can produce them yourself (if you prefer programming in other languages and want to use the scripts to make the graphs).

Using the scripts is quite easy. They require only two parameters, which indicate where are their inputs (the outputs of the evolutionary algorithm) a how they should be named in the plot. Assume, we have outputs of two algorithms in files logs/basic.objective_stats and logs/better.objective_stats. To compare these algorithms, we have to run

createGraphs.ps1 -logFileNames basic,better -legendNames Basic,Better

This yields the comparison of the two algorithms in one plot named output.svg, the names in the legend will read “Basic” and “Better”.

This script has more useful parameters:

  • output specifies the name of the output file (default output.svg)
  • title sets the title of the plot (default “Objective value log”)
  • logScale specifies, which axis use logarithmic scale, possible values are xxyy (default - both axis use linear scale)
  • path specifies the directory with the outputs (default “”)
  • scale sets the scaling of the horizontal axis, e.g. if -scale 1000 is used, all the numbers are divided by 1000 (default 1)
  • barsEvery sets the frequency of the error bars, as the number of lines in the input file (default 20)
  • limit is the upper limit of the horizontal axis (default “” - no limit)

These parameters can be used to make the resulting plot more readable. Use them and experiment with them. Try to make the plots as readable as possible.

A Python script is also available to create the graphs. It does not need gnuplot, but uses numpy and matplotlib instead. The parameters of the script are similar to those of the scripts described above. Check them in the source code.

The following hints may help you to make better graphs

  • If both algorithms converge long before they are stopped, the plot contains long almost horizontal lines and most information is contained on the left side -> use a suitable value of the limit parameter to ignore the right part of the plot
  • If one of the algorithms converges much faster than the other, use logScale x or logScale xy to emphasize the left part of the plot.
  • If there is a large difference in the objective values in the beginning of the evolution and in the end, it can lead to almost vertical lines in the plot, use logScale y or logScale xy to make the small differences in the end more pronounced
  • If error bars are too close to each other, use a larger value of the barsEvery parameter (and vice versa)
Last modified: Monday, 7 October 2019, 3:11 PM