Benchmarked: Optimization Meets Machine Learning

9 - 13 November 2020

Venue: Lorentz Center@Oort

If you are invited or already registered for this workshop, you have received login details by email.

The dramatic increase of computational resources, the availability of enormous data, and the significant advances in machine learning and artificial intelligence are currently shaking up our societies. As part of this change, our use of technology is evolving at a fast pace, with lasting effects on our traditional routines. Decisions that always required considerable expert knowledge can suddenly be taken (or be prepared) by automated procedures. This automation does not only support manual work, but it also drastically changes the objectives of various research domains, including Computer Science itself.

In the Benchmarked: Optimization Meets Machine Learning workshop we will discuss the impact of automated decision-making on an important sub-domain: heuristic optimization. More specifically, we will discuss how the possibility to automatically select and configure optimization heuristics changes the requirements for their benchmarking.

The key objectives of this Lorentz Center workshop are
- to develop a joint vision on the next generation of benchmarking optimization heuristics in the context of automated algorithm selection and configuration, and
- to design a clear road-map guiding the research community towards this vision.

The workshop brings together researchers from different sub-domains in optimization heuristics with colleagues from automated machine learning. Together we will discuss what an ideal benchmarking environment would look like, how such an ``ideal tool'' compares to existing software, and how we can close the gap by improving the compatibility between ongoing and future projects. Concretely, we aim at designing a full benchmarking engine that ranges from modular algorithm frameworks over problem instance generators and landscape analysis tools to automated algorithm configuration and selection techniques, all the way to a statistically sound evaluation of the experimental data.


    Monday 9 November

    09:3010:00 Registration and coffee
    10:0010:15 Workshop opening by Lorentz Center
    10:1510:30 Welcome by organizers, goals of the workshop, organizational matters
    10:3011:30 Short introduction of participants (1 slide each)
    11:3012:00 Plenary discussion: key components of the benchmarking pipeline
    12:0013:30 Lunch break
    13:3014:00 Tutorial: Automated Algorithm Configuration and Selection
    14:0014:30 Presenation of tools I
    14:3015:00 Coffee break
    15:0015:30 Presentation of tools II
    15:3016:30 Brianstorming in small groups: Think Big! Next Generation Benchmarking Environments
    16:3017:00 Wrap-up of the brainstorming session, organizational matters
    17:0018:30 Wine&Cheese welcome reception

    Tuesday 10 November

    09:3010:00 Tutorial: Modular Algorith Frameworks
    10:0010:30 Tutoral: Perfomance Measures and the Role of Statistics
    10:3011:00 Coffee break
    11:0012:00 Breakout sessions
    12:0013:30 Lunch break
    13:3014:15 Individual discussions, flex time
    14:1514:45 Coffee break
    14:4515:15 Tutorial: Landscape Analysis
    15:1516:00 Poster session
    16:0016:45 Wrap up and organizaton of evening gatherings

    Wednesday 11 November

    09:3010:00 Invited presentation
    10:0010:30 Flashlight talks
    10:3011:00 Coffee break
    11:0012:00 Creativity session: challenges and pitfalls
    12:0013:30 Lunch break
    13:3017:00 Individual discussions, flex time
    15:0015:30 Coffee break
    15:3017:00 Individual discussions, flex time
    17:0022:00 Workshopdinner

    Thursday 12 November

    09:3010:00 Invited presentation
    10:0010:30 Contributede presentations
    10:3011:00 Coffee break
    11:0012:00 Breakout sessions
    12:0013:30 Lunch break
    13:3015:30 Individual discussions, flex time
    15:3016:00 Coffee break
    16:0016:45 Plenary discussion: Open Science, Reproducibility, Communication
    16:4517:00 Wrap-up of the day, announcements for evening gatherings

    Friday 13 November

    00:4511:15 Coffee break
    09:3010:00 Invited presentation
    10:0010:45 Breakout session
    11:1511:45 Flash coaching: The last mile is the longest (but yet so important): how to keep up your motivation for the final steps before releasing a research "product"
    11:4512:15 Plenary discussion: what have we archieved? Open questions? Next steps?
    12:1512:30 Wrap-up & feedback
    12:3014:00 Lunch break
    14:0015:00 Individual discussions and departures
    Please log in to view the participants information...

    Carola Doerr, CNRS and Sorbonne University  

    Thomas Stützle, Université Libre de Bruxelles (ULB)  

    Mike Preuss, Leiden University  

    Marc Schoenauer, INRIA Saclay  

    Joaquin Vanschoren, Eindhoven University of Technology  

Michelle Grandia

+31 71-5275579

Follow us on: