Control by Reinforcement Learning of Shear flows (M/F)

The PhD position will take place at LISN within the framework of the ANR-REASON project. The project is coordinated by the LISN. The PhD fellow is expected to interact mainly with the members of the Decipher team at LISN and work in collaboration with the partners of the project at TAU-team and LAMSADE Paris-Dauphine. The experimental part will be led by the IMSIA-ENSTA (Palaiseau). Missions for conferences are financed during the fellowship. The PhD fellow will receive a laptop and accessories.

Environmental needs are invigorating research interest in many engineering fields. A compelling example is provided by carbon-dioxide emissions, widely considered one of the main causes of global warming. This urgency extends to numerous applications including aeronautics, where it is recognized that the optimization of aerodynamic flows may have a deep impact on the reduction of pollutant emissions, mitigation of acoustic noise or control of highly complex conditions such as separation. In principle, flow control strategies optimize the flow in real time; in practice, in realistic cases, this technique is used only in limited numerical and experimental cases. The project ANR-REASON aims at verifying to what extent Reinforcement Learning (RL) is a viable strategy for the control of fluids in realistic conditions. RL algorithms use past data to enhance the future manipulation of a dynamical system by discovering optimal control policy, determined from the exploration of the state-action space. This step replaces physical models; thus these algorithms are fully data-driven and solely rely on the measurements of the system and the way it reacts to prescribed actions. This allows to circumvent some drawbacks of model-based control, as approximate reduced-order models of the physical system can critically lose accuracy when control is applied, resulting in poor performance and lack of robustness. The main goal of the project is to provide major breakthroughs in flow control by bridging it with Reinforcement Learning in order to tackle the challenges which have limited the success of standard control tools in non-linear and complex flows.
The PhD student will focus specifically on the control of fluids using RL. During a first step, the candidate is expected to analyse from a theoretical viewpoint RL, namely the actor-critic algorithms, within the theoretical framework of dynamic programming and optimal control. The methodological efforts will focus in particular on two aspects: I) the improvement of the exploration and exploitation during the training of the optimal RL policies ii) the introduction of physics constraints and/or alternative approximation for the value function and optimal policies. These working areas are meant at reducing the amount of data required for the training and suggesting possible methodologies for the introduction of guarantee margins.
The numerical applications will be of incremental complexity. We will first consider simplified models as benchmarks (i.e. the Kuramoto-Sivashinsky equation or the Ginzburg-Landau equation), for the development and direct comparison of RL tools with optimal control. As a fluid mechanics example, we will consider the linearised boundary layer in two dimensions. Final demonstrators will include numerical simulations of transitional shear flows at moderate Reynolds number, and, coordinated by the team at ENSTA, the control of the bistable dynamics of an experimental 3D bluff body wake-flow.