Tesi etd-03302023-190316
Link copiato negli appunti
Tipo di tesi
Dottorato
Autore
ROBERTAZZI, FEDERICA
URN
etd-03302023-190316
Titolo
Brain-inspired meta-learning for response inhibition during decision-making in artificial agents: a neurorobotics approach
Settore scientifico disciplinare
ING-INF/06
Corso di studi
Istituto di Biorobotica - PHD IN BIOROBOTICA
Commissione
relatore Dott. FALOTICO, EGIDIO
Membro Dott. KHAMASSI, MEHDI
Presidente Prof.ssa MENCIASSI, ARIANNA
Membro Dott. KHAMASSI, MEHDI
Presidente Prof.ssa MENCIASSI, ARIANNA
Parole chiave
- Meta-learning
- Brain-inspired modeling
- Inhibition cognitive control
- Decision-making
- Action cancellation/restraint
- Stop-Signal Paradigm/ NoGo Paradigm
- Basal ganglia
- Prefrontal cortex
- Serotonin-dopamine interaction
- Nigrostriatal pathway
- Mesocorticolimbic pathway
Data inizio appello
07/10/2024;
Disponibilità
completa
Riassunto analitico
Action inhibition – the disengagement from a motor response – represents a crucial aspect of cognitive control, enabling the adaptation to changing demands by prioritizing relevant information over internal urges. While current artificial agents have been becoming extremely efficient in the execution of repetitive tasks, real-world scenarios necessitate the handling of unexpected constraints, such as suppressing unwanted actions (Pfeifer et al., 2007; Spelke & Kinzler, 2007).
Meta-learning, applied to reinforcement learning has the potential to enhance the design of control algorithms where an high-level outer learning system progressively adjusts the operation of a low-level inner learning system, yielding to practical benefits for the learning schema and facilitating adaptive learning in non-stationary environments (Baxter, 1998; Schmidhuber et al., 1996; Thrun & Pratt, 1998; Xu et al., 2018, 2020).
In the first study outlined in this thesis, we developed a brain-inspired meta-learning framework for conflictual inhibition decision-making. This framework: i) leverages meta-learning principles in the neuromodulation theory proposed by Doya (Doya, 2002; Schweighofer & Doya, 2003), ii) relies on a well-established neural architecture encompassing distributed learning systems within the human brain , e.g., cortical areas and basal ganglia circuitry (Khamassi et al., 2011, 2013), and iii) proposes optimization rules for meta-learning hyperparameters that emulate the dynamics of the major neurotransmitters in the brain.
Doya’s neuromodulation theory is grounded on the concept that neurophysiological knowledge about neurotransmitters dynamics and interplay, provides feasible strategies for parameterization, (Doya, 2002; Schweighofer & Doya, 2003), suggesting an equivalence between the neurophysiological role of the four major neurotransmitters (e.g., acetylcholine, serotonin, dopamine, and noradrenaline) in the brain and the computational role of the meta-parameters shaping the meta-learning processes in reinforcement learning.
We tested an artificial agent in inhibiting the action command in two different conflictual tasks, i.e., NoGo and Stop-Signal Paradigms, that require different types of action inhibition (Eagle et al., 2008; Mosher et al., 2021; Pasquereau & Turner, 2017; Schall et al., 2017). In NoGo Paradigm we tested the ability to withdraw a not-yet-initiated action from responding (i.e., action restraint) using a hold signal before the initiation of the movement. In Stop-Signal Paradigm we investigated the ability to cancel an initiated response (i.e., action cancellation) triggering an unpredictable hold signal after a range of delays from the action onset.
In the test phase, the artificial agent learned to react to the hold signal, enabling the successful inhibition of the motor command in both tasks, through the continuous adjustment of the learning meta-parameters. Our findings revealed a significant increase in global accuracy, right inhibition, and a reduction in the latency time required to cancel the action process, i.e., the Stop-signal reaction time. Additionally, we conducted a sensitivity analysis to assess the behavioral impact of the meta-parameters, with a focus on the serotoninergic modulation of dopamine release (open-loop condition), given the pivotal role of serotonin in regulating agent’s impulsive behaviors.
In the second study detailed in this thesis, we expanded the brain-inspired meta-reinforcement learning framework for inhibition cognitive control (Robertazzi et al., 2022) by incorporating the dynamic regulation of serotonin and dopamine neurotransmitters through a closed-loop interaction within the artificial agent's meta-reinforcement cognitive control architecture for adapting neurotransmitter levels based on the agent's learning needs during the Stop-Signal Paradigm.
Our results showed that inhibition performance metrics, including right inhibition, Stop-signal reaction time and accuracy improved in the closed-loop scenario compared to the previous open-loop implementation.
Additionally, in this second study we utilized the framework as an in-silico platform for generating testable hypotheses and novel insights about neurophysiological mechanisms and understanding of mechanistic processes during the inhibition decision-making process. For example, we investigated the effects of changes in concentration and efficacy of the D_1-mesocorticolimbic and D_2-nigrostriatal pathways externally modulated by serotonin release on meta-reinforcement learning rules, thus the extent to which they affect behavioral performance during action cancellation within the Stop-Signal Paradigm. Moreover, we explored scenarios of coupling configurations and interplays between serotonin and dopamine understanding their role in response inhibition and, consequently, how they might be involved in impulsive behaviors.
Hence, in the context of the neurorobotics research, we demonstrated that brain-inspired mechanisms for implementing meta-learning processes enable artificial agents to achieve more robust and flexible behaviors when conflictual inhibitory signals are present in the environment. At the same time, we employed this framework to generate novel insights and testable hypotheses about neural mechanisms/functions underlying high-level cognitive processes within the human brain.
References
Baxter, J. (1998). Theoretical Models of Learning to Learn. In S. Thrun & L. Pratt (Eds.), Learning to Learn (pp. 71–94). Springer US. https://doi.org/10.1007/978-1-4615-5529-2_4
Doya, K. (2002). Metalearning and neuromodulation. Neural Networks: The Official Journal of the International Neural Network Society, 15(4–6), 495–506. https://doi.org/10.1016/s0893-6080(02)00044-8
Eagle, D. M., Bari, A., & Robbins, T. W. (2008). The neuropsychopharmacology of action inhibition: Cross-species translation of the stop-signal and go/no-go tasks. Psychopharmacology, 199(3), 439–456. https://doi.org/10.1007/s00213-008-1127-6
Khamassi, M., Enel, P., Dominey, P. F., & Procyk, E. (2013). Medial prefrontal cortex and the adaptive regulation of reinforcement learning parameters. In Progress in Brain Research (Vol. 202, pp. 441–464). Elsevier. https://doi.org/10.1016/B978-0-444-62604-2.00022-8
Khamassi, M., Lallée, S., Enel, P., Procyk, E., & Dominey, P. F. (2011). Robot Cognitive Control with a Neurophysiologically Inspired Reinforcement Learning Model. Frontiers in Neurorobotics, 5. https://doi.org/10.3389/fnbot.2011.00001
Mosher, C. P., Mamelak, A. N., Malekmohammadi, M., Pouratian, N., & Rutishauser, U. (2021). Distinct roles of dorsal and ventral subthalamic neurons in action selection and cancellation. Neuron, 109(5), 869-881.e6. https://doi.org/10.1016/j.neuron.2020.12.025
Pasquereau, B., & Turner, R. S. (2017). A selective role for ventromedial subthalamic nucleus in inhibitory control. eLife, 6, e31627. https://doi.org/10.7554/eLife.31627
Pfeifer, R., Lungarella, M., & Iida, F. (2007). Self-Organization, Embodiment, and Biologically Inspired Robotics. Science, 318(5853), 1088–1093. https://doi.org/10.1126/science.1145803
Robertazzi, F., Vissani, M., Schillaci, G., & Falotico, E. (2022). Brain-inspired meta-reinforcement learning cognitive control in conflictual inhibition decision-making task for artificial agents. Neural Networks, 154, 283–302. https://doi.org/10.1016/j.neunet.2022.06.020
Schall, J. D., Palmeri, T. J., & Logan, G. D. (2017). Models of inhibitory control. Philosophical Transactions of the Royal Society B: Biological Sciences, 372(1718), 20160193. https://doi.org/10.1098/rstb.2016.0193
Schmidhuber, J., Zhao, J., & Wiering, M. A. (1996). Simple Principles of Metalearning. https://api.semanticscholar.org/CorpusID:14099817
Schweighofer, N., & Doya, K. (2003). Meta-learning in Reinforcement Learning. Neural Networks, 16(1), 5–9. https://doi.org/10.1016/S0893-6080(02)00228-9
Spelke, E. S., & Kinzler, K. D. (2007). Core knowledge. Developmental Science, 10(1), 89–96. https://doi.org/10.1111/j.1467-7687.2007.00569.x
Thrun, S., & Pratt, L. (1998). Learning to Learn: Introduction and Overview. In S. Thrun & L. Pratt (Eds.), Learning to Learn (pp. 3–17). Springer US. https://doi.org/10.1007/978-1-4615-5529-2_1
Xu, Z., van Hasselt, H., Hessel, M., Oh, J., Singh, S., & Silver, D. (2020). Meta-Gradient Reinforcement Learning with an Objective Discovered Online. arXiv:2007.08433 [Cs, Stat]. http://arxiv.org/abs/2007.08433
Xu, Z., van Hasselt, H., & Silver, D. (2018). Meta-Gradient Reinforcement Learning. arXiv:1805.09801 [Cs, Stat]. http://arxiv.org/abs/1805.09801
Meta-learning, applied to reinforcement learning has the potential to enhance the design of control algorithms where an high-level outer learning system progressively adjusts the operation of a low-level inner learning system, yielding to practical benefits for the learning schema and facilitating adaptive learning in non-stationary environments (Baxter, 1998; Schmidhuber et al., 1996; Thrun & Pratt, 1998; Xu et al., 2018, 2020).
In the first study outlined in this thesis, we developed a brain-inspired meta-learning framework for conflictual inhibition decision-making. This framework: i) leverages meta-learning principles in the neuromodulation theory proposed by Doya (Doya, 2002; Schweighofer & Doya, 2003), ii) relies on a well-established neural architecture encompassing distributed learning systems within the human brain , e.g., cortical areas and basal ganglia circuitry (Khamassi et al., 2011, 2013), and iii) proposes optimization rules for meta-learning hyperparameters that emulate the dynamics of the major neurotransmitters in the brain.
Doya’s neuromodulation theory is grounded on the concept that neurophysiological knowledge about neurotransmitters dynamics and interplay, provides feasible strategies for parameterization, (Doya, 2002; Schweighofer & Doya, 2003), suggesting an equivalence between the neurophysiological role of the four major neurotransmitters (e.g., acetylcholine, serotonin, dopamine, and noradrenaline) in the brain and the computational role of the meta-parameters shaping the meta-learning processes in reinforcement learning.
We tested an artificial agent in inhibiting the action command in two different conflictual tasks, i.e., NoGo and Stop-Signal Paradigms, that require different types of action inhibition (Eagle et al., 2008; Mosher et al., 2021; Pasquereau & Turner, 2017; Schall et al., 2017). In NoGo Paradigm we tested the ability to withdraw a not-yet-initiated action from responding (i.e., action restraint) using a hold signal before the initiation of the movement. In Stop-Signal Paradigm we investigated the ability to cancel an initiated response (i.e., action cancellation) triggering an unpredictable hold signal after a range of delays from the action onset.
In the test phase, the artificial agent learned to react to the hold signal, enabling the successful inhibition of the motor command in both tasks, through the continuous adjustment of the learning meta-parameters. Our findings revealed a significant increase in global accuracy, right inhibition, and a reduction in the latency time required to cancel the action process, i.e., the Stop-signal reaction time. Additionally, we conducted a sensitivity analysis to assess the behavioral impact of the meta-parameters, with a focus on the serotoninergic modulation of dopamine release (open-loop condition), given the pivotal role of serotonin in regulating agent’s impulsive behaviors.
In the second study detailed in this thesis, we expanded the brain-inspired meta-reinforcement learning framework for inhibition cognitive control (Robertazzi et al., 2022) by incorporating the dynamic regulation of serotonin and dopamine neurotransmitters through a closed-loop interaction within the artificial agent's meta-reinforcement cognitive control architecture for adapting neurotransmitter levels based on the agent's learning needs during the Stop-Signal Paradigm.
Our results showed that inhibition performance metrics, including right inhibition, Stop-signal reaction time and accuracy improved in the closed-loop scenario compared to the previous open-loop implementation.
Additionally, in this second study we utilized the framework as an in-silico platform for generating testable hypotheses and novel insights about neurophysiological mechanisms and understanding of mechanistic processes during the inhibition decision-making process. For example, we investigated the effects of changes in concentration and efficacy of the D_1-mesocorticolimbic and D_2-nigrostriatal pathways externally modulated by serotonin release on meta-reinforcement learning rules, thus the extent to which they affect behavioral performance during action cancellation within the Stop-Signal Paradigm. Moreover, we explored scenarios of coupling configurations and interplays between serotonin and dopamine understanding their role in response inhibition and, consequently, how they might be involved in impulsive behaviors.
Hence, in the context of the neurorobotics research, we demonstrated that brain-inspired mechanisms for implementing meta-learning processes enable artificial agents to achieve more robust and flexible behaviors when conflictual inhibitory signals are present in the environment. At the same time, we employed this framework to generate novel insights and testable hypotheses about neural mechanisms/functions underlying high-level cognitive processes within the human brain.
References
Baxter, J. (1998). Theoretical Models of Learning to Learn. In S. Thrun & L. Pratt (Eds.), Learning to Learn (pp. 71–94). Springer US. https://doi.org/10.1007/978-1-4615-5529-2_4
Doya, K. (2002). Metalearning and neuromodulation. Neural Networks: The Official Journal of the International Neural Network Society, 15(4–6), 495–506. https://doi.org/10.1016/s0893-6080(02)00044-8
Eagle, D. M., Bari, A., & Robbins, T. W. (2008). The neuropsychopharmacology of action inhibition: Cross-species translation of the stop-signal and go/no-go tasks. Psychopharmacology, 199(3), 439–456. https://doi.org/10.1007/s00213-008-1127-6
Khamassi, M., Enel, P., Dominey, P. F., & Procyk, E. (2013). Medial prefrontal cortex and the adaptive regulation of reinforcement learning parameters. In Progress in Brain Research (Vol. 202, pp. 441–464). Elsevier. https://doi.org/10.1016/B978-0-444-62604-2.00022-8
Khamassi, M., Lallée, S., Enel, P., Procyk, E., & Dominey, P. F. (2011). Robot Cognitive Control with a Neurophysiologically Inspired Reinforcement Learning Model. Frontiers in Neurorobotics, 5. https://doi.org/10.3389/fnbot.2011.00001
Mosher, C. P., Mamelak, A. N., Malekmohammadi, M., Pouratian, N., & Rutishauser, U. (2021). Distinct roles of dorsal and ventral subthalamic neurons in action selection and cancellation. Neuron, 109(5), 869-881.e6. https://doi.org/10.1016/j.neuron.2020.12.025
Pasquereau, B., & Turner, R. S. (2017). A selective role for ventromedial subthalamic nucleus in inhibitory control. eLife, 6, e31627. https://doi.org/10.7554/eLife.31627
Pfeifer, R., Lungarella, M., & Iida, F. (2007). Self-Organization, Embodiment, and Biologically Inspired Robotics. Science, 318(5853), 1088–1093. https://doi.org/10.1126/science.1145803
Robertazzi, F., Vissani, M., Schillaci, G., & Falotico, E. (2022). Brain-inspired meta-reinforcement learning cognitive control in conflictual inhibition decision-making task for artificial agents. Neural Networks, 154, 283–302. https://doi.org/10.1016/j.neunet.2022.06.020
Schall, J. D., Palmeri, T. J., & Logan, G. D. (2017). Models of inhibitory control. Philosophical Transactions of the Royal Society B: Biological Sciences, 372(1718), 20160193. https://doi.org/10.1098/rstb.2016.0193
Schmidhuber, J., Zhao, J., & Wiering, M. A. (1996). Simple Principles of Metalearning. https://api.semanticscholar.org/CorpusID:14099817
Schweighofer, N., & Doya, K. (2003). Meta-learning in Reinforcement Learning. Neural Networks, 16(1), 5–9. https://doi.org/10.1016/S0893-6080(02)00228-9
Spelke, E. S., & Kinzler, K. D. (2007). Core knowledge. Developmental Science, 10(1), 89–96. https://doi.org/10.1111/j.1467-7687.2007.00569.x
Thrun, S., & Pratt, L. (1998). Learning to Learn: Introduction and Overview. In S. Thrun & L. Pratt (Eds.), Learning to Learn (pp. 3–17). Springer US. https://doi.org/10.1007/978-1-4615-5529-2_1
Xu, Z., van Hasselt, H., Hessel, M., Oh, J., Singh, S., & Silver, D. (2020). Meta-Gradient Reinforcement Learning with an Objective Discovered Online. arXiv:2007.08433 [Cs, Stat]. http://arxiv.org/abs/2007.08433
Xu, Z., van Hasselt, H., & Silver, D. (2018). Meta-Gradient Reinforcement Learning. arXiv:1805.09801 [Cs, Stat]. http://arxiv.org/abs/1805.09801
File
| Nome file | Dimensione |
|---|---|
| PhD_thes...ce_up.pdf | 21.87 Mb |
Contatta l'autore |
|