Hierarchical control over foraging behavior by anterior cingulate cortex

Foraging is a natural behavior that involves making sequential decisions to maximize rewards while minimizing the costs incurred when doing so. The prevalence of foraging across species suggests that a common brain computation underlies its implementation. Although anterior cingulate cortex is believed to contribute to foraging behavior, its specific role has been contentious, with predominant theories arguing either that it encodes environmental value or choice difficulty. Additionally, recent attempts to characterize foraging have taken place within the reinforcement learning framework, with increasingly complex models scaling with task complexity. Here we review reinforcement learning foraging models, highlighting the hierarchical structure of many foraging problems. We extend this literature by proposing that ACC guides foraging according to principles of model-based hierarchical reinforcement learning. This idea holds that ACC function is organized hierarchically along a rostral-caudal gradient, with rostral structures monitoring the status and completion of high-level task goals (like finding food), and midcingulate structures overseeing the execution of task options (subgoals, like harvesting fruit) and lower-level actions (such as grabbing an apple).

Optimizing warfarin dosing using deep reinforcement learning

Warfarin is a widely used anticoagulant, and has a narrow therapeutic range. Dosing of warfarin should be individualized, since slight overdosing or underdosing can have catastrophic or even fatal consequences. Despite much research on warfarin dosing, current dosing protocols do not live up to expectations, especially for patients sensitive to warfarin. We propose a deep reinforcement learning-based dosing model for warfarin. To overcome the issue of relatively small sample sizes in dosing trials, we use a Pharmacokinetic/ Pharmacodynamic (PK/PD) model of warfarin to simulate dose-responses of virtual patients. Applying the proposed algorithm on virtual test patients shows that this model outperforms a set of clinically accepted dosing protocols by a wide margin. We tested the robustness of our dosing protocol on a second PK/PD model and showed that its performance is comparable to the set of baseline protocols.

Forward planning driven by context-dependant conflict processing in anterior cingulate cortex

Cognitive control and forward planning in particular is costly, and therefore must be regulated such that the amount of cognitive resources invested is adequate to the current situation. However, knowing in advance how beneficial forward planning will be in a given situation is hard. A way to know the exact value of planning would be to actually do it, which would ab initio defeat the purpose of regulating planning, i.e. the reduction of computational and time costs. One possible solution to this dilemma is that planning is regulated by learned associations between stimuli and the expected demand for planning. Such learning might be based on generalisation processes that cluster together stimulus states with similar control relevant properties into more general control contexts. In this way, the brain could infer the demand for planning, based on previous experience with situations that share some structural properties with the current situation. Here, we used a novel sequential task to test the hypothesis that people use control contexts to efficiently regulate their forward planning, using behavioural and functional magnetic resonance imaging data. Consistent with our hypothesis, reaction times increased with trial-by-trial conflict, where this increase was more pronounced in a context with a learned high demand for planning. Similarly, we found that fMRI activity in the dorsal anterior cingulate cortex (dACC) increased with conflict, and this increase was more pronounced in a context with generally high demand for planning. Taken together, the results indicate that the dACC integrates representations of planning demand at different levels of abstraction to regulate planning in an efficient and situation-appropriate way.

Brain signals of a Surprise-Actor-Critic model: Evidence for multiple learning modules in human decision making

Learning how to reach a reward over long series of actions is a remarkable capability of humans, and potentially guided by multiple parallel learning modules. Current brain imaging of learning modules is limited by (i) simple experimental paradigms, (ii) entanglement of brain signals of different learning modules, and (iii) a limited number of computational models considered as candidates for explaining behavior. Here, we address these three limitations and (i) introduce a complex sequential decision making task with surprising events that allows us to (ii) dissociate correlates of reward prediction errors from those of surprise in functional magnetic resonance imaging (fMRI); and (iii) we test behavior against a large repertoire of model-free, model-based, and hybrid reinforcement learning algorithms, including a novel surprise-modulated actor-critic algorithm. Surprise, derived from an approximate Bayesian approach for learning the world-model, is extracted in our algorithm from a state prediction error. Surprise is then used to modulate the learning rate of a model-free actor, which itself learns via the reward prediction error from model-free value estimation by the critic. We find that action choices are well explained by pure model-free policy gradient, but reaction times and neural data are not. We identify signatures of both model-free and surprise-based learning signals in blood oxygen level dependent (BOLD) responses, supporting the existence of multiple parallel learning modules in the brain. Our results extend previous fMRI findings to a multi-step setting and emphasize the role of policy gradient and surprise signalling in human learning.

A MEA is a MEA is a MEA? Sequential decision making and the impact of different managed entry agreements at the manufacturer and payer level, using a case study for an oncology drug in England

BACKGROUND

In a typical single-payer setting that uses an explicit cost-effectiveness (CE) threshold in its decision-making, the payer aims to maximize the net-monetary-benefit (NMB) given the CE threshold, whilst the manufacturer aims to maximize the expected discounted-cash-flow (DCF) resulting from the sales of that technology. Managed entry agreements (MEAs) are tools that are used to improve access to expensive technologies that would otherwise not be deemed to be cost-effective to payers. While simple discount on the list price is the most commonly applied MEA type, there are different forms, each having a different impact on the cost-effectiveness of the technology, on the lifetime DCF-per-patient and on the decision uncertainty. We aim to analyze the sequential decision-making (SDM) of different MEAs (i.e. simple discount, free treatment initiation, lifetime treatment acquisition cost-capping [LTTACC], performance-based money-back guarantee [MBG]) at the manufacturer and at the payer level, respectively.

METHODS

We first model the SDM of the manufacturer and the payer as a sequential game and explain the challenges to find an equilibrium analytically. Then we propose a heuristic computational method to follow for each of the MEA types, based on practice. To demonstrate this SDM on a case study, a UK-based cost-utility analysis using a three-state, partitioned-survival-model was constructed to determine the cost-effectiveness of regorafenib versus best-supportive-care for the second-line treatment of hepatocellular carcinoma. The optimal agreement terms that would maximise the lifetime DCF-per-patient for each MEA, whilst remaining below the CE-threshold (£50,000/QALY gained) were obtained in the deterministic base-case. Robustness for each optimized MEA was then assessed using probabilistic sensitivity and scenario analyses, the value of information (VoI), and HTA-risk analyses.

RESULTS

As expected, the introduction of all MEAs improved the probabilistic ICER and NMB values to (almost) acceptable levels, compared to the "no-MEA" case (ICER ~ £78,000/QALY-gained). The expected DCFs across the explored MEAs were all similar, whilst the payer strategy & uncertainty burden (PSUB) for regorafenib decreased in all MEAs explored. VoI analyses revealed that regorafenib mean-dose-intensity and time-on-treatment (ToT) parameters attributed most to the decision uncertainty. LTTACC provided the smallest PSUB and the most robust NMB estimates under parametric uncertainty. For scenarios assuming increased regorafenib ToT or mean-dose-intensity, LTACC again provided acceptable cost-effectiveness outcomes, whereas for scenarios assuming decreased regorafenib progression-free/overall survival effectiveness, only MBG resulted in plausible ICER values. In scenarios, where the source of uncertainty was not targeted by MEA parameters (e.g. the scenario assuming higher progressed disease resource utilization), all investigated MEA types resulted in unacceptable cost-effectiveness outcomes.

CONCLUSION

Each MEA type has a different implication. The impact of different MEAs on the NMB is more noteworthy than on the DCF, in relative terms, hence payers will benefit from the early participation of the MEA design rather than leaving this up to the prerogative of the manufacturer. While simple discount might be practical for implementation purposes, other MEAs can provide additional benefits to the payer in terms of increased NMB, reduced decision risk and reduced uncertainty. MEA performance should be investigated not only under parametric uncertainty, but also under-identified structural uncertainty, and the barriers of implementation should be considered thoroughly before choosing the most appropriate MEA type.