One Period Rewards

Loading...

From the course by New York University Tandon School of Engineering

Reinforcement Learning in Finance

17 ratings

New York University Tandon School of Engineering

Reinforcement Learning in Finance

17 ratings

Course 3 of 4 in the Specialization Machine Learning and Reinforcement Learning in Finance

This course aims at introducing the fundamental concepts of Reinforcement Learning (RL), and develop use cases for applications of RL for option valuation, trading, and asset management. Prerequisites are the courses "Guided Tour of Machine Learning in Finance" and "Fundamentals of Machine Learning in Finance". Students are expected to know the lognormal process and how it can be simulated. Knowledge of option pricing is not assumed but desirable.

From the lesson

RL and INVERSE RL for Portfolio Stock Trading

Week Welcome Video2:17

Introduction to RL for Trading12:59

Portfolio Model8:08

One Period Rewards6:26

Forward and Inverse Optimisation10:05

Reinforcement Learning for Portfolios9:02

Entropy Regularized RL8:41

RL Equations10:04

RL and Inverse Reinforcement Learning Solutions10:51

Course Summary3:07

Meet the Instructors

Igor Halperin

Next, we want to specify one period reward, or negative costs in our module.

An instantaneous random reward received upon taking such action is obtained by

substituting the equation for return, shown as the first equation here,

into the equation for the portfolio change that we derived earlier.

This produces an instantaneous random reward,

received upon taking action ut, given by equation 13.

Please note that because of the market temperature in proportional to matrix Mt,

this equation is quadratic rather than linear in action ut.

The other thing that we can see from this equation is that rewards are risky,

as they depend on a noise term, epsilon t.

Therefore, we will add the certain risk penalty to compensate for this risk.

A risk penalty is simply a negative reward received for taking risk in trades.

We will use a very simple specification for such risk penalty, and

we'll take it to be proportional to the variance of the instantaneous reward

are zero, conditional on the given values of xt and

ut, as shown in equation 14 on this slide.

Here, lambda is a risk aversion parameter.

This variance is easily calculated, and this produces the last expression here.

We can see that this is a quadratic function of the sum xt plus ut.

The next element we have to include in the one-step reward is the penalty for

fees, or transaction costs.

The next element we have to include in the one-step reward is the penalty for

fees, or transaction costs.

Now, transaction costs depend on the sign of ut because traders buy for

a bid price, and sell for an ask price.

This can be taken into account by using different proportional transactional

costs for positive and negative values of ut.

And to handle this we represent each component uit,

as a difference of two non-negative values uit plus and uit minus.

Then ut becomes a difference of these two values,

while the absolute value of ut becomes their sum.

In other words, ut is equal to ut plus when it's positive,

and it's equal to ut minus when it's negative.

Now, with these definitions, we can specify,

Shown in equation 17 on this slide, where kappa plus and

kappa minus are different transaction cost parameters for buy and sell orders.

And finally, we have to incorporate market input effects from trading in stocks.

Here we will assume a proportional market input

that is proportional to the total position in all stocks.

It has a few terms.

Two terms are proportional to the size of the order, and

they can in general have different coefficients, theta plus and theta minus,

That would describe such inputs, for positive and negative values of u.

In addition, marketing can depend on signals, therefore,

we add a third term here, with a weight given by some metrics, feet.

Finally, we collect all this contributions together and

define the total one-step reward, their sum, as shown in equation 18.

Please note that this is a random reward,

as the first term here depends on the noise epsilon t.

The rest of the terms is non-random for given values of xt and ut.

Now, when we are given an action ut, which is equal to the difference of ut plus and

ut minus, we can define an expected one-step reward as an expectation

of the expression, given by equation 18.

This produces equation 19, for the expected reward R hat t.

Here our 0 hat is an expectation of the instantaneous reward R0 conditional

on a given state and action, as shown in equation 20.

We can now simplify this expression and bring it to a more compact form,

which will be shown next.

So here we use a vector notation to descend, and

introduce a vector AT of size two M, which is made

by stacking together components ut plus and ut minus.

Then the whole expression for the one-step reward can be expressed as a quadratic

functional of xt and at, which is shown in equation 21.

We have three second order terms here and two first order terms,

proportional to xt and at respectively.

There is no constant term here because

there is no reward without taking an action at, or having a position xt.

Please note that matrices Raat, Rxxt,

and Raxt depend on risk aversion lambda.

They also depend on market input metrics, M and efficients theta.

If all of them are equal zero, then the one-step reward becomes a linear

functional, instead of a quadratic functional.

Explore our Catalog

Join for free and get personalized recommendations, updates and offers.

Coursera provides universal access to the world’s best education, partnering with top universities and organizations to offer courses online.

© 2018 Coursera Inc. All rights reserved.

Download on the App Store

Get it on Google Play