State-action value function (Q function)
A state-action value function is also called the Q function. It specifies how good it is for an agent to perform a particular action in a state with a policy π. The Q function is denoted by Q(s, a). It denotes the value of taking an action in a state following a policy π.
We can define Q function as follows:
![](https://epubservercos.yuewen.com/F4348E/19470379901496006/epubprivate/OEBPS/Images/36a6e230-3432-4477-810c-5dca66d8669d.png?sign=1734445552-JRNRwB7q4vZiG2ktlIpSoJUD1zPwn231-0-c7a8c1017471096659d5095d19f1f37b)
This specifies the expected return starting from state s with the action a according to policy π. We can substitute the value of Rt in the Q function from (2) as follows:
![](https://epubservercos.yuewen.com/F4348E/19470379901496006/epubprivate/OEBPS/Images/0424410a-24d5-4fc3-859d-f201929b41c3.png?sign=1734445552-MAd4B6dqkIVtggdmXlQZnMrWxB1OcY04-0-673d13f4cc0fd146af35038c775f2cbb)
The difference between the value function and the Q function is that the value function specifies the goodness of a state, while a Q function specifies the goodness of an action in a state.
Like state value functions, Q functions can be viewed in a table. It is also called a Q table. Let us say we have two states and two actions; our Q table looks like the following:
![](https://epubservercos.yuewen.com/F4348E/19470379901496006/epubprivate/OEBPS/Images/3.jpg?sign=1734445552-6ZAmOfu4zECIP9LDn0EeKYxCaeI8GeJm-0-7912fbb3e5f2cde29571ca091c4466b4)
Thus, the Q table shows the value of all possible state action pairs. So, by looking at this table, we can come to the conclusion that performing action 1 in state 1 and action 2 in state 2 is the better option as it has high value.
Whenever we say value function V(S) or Q function Q( S, a), it actually means the value table and Q table, as shown previously.