1. Purpose

Overall project topics:

1. Multi-modality and integration of eye tracking for interactive IR.
2. Reinforcement learning for user interest/attention modeling and assistance in information seeking tasks.
3. Information presentation and visualization for intelligent inforamtion retrieval.

Specific goals in this study, for interactive IR:

1. Learn to track eye movement.
2. Learn to identify eye attention.
3. Learn to identify and following (changing) interests.
4. Learn to predict and provide relevant assistance to foster interactive retrieval.

2. Eye Movement and Attention

Signals from eye tracking device:

1. Gaze: position $(x,y)$ and duration $t$
2. Dilation

Signals from other input devices:

1. Mouse over $(x,y)$
2. Mouse click $(x,y)$
3. Positions of UI components and visualization

3. Reinforcement Learning

Given the reinforcement learning diagram below:

Now consider the first task to track and follow eye movement:

1. Agent is the eye-tracking model that attempts to follow eye movement and identify interests.
2. Environment refers to the entire dynamically changing setting where the eye moves and UI (visualization) changes.
3. State of the environment such as relative positions of the agent (tracker), eye, and mouse; and their current moving directions/speeds.
4. Reward or Penalty as a result of the agent's action. For example, while the agent consumes energy in each move (penalty), it gets rewarded by following the eye closely. In addition, it may receive a great reward for predicting a validated click after sufficient eye attention.

Q-Learning Policy

Through actions and interactions (state and reward) with the environment, it is possible for an agent to gain knowledge and improve its actions. We refer to as

The Policy: the method (strategy) an agent uses to act on its knowledge to maximize its long-term rewards.

Q-Learning is a reinforcement learning policy, in which the agent keeps track of rewards associated with each state and action.

Q-Table

Specifically, the agent stores rewards in a so-called Q-table like this:

	$A_1$	$A_2$	$A_3$	..	$A_m$
$S_1$	0	0	0		0
$S_2$	0	0	0		0
..	0	0	0		0
$S_n$	0	0	0		0

where:

• $S_1, S_2, .., S_n$ are all possible states (rows).
• $A_1, A_2, .., A_m$ are all possible actions (columns) for each state.

The Q-table is initiated with $0$ values and gets updated whenever the agent performs an action and receives a reward (or penalty) from the environment.

Suppose at time $t$, the agent is at the state of $s_t$, performs an action $a_t$ and receives a reward of $r_t$.

Here is one strategy to update the Q-table:

\begin{eqnarray} Q'(s_t, a_t) & \leftarrow & (1-\beta) Q(s_t, a_t) + \beta (r_t + \gamma \max_a{Q(s_{t+1},a)}) \end{eqnarray}

where:

• $\beta \in [0,1]$ is the learning rate. A greater $\beta$ leads to more aggressive updating.
• $\gamma \in [0,1]$ is the discount factor for future rewards. A greater $\gamma$ tends to put more emphasis on long-term rewards.

Exploration vs. Exploitation

An agent learns from trials and errors. So learning requires (random) exploration in the environment. At some points, the agent may gain "sufficient" knowledge to make wiser decisions based on a trained Q-table. However, in order to allow opportunities to continued learning, one should avoid complete exploitation of Q-table for action selection.

For the tradeoff, we introduce a third parameter $\epsilon$ (probability), at time $t$:

1. With $\epsilon$ probability, the agent selects a random action.
2. With $1 - \epsilon$ probability, the agent selects the action with the maximum reward for state $s_t$ in the Q-table: $a_t = \underset{a}{\operatorname{argmax}}{r(s_t,a)}$

4. Learning to Track and Interact

Problem

Major tasks of the tracker agent include:

1. Adjust its speed to move closer to the eye.
2. Stay in proximity with the eye and move as little (smooth) as possible.
3. Predict a click and get rewarded if validated.

States

The entire screen is divided into a grid of:

• $X$ blocks horizontally, e.g. $X=12$ of $100$-pixel blocks
• $Y$ blocks vertically, e.g. $Y=8$ of $100$-pixel blocks

States are combintations of the following:

1. Eye position in the grid $(e_x,e_y)$.
2. Latest eye move $(\Delta_x, \Delta_y)$, or latest moving average.
3. Tracker position in the grid $(x,y)$.
4. Tracker current speed and direction, $(v_x, v_y)$, where $v_x, v_y \in [-2, -1, 0, 1, 2]$.
5. Tracker average speed in the latest iterations $(\bar{v_x}, \bar{v_y})$

Actions

In each state, the cab can take any of the following actions:

1. Accelerate or deccelerate on the $X$ dimension, $a_x \in [-1, 0, +1]$.
2. Accelerate or deccelerate on the $Y$ dimension, $a_y \in [-1, 0, +1]$.
3. Click and select (assist), e.g. after tracker speed remains slow in the latest iterations.

Rewards (or Penalty)

There are related rewards and penalties for each action:

• $-1$ point penalty for each action (e.g. change of speed).
• $20$ points reward for staying in the same block with the eye.
• $40$ points reward for predicting a correct mouse click, e.g. triggered by low speed in the latest iterations.

References

• Wlfgang Ertel (2017). Introduction to Artificial Intelligence. Chapter 10 Reinforcement Learning. Springer.
• Miroslav Kubat (2017). An Introduction to Machine Learning. Chapter 17. Reinforcement learning. Springer.