# Processing Pipeline 
## Conda Environment Setup
```
conda env create -f conan_windows.yml
conda activate conan_windows_env
```
## Usage 
Run [ConAn_RunProcessing.ipynb](ConAn_RunProcessing.ipynb) to extract all frames from video and run processing models. 
### Body Movement
For body movement detection we selected [OpenPose](https://github.com/CMU-Perceptual-Computing-Lab/openpose). For our case, we used the 18-keypoint model, 
which takes the full frame as input and jointly predicts anatomical keypoints and a measurement 
for the degree of association between them.<br>
If you're using this processing step in your research please cite: 
```
@article{8765346,
  author = {Z. {Cao} and G. {Hidalgo Martinez} and T. {Simon} and S. {Wei} and Y. A. {Sheikh}},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  title = {OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields},
  year = {2019}
}
```
### Eye Gaze 
For eye gaze estimation we selected [RT-GENE](https://github.com/Tobias-Fischer/rt_gene). In addition to feeding each video frame to the model, 
we also input a version of the frame where the left side and the right side are wrapped together. 
This enables us to detect when a person moves over the edge of the video, as none of the models account for this. 
As this is a single frame estimation, we then track all subjects throughout the video using a minimal euclidean distance heuristic. <br>
<br>
If you're using this processing step in your research please cite: 
```
@inproceedings{FischerECCV2018,
author = {Tobias Fischer and Hyung Jin Chang and Yiannis Demiris},
title = {{RT-GENE: Real-Time Eye Gaze Estimation in Natural Environments}},
booktitle = {European Conference on Computer Vision},
year = {2018},
month = {September},
pages = {339--357}
}
```
Notes: 
- Before using [process_RTGene.py](process_RTGene.py) you need to run [install_RTGene.py](install_RTGene.py)!
- [OPTIONAL] You can provide a camera calibration file calib.pkl to improve detections. 
- You need to provide maximum number of people in the video for the sorting algorithm. 
### Facial Expression
Under construction 
### Speaking Activity 
Under construction 
### Object Tracking
We assume that you are most likely able to define your own study procedure, 
therefore we decided to simplify object tracking by employing the visual fiducial system [AprilTag 2](https://github.com/AprilRobotics/apriltag), 
where the tag positions are extracted with their tailored detector.

Note: For Windows we use [pupil_apriltags](https://github.com/pupil-labs/apriltags).