public-projects/gazesim
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity

migrated code to public repository

Browse source
This commit is contained in:
mohsen-mansouryar 2016-03-09 19:52:35 +01:00
parent a7df82d7a4
commit f34dc653e5
233 changed files with 16279 additions and 186 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

23
README.md
View file

@ -1 +1,22 @@
# etra2016_3dgaze
### 3D Gaze Estimation from 2D Pupil Positions on Monocular Head-Mounted Eye Trackers
**Mohsen Mansouryar, Julian Steil, Yusuke Sugano and Andreas Bulling**
published at ETRA 2016
code -> Contains main scripts. below you can see a list of commands and the results they produce:
- cmd: python parallax_analysis.py pts |-> result: plot of calibration and test points.
- cmd: python parallax_analysis.py 2d3d |-> result: plot of 2D-to-2D againt 3D-to-3D mapping over all number of calibration depths.
- cmd: python parallax_analysis.py 2d2d_2d3d |-> result: plot comparing parallax error over five different test depths for three calibration depths of 1.0m, 1.5m, and 2.0m between 2D-to-2D and 3D-to-3D mapping.
- cmd: python parallax_2D3D_3Cdepths.py |-> result: plot comparing average angular error of the two mapping techniques when 3 calibration depths are used together. (depths 1 to 5 correspond to test depths 1.0m to 2.0m)
code/pupil -> Modules directly used from PUPIL source code for baseline 2D-to-2D mapping and data stream correlation.
code/recording -> Scripts related to dataset recording and marker visualization and tracking. script dependencies are python 2's openCV and ArUco library. more information regarding each module is documented where required.
code/results -> Contains gaze estimation results for both 2D-to-2D and 2D-to-3D mapping approaches with multiple calibration depths on data from participants. data files in the root directory of each method correspond to single depth calibration results. data format is described inside README.txt inside each method directory. the results are also available via /BS/3D_Gaze_Tracking/work/results
code/Visualization -> Creation of figures for the paper
1CalibrationDepth.py -> 2D-to-2D vs. 2D-to-3D with one calibration depth
3CalibrationDepths.py -> 2D-to-2D vs. 2D-to-3D with three calibration depth
EffectDistanceDifference1CalibrationDepth.py -> Effect of different distances to the original calibration depth
EffectNumberofClusters.py -> Effect of the number of clusters

4
code/.gitignore vendored
View file

@ -493,3 +493,7 @@ FakesAssemblies/
**/*.Server/GeneratedArtifacts
**/*.Server/ModelManifest.xml
_Pvt_Extensions
# Custom ignore rules
*.npy

37
code/README.md
View file

@ -1 +1,38 @@
- Participants data is accessible under /BS/3D_Gaze_Tracking/archive00/participants/
- only the data for the following 14 participants is used in the study:
'p10', 'p16', 'p13', 'p24', 'p5', 'p14', 'p26', 'p12', 'p20', 'p7', 'p15', 'p11', 'p21', 'p25'
- Eye camera's intrinsic parameters are accessible under /BS/3D_Gaze_Tracking/archive00/eye_camera_images/
- Scene camera's intrinsic parameters are accessible under /BS/3D_Gaze_Tracking/archive00/scene_camera/
- geom.py contains the Pinhole camera model along with a couple of geometry related methods.
- minimize.py contains code on least square minimization using numpy.
- vector.py contains a fully python version of VPython's vector object used for vector processing.
- sim.py contains the main GazeSimulation object used for simulation and visualization. this object also
contains methods for handling real world data.
- parallax_analysis.py mainly uses GazeSimulation object to perform different experiments on simulation/real-
world data. (parallax_2D3D_3Cdepths.py is just a single experiment separated from this module)
- Scripts regarding real-world data processing including marker tracking, marker movement detection, frame
extraction, ... are inside recording package.
- recording/tracker.py uses a slightly modified version of aruco_test script to silently track markers in
the scene, log the output of ArUco, and compute 2D and 3D position of the center of marker given the ArUco output. (the modified aruco_test is included in recording package)
- recording/process_recordings.py contains the main code for detecting intervals in which marker is not
moving using sklearn's AgglomerativeClustering. the method performs well almost always but in a few cases manual annotation was required to get the correct output. output of this detection is depicted in marker_motion.png inside each recording's data files. a sample output for a test recording (16 points) is located under /BS/3D_Gaze_Tracking/work/marker_motion.png
- recording/util/check_pupil_positions.py is used during recording to ensure accurate pupil detection.
example usage:
python check_pupil_positions.py /BS/3D_Gaze_Tracking/archive00/participants/p16/2015_09_29/000/
- recording/util/SingleMarkerVisualizer/ contains the main Processing script for visualizing a moving marker
during the recording experiments. the code uses Processing version 3.

944
code/Visualization/1CalibrationDepth.py Normal file
View file

@ -0,0 +1,944 @@
import os, sys
import seaborn
from pylab import rcParams
import cv2
import numpy as np
from numpy import linalg as LA
from time import time
from itertools import combinations
import matplotlib.pyplot as plt
from matplotlib.pyplot import *
import matplotlib.patches as mpatches
# activate latex text rendering
#rc('text', usetex=True)
def main(argv):
# path = str(argv[0])
path ="/home/mmbrian/3D_Gaze_Tracking/work/results/2D2D/"
Data1 = np.load(path + "p5_2d2d_all.npy")
Data2 = np.load(path + "p7_2d2d_all.npy")
Data3 = np.load(path + "p10_2d2d_all.npy")
Data4 = np.load(path + "p11_2d2d_all.npy")
Data5 = np.load(path + "p12_2d2d_all.npy")
Data6 = np.load(path + "p13_2d2d_all.npy")
Data7 = np.load(path + "p14_2d2d_all.npy")
Data8 = np.load(path + "p15_2d2d_all.npy")
Data9 = np.load(path + "p16_2d2d_all.npy")
Data10 = np.load(path + "p20_2d2d_all.npy")
Data11 = np.load(path + "p21_2d2d_all.npy")
Data12 = np.load(path + "p24_2d2d_all.npy")
Data13 = np.load(path + "p25_2d2d_all.npy")
Data14 = np.load(path + "p26_2d2d_all.npy")
Data = [Data1,Data2,Data3,Data4,Data5,Data6,Data7,Data8,Data9,Data10,Data11,Data12,Data13,Data14]
Participantmean = []
for i in xrange(5):
Participantmean.append(float(0))
yerrup = []
for i in xrange(5):
yerrup.append(float(0))
yerrdown = []
for i in xrange(5):
yerrdown.append(float(0))
maxvalue = []
for i in xrange(5):
maxvalue.append(float(0))
minvalue = []
for i in xrange(5):
minvalue.append(float(0))
Activitymax = []
for i in xrange(5):
Activitymax.append([])
for j in xrange(15):
Activitymax[i].append(float(0))
Activitymin = []
for i in xrange(5):
Activitymin.append([])
for j in xrange(15):
Activitymin[i].append(float(0))
AngularerrorC1 = []
for i in xrange(5):
AngularerrorC1.append([])
for j in xrange(14):
AngularerrorC1[i].append(float(0))
AngularerrorC2 = []
for i in xrange(5):
AngularerrorC2.append([])
for j in xrange(14):
AngularerrorC2[i].append(float(0))
AngularerrorC3 = []
for i in xrange(5):
AngularerrorC3.append([])
for j in xrange(14):
AngularerrorC3[i].append(float(0))
AngularerrorC4 = []
for i in xrange(5):
AngularerrorC4.append([])
for j in xrange(14):
AngularerrorC4[i].append(float(0))
AngularerrorC5 = []
for i in xrange(5):
AngularerrorC5.append([])
for j in xrange(14):
AngularerrorC5[i].append(float(0))
# C1
distance = 1.0
i = 0
while i < 14:
j = 0
while j < 25:
if Data[i][j][1] == 1.0 and Data[i][j][0] == distance:
AngularerrorC1[0][i] = Data[i][j][7]
break
else:
j += 1
j = 0
while j < 25:
print "i: ", i," j: ", j
if Data[i][j][1] == 1.25 and Data[i][j][0] == distance:
print Data[i][j][7]
AngularerrorC1[1][i] = Data[i][j][7]
break
else:
j += 1
j = 0
while j < 25:
if Data[i][j][1] == 1.5 and Data[i][j][0] == distance:
AngularerrorC1[2][i] = Data[i][j][7]
j = 25
else:
j += 1
j = 0
while j < 25:
if Data[i][j][1] == 1.75 and Data[i][j][0] == distance:
AngularerrorC1[3][i] = Data[i][j][7]
j = 25
else:
j += 1
j = 0
while j < 25:
if Data[i][j][1] == 2.0 and Data[i][j][0] == distance:
AngularerrorC1[4][i] = Data[i][j][7]
j = 25
else:
j += 1
i += 1
print "AngularerrorC1: ", AngularerrorC1[0]
print "AngularerrorC1: ", AngularerrorC1[1]
print "AngularerrorC1: ", AngularerrorC1[2]
print "AngularerrorC1: ", AngularerrorC1[3]
print "AngularerrorC1: ", AngularerrorC1[4]
meanC1D1 = np.mean(AngularerrorC1[0])
meanC1D2 = np.mean(AngularerrorC1[1])
meanC1D3 = np.mean(AngularerrorC1[2])
meanC1D4 = np.mean(AngularerrorC1[3])
meanC1D5 = np.mean(AngularerrorC1[4])
stdC1D1 = np.std(AngularerrorC1[0])
stdC1D2 = np.std(AngularerrorC1[1])
stdC1D3 = np.std(AngularerrorC1[2])
stdC1D4 = np.std(AngularerrorC1[3])
stdC1D5 = np.std(AngularerrorC1[4])
meanC1 = [meanC1D1,meanC1D2,meanC1D3,meanC1D4,meanC1D5]
stdC1 = [stdC1D1,stdC1D2,stdC1D3,stdC1D4,stdC1D5]
# C2
distance = 1.25
i = 0
while i < 14:
j = 0
while j < 25:
if Data[i][j][1] == 1.0 and Data[i][j][0] == distance:
AngularerrorC2[0][i] = Data[i][j][7]
break
else:
j += 1
j = 0
while j < 25:
print "i: ", i," j: ", j
if Data[i][j][1] == 1.25 and Data[i][j][0] == distance:
print Data[i][j][7]
AngularerrorC2[1][i] = Data[i][j][7]
break
else:
j += 1
j = 0
while j < 25:
if Data[i][j][1] == 1.5 and Data[i][j][0] == distance:
AngularerrorC2[2][i] = Data[i][j][7]
j = 25
else:
j += 1
j = 0
while j < 25:
if Data[i][j][1] == 1.75 and Data[i][j][0] == distance:
AngularerrorC2[3][i] = Data[i][j][7]
j = 25
else:
j += 1
j = 0
while j < 25:
if Data[i][j][1] == 2.0 and Data[i][j][0] == distance:
AngularerrorC2[4][i] = Data[i][j][7]
j = 25
else:
j += 1
i += 1
print "AngularerrorC2: ", AngularerrorC2[0]
print "AngularerrorC2: ", AngularerrorC2[1]
print "AngularerrorC2: ", AngularerrorC2[2]
print "AngularerrorC2: ", AngularerrorC2[3]
print "AngularerrorC2: ", AngularerrorC2[4]
meanC2D1 = np.mean(AngularerrorC2[0])
meanC2D2 = np.mean(AngularerrorC2[1])
meanC2D3 = np.mean(AngularerrorC2[2])
meanC2D4 = np.mean(AngularerrorC2[3])
meanC2D5 = np.mean(AngularerrorC2[4])
stdC2D1 = np.std(AngularerrorC2[0])
stdC2D2 = np.std(AngularerrorC2[1])
stdC2D3 = np.std(AngularerrorC2[2])
stdC2D4 = np.std(AngularerrorC2[3])
stdC2D5 = np.std(AngularerrorC2[4])
meanC2 = [meanC2D1,meanC2D2,meanC2D3,meanC2D4,meanC2D5]
stdC2 = [stdC2D1,stdC2D2,stdC2D3,stdC2D4,stdC2D5]
# C3
distance = 1.5
i = 0
while i < 14:
j = 0
while j < 25:
if Data[i][j][1] == 1.0 and Data[i][j][0] == distance:
AngularerrorC3[0][i] = Data[i][j][7]
break
else:
j += 1
j = 0
while j < 25:
print "i: ", i," j: ", j
if Data[i][j][1] == 1.25 and Data[i][j][0] == distance:
print Data[i][j][7]
AngularerrorC3[1][i] = Data[i][j][7]
break
else:
j += 1
j = 0
while j < 25:
if Data[i][j][1] == 1.5 and Data[i][j][0] == distance:
AngularerrorC3[2][i] = Data[i][j][7]
j = 25
else:
j += 1
j = 0
while j < 25:
if Data[i][j][1] == 1.75 and Data[i][j][0] == distance:
AngularerrorC3[3][i] = Data[i][j][7]
j = 25
else:
j += 1
j = 0
while j < 25:
if Data[i][j][1] == 2.0 and Data[i][j][0] == distance:
AngularerrorC3[4][i] = Data[i][j][7]
j = 25
else:
j += 1
i += 1
print "AngularerrorC3: ", AngularerrorC3[0]
print "AngularerrorC3: ", AngularerrorC3[1]
print "AngularerrorC3: ", AngularerrorC3[2]
print "AngularerrorC3: ", AngularerrorC3[3]
print "AngularerrorC3: ", AngularerrorC3[4]
meanC3D1 = np.mean(AngularerrorC3[0])
meanC3D2 = np.mean(AngularerrorC3[1])
meanC3D3 = np.mean(AngularerrorC3[2])
meanC3D4 = np.mean(AngularerrorC3[3])
meanC3D5 = np.mean(AngularerrorC3[4])
stdC3D1 = np.std(AngularerrorC3[0])
stdC3D2 = np.std(AngularerrorC3[1])
stdC3D3 = np.std(AngularerrorC3[2])
stdC3D4 = np.std(AngularerrorC3[3])
stdC3D5 = np.std(AngularerrorC3[4])
meanC3 = [meanC3D1,meanC3D2,meanC3D3,meanC3D4,meanC3D5]
stdC3 = [stdC3D1,stdC3D2,stdC3D3,stdC3D4,stdC3D5]
# C4
distance = 1.75
i = 0
while i < 14:
j = 0
while j < 25:
if Data[i][j][1] == 1.0 and Data[i][j][0] == distance:
AngularerrorC4[0][i] = Data[i][j][7]
break
else:
j += 1
j = 0
while j < 25:
print "i: ", i," j: ", j
if Data[i][j][1] == 1.25 and Data[i][j][0] == distance:
print Data[i][j][7]
AngularerrorC4[1][i] = Data[i][j][7]
break
else:
j += 1
j = 0
while j < 25:
if Data[i][j][1] == 1.5 and Data[i][j][0] == distance:
AngularerrorC4[2][i] = Data[i][j][7]
j = 25
else:
j += 1
j = 0
while j < 25:
if Data[i][j][1] == 1.75 and Data[i][j][0] == distance:
AngularerrorC4[3][i] = Data[i][j][7]
j = 25
else:
j += 1
j = 0
while j < 25:
if Data[i][j][1] == 2.0 and Data[i][j][0] == distance:
AngularerrorC4[4][i] = Data[i][j][7]
j = 25
else:
j += 1
i += 1
print "AngularerrorC4: ", AngularerrorC4[0]
print "AngularerrorC4: ", AngularerrorC4[1]
print "AngularerrorC4: ", AngularerrorC4[2]
print "AngularerrorC4: ", AngularerrorC4[3]
print "AngularerrorC4: ", AngularerrorC4[4]
meanC4D1 = np.mean(AngularerrorC4[0])
meanC4D2 = np.mean(AngularerrorC4[1])
meanC4D3 = np.mean(AngularerrorC4[2])
meanC4D4 = np.mean(AngularerrorC4[3])
meanC4D5 = np.mean(AngularerrorC4[4])
stdC4D1 = np.std(AngularerrorC4[0])
stdC4D2 = np.std(AngularerrorC4[1])
stdC4D3 = np.std(AngularerrorC4[2])
stdC4D4 = np.std(AngularerrorC4[3])
stdC4D5 = np.std(AngularerrorC4[4])
meanC4 = [meanC4D1,meanC4D2,meanC4D3,meanC4D4,meanC4D5]
stdC4 = [stdC4D1,stdC4D2,stdC4D3,stdC4D4,stdC4D5]
# C5
distance = 2.0
i = 0
while i < 14:
j = 0
while j < 25:
if Data[i][j][1] == 1.0 and Data[i][j][0] == distance:
AngularerrorC5[0][i] = Data[i][j][7]
break
else:
j += 1
j = 0
while j < 25:
print "i: ", i," j: ", j
if Data[i][j][1] == 1.25 and Data[i][j][0] == distance:
print Data[i][j][7]
AngularerrorC5[1][i] = Data[i][j][7]
break
else:
j += 1
j = 0
while j < 25:
if Data[i][j][1] == 1.5 and Data[i][j][0] == distance:
AngularerrorC5[2][i] = Data[i][j][7]
j = 25
else:
j += 1
j = 0
while j < 25:
if Data[i][j][1] == 1.75 and Data[i][j][0] == distance:
AngularerrorC5[3][i] = Data[i][j][7]
j = 25
else:
j += 1
j = 0
while j < 25:
if Data[i][j][1] == 2.0 and Data[i][j][0] == distance:
AngularerrorC5[4][i] = Data[i][j][7]
j = 25
else:
j += 1
i += 1
print "AngularerrorC5: ", AngularerrorC5[0]
print "AngularerrorC5: ", AngularerrorC5[1]
print "AngularerrorC5: ", AngularerrorC5[2]
print "AngularerrorC5: ", AngularerrorC5[3]
print "AngularerrorC5: ", AngularerrorC5[4]
meanC5D1 = np.mean(AngularerrorC5[0])
meanC5D2 = np.mean(AngularerrorC5[1])
meanC5D3 = np.mean(AngularerrorC5[2])
meanC5D4 = np.mean(AngularerrorC5[3])
meanC5D5 = np.mean(AngularerrorC5[4])
stdC5D1 = np.std(AngularerrorC5[0])
stdC5D2 = np.std(AngularerrorC5[1])
stdC5D3 = np.std(AngularerrorC5[2])
stdC5D4 = np.std(AngularerrorC5[3])
stdC5D5 = np.std(AngularerrorC5[4])
meanC5 = [meanC5D1,meanC5D2,meanC5D3,meanC5D4,meanC5D5]
stdC5 = [stdC5D1,stdC5D2,stdC5D3,stdC5D4,stdC5D5]
#######################################################################################
path ="/home/mmbrian/3D_Gaze_Tracking/work/results/2D3D/"
Datatwo1 = np.load(path + "p5_2d3d_all.npy")
Datatwo2 = np.load(path + "p7_2d3d_all.npy")
Datatwo3 = np.load(path + "p10_2d3d_all.npy")
Datatwo4 = np.load(path + "p11_2d3d_all.npy")
Datatwo5 = np.load(path + "p12_2d3d_all.npy")
Datatwo6 = np.load(path + "p13_2d3d_all.npy")
Datatwo7 = np.load(path + "p14_2d3d_all.npy")
Datatwo8 = np.load(path + "p15_2d3d_all.npy")
Datatwo9 = np.load(path + "p16_2d3d_all.npy")
Datatwo10 = np.load(path + "p20_2d3d_all.npy")
Datatwo11 = np.load(path + "p21_2d3d_all.npy")
Datatwo12 = np.load(path + "p24_2d3d_all.npy")
Datatwo13 = np.load(path + "p25_2d3d_all.npy")
Datatwo14 = np.load(path + "p26_2d3d_all.npy")
Datatwo = [Datatwo1,Datatwo2,Datatwo3,Datatwo4,Datatwo5,Datatwo6,Datatwo7,Datatwo8,Datatwo9,Datatwo10,Datatwo11,Datatwo12,Datatwo13,Datatwo14]
Participantmean = []
for i in xrange(5):
Participantmean.append(float(0))
yerrup = []
for i in xrange(5):
yerrup.append(float(0))
yerrdown = []
for i in xrange(5):
yerrdown.append(float(0))
maxvalue = []
for i in xrange(5):
maxvalue.append(float(0))
minvalue = []
for i in xrange(5):
minvalue.append(float(0))
Activitymax = []
for i in xrange(5):
Activitymax.append([])
for j in xrange(15):
Activitymax[i].append(float(0))
Activitymin = []
for i in xrange(5):
Activitymin.append([])
for j in xrange(15):
Activitymin[i].append(float(0))
AngularerrortwoC1 = []
for i in xrange(5):
AngularerrortwoC1.append([])
for j in xrange(14):
AngularerrortwoC1[i].append(float(0))
AngularerrortwoC2 = []
for i in xrange(5):
AngularerrortwoC2.append([])
for j in xrange(14):
AngularerrortwoC2[i].append(float(0))
AngularerrortwoC3 = []
for i in xrange(5):
AngularerrortwoC3.append([])
for j in xrange(14):
AngularerrortwoC3[i].append(float(0))
AngularerrortwoC4 = []
for i in xrange(5):
AngularerrortwoC4.append([])
for j in xrange(14):
AngularerrortwoC4[i].append(float(0))
AngularerrortwoC5 = []
for i in xrange(5):
AngularerrortwoC5.append([])
for j in xrange(14):
AngularerrortwoC5[i].append(float(0))
# C1
distance = 1.0
i = 0
while i < 14:
j = 0
while j < 25:
if Datatwo[i][j][1] == 1.0 and Datatwo[i][j][0] == distance:
AngularerrortwoC1[0][i] = Datatwo[i][j][2]
break
else:
j += 1
j = 0
while j < 25:
print "i: ", i," j: ", j
if Datatwo[i][j][1] == 1.25 and Datatwo[i][j][0] == distance:
print Datatwo[i][j][7]
AngularerrortwoC1[1][i] = Datatwo[i][j][2]
break
else:
j += 1
j = 0
while j < 25:
if Datatwo[i][j][1] == 1.5 and Datatwo[i][j][0] == distance:
AngularerrortwoC1[2][i] = Datatwo[i][j][2]
j = 25
else:
j += 1
j = 0
while j < 25:
if Datatwo[i][j][1] == 1.75 and Datatwo[i][j][0] == distance:
AngularerrortwoC1[3][i] = Datatwo[i][j][2]
j = 25
else:
j += 1
j = 0
while j < 25:
if Datatwo[i][j][1] == 2.0 and Datatwo[i][j][0] == distance:
AngularerrortwoC1[4][i] = Datatwo[i][j][2]
j = 25
else:
j += 1
i += 1
print "AngularerrortwoC1: ", AngularerrortwoC1[0]
print "AngularerrortwoC1: ", AngularerrortwoC1[1]
print "AngularerrortwoC1: ", AngularerrortwoC1[2]
print "AngularerrortwoC1: ", AngularerrortwoC1[3]
print "AngularerrortwoC1: ", AngularerrortwoC1[4]
meantwoC1D1 = np.mean(AngularerrortwoC1[0])
meantwoC1D2 = np.mean(AngularerrortwoC1[1])
meantwoC1D3 = np.mean(AngularerrortwoC1[2])
meantwoC1D4 = np.mean(AngularerrortwoC1[3])
meantwoC1D5 = np.mean(AngularerrortwoC1[4])
stdtwoC1D1 = np.std(AngularerrortwoC1[0])
stdtwoC1D2 = np.std(AngularerrortwoC1[1])
stdtwoC1D3 = np.std(AngularerrortwoC1[2])
stdtwoC1D4 = np.std(AngularerrortwoC1[3])
stdtwoC1D5 = np.std(AngularerrortwoC1[4])
meantwoC1 = [meantwoC1D1,meantwoC1D2,meantwoC1D3,meantwoC1D4,meantwoC1D5]
stdtwoC1 = [stdtwoC1D1,stdtwoC1D2,stdtwoC1D3,stdtwoC1D4,stdtwoC1D5]
# C2
distance = 1.25
i = 0
while i < 14:
j = 0
while j < 25:
if Datatwo[i][j][1] == 1.0 and Datatwo[i][j][0] == distance:
AngularerrortwoC2[0][i] = Datatwo[i][j][2]
break
else:
j += 1
j = 0
while j < 25:
print "i: ", i," j: ", j
if Datatwo[i][j][1] == 1.25 and Datatwo[i][j][0] == distance:
print Datatwo[i][j][7]
AngularerrortwoC2[1][i] = Datatwo[i][j][2]
break
else:
j += 1
j = 0
while j < 25:
if Datatwo[i][j][1] == 1.5 and Datatwo[i][j][0] == distance:
AngularerrortwoC2[2][i] = Datatwo[i][j][2]
j = 25
else:
j += 1
j = 0
while j < 25:
if Datatwo[i][j][1] == 1.75 and Datatwo[i][j][0] == distance:
AngularerrortwoC2[3][i] = Datatwo[i][j][2]
j = 25
else:
j += 1
j = 0
while j < 25:
if Datatwo[i][j][1] == 2.0 and Datatwo[i][j][0] == distance:
AngularerrortwoC2[4][i] = Datatwo[i][j][2]
j = 25
else:
j += 1
i += 1
print "AngularerrortwoC2: ", AngularerrortwoC2[0]
print "AngularerrortwoC2: ", AngularerrortwoC2[1]
print "AngularerrortwoC2: ", AngularerrortwoC2[2]
print "AngularerrortwoC2: ", AngularerrortwoC2[3]
print "AngularerrortwoC2: ", AngularerrortwoC2[4]
meantwoC2D1 = np.mean(AngularerrortwoC2[0])
meantwoC2D2 = np.mean(AngularerrortwoC2[1])
meantwoC2D3 = np.mean(AngularerrortwoC2[2])
meantwoC2D4 = np.mean(AngularerrortwoC2[3])
meantwoC2D5 = np.mean(AngularerrortwoC2[4])
stdtwoC2D1 = np.std(AngularerrortwoC2[0])
stdtwoC2D2 = np.std(AngularerrortwoC2[1])
stdtwoC2D3 = np.std(AngularerrortwoC2[2])
stdtwoC2D4 = np.std(AngularerrortwoC2[3])
stdtwoC2D5 = np.std(AngularerrortwoC2[4])
meantwoC2 = [meantwoC2D1,meantwoC2D2,meantwoC2D3,meantwoC2D4,meantwoC2D5]
stdtwoC2 = [stdtwoC2D1,stdtwoC2D2,stdtwoC2D3,stdtwoC2D4,stdtwoC2D5]
# C3
distance = 1.5
i = 0
while i < 14:
j = 0
while j < 25:
if Datatwo[i][j][1] == 1.0 and Datatwo[i][j][0] == distance:
AngularerrortwoC3[0][i] = Datatwo[i][j][2]
break
else:
j += 1
j = 0
while j < 25:
print "i: ", i," j: ", j
if Datatwo[i][j][1] == 1.25 and Datatwo[i][j][0] == distance:
print Datatwo[i][j][7]
AngularerrortwoC3[1][i] = Datatwo[i][j][2]
break
else:
j += 1
j = 0
while j < 25:
if Datatwo[i][j][1] == 1.5 and Datatwo[i][j][0] == distance:
AngularerrortwoC3[2][i] = Datatwo[i][j][2]
j = 25
else:
j += 1
j = 0
while j < 25:
if Datatwo[i][j][1] == 1.75 and Datatwo[i][j][0] == distance:
AngularerrortwoC3[3][i] = Datatwo[i][j][2]
j = 25
else:
j += 1
j = 0
while j < 25:
if Datatwo[i][j][1] == 2.0 and Datatwo[i][j][0] == distance:
AngularerrortwoC3[4][i] = Datatwo[i][j][2]
j = 25
else:
j += 1
i += 1
print "AngularerrortwoC3: ", AngularerrortwoC3[0]
print "AngularerrortwoC3: ", AngularerrortwoC3[1]
print "AngularerrortwoC3: ", AngularerrortwoC3[2]
print "AngularerrortwoC3: ", AngularerrortwoC3[3]
print "AngularerrortwoC3: ", AngularerrortwoC3[4]
meantwoC3D1 = np.mean(AngularerrortwoC3[0])
meantwoC3D2 = np.mean(AngularerrortwoC3[1])
meantwoC3D3 = np.mean(AngularerrortwoC3[2])
meantwoC3D4 = np.mean(AngularerrortwoC3[3])
meantwoC3D5 = np.mean(AngularerrortwoC3[4])
stdtwoC3D1 = np.std(AngularerrortwoC3[0])
stdtwoC3D2 = np.std(AngularerrortwoC3[1])
stdtwoC3D3 = np.std(AngularerrortwoC3[2])
stdtwoC3D4 = np.std(AngularerrortwoC3[3])
stdtwoC3D5 = np.std(AngularerrortwoC3[4])
meantwoC3 = [meantwoC3D1,meantwoC3D2,meantwoC3D3,meantwoC3D4,meantwoC3D5]
stdtwoC3 = [stdtwoC3D1,stdtwoC3D2,stdtwoC3D3,stdtwoC3D4,stdtwoC3D5]
# C4
distance = 1.75
i = 0
while i < 14:
j = 0
while j < 25:
if Datatwo[i][j][1] == 1.0 and Datatwo[i][j][0] == distance:
AngularerrortwoC4[0][i] = Datatwo[i][j][2]
break
else:
j += 1
j = 0
while j < 25:
print "i: ", i," j: ", j
if Datatwo[i][j][1] == 1.25 and Datatwo[i][j][0] == distance:
print Datatwo[i][j][7]
AngularerrortwoC4[1][i] = Datatwo[i][j][2]
break
else:
j += 1
j = 0
while j < 25:
if Datatwo[i][j][1] == 1.5 and Datatwo[i][j][0] == distance:
AngularerrortwoC4[2][i] = Datatwo[i][j][2]
j = 25
else:
j += 1
j = 0
while j < 25:
if Datatwo[i][j][1] == 1.75 and Datatwo[i][j][0] == distance:
AngularerrortwoC4[3][i] = Datatwo[i][j][2]
j = 25
else:
j += 1
j = 0
while j < 25:
if Datatwo[i][j][1] == 2.0 and Datatwo[i][j][0] == distance:
AngularerrortwoC4[4][i] = Datatwo[i][j][2]
j = 25
else:
j += 1
i += 1
print "AngularerrortwoC4: ", AngularerrortwoC4[0]
print "AngularerrortwoC4: ", AngularerrortwoC4[1]
print "AngularerrortwoC4: ", AngularerrortwoC4[2]
print "AngularerrortwoC4: ", AngularerrortwoC4[3]
print "AngularerrortwoC4: ", AngularerrortwoC4[4]
meantwoC4D1 = np.mean(AngularerrortwoC4[0])
meantwoC4D2 = np.mean(AngularerrortwoC4[1])
meantwoC4D3 = np.mean(AngularerrortwoC4[2])
meantwoC4D4 = np.mean(AngularerrortwoC4[3])
meantwoC4D5 = np.mean(AngularerrortwoC4[4])
stdtwoC4D1 = np.std(AngularerrortwoC4[0])
stdtwoC4D2 = np.std(AngularerrortwoC4[1])
stdtwoC4D3 = np.std(AngularerrortwoC4[2])
stdtwoC4D4 = np.std(AngularerrortwoC4[3])
stdtwoC4D5 = np.std(AngularerrortwoC4[4])
meantwoC4 = [meantwoC4D1,meantwoC4D2,meantwoC4D3,meantwoC4D4,meantwoC4D5]
stdtwoC4 = [stdtwoC4D1,stdtwoC4D2,stdtwoC4D3,stdtwoC4D4,stdtwoC4D5]
# C5
distance = 2.0
i = 0
while i < 14:
j = 0
while j < 25:
if Datatwo[i][j][1] == 1.0 and Datatwo[i][j][0] == distance:
AngularerrortwoC5[0][i] = Datatwo[i][j][2]
break
else:
j += 1
j = 0
while j < 25:
print "i: ", i," j: ", j
if Datatwo[i][j][1] == 1.25 and Datatwo[i][j][0] == distance:
print Datatwo[i][j][7]
AngularerrortwoC5[1][i] = Datatwo[i][j][2]
break
else:
j += 1
j = 0
while j < 25:
if Datatwo[i][j][1] == 1.5 and Datatwo[i][j][0] == distance:
AngularerrortwoC5[2][i] = Datatwo[i][j][2]
j = 25
else:
j += 1
j = 0
while j < 25:
if Datatwo[i][j][1] == 1.75 and Datatwo[i][j][0] == distance:
AngularerrortwoC5[3][i] = Datatwo[i][j][2]
j = 25
else:
j += 1
j = 0
while j < 25:
if Datatwo[i][j][1] == 2.0 and Datatwo[i][j][0] == distance:
AngularerrortwoC5[4][i] = Datatwo[i][j][2]
j = 25
else:
j += 1
i += 1
print "AngularerrortwoC5: ", AngularerrortwoC5[0]
print "AngularerrortwoC5: ", AngularerrortwoC5[1]
print "AngularerrortwoC5: ", AngularerrortwoC5[2]
print "AngularerrortwoC5: ", AngularerrortwoC5[3]
print "AngularerrortwoC5: ", AngularerrortwoC5[4]
meantwoC5D1 = np.mean(AngularerrortwoC5[0])
meantwoC5D2 = np.mean(AngularerrortwoC5[1])
meantwoC5D3 = np.mean(AngularerrortwoC5[2])
meantwoC5D4 = np.mean(AngularerrortwoC5[3])
meantwoC5D5 = np.mean(AngularerrortwoC5[4])
stdtwoC5D1 = np.std(AngularerrortwoC5[0])
stdtwoC5D2 = np.std(AngularerrortwoC5[1])
stdtwoC5D3 = np.std(AngularerrortwoC5[2])
stdtwoC5D4 = np.std(AngularerrortwoC5[3])
stdtwoC5D5 = np.std(AngularerrortwoC5[4])
meantwoC5 = [meantwoC5D1,meantwoC5D2,meantwoC5D3,meantwoC5D4,meantwoC5D5]
stdtwoC5 = [stdtwoC5D1,stdtwoC5D2,stdtwoC5D3,stdtwoC5D4,stdtwoC5D5]
######################################################################################
N = 5
ind = np.asarray([0.25,1.25,2.25,3.25,4.25])
width = 0.5 # the width of the bars
x1 = [0.375,1.375,2.375,3.375,4.375]
x2 = [0.425,1.425,2.425,3.425,4.425]
x3 = [0.475,1.475,2.475,3.475,4.475]
x4 = [0.525,1.525,2.525,3.525,4.525]
x5 = [0.575,1.575,2.575,3.575,4.575]
x6 = [0.625,1.625,2.625,3.625,4.625]
fig = plt.figure(figsize=(14.0, 10.0))
ax = fig.add_subplot(111)
# rects1 = ax.bar(ind, Participantmean,width, color='r',edgecolor='black',)#, hatch='//')
rects1 = ax.errorbar(x1, meanC1,yerr=[stdC1,stdC1],fmt='o',color='blue',ecolor='blue',lw=3, capsize=5, capthick=2)
plt.plot(x1, meanC1, marker="o", linestyle='-',lw=3,color='blue',label = r'2D-to-2D Calibration Depth 1')
# rects2 =ax.errorbar(x2, meanC2,yerr=[stdC2,stdC2],fmt='o',color='red',ecolor='red',lw=3, capsize=5, capthick=2)
# plt.plot(x2, meanC2, marker="o", linestyle='-',lw=3,color='red')
rects3 = ax.errorbar(x3, meanC3,yerr=[stdC3,stdC3],fmt='o',color='orange',ecolor='orange',lw=3, capsize=5, capthick=2)
plt.plot(x3, meanC3, marker="o", linestyle='-',lw=3,color='orange',label = r'2D-to-2D Calibration Depth 3')
#
# rects4 =ax.errorbar(x4, meanC4,yerr=[stdC4,stdC4],fmt='o',color='green',ecolor='green',lw=3, capsize=5, capthick=2)
# plt.plot(x4, meanC4, marker="o", linestyle='-',lw=3,color='green')
rects5 =ax.errorbar(x5, meanC5,yerr=[stdC5,stdC5],fmt='o',color='red',ecolor='red',lw=3, capsize=5, capthick=2)
plt.plot(x5, meanC5, marker="o", linestyle='-',lw=3,color='red',label = r'2D-to-2D Calibration Depth 5')
# rects1 = ax.bar(ind, Participantmean,width, color='r',edgecolor='black',)#, hatch='//')
rects2 = ax.errorbar(x2, meantwoC1,yerr=[stdtwoC1,stdtwoC1],fmt='o',color='blue',ecolor='blue',lw=3, capsize=5, capthick=2)
plt.plot(x2, meantwoC1, marker="o", linestyle='--',lw=3,color='blue',label = r'2D-to-3D Calibration Depth 1')
# rects2 =ax.errorbar(x2, meanC2,yerr=[stdC2,stdC2],fmt='o',color='red',ecolor='red',lw=3, capsize=5, capthick=2)
# plt.plot(x2, meanC2, marker="o", linestyle='-',lw=3,color='red')
rects4 = ax.errorbar(x4, meantwoC3,yerr=[stdtwoC3,stdtwoC3],fmt='o',color='orange',ecolor='orange',lw=3, capsize=5, capthick=2)
plt.plot(x4, meantwoC3, marker="o", linestyle='--',lw=3,color='orange',label = r'2D-to-3D Calibration Depth 3')
#
# rects4 =ax.errorbar(x4, meanC4,yerr=[stdC4,stdC4],fmt='o',color='green',ecolor='green',lw=3, capsize=5, capthick=2)
# plt.plot(x4, meanC4, marker="o", linestyle='-',lw=3,color='green')
rects6 =ax.errorbar(x6, meantwoC5,yerr=[stdtwoC5,stdtwoC5],fmt='o',color='red',ecolor='red',lw=3, capsize=5, capthick=2)
plt.plot(x6, meantwoC5, marker="o", linestyle='--',lw=3,color='red',label = r'2D-to-3D Calibration Depth 5')
ax.set_ylabel('Angular Error',fontsize=22)
ax.set_xlabel('Depth',fontsize=22)
ax.set_xticks(ind+0.25)
ax.set_xticklabels( ('D1', 'D2', 'D3','D4', 'D5') ,fontsize=16)
TOPICs = [0.0,0.5,1.5,2.5,3.5,4.5,5.0]#,110]#,120]
print TOPICs
LABELs = ["",r'D1 - 1m',r'D2 - 1.25m', r'D3 - 1.5m', r'D4 - 1.75m', r'D5 - 2.0m', ""]#, ""]#, ""]
# fig.canvas.set_window_title('Distance Error Correlation')
plt.xticks(TOPICs, LABELs,fontsize=18)
# legend([rects1,rects2,rects3,rects4,rects5], [r'\LARGE\textbf{Calibration Distance 1}', r'\LARGE\textbf{Calibration Distance 2}',r'\LARGE\textbf{Calibration Distance 3}', r'\LARGE\textbf{Calibration Distance 4}',r'\LARGE\textbf{Calibration Distance 5}'], loc='lower right')
legend(fontsize=20,loc='best')
TOPICS = [-4.0,-2.0, 0.0,2.0,4.0,6.0,8.0,10.0,12,14,16,18,20,22,24]#,110]#,120]
print TOPICS
LABELS = [r'', r'',r'0',r'2', r'4', r'6', r'8', r'10', r'12', r'14', r'16', r'18', r'20', r'22', r'24']#, ""]#, ""]
# fig.canvas.set_window_title('Accuracy - Activity Statistics')
plt.yticks(TOPICS, LABELS,fontsize=18)
def autolabel(rects):
# attach some text labels
for rect in rects:
height = rect.get_height()
ax.text(0.26+rect.get_x()+rect.get_width()/2., height +0.35, "%.2f"%float(height),
ha='center', va='bottom',fontweight='bold',fontsize=13.5)
# autolabel(rects1)
left = 0.1 # the left side of the subplots of the figure
right = 0.975 # the right side of the subplots of the figure
bottom = 0.075 # the bottom of the subplots of the figure
top = 0.925 # the top of the subplots of the figure
wspace = 0.2 # the amount of width reserved for blank space between subplots
hspace = 0.4 # the amount of height reserved for white space between subplots
plt.subplots_adjust(left=left, bottom=bottom, right=right, top=top, wspace=wspace, hspace=hspace)
plt.show()
means = [meanC1, meanC2, meanC3, meanC4, meanC5]
print meanC1
print meanC2
print meanC3
print meanC4
print meanC5
fixationinfos_list_path = "MeansC1D2D2.npy"
fixationinfos_list_csv_path = "MeansC1D2D2.csv"
np.save(fixationinfos_list_path,np.asarray(means))
np.savetxt(fixationinfos_list_csv_path,np.asarray(means), delimiter=",", fmt="%f")
## fixationinfos_list_path = "Activitymin_"+str(activity)+".npy"
## fixationinfos_list_csv_path = "Activitymin_"+str(activity)+".csv"
## np.save(fixationinfos_list_path,np.asarray(Activitymin))
## np.savetxt(fixationinfos_list_csv_path,np.asarray(Activitymin), delimiter=",", fmt="%s")
if __name__ == "__main__":
main(sys.argv[1:])

1980
code/Visualization/3CalibrationDepths.py Normal file
View file

File diff suppressed because it is too large Load diff

200
code/Visualization/EffectDistanceDifference1CalibrationDepth.py Normal file
View file

@ -0,0 +1,200 @@
import os, sys
import seaborn
from pylab import rcParams
import cv2
import numpy as np
from numpy import linalg as LA
from time import time
from itertools import combinations
import matplotlib.pyplot as plt
from matplotlib.pyplot import *
import matplotlib.patches as mpatches
# activate latex text rendering
#rc('text', usetex=True)
def main(argv):
# path = str(argv[0])
# path ="/home/Julian/3D_Pupil_Project/work/results/2D2D/"
# Data1 = np.load(path + "p5_2d2d_all.npy")
# Data2 = np.load(path + "p7_2d2d_all.npy")
# Data3 = np.load(path + "p10_2d2d_all.npy")
# Data4 = np.load(path + "p11_2d2d_all.npy")
# Data5 = np.load(path + "p12_2d2d_all.npy")
# Data6 = np.load(path + "p13_2d2d_all.npy")
# Data7 = np.load(path + "p14_2d2d_all.npy")
# Data8 = np.load(path + "p15_2d2d_all.npy")
# Data9 = np.load(path + "p16_2d2d_all.npy")
# Data10 = np.load(path + "p20_2d2d_all.npy")
# Data11 = np.load(path + "p21_2d2d_all.npy")
# Data12 = np.load(path + "p24_2d2d_all.npy")
# Data13 = np.load(path + "p25_2d2d_all.npy")
D2D2 = np.load("MeansC1D2D2.npy")
D2D3 = np.load("MeansC1D2D3.npy")
mean02D2D = np.mean([D2D2[0][0],D2D2[1][1], D2D2[2][2], D2D2[3][3], D2D2[4][4]])
mean12D2D = np.mean([D2D2[0][1],D2D2[1][2], D2D2[2][3], D2D2[3][4]])
mean22D2D = np.mean([D2D2[0][2],D2D2[1][3], D2D2[2][4]])
mean32D2D = np.mean([D2D2[0][3],D2D2[1][4]])
mean42D2D = np.mean([D2D2[0][4]])
# mean2D2D = [mean02D2D,mean12D2D,mean22D2D,mean32D2D,mean42D2D]
minmean02D2D = np.mean([D2D2[1][0],D2D2[2][1], D2D2[3][2], D2D2[4][3]])
minmean12D2D = np.mean([D2D2[2][0],D2D2[3][1], D2D2[4][2]])
minmean22D2D = np.mean([D2D2[3][0],D2D2[4][1]])
minmean32D2D = np.mean([D2D2[4][0]])
# minmean2D2D = [minmean02D2D,minmean12D2D,minmean22D2D,minmean32D2D]
std02D2D = np.std([D2D2[0][0],D2D2[1][1], D2D2[2][2], D2D2[3][3], D2D2[4][4]])
std12D2D = np.std([D2D2[0][1],D2D2[1][2], D2D2[2][3], D2D2[3][4]])
std22D2D = np.std([D2D2[0][2],D2D2[1][3], D2D2[2][4]])
std32D2D = np.std([D2D2[0][3],D2D2[1][4]])
std42D2D = np.std([D2D2[0][4]])
# std2D2D = [std02D2D,std12D2D,std22D2D,std32D2D,std42D2D]
minstd02D2D = np.std([D2D2[1][0],D2D2[2][1], D2D2[3][2], D2D2[4][3]])
minstd12D2D = np.std([D2D2[2][0],D2D2[3][1], D2D2[4][2]])
minstd22D2D = np.std([D2D2[3][0],D2D2[4][1]])
minstd32D2D = np.std([D2D2[4][0]])
# minstd2D2D = [minstd02D2D,minstd12D2D,minstd22D2D,minstd32D2D]
mean2D2D = [minmean32D2D,minmean22D2D,minmean12D2D,minmean02D2D,mean02D2D,mean12D2D,mean22D2D,mean32D2D,mean42D2D]
std2D2D = [minstd32D2D,minstd22D2D,minstd12D2D,minstd02D2D,std02D2D,std12D2D,std22D2D,std32D2D,std42D2D]
mean02D3D = np.mean([D2D3[0][0],D2D3[1][1], D2D3[2][2], D2D3[3][3], D2D3[4][4]])
mean12D3D = np.mean([D2D3[0][1],D2D3[1][2], D2D3[2][3], D2D3[3][4]])
mean22D3D = np.mean([D2D3[0][2],D2D3[1][3], D2D3[2][4]])
mean32D3D = np.mean([D2D3[0][3],D2D3[1][4]])
mean42D3D = np.mean([D2D3[0][4]])
# mean2D3D = [mean02D3D,mean12D3D,mean22D3D,mean32D3D,mean42D3D]
std02D3D = np.std([D2D3[0][0],D2D3[1][1], D2D3[2][2], D2D3[3][3], D2D3[4][4]])
std12D3D = np.std([D2D3[0][1],D2D3[1][2], D2D3[2][3], D2D3[3][4]])
std22D3D = np.std([D2D3[0][2],D2D3[1][3], D2D3[2][4]])
std32D3D = np.std([D2D3[0][3],D2D3[1][4]])
std42D3D = np.std([D2D3[0][4]])
# std2D3D = [std02D3D,std12D3D,std22D3D,std32D3D,std42D3D]
minmean02D3D = np.mean([D2D3[1][0],D2D3[2][1], D2D3[3][2], D2D3[4][3]])
minmean12D3D = np.mean([D2D3[2][0],D2D3[3][1], D2D3[4][2]])
minmean22D3D = np.mean([D2D3[3][0],D2D3[4][1]])
minmean32D3D = np.mean([D2D3[4][0]])
# minmean2D3D = [minmean02D3D,minmean12D3D,minmean22D3D,minmean32D3D]
minstd02D3D = np.std([D2D3[1][0],D2D3[2][1], D2D3[3][2], D2D3[4][3]])
minstd12D3D = np.std([D2D3[2][0],D2D3[3][1], D2D3[4][2]])
minstd22D3D = np.std([D2D3[3][0],D2D3[4][1]])
minstd32D3D = np.std([D2D3[4][0]])
# minstd2D3D = [minstd02D3D,minstd12D3D,minstd22D3D,minstd32D3D]
mean2D3D = [minmean32D3D,minmean22D3D,minmean12D3D,minmean02D3D,mean02D3D,mean12D3D,mean22D3D,mean32D3D,mean42D3D]
std2D3D = [minstd32D3D,minstd22D3D,minstd12D3D,minstd02D3D,std02D3D,std12D3D,std22D3D,std32D3D,std42D3D]
N = 5
ind = np.asarray([0.25,1.25,2.25,3.25,4.25])
width = 0.5 # the width of the bars
# x1 = [0.4,1.4,2.4,3.4,4.4]
x2 = [-3.55,-2.55,-1.55,-0.55,0.45,1.45,2.45,3.45,4.45]
# x3 = [0.5,1.5,2.5,3.5,4.5]
x4 = [-3.45,-2.45,-1.45,-0.45,0.55,1.55,2.55,3.55,4.55]
# x5 = [0.6,1.6,2.6,3.6,4.6]
fig = plt.figure(figsize=(14.0, 10.0))
ax = fig.add_subplot(111)
# print mean2D2D
# print mean2D3D
# ax.axhline(linewidth=2, y = np.mean(mean2D2D),color='r')
# ax.axhline(linewidth=2, y = np.mean(mean2D3D),color='blue')
# ax.axhline(linewidth=2, y = minvaluevalue,color='black')
# ax.text(0.98, Participantmeanvalue+0.5, "Mean %.2f" % Participantmeanvalue,fontsize=12, fontweight='bold',color='r')
# ax.text(0.98, maxvaluevalue+0.5, "Maximum %.2f" % maxvaluevalue,fontsize=12, fontweight='bold',color='black')
# ax.text(0.98, minvaluevalue+0.5, "Minimum %.2f" % minvaluevalue,fontsize=12, fontweight='bold', color='black')
# rects1 = ax.bar(ind, Participantmean,width, color='r',edgecolor='black',)#, hatch='//')
rects1 = ax.errorbar(x2, mean2D2D,yerr=[std2D2D,std2D2D],fmt='o',color='red',ecolor='red',lw=3, capsize=5, capthick=2)
plt.plot(x2, mean2D2D, marker="o", linestyle='-',lw=3,color='red',label = r'2D-to-2D')
rects2 =ax.errorbar(x4, mean2D3D,yerr=[std2D3D,std2D3D],fmt='o',color='blue',ecolor='blue',lw=3, capsize=5, capthick=2)
plt.plot(x4, mean2D3D, marker="o", linestyle='-',lw=3,color='blue', label = r'2D-to-3D')
legend(fontsize=20,loc='upper right')
# rects3 = ax.errorbar(x3, meanC3,yerr=[stdC3,stdC3],fmt='o',color='black',ecolor='black',lw=3, capsize=5, capthick=2)
# plt.plot(x3, meanC3, marker="o", linestyle='-',lw=3,color='black')
#
# rects4 =ax.errorbar(x4, meanC4,yerr=[stdC4,stdC4],fmt='o',color='green',ecolor='green',lw=3, capsize=5, capthick=2)
# plt.plot(x4, meanC4, marker="o", linestyle='-',lw=3,color='green')
#
# rects5 =ax.errorbar(x5, meanC5,yerr=[stdC5,stdC5],fmt='o',color='orange',ecolor='orange',lw=3, capsize=5, capthick=2)
# plt.plot(x5, meanC5, marker="o", linestyle='-',lw=3,color='orange')
ax.set_ylabel(r'Angular Error',fontsize=22)
ax.set_xlabel(r'Distance Difference',fontsize=22)
ax.set_xticks(ind+0.25)
ax.set_xticklabels( ('D1', 'D2', 'D3','D4', 'D5') ,fontsize=18)
TOPICs = [-3.5,-2.5,-1.5,-0.5,0.5,1.5,2.5,3.5,4.5]#,110]#,120]
print TOPICs
LABELs = [r'-1m',r'-0.75m', r'-0.5m', r'-0.25m', r'0m',r'0.25m', r'0.5m', r'0.75m', r'1.0m', ""]#, ""]#, ""]
# fig.canvas.set_window_title('Distance Error Correlation')
plt.xticks(TOPICs, LABELs,fontsize=18)
# legend([rects1,rects2], [r'\LARGE\textbf{2D2D}', r'\LARGE\textbf{2D3D}'], loc='lower right')
TOPICS = [0.5,1.0,1.5,2.0,2.5,3.0,3.5,4.0,4.5,5.0,5.5,6.0,6.5,7.0,7.5,8.0,8.5]#,110]#,120]
print TOPICS
LABELS = [r'0.5',r'1', r'1.5', r'2', r'2.5', r'3', r'3.5', r'4', r'4.5', r'5', r'5.5', r'6', r'6.5', r'7', r'7.5', r'8', r'8.5']#, ""]#, ""]
# fig.canvas.set_window_title('Accuracy - Activity Statistics')
plt.yticks(TOPICS, LABELS,fontsize=18)
def autolabel(rects):
# attach some text labels
for rect in rects:
height = rect.get_height()
ax.text(0.26+rect.get_x()+rect.get_width()/2., height +0.35, "%.2f"%float(height),
ha='center', va='bottom',fontweight='bold',fontsize=13.5)
# autolabel(rects1)
left = 0.1 # the left side of the subplots of the figure
right = 0.975 # the right side of the subplots of the figure
bottom = 0.075 # the bottom of the subplots of the figure
top = 0.925 # the top of the subplots of the figure
wspace = 0.2 # the amount of width reserved for blank space between subplots
hspace = 0.4 # the amount of height reserved for white space between subplots
plt.subplots_adjust(left=left, bottom=bottom, right=right, top=top, wspace=wspace, hspace=hspace)
plt.show()
if __name__ == "__main__":
main(sys.argv[1:])

350
code/Visualization/EffectNumberofClusters.py Normal file
View file

@ -0,0 +1,350 @@
import os, sys
import seaborn
from pylab import rcParams
import cv2
import numpy as np
from numpy import linalg as LA
from time import time
from itertools import combinations
import matplotlib.pyplot as plt
from matplotlib.pyplot import *
import matplotlib.patches as mpatches
# activate latex text rendering
#rc('text', usetex=True)
def main(argv):
C12D2D = np.load("MeansC1D2D2.npy")
C12D3D = np.load("MeansC1D2D3.npy")
C22D2D = np.load("MeansC2D2D2.npy")
C22D3D = np.load("MeansC2D2D3.npy")
C32D2D = np.load("MeansC3D2D2.npy")
C32D3D = np.load("MeansC3D2D3.npy")
C42D2D = np.load("MeansC4D2D2.npy")
C42D3D = np.load("MeansC4D2D3.npy")
C52D2D = np.load("MeansC5D2D2.npy")
C52D3D = np.load("MeansC5D2D3.npy")
summeC12D2D = []
summeC22D2D = []
summeC32D2D = []
summeC42D2D = []
summeC52D2D = []
summeC12D3D = []
summeC22D3D = []
summeC32D3D = []
summeC42D3D = []
summeC52D3D = []
i = 0
while i < len(C12D2D):
j = 0
while j < len(C12D2D[0]):
summeC12D2D.append(C12D2D[i][j])
j += 1
i += 1
i = 0
while i < len(C22D2D):
j = 0
while j < len(C22D2D[0]):
summeC22D2D.append(C22D2D[i][j])
j += 1
i += 1
i = 0
while i < len(C32D2D):
j = 0
while j < len(C32D2D[0]):
summeC32D2D.append(C32D2D[i][j])
j += 1
i += 1
i = 0
while i < len(C42D2D):
j = 0
while j < len(C42D2D[0]):
summeC42D2D.append(C42D2D[i][j])
j += 1
i += 1
i = 0
while i < len(C52D2D):
j = 0
while j < len(C52D2D[0]):
summeC52D2D.append(C52D2D[i][j])
j += 1
i += 1
i = 0
while i < len(C12D3D):
j = 0
while j < len(C12D3D[0]):
summeC12D3D.append(C12D3D[i][j])
j += 1
i += 1
i = 0
while i < len(C22D3D):
j = 0
while j < len(C22D3D[0]):
summeC22D3D.append(C22D3D[i][j])
j += 1
i += 1
i = 0
while i < len(C32D3D):
j = 0
while j < len(C32D3D[0]):
summeC32D3D.append(C32D3D[i][j])
j += 1
i += 1
i = 0
while i < len(C42D3D):
j = 0
while j < len(C42D3D[0]):
summeC42D3D.append(C42D3D[i][j])
j += 1
i += 1
i = 0
while i < len(C52D3D):
j = 0
while j < len(C52D3D[0]):
summeC52D3D.append(C52D3D[i][j])
j += 1
i += 1
mean1 = np.mean(summeC12D2D)
mean2 = np.mean(summeC22D2D)
mean3 = np.mean(summeC32D2D)
mean4 = np.mean(summeC42D2D)
mean5 = np.mean(summeC52D2D)
mean6 = np.mean(summeC12D3D)
mean7 = np.mean(summeC22D3D)
mean8 = np.mean(summeC32D3D)
mean9 = np.mean(summeC42D3D)
mean10 = np.mean(summeC52D3D)
std1 = np.std(summeC12D2D)
std2 = np.std(summeC22D2D)
std3 = np.std(summeC32D2D)
std4 = np.std(summeC42D2D)
std5 = np.std(summeC52D2D)
std6 = np.std(summeC12D3D)
std7 = np.std(summeC22D3D)
std8 = np.std(summeC32D3D)
std9 = np.std(summeC42D3D)
std10 = np.std(summeC52D3D)
# i = 0
# while i < len(C12D2D):
# j = 0
# while j < len(C12D2D[0]):
# summeC12D2D.append(C12D2D[i][j])
# j += 1
# i += 1
# print summeC12D2D
# i = 0
# minimum2 = 100
# while i < len(C22D2D):
# print np.mean(C22D2D[i])
# if np.mean(C22D2D[i]) < minimum2:
# minimum2 = np.mean(C22D2D[i])
# i += 1
#
# i = 0
# minimum3 = 100
# while i < len(C32D2D):
# print np.mean(C32D2D[i])
# if np.mean(C32D2D[i]) < minimum3:
# minimum3 = np.mean(C32D2D[i])
# i += 1
#
# i = 0
# minimum4 = 100
# while i < len(C42D2D):
# print np.mean(C42D2D[i])
# if np.mean(C42D2D[i]) < minimum4:
# minimum4 = np.mean(C42D2D[i])
# i += 1
#
# i = 0
# minimum5 = 100
# while i < len(C52D2D):
# print np.mean(C52D2D[i])
# if np.mean(C52D2D[i]) < minimum5:
# minimum5 = np.mean(C52D2D[i])
# i += 1
#
# i = 0
# minimum6 = 100
# while i < len(C12D3D):
# print np.mean(C12D3D[i])
# if np.mean(C12D3D[i]) < minimum6:
# minimum6 = np.mean(C12D3D[i])
# i += 1
#
# i = 0
# minimum7 = 100
# while i < len(C22D3D):
# print np.mean(C22D3D[i])
# if np.mean(C22D3D[i]) < minimum7:
# minimum7 = np.mean(C22D3D[i])
# i += 1
#
# i = 0
# minimum8 = 100
# while i < len(C32D3D):
# print np.mean(C32D3D[i])
# if np.mean(C32D3D[i]) < minimum8:
# minimum8 = np.mean(C32D3D[i])
# i += 1
#
# i = 0
# minimum9 = 100
# while i < len(C42D3D):
# print np.mean(C42D3D[i])
# if np.mean(C42D3D[i]) < minimum9:
# minimum9 = np.mean(C42D3D[i])
# i += 1
# i = 0
# minimum10 = 100
# while i < len(C52D3D):
# print np.mean(C52D3D[i])
# if np.mean(C52D3D[i]) < minimum10:
# minimum10 = np.mean(C52D3D[i])
# i += 1
#
mean2D2D = [mean1,mean2,mean3,mean4,mean5]
mean2D3D = [mean6,mean7,mean8,mean9,mean10]
std2D2D = [std1,std2,std3,std4,std5]
std2D3D = [std6,std7,std8,std9,std10]
# print minimum1
# i = 0
# minimum2 = 100
# while i < 5:
# if np.mean(C12D2D[i]) < minimum:
# minimum1 = np.mean(C12D2D[i])
# i += 1
N = 5
ind = np.asarray([0.25,1.25,2.25,3.25,4.25])
width = 0.5 # the width of the bars
# x1 = [0.4,1.4,2.4,3.4,4.4]
x2 = [0.45,1.45,2.45,3.45,4.45]
# x3 = [0.5,1.5,2.5,3.5,4.5]
x4 = [0.55,1.55,2.55,3.55,4.55]
# x5 = [0.6,1.6,2.6,3.6,4.6]
fig = plt.figure(figsize=(14.0, 10.0))
ax = fig.add_subplot(111)
# print mean2D2D
# print mean2D3D
# ax.axhline(linewidth=2, y = np.mean(mean2D2D),color='r')
# ax.axhline(linewidth=2, y = np.mean(mean2D3D),color='blue')
# ax.axhline(linewidth=2, y = minvaluevalue,color='black')
# ax.text(0.98, Participantmeanvalue+0.5, "Mean %.2f" % Participantmeanvalue,fontsize=12, fontweight='bold',color='r')
# ax.text(0.98, maxvaluevalue+0.5, "Maximum %.2f" % maxvaluevalue,fontsize=12, fontweight='bold',color='black')
# ax.text(0.98, minvaluevalue+0.5, "Minimum %.2f" % minvaluevalue,fontsize=12, fontweight='bold', color='black')
# rects1 = ax.bar(ind, Participantmean,width, color='r',edgecolor='black',)#, hatch='//')
rects1 = ax.errorbar(x2, mean2D2D,yerr=[std2D2D,std2D2D],fmt='o',color='red',ecolor='red',lw=3, capsize=5, capthick=2)
plt.plot(x2, mean2D2D, marker="o", linestyle='-',lw=3,color='red',label = r'2D-to-2D')
rects2 =ax.errorbar(x4, mean2D3D,yerr=[std2D3D,std2D3D],fmt='o',color='blue',ecolor='blue',lw=3, capsize=5, capthick=2)
plt.plot(x4, mean2D3D, marker="o", linestyle='-',lw=3,color='blue', label = r'2D-to-3D')
legend(fontsize=20,loc='upper right')
# rects3 = ax.errorbar(x3, meanC3,yerr=[stdC3,stdC3],fmt='o',color='black',ecolor='black',lw=3, capsize=5, capthick=2)
# plt.plot(x3, meanC3, marker="o", linestyle='-',lw=3,color='black')
#
# rects4 =ax.errorbar(x4, meanC4,yerr=[stdC4,stdC4],fmt='o',color='green',ecolor='green',lw=3, capsize=5, capthick=2)
# plt.plot(x4, meanC4, marker="o", linestyle='-',lw=3,color='green')
#
# rects5 =ax.errorbar(x5, meanC5,yerr=[stdC5,stdC5],fmt='o',color='orange',ecolor='orange',lw=3, capsize=5, capthick=2)
# plt.plot(x5, meanC5, marker="o", linestyle='-',lw=3,color='orange')
ax.set_ylabel(r'Angular Error',fontsize=22)
ax.set_xlabel(r'Number of Calibration Depths',fontsize=22)
ax.set_xticks(ind+0.25)
ax.set_xticklabels( ('D1', 'D2', 'D3','D4', 'D5') ,fontsize=18)
TOPICs = [0.0,0.5,1.5,2.5,3.5,4.5,5.0]#,110]#,120]
print TOPICs
LABELs = ["",r'1',r'2', r'3', r'4', r'5', ""]#, ""]#, ""]
# fig.canvas.set_window_title('Distance Error Correlation')
plt.xticks(TOPICs, LABELs,fontsize=18)
# legend([rects1,rects2], [r'\LARGE\textbf{2D2D}', r'\LARGE\textbf{2D3D}'], loc='lower right')
TOPICS = [0.5,1,1.5,2,2.5,3,3.5,4,4.5,5]#,110]#,120]
print TOPICS
LABELS = [r'0.5', r'1',r'1.5', r'2',r'2.5', r'3',r'3.5', r'4',r'4.5',r'5']#, ""]#, ""]
# fig.canvas.set_window_title('Accuracy - Activity Statistics')
plt.yticks(TOPICS, LABELS,fontsize=18)
def autolabel(rects):
# attach some text labels
for rect in rects:
height = rect.get_height()
ax.text(0.26+rect.get_x()+rect.get_width()/2., height +0.35, "%.2f"%float(height),
ha='center', va='bottom',fontweight='bold',fontsize=13.5)
# autolabel(rects1)
left = 0.1 # the left side of the subplots of the figure
right = 0.975 # the right side of the subplots of the figure
bottom = 0.075 # the bottom of the subplots of the figure
top = 0.925 # the top of the subplots of the figure
wspace = 0.2 # the amount of width reserved for blank space between subplots
hspace = 0.4 # the amount of height reserved for white space between subplots
plt.subplots_adjust(left=left, bottom=bottom, right=right, top=top, wspace=wspace, hspace=hspace)
plt.show()
if __name__ == "__main__":
main(sys.argv[1:])

418
code/Visualization/EffectNumberofClusters_plus_Simulation.py Normal file
View file

@ -0,0 +1,418 @@
from __future__ import division
import os, sys
import seaborn
from pylab import rcParams
import cv2
import numpy as np
from numpy import linalg as LA
from time import time
from itertools import combinations
import matplotlib.pyplot as plt
from matplotlib.pyplot import *
import matplotlib.patches as mpatches
sys.path.append('..') # so we can import modules from `code` directory
from minimize import findInitialW, _q, g, minimizeEnergy, g3D3D, findW3D3D
from minimize import g as gaze_ray
from geom import getSphericalCoords, getAngularDiff
from vector import Vector as v
from sim import GazeSimulation
from recording.util.tools import is_outlier
from recording.tracker import readCameraParams
from geom import getRotationMatrix
from parallax_analysis import Experiment
'''
Change lines 48 to 61 accordingly
'''
class Parallax2Dto3DMapping(Experiment):
'''
IMPORTANT!
In all experiments, scene camera's rvec = (0, 0, 0) i.e. the corresponding rotation matrix is the identity matrix therefore
I have not included the dot production with this rotation matrix to convert points in world coordinates
into scene camera coordinates. however, one should know that if the scene camera is rotated differentl7y
this transformation is essential. I would add the corresponding computations later on.
'''
def __run__(self):
# Processing real world data
######################################################################################################
print '> Processing real world data...'
C12D2D = np.load("../results/MeansC1D2D2.npy")
C12D3D = np.load("../results/MeansC1D2D3.npy")
C22D2D = np.load("../results/MeansC2D2D2.npy")
C22D3D = np.load("../results/MeansC2D2D3.npy")
C32D2D = np.load("../results/MeansC3D2D2.npy")
C32D3D = np.load("../results/MeansC3D2D3.npy")
C42D2D = np.load("../results/MeansC4D2D2.npy")
C42D3D = np.load("../results/MeansC4D2D3.npy")
C52D2D = np.load("../results/MeansC5D2D2.npy")
C52D3D = np.load("../results/MeansC5D2D3.npy")
summeC12D2D = []
summeC22D2D = []
summeC32D2D = []
summeC42D2D = []
summeC52D2D = []
summeC12D3D = []
summeC22D3D = []
summeC32D3D = []
summeC42D3D = []
summeC52D3D = []
i = 0
while i < len(C12D2D):
j = 0
while j < len(C12D2D[0]):
summeC12D2D.append(C12D2D[i][j])
j += 1
i += 1
i = 0
while i < len(C22D2D):
j = 0
while j < len(C22D2D[0]):
summeC22D2D.append(C22D2D[i][j])
j += 1
i += 1
i = 0
while i < len(C32D2D):
j = 0
while j < len(C32D2D[0]):
summeC32D2D.append(C32D2D[i][j])
j += 1
i += 1
i = 0
while i < len(C42D2D):
j = 0
while j < len(C42D2D[0]):
summeC42D2D.append(C42D2D[i][j])
j += 1
i += 1
i = 0
while i < len(C52D2D):
j = 0
while j < len(C52D2D[0]):
summeC52D2D.append(C52D2D[i][j])
j += 1
i += 1
i = 0
while i < len(C12D3D):
j = 0
while j < len(C12D3D[0]):
summeC12D3D.append(C12D3D[i][j])
j += 1
i += 1
i = 0
while i < len(C22D3D):
j = 0
while j < len(C22D3D[0]):
summeC22D3D.append(C22D3D[i][j])
j += 1
i += 1
i = 0
while i < len(C32D3D):
j = 0
while j < len(C32D3D[0]):
summeC32D3D.append(C32D3D[i][j])
j += 1
i += 1
i = 0
while i < len(C42D3D):
j = 0
while j < len(C42D3D[0]):
summeC42D3D.append(C42D3D[i][j])
j += 1
i += 1
i = 0
while i < len(C52D3D):
j = 0
while j < len(C52D3D[0]):
summeC52D3D.append(C52D3D[i][j])
j += 1
i += 1
mean1 = np.mean(summeC12D2D)
mean2 = np.mean(summeC22D2D)
mean3 = np.mean(summeC32D2D)
mean4 = np.mean(summeC42D2D)
mean5 = np.mean(summeC52D2D)
mean6 = np.mean(summeC12D3D)
mean7 = np.mean(summeC22D3D)
mean8 = np.mean(summeC32D3D)
mean9 = np.mean(summeC42D3D)
mean10 = np.mean(summeC52D3D)
std1 = np.std(summeC12D2D)
std2 = np.std(summeC22D2D)
std3 = np.std(summeC32D2D)
std4 = np.std(summeC42D2D)
std5 = np.std(summeC52D2D)
std6 = np.std(summeC12D3D)
std7 = np.std(summeC22D3D)
std8 = np.std(summeC32D3D)
std9 = np.std(summeC42D3D)
std10 = np.std(summeC52D3D)
mean2D2D_real = [mean1,mean2,mean3,mean4,mean5]
mean2D3D_real = [mean6,mean7,mean8,mean9,mean10]
std2D2D_real = [std1,std2,std3,std4,std5]
std2D3D_real = [std6,std7,std8,std9,std10]
######################################################################################################
# Simulation
print '> Processing simulation data...'
######################################################################################################
sim = GazeSimulation(log = False)
sim.place_eyeball_on_scene_camera = False
sim.setEyeRelativeToSceneCamera(v(-65, -33, -73))
# sim.setEyeRelativeToSceneCamera(v(-65, -33, 0)) # assuming eyeball and scene camera are coplanar i.e. e = (e.x, e.y, 0)
sim.setCalibrationDepth(1 * 1000) # mm, wrt scene camera
sim.setTestDepth(1.5 * 1000)
sim.calibration_grid = True
sim.calibration_random_depth = False
sim.test_grid = True
sim.test_random_depth = False
sim.test_random_fixed_depth = False
depths = map(lambda d:d*1000, [1, 1.25, 1.5, 1.75, 2.0])
print '> Computing results for multiple calibration depths...'
results, results_std = [], []
for num_of_calibration_depths in xrange(1, 6): # from 1 calibration depths to 5
print '> Considering only %s calibration depth(s)...' %num_of_calibration_depths
sim.reset()
aae_2ds_aae = []
aae_2ds_phe = []
aae_3ds_aae = []
aae_3ds_phe = []
aae_3D3Ds = [] # angular error
for calibs in combinations(depths, num_of_calibration_depths):
# Now calibs is a set of depths from each of which we need calibration data
print '> Current calibration depths', calibs
calibs = list(calibs)
cp, ct = [], []
sim.reset()
sim.setCalibrationDepth(calibs)
# Perform calibration
sim.runCalibration()
cp, ct, p3d = sim.tr_pupil_locations, sim.calibration_points, sim.tr_3d_pupil_locations
# target positions are computed relative to the scene CCS
ti = map(lambda target: v(target) - v(sim.scene_camera.t), ct)
# Computing pupil pose for each gaze
ni = map(lambda p: (v(p)-v(sim.sclera_pos)).norm(), p3d) # ground truth gaze vectors
w, e, w0 = minimizeEnergy(cp, ti)
e = v(e)
# transforming pupil pose to eye camera CS
eyeR = np.array(sim.eye_camera.R[:3])
ni = map(lambda pose: eyeR.dot(np.array(pose)), ni)
R, e3d3d = minimizeEnergy(ni, ti, pose_given=True)
e3d3d = v(e3d3d)
# R = LA.inv(R)
# Now we have calibration data from multiple depths, we can test on all depths
for test_depth in depths:
sim.setTestDepth(test_depth)
aae_2d_aae, aae_2d_phe, aae_2d_std, _ = sim.runTest() # last one is PHE std
aae_2ds_aae.append((aae_2d_aae, aae_2d_std))
aae_2ds_phe.append(aae_2d_phe)
# Fetching test points
t, p, p3d = sim.test_points, sim.te_pupil_locations, sim.te_3d_pupil_locations
t = map(lambda target: v(target) - v(sim.scene_camera.t), t) # target coords in scene CCS
# 3D3D
t_3d3d = t[:]
ni = map(lambda p: v(v(p)-v(sim.sclera_pos)).norm(), p3d) # ground truth gaze vectors
# transforming pupil pose to eye camera CS
ni = map(lambda r: v(eyeR.dot(np.array(r))), ni)
# applying estimated rotation to pose vector in eye camera coordinates (Rn)
# R is estimated rotation between scene camera and eye coordinate system (not eye camera!)
# in other words, R is the rotation part of e
Rni = map(lambda n: v(R.dot(np.array(n))), ni) # now ready to compare Rn with t-e
# Intersecting gaze rays originating from the eye with the planes defined by each
# target. then we can simply compute angular error between each intersection and
# the corresponding 3D target
gis = map(lambda vec: v(vec), Rni) # gaze rays originating from eyeball
# we multiply g such that it hits t's z-plane i.e. multiply all coordinates by factor (t.z-e.z)/g.z
# then we add e to the final g so that it originates from scene camera. now both g and t are in the
# same coordinate system and originate from the same point, so we can compare them
gprimes = map(lambda tg: v(((tg[0].z - e3d3d.z)/tg[1].z)*tg[1] + e3d3d), zip(t_3d3d, gis))
AE = list(np.degrees(np.arctan((v(p[0]).cross(p[1])/(v(p[0]).dot(p[1]))).mag)) for p in zip(gprimes, t_3d3d))
N = len(t)
AAE = np.mean(AE)
STD = np.std(AE)
m, M = min(AE), max(AE)
aae_3D3Ds.append((AAE, STD))
qi = map(_q, p) # computing feature vectors from raw pupil coordinates in 2D
# computing unit gaze vectors corresponding to pupil positions
# here we use the computed mapping matrix w
gis = map(lambda q: g(q, w), qi)
# Intersecting gaze rays originating from the eye with the planes defined by each
# target. then we can simply compute angular error between each intersection and
# the corresponding 3D target
t = map(lambda vec: v(vec), t)
gis = map(lambda vec: v(vec), gis)
gprimes = map(lambda tg: v(((tg[0].z - e.z)/tg[1].z)*tg[1] + e), zip(t, gis))
AE = list(np.degrees(np.arctan((v(p[0]).cross(p[1])/(v(p[0]).dot(p[1]))).mag)) for p in zip(gprimes, t))
N = len(t)
AAE = np.mean(AE)
STD = np.std(AE)
m, M = min(AE), max(AE)
# Computing physical distance error (in meters)
PHE = list((u-v).mag/1000 for u,v in zip(t, gprimes))
N = len(t)
APHE = np.mean(PHE)
PHE_STD = np.std(PHE)
PHE_m, PHE_M = min(PHE), max(PHE)
aae_3ds_aae.append((AAE, STD))
aae_3ds_phe.append((PHE, PHE_STD))
# results only contains AAE
results.append([np.mean(np.array(aae_2ds_aae)[:,0]), np.mean(np.array(aae_3ds_aae)[:,0]), np.mean(np.array(aae_3D3Ds)[:,0])])
results_std.append([np.std(np.array(aae_2ds_aae)[:,0]), np.std(np.array(aae_3ds_aae)[:,0]), np.std(np.array(aae_3D3Ds)[:,0])])
######################################################################################################
# Plotting
print '> Plotting...'
######################################################################################################
# New plot code based on EffectNumberofClusters.py
mean2D2D = [res[0] for res in results]
mean2D3D = [res[1] for res in results]
mean3D3D = [res[2] for res in results]
std2D2D = [res[0] for res in results_std]
std2D3D = [res[1] for res in results_std]
std3D3D = [res[2] for res in results_std]
N = 5
ind = np.asarray([0.25,1.25,2.25,3.25,4.25])
width = 0.5 # the width of the bars
# x1 = [0.4,1.4,2.4,3.4,4.4]
x2 = [0.45,1.45,2.45,3.45,4.45]
# x3 = [0.5,1.5,2.5,3.5,4.5]
x4 = [0.55,1.55,2.55,3.55,4.55]
# x5 = [0.6,1.6,2.6,3.6,4.6]
x6 = [0.50,1.50,2.50,3.50,4.50]
fig = plt.figure(figsize=(14.0, 10.0))
ax = fig.add_subplot(111)
rrects1 = ax.errorbar(x2, mean2D2D_real,yerr=[std2D2D_real,std2D2D_real],fmt='o',color='red',ecolor='red',lw=3, capsize=8, capthick=3)
plt.plot(x2, mean2D2D_real, marker="o", linestyle='-',lw=3,color='red',label = r'2D-to-2D')
rrects2 =ax.errorbar(x4, mean2D3D_real,yerr=[std2D3D_real,std2D3D_real],fmt='o',color='blue',ecolor='blue',lw=3, capsize=8, capthick=3)
plt.plot(x4, mean2D3D_real, marker="o", linestyle='-',lw=3,color='blue', label = r'2D-to-3D')
rects1 = ax.errorbar(x2, mean2D2D,yerr=[std2D2D,std2D2D],fmt='o',color='red',ecolor='red',lw=3, capsize=5, capthick=2)
plt.plot(x2, mean2D2D, marker="o", linestyle='--',lw=3,color='red',label = r'2D-to-2D Simulation')
rects2 =ax.errorbar(x4, mean2D3D,yerr=[std2D3D,std2D3D],fmt='o',color='blue',ecolor='blue',lw=3, capsize=5, capthick=2)
plt.plot(x4, mean2D3D, marker="o", linestyle='--',lw=3,color='blue', label = r'2D-to-3D Simulation')
rects3 =ax.errorbar(x6, mean3D3D,yerr=[std3D3D,std3D3D],fmt='o',color='orange',ecolor='orange',lw=3, capsize=5, capthick=2)
plt.plot(x6, mean3D3D, marker="o", linestyle='--',lw=3,color='orange', label = r'3D-to-3D Simulation')
legend(fontsize=20,loc='upper right')
# rects3 = ax.errorbar(x3, meanC3,yerr=[stdC3,stdC3],fmt='o',color='black',ecolor='black',lw=3, capsize=5, capthick=2)
# plt.plot(x3, meanC3, marker="o", linestyle='-',lw=3,color='black')
#
# rects4 =ax.errorbar(x4, meanC4,yerr=[stdC4,stdC4],fmt='o',color='green',ecolor='green',lw=3, capsize=5, capthick=2)
# plt.plot(x4, meanC4, marker="o", linestyle='-',lw=3,color='green')
#
# rects5 =ax.errorbar(x5, meanC5,yerr=[stdC5,stdC5],fmt='o',color='orange',ecolor='orange',lw=3, capsize=5, capthick=2)
# plt.plot(x5, meanC5, marker="o", linestyle='-',lw=3,color='orange')
ax.set_ylabel(r'Angular Error',fontsize=22)
ax.set_xlabel(r'Number of Calibration Depths',fontsize=22)
ax.set_xticks(ind+0.25)
ax.set_xticklabels( ('D1', 'D2', 'D3','D4', 'D5') ,fontsize=18)
TOPICs = [0.0,0.5,1.5,2.5,3.5,4.5,5.0]#,110]#,120]
print TOPICs
LABELs = ["",r'1',r'2', r'3', r'4', r'5', ""]#, ""]#, ""]
# fig.canvas.set_window_title('Distance Error Correlation')
plt.xticks(TOPICs, LABELs,fontsize=18)
# legend([rects1,rects2], [r'\LARGE\textbf{2D2D}', r'\LARGE\textbf{2D3D}'], loc='lower right')
TOPICS = [0.0, 0.5,1,1.5,2,2.5,3,3.5,4,4.5,5]#,110]#,120]
print TOPICS
LABELS = [r'0.0', r'0.5', r'1',r'1.5', r'2',r'2.5', r'3',r'3.5', r'4',r'4.5',r'5']#, ""]#, ""]
# fig.canvas.set_window_title('Accuracy - Activity Statistics')
plt.yticks(TOPICS, LABELS,fontsize=18)
def autolabel(rects):
# attach some text labels
for rect in rects:
height = rect.get_height()
ax.text(0.26+rect.get_x()+rect.get_width()/2., height +0.35, "%.2f"%float(height),
ha='center', va='bottom',fontweight='bold',fontsize=13.5)
# autolabel(rects1)
left = 0.1 # the left side of the subplots of the figure
right = 0.975 # the right side of the subplots of the figure
bottom = 0.075 # the bottom of the subplots of the figure
top = 0.925 # the top of the subplots of the figure
wspace = 0.2 # the amount of width reserved for blank space between subplots
hspace = 0.4 # the amount of height reserved for white space between subplots
plt.subplots_adjust(left=left, bottom=bottom, right=right, top=top, wspace=wspace, hspace=hspace)
plt.show()
######################################################################################################
def main():
ex = Parallax2Dto3DMapping()
ex.performExperiment()
if __name__ == "__main__":
main()

36
code/copy_missing_data.py Normal file
View file

@ -0,0 +1,36 @@
'''
Copies all missing files from ROOT to DEST for selected PARTICIPANTS
'''
import os, sys
import shutil
PARTICIPANTS = ['p10', 'p16', 'p13', 'p24', 'p5', 'p14', 'p26', 'p12', 'p20', 'p7', 'p15', 'p11', 'p21', 'p25']
DEST = '/home/mmbrian/3D_Gaze_Tracking/archive00/participants/'
ROOT = '/home/mmbrian/HiWi/etra2016_mohsen/code/recording/data/participants/'
def main():
processed = 0
for p in os.listdir(ROOT):
if p in PARTICIPANTS:
print '> Copying missing files for participant', p
d1 = os.path.join(ROOT, p)
d1 = os.path.join(d1, os.listdir(d1)[0]) # ../p_i/../
for d2 in os.listdir(d1): # per recording
root_path = os.path.join(d1, d2)
dest_path = os.path.join(DEST, p)
dest_path = os.path.join(dest_path, os.listdir(dest_path)[0], d2)
print '> From:', root_path, 'To:', dest_path
processPath(root_path, dest_path)
processed+=1
print '> Processed %s participants.' % processed
def processPath(root_path, dest_path):
dest_files = os.listdir(dest_path)
for f in os.listdir(root_path): # every file from root path
if not f in dest_files: # copy if file does not exist in destination
print '> Copying', f
shutil.copyfile(os.path.join(root_path, f), os.path.join(dest_path, f))
if __name__ == '__main__':
main()

441
code/experiment.py Normal file
View file

@ -0,0 +1,441 @@
from __future__ import division
import numpy as np
from numpy import linalg as LA
from time import time
from sklearn import linear_model
from sim import GazeSimulation
from svis import Ray
from minimize import findInitialW, _q, g, minimizeEnergy, minimizeEnergyGivenE, minimizeUsingScaleVector
from geom import getSphericalCoords, getAngularDiff
import visual as vs
# from visual import vector as v # for vector operations
from vector import Vector as v # for vector operations
class Experiment:
def performExperiment(self):
print 'Running Experiment...'
start = time()
self.__run__()
self.runningTime = time() - start
print 'Running Time:', self.runningTime
if self.minimizationTime:
print 'Minimization Time:', self.minimizationTime
def __run__(self):
pass
class Experiment1(Experiment):
def __run__(self):
'''
Simply evaluates 2D gaze estimation (using pupil calibration method)
'''
sim = GazeSimulation(log = False)
## Uncomment for shifting eyeball to scene camera coordinates
sim.scene_eye_distance_factor = 0.6 # controls how much to move eyeball towards scene camera [0-1]
sim.place_eyeball_on_scene_camera = False
# calibration_grid, calibration_random_depth, test_grid, test_random_depth, test_random_fixed_depth
experiments = [
[True, False, False, True, False],
[True, False, False, False, False],
[True, False, False, False, True],
[False, True, False, True, False],
[False, True, False, False, False],
[False, True, False, False, True],
[False, False, False, True, False],
[False, False, False, False, False],
[False, False, False, False, True],
# [True, False, True, False, False],
]
nsim = 100
for experiment in experiments:
res = [sim.runCustomSimulation(experiment) for n in xrange(nsim)]
res = filter(lambda aae: aae>=0, res)
M = max(res)
m = min(res)
aae = sum(res) / len(res)
print 'AAE: %2.5f - Min AAE: %2.5f | Max AAE: %2.5f' % (aae, m, M)
eye_scene_diff = v(sim.sclera_pos) - v(sim.scene_camera.t)
print "Eyeball-SceneCamera distance was", eye_scene_diff.mag
self.sim = sim
self.minimizationTime = -1
class Experiment1p5(Experiment):
def __run__(self):
'''
Read 3D gaze estimation experiment
'''
sim = GazeSimulation(log = False)
## Uncomment for shifting eyeball to scene camera coordinates
sim.place_eyeball_on_scene_camera = True
sim.scene_eye_distance_factor = 0.6
# sim.scene_eye_distance_factor = 1.0
# calibration_grid, calibration_random_depth, test_grid, test_random_depth, test_random_fixed_depth
# -> grid setting
# con = [True, False, True, False, True]
# -> general setting with grid learning
# con = [True, False, False, True, False]
con = [True, False, False, False, False]
# -> general setting with random learning
# con = [False, True, False, True, False]
sim.num_calibration = 36
sim.num_test = 25
aae_2d = sim.runCustomSimulation(con)
## Fetching the calibration points
t, p = sim.calibration_points, sim.tr_pupil_locations
# target positions are computed relative to the scene CCS
t = map(lambda target: v(target) - v(sim.scene_camera.t), t) # verified
# p = sim.eye_camera.getNormalizedPts(p)
start = time()
eye_scene_diff = v(sim.sclera_pos) - v(sim.scene_camera.t)
e = np.array(eye_scene_diff) # eyeball coords in scene CCS
print 'Real e', e
e_org = e[:]
print 'Eye-Scene distance', eye_scene_diff.mag
# w, w0 = minimizeEnergyGivenE(p, t, e)
## Finding the optimal transformation matrix by minimizing the nonlinear energy
# w0 is the initial w by solving the leastsq with e=(0,0,0)
# w is by solving the leastsq again optimizing for both e and w
w, e, w0 = minimizeEnergy(p, t)
# here were are initializing minimization with an e close to the ground
# truth e by adding random perturbations to it
# w, e, w0 = minimizeEnergy(p, t, e + np.random.rand(1, 3)[0])
print 'Estimated e', e, '( Distance:', LA.norm(e - e_org) , ')'
self.minimizationTime = time() - start
print
sim.w = w
## Fetching test points
t, p = sim.test_points, sim.te_pupil_locations
t = map(lambda target: v(target) - v(sim.scene_camera.t), t) # target coords in scene CCS
# closest point distance to scene camera
cDist = min(v(pt).mag for pt in t)
# farthest point distance to scene camera
fDist = max(v(pt).mag for pt in t)
# average point distance to scene camera
avgDist = sum(v(pt).mag for pt in t)/len(t)
# p = sim.eye_camera.getNormalizedPts(p)
# print
# print p
# print
qi = map(_q, p) # computing feature vectors from raw pupil coordinates in 2D
# computing unit gaze vectors corresponding to pupil positions
# here we use the computed mapping matrix w
gis = map(lambda q: g(q, w), qi)
gis0 = map(lambda q: g(q, w0), qi)
# now we can compare unit gaze vectors with their corresponding gaze rays t
# normalizing gaze rays first
t = map(lambda vec: v(vec).norm(), t)
# TODO: compare spherical coordinates instead
AE = list(np.degrees(np.arctan((v(p[0]).cross(p[1])/(v(p[0]).dot(p[1]))).mag)) for p in zip(gis, t))
N = len(t)
AAE = sum(AE)/N
VAR = sum((ae - AAE)**2 for ae in AE)/N
print 'AAE:', AAE, '\nVariance:', VAR, 'STD:', np.sqrt(VAR), '\nMin:', min(AE), 'Max:', max(AE), '(N=' + str(N) + ')'
print 'Target Distances: m=%s M=%s Avg=%s' % (cDist, fDist, avgDist)
AE0 = list(np.degrees(np.arctan((v(p[0]).cross(p[1])/(v(p[0]).dot(p[1]))).mag)) for p in zip(gis0, t))
AAE0 = sum(AE0)/N
print 'AAE (only optimizing W for e=(0,0,0)):', AAE0
print 'AAE (2D-to-2D mapping):', aae_2d
print ('Improvement (2D-2D vs 2D-3D): %s' % round((aae_2d - AAE)*100/aae_2d, 2)) + '%'
print ('Improvement (2D-2D vs 2D-2D): %s' % round((aae_2d - AAE0)*100/aae_2d, 2)) + '%'
sim.display_test_points = True
sim.display_calibration_points = False
## Visualizing gaze vector
ray_scale_factor = 120
ground_truth_gaze_rays = []
for pt in t:
ground_truth_gaze_rays.append(Ray(sim.scene_camera.t, ray_scale_factor, v(pt), vs.color.green))
sim.rays = ground_truth_gaze_rays
estimated_gaze_rays = []
for pt in gis:
estimated_gaze_rays.append(Ray(sim.scene_camera.t, ray_scale_factor, v(pt), vs.color.red))
sim.rays.extend(estimated_gaze_rays)
AE = []
for pair in zip(t, gis):
gt_r, e_r = pair
base = v(sim.scene_camera.t)
gt_ray_endpoint = base + (v(gt_r) * ray_scale_factor)
e_ray_endpoint = base + (v(e_r) * ray_scale_factor)
AE.append(getAngularDiff(gt_ray_endpoint, e_ray_endpoint, base))
diff = gt_ray_endpoint - e_ray_endpoint
sim.rays.append(Ray(e_ray_endpoint, diff.mag, diff.norm(), vs.color.orange))
# Recomputing AAE (using law of cosines) sum(AE)/N
assert(round(sum(AE)/N - AAE) == 0)
self.sim = sim
print
def p3dEx():
ex = Experiment1p5()
ex.performExperiment()
return ex
def experiment2(s = 1):
sim = GazeSimulation(log = False)
sim.place_eyeball_on_scene_camera = True
# 2/3 for calibration, 1/3 for test
# sim.num_calibration = 66
# sim.num_test = 33
# sim.reset()
# calibration_grid, calibration_random_depth, test_grid, test_random_depth, test_random_fixed_depth
# con = [True, False, False, False, True]
# con = [False, True, False, True, False]
con = [True, False, True, False, True]
# sim.runSimulation()
# sim.num_calibration = 36
sim.num_calibration = 25
sim.num_test = 25
sim.runCustomSimulation(con)
# retrieving calibration points (t for 3D target positions and p for 2D pupil locations)
t, p = sim.calibration_points, sim.tr_pupil_locations
# t = sim.tr_target_projections # projections of target points
# target positions are computed relative to the scene CCS
t = map(lambda target: v(target) - v(sim.scene_camera.t), t) # verified
# p = sim.eye_camera.getNormalizedPts(p)
# computing the initial mapping matrix w by solving the energy minimization
# for e=(0, 0, 0)
# w = findInitialW(p, t, True)
# print w
# w, e = minimizeEnergy(p, t)
# print w
w = np.array([[s, 0],
[0, s]])
# w = minimizeUsingScaleVector(p, t)
# e = np.array(v(sim.sclera_pos) - v(sim.scene_camera.t)) # eyeball coords in scene CCS
# w = minimizeEnergyGivenE(p, t, e)
t, p = sim.test_points, sim.te_pupil_locations
t = map(lambda target: v(target) - v(sim.scene_camera.t), t) # target coords in scene CCS
# TODO, is this necessary? (due to eye camera rotation)
p = map(lambda pt: np.array([-pt[0], pt[1]]), p)
# p = sim.eye_camera.getNormalizedPts(p)
# this w can be used for 2D-2D mapping (apparently!) testing it
qi = map(_q, p) # computing feature vectors from raw pupil coordinates in 2D
# computing unit gaze vectors corresponding to pupil positions
# here we use the computed mapping matrix w
gis = map(lambda q: g(q, w), qi)
# now we can compare unit gaze vectors with their corresponding gaze rays t
# normalizing gaze rays
t = map(lambda vec: v(vec).norm(), t)
SAE = sum(np.degrees(np.arctan((v(p[0]).cross(p[1])/(v(p[0]).dot(p[1]))).mag)) for p in zip(gis, t))
# return SAE / len(t)
sim.display_test_points = True
sim.display_calibration_points = False
## Visualizing gaze vector
ray_scale_factor = 120
ground_truth_gaze_rays = []
for pt in t:
ground_truth_gaze_rays.append(Ray(sim.scene_camera.t, ray_scale_factor, v(pt), vs.color.green))
sim.rays = ground_truth_gaze_rays
estimated_gaze_rays = []
for pt in gis:
estimated_gaze_rays.append(Ray(sim.scene_camera.t, ray_scale_factor, v(pt), vs.color.red))
sim.rays.extend(estimated_gaze_rays)
AE = []
for pair in zip(t, gis):
gt_r, e_r = pair
base = v(sim.scene_camera.t)
gt_ray_endpoint = base + (v(gt_r) * ray_scale_factor)
e_ray_endpoint = base + (v(e_r) * ray_scale_factor)
AE.append(getAngularDiff(gt_ray_endpoint, e_ray_endpoint, base))
diff = gt_ray_endpoint - e_ray_endpoint
sim.rays.append(Ray(gt_ray_endpoint, diff.mag, -diff.norm(), vs.color.orange))
return sim
# print e
def experiment3():
sim = GazeSimulation(log = False)
# 2/3 for calibration, 1/3 for test
# sim.num_calibration = 66
# sim.num_test = 33
# sim.reset()
# calibration_grid, calibration_random_depth, test_grid, test_random_depth, test_random_fixed_depth
con = [True, False, False, True, False]
# con = [False, True, False, True, False]
# sim.runSimulation()
sim.runCustomSimulation(con)
# retrieving calibration points (t for 3D target positions and p for 2D pupil locations)
t, p = sim.calibration_points, sim.tr_pupil_locations
# t = sim.tr_target_projections # projections of target points
# target positions are computed relative to the scene CCS
t = map(lambda target: v(target) - v(sim.scene_camera.t), t)
# p = sim.eye_camera.getNormalizedPts(p)
# Applying regression using LARS (Least Angle Regression)
qi = map(_q, p)
clf = linear_model.Lars(n_nonzero_coefs=np.inf) # n_nonzero_coefs=1
t = map(lambda vec: v(vec).norm(), t)
clf.fit(qi, t)
t, p = sim.test_points, sim.te_pupil_locations
t = map(lambda target: v(target) - v(sim.scene_camera.t), t)
# p = sim.eye_camera.getNormalizedPts(p)
# this w can be used for 2D-2D mapping (apparently!) testing it
qi = map(_q, p) # computing feature vectors from raw pupil coordinates in 2D
# computing unit gaze vectors corresponding to pupil positions
# here we use the computed mapping matrix w
gis = map(lambda q: clf.predict(q)[0], qi)
# now we can compare unit gaze vectors with their corresponding gaze rays t
# normalizing gaze rays
t = map(lambda vec: v(vec).norm(), t)
SAE = sum(np.degrees(np.arctan((v(p[0]).cross(p[1])/(v(p[0]).dot(p[1]))).mag)) for p in zip(gis, t))
print SAE / len(t)
# print e
def experimentGT():
## Here we are using the ground truth gaze vectors (from the eye) and we compare
## them with ground truth vectors initiating from the scene camera
## In case eyeball and scene camera are located at the same point, the AAE should
## be reported as zero, this is only to make sure we're doing the right thing
sim = GazeSimulation(log = False)
# Uncomment to shift eyeball to scene camera
# sim.place_eyeball_on_scene_camera = True
# 2/3 for calibration, 1/3 for test
# sim.num_calibration = 66
# sim.num_test = 33
# sim.reset()
# calibration_grid, calibration_random_depth, test_grid, test_random_depth, test_random_fixed_depth
con = [True, False, False, True, False]
# con = [False, True, False, True, False]
# sim.runSimulation()
sim.runCustomSimulation(con)
# retrieving calibration points (t for 3D target positions and p for 2D pupil locations)
t, p, tp = sim.calibration_points, sim.tr_pupil_locations, sim.tr_target_projections
# t = sim.tr_target_projections # projections of target points
# target positions are computed relative to the scene CCS
t_org = t[:]
t = map(lambda target: v(target) - v(sim.scene_camera.t), t)
rays_from_scene = []
for pt in t:
rays_from_scene.append(Ray(sim.scene_camera.t, v(pt).mag, v(pt).norm(), vs.color.red))
sim.rays = rays_from_scene
rays_from_scene_to_target_projections = []
base = v(sim.scene_camera.t) + v(sim.scene_camera.direction) * sim.scene_camera.f
for pp in tp:
# print pp
# since we wanna draw rays starting from scene camera,
# ray is relative to scene CCS
ray = base + v(pp[0], pp[1], 0) - v(sim.scene_camera.t)
rays_from_scene_to_target_projections.append(Ray(v(sim.scene_camera.t), ray.mag * 4, ray.norm(), vs.color.orange))
sim.rays.extend(rays_from_scene_to_target_projections)
t = map(lambda vec: v(vec).norm(), t)
alpha_GT = []
_deg = lambda triple: (np.degrees(triple[0]), np.degrees(triple[1]), triple[2])
print
for pt in t:
alpha_GT.append(_deg(getSphericalCoords(pt[0], pt[1], pt[2])))
# print _deg(getSphericalCoords(pt[0], pt[1], pt[2])), pt[2]
print
# starting from eyeball
base = v(sim.sclera_pos)
# direction vector from eyeball towards eye camera
d = -v(sim.eye_camera.direction).norm()
# calculating distance between eyeball and image plane of the eye camera
dist = (v(sim.eye_camera.t) - v(sim.sclera_pos)).mag - sim.eye_camera.f
p = map(lambda p: np.array((-p[0], p[1])), p) # -x due to orientation
sf = 4
p = map(lambda p: p.dot(np.array([[sf, 0],
[0 , sf]])), p)
gis = map(lambda p: (d*dist + v(p[0], p[1], 0)).norm(), p)
alpha_test = []
rays_from_eye = []
for pt in gis:
alpha_test.append(_deg(getSphericalCoords(pt[0], pt[1], pt[2])))
rays_from_eye.append(Ray(sim.sclera_pos, 200, pt, vs.color.yellow))
sim.rays.extend(rays_from_eye)
# gis = map(lambda t: (v(t) - v(sim.sclera_pos)).norm(), t_org)
# now we can compare unit gaze vectors with their corresponding gaze rays t
# normalizing gaze rays
# for _t, _gis in zip(t, gis): print _t, _gis
SAE = sum(np.degrees(np.arctan((v(p[0]).cross(p[1])/(v(p[0]).dot(p[1]))).mag)) for p in zip(gis, t))
print SAE / len(t)
return sim
# experiment1()
# for s in np.arange(1, 20, 0.25):
# print experiment2(s), s
# experiment3()
# experimentGT()
# experiment1p5()

288
code/geom.py Normal file
View file

@ -0,0 +1,288 @@
from __future__ import division
import numpy as np
from random import sample
def_fov = np.pi * 2./3
def generatePoints(n, min_xyz, max_xyz, grid=False, randomZ=True, randFixedZ=False, depth=None, offset=0.5, xoffset=0, yoffset=0, zoffset=0):
if randFixedZ: # means all points have the same z but z is chosen at random between max and min z
z = min_xyz[2] + np.random.random() * (max_xyz[2] - min_xyz[2])
else: # same depth
if not isinstance(depth, list) and not depth: # depth is exactly the middle of min and max z
z = min_xyz[2] + (max_xyz[2] - min_xyz[2]) / 2
else:
z = depth
if not grid: # compute randomly
xr, yr, zr = max_xyz[0] - min_xyz[0], max_xyz[1] - min_xyz[1], max_xyz[2] - min_xyz[2]
xr = np.random.rand(1, n)[0] * xr + min_xyz[0]
yr = np.random.rand(1, n)[0] * yr + min_xyz[1]
if randomZ:
zr = np.random.rand(1, n)[0] * zr + min_xyz[2]
else:
zr = np.ones((1, n))[0] * z
return zip(xr, yr, zr)
else: # compute points on a mXm grid when m = sqrt(n)
m = int(np.sqrt(n))
gwx = (max_xyz[0] - min_xyz[0]) / m
gwy = (max_xyz[1] - min_xyz[1]) / m
zr = max_xyz[2] - min_xyz[2]
if randomZ:
return [(min_xyz[0] + (i+offset) * gwx + xoffset,
min_xyz[1] + (j+offset) * gwy + yoffset,
np.random.random() * zr + min_xyz[2] + zoffset) for i in xrange(m) for j in xrange(m)]
else:
if not isinstance(depth, list):
ret = [(min_xyz[0] + (i+offset) * gwx + xoffset, # offset .5
min_xyz[1] + (j+offset) * gwy + yoffset, # offset .5
z + zoffset) for i in xrange(m) for j in xrange(m)]
# return ret
return sample(ret, len(ret)) # this shuffles the points
else:
ret = []
for dz in depth:
ret.extend([(min_xyz[0] + (i+offset) * gwx + xoffset,
min_xyz[1] + (j+offset) * gwy + yoffset,
dz + zoffset) for i in xrange(m) for j in xrange(m)])
# return ret
return sample(ret, len(ret)) # this shuffles the points
def getSphericalCoords(x, y, z):
'''
According to our coordinate system, this returns the
spherical coordinates of a 3D vector.
A vector originating from zero and pointing to the positive Z direction (no X or Y deviation)
will correspond to (teta, phi) = (0, 90) (in degrees)
The coordinate system we are using is similar to https://en.wikipedia.org/wiki/File:3D_Spherical_2.svg
Y
|
|
|______X
/
/
/
Z
with a CounterClockwise rotation of the axis vectors
'''
r = np.sqrt(x*x + y*y + z*z)
teta = np.arctan(x/z)
phi = np.arccos(y/r)
return teta, phi, r
def getAngularDiff(T, E, C):
'''
T is the target point
E is the estimated target
C is camera center
Returns angular error
(using law of cosines: http://mathcentral.uregina.ca/QQ/database/QQ.09.07/h/lucy1.html)
'''
t = (E - C).mag
e = (C - T).mag
c = (T - E).mag
return np.degrees(np.arccos((e*e + t*t - c*c)/(2*e*t)))
def getRotationMatrixFromAngles(r):
'''
Returns a rotation matrix by combining elemental rotations
around x, y', and z''
It also appends a zero row, so the end result looks like:
[R_11 R_12 R_13]
[R_11 R_12 R_13]
[R_11 R_12 R_13]
[0 0 0 ]
'''
cos = map(np.cos, r)
sin = map(np.sin, r)
Rx = np.array([
[1, 0, 0],
[0, cos[0], -sin[0]],
[0, sin[0], cos[0]]])
Ry = np.array([
[ cos[1], 0, sin[1]],
[ 0 , 1, 0],
[-sin[1], 0, cos[1]]])
Rz = np.array([
[cos[2], -sin[2], 0],
[sin[2], cos[2], 0],
[0 , 0, 1]])
R = Rz.dot(Ry.dot(Rx))
return np.concatenate((R, [[0, 0, 0]]))
# import cv2
# def getRotationMatrix(a, b):
# y = a[1] - b[1]
# z = a[2] - b[2]
# x = a[0] - b[0]
# rotx = np.arctan(y/z)
# roty = np.arctan(x*np.cos(rotx)/z)
# rotz = np.arctan(np.cos(rotx)/(np.sin(rotx)*np.sin(roty)))
# return cv2.Rodrigues(np.array([rotx, roty, rotz]))[0]
def getRotationMatrix(a, b):
'''
Computes the rotation matrix that maps unit vector a to unit vector b
It also augments a zero row, so the end result looks like:
[R_11 R_12 R_13]
[R_11 R_12 R_13]
[R_11 R_12 R_13]
[0 0 0 ]
(simply slice the output like R = output[:3] to get only the rotation matrix)
based on the solution here:
https://math.stackexchange.com/questions/180418/calculate-rotation-matrix-to-align-vector-a-to-vector-b-in-3d
'''
a, b = np.array(a), np.array(b)
v = np.cross(a, b, axis=0)
s = np.linalg.norm(v) # sine of angle
c = a.dot(b) # cosine of angle
vx = np.array([
[0 , -v[2], v[1]],
[v[2] , 0, -v[0]],
[-v[1], v[0], 0]])
if s == 0: # a == b
return np.concatenate((np.eye(3), [[0, 0, 0]]))
if c == 1:
return np.concatenate((np.eye(3) + vx, [[0, 0, 0]]))
return np.concatenate((np.eye(3) + vx + vx.dot(vx)*((1-c)/s/s), [[0, 0, 0]]))
class PinholeCamera:
'''
Models a basic Pinhole Camera with 9 degrees of freedom
'''
# Intrinsic parameters
f = 1 # focal length
p = (0, 0) # position of principal point in the image plane
# Extrinsic parameters
# this rotation corresponds to a camera setting pointing towards (0, 0, -1)
r = (0, 0, 0) # rotations in x, y', and z'' planes respectively
t = (0, 0, 0) # camera center translation w.r.t world coordinate system (with no rotation)
#
# Using the above parameters we can construct the camera matrix
#
# [f 0 p.x 0]
# P = K[R|t] where K = [0 f p.y 0] is the camera calibration matrix (full projection matrix)
# [0 0 1 0]
#
# and thus we have x = PX for every point X in the word coordinate system
# and its corresponding projection in the camera image plane x
# NOTE: points are assumed to be represented by homogeneous vectors
#
# Other parameters
label = ''
direction = (0, 0, 1) # camera direction
fov = def_fov # field of view (both horizontal and vertical)
image_width = 2*f*np.tan(fov/2.)
################################################
def __init__(self, label, f = 1, r = (0, 0, 0), t = (0, 0, 0), direction = (0, 0, 1), fov=def_fov):
self.label = label
self.f = f
self.r = r
self.direction = direction
self.t = t
self.setFOV(fov)
self.recomputeCameraMatrix(True, True)
def recomputeCameraMatrix(self, changedRotation, changedIntrinsic):
if changedRotation:
# # Computing rotation matrix using elemental rotations
self.R = getRotationMatrixFromAngles(self.r)
# by default if rotation is 0 then camera optical axis points to positive Z
self.direction = np.array([0, 0, 1, 0]).dot(self.R)
# Computing the extrinsic matrix
_t = -self.R.dot(np.array(self.t)) # t = -RC
self.Rt = np.concatenate((self.R, np.array([[_t[0], _t[1], _t[2], 1]]).T), axis=1)
# instead of the above, we could also represent translation matrix as [I|-C] and
# [R -RC]
# then compute Rt as R[I|-C] = [0 1] [R] [-RC]
# but we're basically do the same thing by concatenating [0] with [ 1]
if changedIntrinsic:
# Computing intrinsic matrix
f, px, py = self.f, self.p[0], self.p[1]
self.K = np.array([
[f, 0, px, 0],
[0, f, py, 0],
[0, 0, 1, 0]])
# Full Camera Projection Matrix
self.P = self.K.dot(self.Rt)
################################################
## Intrinsic parameter setters
def setF(self, f, auto_adjust = True):
self.f = f
if auto_adjust:
self.setFOV(self.fov)
self.recomputeCameraMatrix(False, True)
def setP(self, p, auto_adjust = True):
self.p = p
if auto_adjust:
self.recomputeCameraMatrix(False, True)
def setFOV(self, fov):
self.fov = fov
self.image_width = 2*self.f*np.tan(fov/2.)
################################################
## Extrinsic parameter setters
def setT(self, t, auto_adjust = True):
self.t = t
if auto_adjust:
self.recomputeCameraMatrix(False, False)
def setR(self, r, auto_adjust = True):
self.r = r
if auto_adjust:
self.recomputeCameraMatrix(True, False)
################################################
def project(self, p):
'''
Computes projection of a point p in world coordinate system
using its homogeneous vector coordinates [x, y, z, 1]
'''
if len(p) < 4: p = (p[0], p[1], p[2], 1)
projection = self.P.dot(np.array(p))
# dividing by the Z value to get a 2D point from homogeneous coordinates
return np.array((projection[0], projection[1]))/projection[2]
def getNormalizedPts(self, pts):
'''
Returns normalized x and y coordinates in range [0, 1]
'''
px, py = self.p[0], self.p[1]
return map(lambda p:np.array([p[0] - px + self.image_width/2,
p[1] - py + self.image_width/2]) / self.image_width, pts)
def getDenormalizedPts(self, pts):
'''
Returns original points in the camera image coordinate plane from normalized points
'''
px, py = self.p[0], self.p[1]
offset = np.array([self.image_width/2-px, self.image_width/2-py])
return map(lambda p:(p*self.image_width)-offset, pts)
class Camera(PinholeCamera):
default_radius = 2.0
def updateShape(self):
c = v(self.t)
# Update camera center
self.center.pos = c
# Update the arrow
self.dir.pos = c
self.dir.axis = v(self.direction) * self.f
# Update image plane
self.img_plane.pos = c + self.dir.axis
self.img_plane.length = self.image_width
self.img_plane.height = self.image_width
# TODO: handle rotation of image plane

131
code/minimize.py Normal file
View file

@ -0,0 +1,131 @@
from __future__ import division
import numpy as np
from numpy import linalg as LA
from scipy import optimize
from sklearn.preprocessing import PolynomialFeatures as P
from vector import Vector as v
import cv2
M = 7 # this is the size of the feature vector
_q = lambda p: np.array([p[0], p[1], p[0]*p[0], p[1]*p[1], p[0]*p[1], p[0]*p[1]*p[0]*p[1], 1])
def alpha(q, w):
'''
q is the polynomial representation of a 2D pupil point p
w is the Mx2 parameter matrix (M is the length of our feature vector)
(could be initialized as a zero matrix np.zeros((M, 2)))
returns alpha_i = (theta_i, phi_i)
'''
theta = np.dot(w[:, 0], q)
phi = np.dot(w[:, 1], q)
return theta, phi
def g(q, w):
'''
computes the unit gaze ray originating from the eye corresponding to q
'''
theta, phi = alpha(q, w)
sin = map(np.sin, (theta, phi))
cos = map(np.cos, (theta, phi))
## This is THE definition, maps (0,0) to (0,0,1) and it produces a unit vector
return np.array([sin[0],
cos[0]*sin[1],
cos[0]*cos[1]])
def g3D3D(pose):
theta, phi = pose
sin = map(np.sin, (theta, phi))
cos = map(np.cos, (theta, phi))
return np.array([sin[0],
cos[0]*sin[1],
cos[0]*cos[1]])
####################################################################################
## Initial attempt to solve E(w, 0) Using a similiar idea to:
## http://scipy-lectures.github.io/advanced/mathematical_optimization/#id28
def getWfromRowRepr(w):
w1 = w[:M] # 1st col
w2 = w[M:] # 2nd col
return np.array([w1, w2]).transpose()
def getRfromRowRepr(w):
return np.array([w[:3], w[3:6], w[6:]])
def f(w, qi, ti):
'''
E(\bm{w}) = \sum_{i=1}^N | \bm{(\theta_i, \phi_i)} - \bm{q}_i\bm{w}|^2
'''
w = getWfromRowRepr(w)
alpha_i = map(lambda q: alpha(q, w), qi) # theta, phi
# we compare the estimated polar coordinates with polar coordinates corresponding
# to ti as in the 2D to 2D case
p_ti = map(lambda g: [np.arctan(g.x/g.z), np.arctan(g.y/g.z)], map(v,ti))
return map(lambda pair: np.sqrt((pair[0][0]-pair[1][0])**2 + (pair[0][1]-pair[1][1])**2), zip(p_ti, alpha_i))
def findInitialW(p, t, retMatrix = True):
ti = map(np.array, t)
qi = map(_q, p)
w0 = np.zeros(2*M)
wr = optimize.leastsq(f, w0[:], args=(qi, ti))
if retMatrix:
return getWfromRowRepr(wr[0])
else:
return wr[0]
# tub function > 0 in -pi:pi, positive outside
tub = lambda x: max(-x-np.pi, 0, x-np.pi)
tub_weight = 10000
def f3D3D(w, ni, ti):
'''
E_{\textmd{3Dto3D}}(\bm{R}, \bm{e}) = \sum_{i=1}^N |\bm{R} \bm{n}_i \times (\bm{t}_i - \bm{e}) |^2
'''
rvec = w[:3]
tvec = w[3:] # e
R = cv2.Rodrigues(np.array(rvec))[0]
return map(lambda (t, n): LA.norm(np.cross(R.dot(n), t-tvec)), zip(ti, ni)) + \
[tub_weight * tub(w[0]), tub_weight * tub(w[1]), tub_weight * tub(w[2])]
# this distance measure works if we initialize the minimization by np.array([0.,0.,0.,0.,0.,0.])
# return map(lambda (t, n): -np.dot(R.dot(n), t-tvec)/LA.norm(t-tvec) + 1, zip(ti, ni))
def findW3D3D(po, t):
'''
po stands for pose i.e. a vector in the direction of gaze for each pupil
'''
ti = map(np.array, t) # ground truth data in scene camera coordinate system
ni = map(np.array, po) # pupil pose in eye coordinate system (normal to pupil disc)
# w0 = np.array([0.,0.,0.,0.,0.,0.]) # rvec (roll, pitch, yaw), tvec (x, y, z)
w0 = np.array([0.,np.pi,0.,0.,0.,0.]) # rvec (roll, pitch, yaw), tvec (x, y, z)
wr = optimize.leastsq(f3D3D, w0[:], args=(ni, ti))
return wr[0]
####################################################################################
## Solving target energy using e0 = (0, 0, 0) and w0 from above
def minimizeEnergy(p, t, e0 = None, pose_given = False):
if e0 is None:
e0 = np.array([0, 0, 0])
if pose_given:
w = findW3D3D(p, t)
return cv2.Rodrigues(np.array(w[:3]))[0], w[3:] # R, e
else:
w0 = findInitialW(p, t, False)
qi = map(_q, p)
ti = map(np.array, t)
we0 = np.concatenate((w0, e0))
wer = optimize.leastsq(f2D3D, we0[:], args=(qi, ti))[0]
w = getWfromRowRepr(wer[:2*M])
e = wer[2*M:]
return w, e, getWfromRowRepr(w0)
def f2D3D(we, qi, ti):
'''
E_{\textmd{2Dto3D}}(\bm{w}, \bm{e}) = \sum_{i=1}^N |\bm{g}(\bm{q}_i\bm{w}) \times (\bm{t}_i - \bm{e})|^2
'''
# extracting parameters from the combined vector
w = getWfromRowRepr(we[:2*M])
e = we[2*M:]
gis = map(lambda q: g(q, w), qi)
return map(lambda pair: LA.norm(np.cross(pair[1], pair[0]-e)), zip(ti, gis))

249
code/parallax_2D3D_3Cdepths.py Normal file
View file

@ -0,0 +1,249 @@
from __future__ import division
import os
import seaborn
from pylab import rcParams
import cv2
import numpy as np
from numpy import linalg as LA
from time import time
from itertools import combinations
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
from minimize import findInitialW, _q, g, minimizeEnergy, g3D3D, findW3D3D
from minimize import g as gaze_ray
from geom import getSphericalCoords, getAngularDiff
from vector import Vector as v
# from experiment import Experiment
from sim import GazeSimulation
from recording.util.tools import is_outlier
from recording.tracker import readCameraParams
from geom import getRotationMatrix
class Experiment:
def performExperiment(self):
print 'Running Experiment...'
start = time()
self.__run__()
self.runningTime = time() - start
print 'Running Time:', self.runningTime
def __run__(self):
pass
class Parallax2Dto3DMapping(Experiment):
def __run__(self):
sim = GazeSimulation(log = False)
sim.place_eyeball_on_scene_camera = False
sim.setEyeRelativeToSceneCamera(v(-65, -33, -73))
# sim.setEyeRelativeToSceneCamera(v(-65, -33, 0)) # assuming eyeball and scene camera are coplanar i.e. e = (e.x, e.y, 0)
sim.setCalibrationDepth(1 * 1000) # mm, wrt scene camera
sim.setTestDepth(1.5 * 1000)
sim.calibration_grid = True
sim.calibration_random_depth = False
sim.test_grid = True
sim.test_random_depth = False
sim.test_random_fixed_depth = False
depths = map(lambda d:d*1000, [1, 1.25, 1.5, 1.75, 2.0])
print '> Computing results for multiple calibration depths...'
results = []
for num_of_calibration_depths in xrange(1, 6): # from 1 calibration depths to 5
print '> Considering only %s calibration depth(s)...' %num_of_calibration_depths
sim.reset()
if num_of_calibration_depths != 3: continue
for calibs in combinations(depths, num_of_calibration_depths):
aae_2ds_aae = []
aae_2ds_phe = []
aae_3ds_aae = []
aae_3ds_phe = []
aae_3D3Ds = [] # angular error
# Now calibs is a set of depths, from each of those we need calibration data
# print calibs
if not calibs in [(1000., 1250., 1500.), (1250., 1500., 1750.), (1500., 1750., 2000.)]:
continue
print 'curr calibs', calibs
calibs = list(calibs)
cp, ct = [], []
sim.reset()
sim.setCalibrationDepth(calibs)
# Perform calibration
sim.runCalibration()
cp, ct, p3d = sim.tr_pupil_locations, sim.calibration_points, sim.tr_3d_pupil_locations
# target positions are computed relative to the scene CCS
ti = map(lambda target: v(target) - v(sim.scene_camera.t), ct)
# Computing pupil pose for each gaze
ni = map(lambda p: (v(p)-v(sim.sclera_pos)).norm(), p3d) # ground truth gaze vectors
w, e, w0 = minimizeEnergy(cp, ti)
e = v(e)
# transforming pupil pose to eye camera CS
eyeR = np.array(sim.eye_camera.R[:3])
ni = map(lambda pose: eyeR.dot(np.array(pose)), ni)
R, e3d3d = minimizeEnergy(ni, ti, pose_given=True)
e3d3d = v(e3d3d)
# R = LA.inv(R)
# Now we have calibration data from multiple depths, we can test on all depths
for test_depth in depths:
sim.setTestDepth(test_depth)
aae_2d_aae, aae_2d_phe, aae_2d_std, _ = sim.runTest() # 2nd one is PHE
aae_2ds_aae.append((aae_2d_aae, aae_2d_std))
aae_2ds_phe.append(aae_2d_phe)
# Fetching test points
t, p, p3d = sim.test_points, sim.te_pupil_locations, sim.te_3d_pupil_locations
t = map(lambda target: v(target) - v(sim.scene_camera.t), t) # target coords in scene CCS
# 3D3D
t_3d3d = t[:]
ni = map(lambda p: v(v(p)-v(sim.sclera_pos)).norm(), p3d) # ground truth gaze vectors
# transforming pupil pose to eye camera CS
ni = map(lambda r: v(eyeR.dot(np.array(r))), ni)
# applying estimated rotation to pose vector in eye camera coordinates (Rn)
# R is estimated rotation between scene camera and eye coordinate system (not eye camera!)
# in other words, R is the rotation part of e
Rni = map(lambda n: v(R.dot(np.array(n))), ni) # now ready to compare Rn with t-e
# Intersecting gaze rays originating from the eye with the planes defined by each
# target. then we can simply compute angular error between each intersection and
# the corresponding 3D target
gis = map(lambda vec: v(vec), Rni) # gaze rays originating from eyeball
# we multiply g such that it hits t's z-plane i.e. multiply all coordinates by factor (t.z-e.z)/g.z
# then we add e to the final g so that it originates from scene camera. now both g and t are in the
# same coordinate system and originate from the same point, so we can compare them
gprimes = map(lambda tg: v(((tg[0].z - e3d3d.z)/tg[1].z)*tg[1] + e3d3d), zip(t_3d3d, gis))
AE = list(np.degrees(np.arctan((v(p[0]).cross(p[1])/(v(p[0]).dot(p[1]))).mag)) for p in zip(gprimes, t_3d3d))
N = len(t)
AAE = np.mean(AE)
STD = np.std(AE)
m, M = min(AE), max(AE)
aae_3D3Ds.append((AAE, STD))
qi = map(_q, p) # computing feature vectors from raw pupil coordinates in 2D
# computing unit gaze vectors corresponding to pupil positions
# here we use the computed mapping matrix w
gis = map(lambda q: g(q, w), qi)
# Intersecting gaze rays originating from the eye with the planes defined by each
# target. then we can simply compute angular error between each intersection and
# the corresponding 3D target
t = map(lambda vec: v(vec), t)
gis = map(lambda vec: v(vec), gis)
gprimes = map(lambda tg: v(((tg[0].z - e.z)/tg[1].z)*tg[1] + e), zip(t, gis))
AE = list(np.degrees(np.arctan((v(p[0]).cross(p[1])/(v(p[0]).dot(p[1]))).mag)) for p in zip(gprimes, t))
N = len(t)
AAE = np.mean(AE)
STD = np.std(AE)
m, M = min(AE), max(AE)
# Computing physical distance error (in meters)
PHE = list((u-v).mag/1000 for u,v in zip(t, gprimes))
N = len(t)
APHE = np.mean(PHE)
PHE_STD = np.std(PHE)
PHE_m, PHE_M = min(PHE), max(PHE)
aae_3ds_aae.append((AAE, STD))
aae_3ds_phe.append((PHE, PHE_STD))
results.append([calibs, aae_2ds_aae, aae_3ds_aae, aae_3D3Ds])
plt.ylabel('Angular Error')
plt.xlabel('Depth')
fig = plt.figure(figsize=(14.0, 10.0))
ax = fig.add_subplot(111)
clrs = ['blue', 'red', 'orange']
depth_map = {1000.:1, 1250.:2, 1500.:3, 1750.:4, 2000.:5}
cdlabel = lambda c: ' + '.join(map(str, map(lambda d: depth_map[d], c)))
x1 = [0.375,1.375,2.375,3.375,4.375]
x2 = [0.425,1.425,2.425,3.425,4.425]
x3 = [0.475,1.475,2.475,3.475,4.475]
x4 = [0.525,1.525,2.525,3.525,4.525]
x5 = [0.575,1.575,2.575,3.575,4.575]
x6 = [0.625,1.625,2.625,3.625,4.625]
xrange_2d2d = [x1, x3, x5]
xrange_2d3d = [x2, x4, x6]
for i in [0, 1, 2]:
res = results[i]
calibs = res[0]
_xrange = xrange_2d2d[i]
aae_2d2d = np.array(res[1])[:,0]
std_2d2d = np.array(res[1])[:,1]
rects1 = ax.errorbar(_xrange, aae_2d2d,yerr=[std_2d2d, std_2d2d],fmt='o',color=clrs[i],ecolor=clrs[i],lw=3, capsize=5, capthick=2)
plt.plot(_xrange, aae_2d2d, marker="o", linestyle='-',lw=3,color=clrs[i],label = '2D-to-2D Calibration Depth '+cdlabel(calibs))
for i in [0, 1, 2]:
res = results[i]
calibs = res[0]
_xrange = xrange_2d3d[i]
aae_2d3d = np.array(res[2])[:,0]
std_2d3d = np.array(res[2])[:,1]
rects2 = ax.errorbar(_xrange, aae_2d3d,yerr=[std_2d3d, std_2d3d],fmt='o',color=clrs[i],ecolor=clrs[i],lw=3, capsize=5, capthick=2)
plt.plot(_xrange, aae_2d3d, marker="o", linestyle='--',lw=3,color=clrs[i],label ='2D-to-3D Calibration Depth '+cdlabel(calibs))
for i in [0, 1, 2]:
res = results[i]
calibs = res[0]
_xrange = xrange_2d2d[i]
aae_3d3d = np.array(res[3])[:,0]
std_3d3d = np.array(res[3])[:,1]
rects3 = ax.errorbar(_xrange, aae_3d3d,yerr=[std_3d3d, std_3d3d],fmt='o',color=clrs[i],ecolor=clrs[i],lw=3, capsize=5, capthick=2)
plt.plot(_xrange, aae_3d3d, marker="o", linestyle='-.',lw=3,color=clrs[i],label ='3D-to-3D Calibration Depth '+cdlabel(calibs))
ax.set_ylim(0, 1.5)
ax.set_ylabel(r'Angular Error',fontsize=22, fontweight='bold')
ax.set_xlabel(r'Depth',fontsize=22, fontweight='bold')
TOPICS = [-0.2, 0, 0.2, 0.4,0.6,0.8,1.0,1.2,1.4,1.6,1.8,2.0,2.2,2.4]#,110]#,120]
LABELS = [r'', r'0', r'0.2',r'0.4',r'0.6', r'0.8', r'1.0', r'1.2', r'1.4', r'1.6', r'1.8', r'2.0', r'2.2', r'2.4']#, ""]#, ""]
plt.yticks(TOPICS, LABELS,fontsize=18)
# plt.legend(fontsize=20)
plt.legend(bbox_to_anchor=(0., 1.02, 1., .102), loc=3,
ncol=3, mode="expand", borderaxespad=0., fontsize=15)
plt.yticks(fontsize='18')
TOPICs = [0.0,0.5,1.5,2.5,3.5,4.5,5.0]
LABELs = ['', 'D1 - 1m', 'D2 - 1.25m', 'D3 - 1.5m', 'D4 - 1.75m', 'D5 - 2.0m', '']
# ax.set_xticklabels(LABELs,fontsize=18)
plt.xticks(TOPICs, LABELs,fontsize=18)
left = 0.1 # the left side of the subplots of the figure
right = 0.975 # the right side of the subplots of the figure
bottom = 0.075 # the bottom of the subplots of the figure
top = 0.925 # the top of the subplots of the figure
wspace = 0.2 # the amount of width reserved for blank space between subplots
hspace = 0.4 # the amount of height reserved for white space between subplots
plt.subplots_adjust(left=left, bottom=bottom, right=right, top=top, wspace=wspace, hspace=hspace)
plt.show()
if __name__ == '__main__':
# This also does 3D gaze estimation and plots estimation results for both 3D and 2D estimation
ex = Parallax2Dto3DMapping()
ex.performExperiment()

996
code/parallax_analysis.py Normal file
View file

@ -0,0 +1,996 @@
from __future__ import division
import os, sys
import seaborn
from pylab import rcParams
import cv2
import numpy as np
from numpy import linalg as LA
from time import time
from itertools import combinations
import matplotlib.pyplot as plt
from matplotlib.pyplot import legend
import matplotlib.patches as mpatches
from minimize import findInitialW, _q, g, minimizeEnergy, g3D3D, findW3D3D
from minimize import g as gaze_ray
from geom import getSphericalCoords, getAngularDiff
from vector import Vector as v
from sim import GazeSimulation
from recording.util.tools import is_outlier
from recording.tracker import readCameraParams
from geom import getRotationMatrix
import pdb
PARTICIPANTS = ['p10', 'p16', 'p13', 'p24', 'p5', 'p14', 'p26', 'p12', 'p20', 'p7', 'p15', 'p11', 'p21', 'p25']
class Experiment:
def performExperiment(self):
print 'Running Experiment...'
start = time()
self.__run__()
self.runningTime = time() - start
print 'Running Time:', self.runningTime
def __run__(self):
pass
class Parallax2Dto2DMapping(Experiment):
def __run__(self):
sim = GazeSimulation(log = False)
sim.place_eyeball_on_scene_camera = False
sim.setEyeRelativeToSceneCamera(v(-65, -33, -73))
sim.setCalibrationDepth(1 * 1000) # mm, wrt scene camera
sim.setTestDepth(1.5 * 1000)
sim.calibration_grid = True
sim.calibration_random_depth = False
sim.test_grid = True
sim.test_random_depth = False
sim.test_random_fixed_depth = False
sim.reset()
print 'scene_camera', sim.scene_camera.t
print 'calibration'
print len(sim.calibration_points)
# print sim.calibration_points
print min(np.array(sim.calibration_points)[:,0]) - sim.scene_camera.t[0], max(np.array(sim.calibration_points)[:,0]) - sim.scene_camera.t[0]
print min(np.array(sim.calibration_points)[:,1]) - sim.scene_camera.t[1], max(np.array(sim.calibration_points)[:,1]) - sim.scene_camera.t[1]
print 'depths', set(np.array(sim.calibration_points)[:,2])
print 'test'
print len(sim.test_points)
print min(np.array(sim.test_points)[:,0]) - sim.scene_camera.t[0], max(np.array(sim.test_points)[:,0]) - sim.scene_camera.t[0]
print min(np.array(sim.test_points)[:,1]) - sim.scene_camera.t[1], max(np.array(sim.test_points)[:,1]) - sim.scene_camera.t[1]
print 'depths', set(np.array(sim.test_points)[:,2])
plt.ylabel('Y (mm)')
plt.xlabel('X (mm)')
plt.plot(np.array(sim.calibration_points)[:,0], np.array(sim.calibration_points)[:,1], 'bo')
plt.plot(np.array(sim.test_points)[:,0], np.array(sim.test_points)[:,1], 'ro')
plt.show()
# self.sim = sim
# visualize the setting in windows
class Parallax2Dto3DMapping(Experiment):
'''
IMPORTANT!
In all experiments, scene camera's rvec = (0, 0, 0) i.e. the corresponding rotation matrix is the identity matrix therefore
I have not included the dot production with this rotation matrix to convert points in world coordinates
into scene camera coordinates. however, one should know that if the scene camera is rotated differentl7y
this transformation is essential. I would add the corresponding computations later on.
'''
def __run__(self):
sim = GazeSimulation(log = False)
sim.place_eyeball_on_scene_camera = False
sim.setEyeRelativeToSceneCamera(v(-65, -33, -73))
# sim.setEyeRelativeToSceneCamera(v(-65, -33, 0)) # assuming eyeball and scene camera are coplanar i.e. e = (e.x, e.y, 0)
sim.setCalibrationDepth(1 * 1000) # mm, wrt scene camera
sim.setTestDepth(1.5 * 1000)
sim.calibration_grid = True
sim.calibration_random_depth = False
sim.test_grid = True
sim.test_random_depth = False
sim.test_random_fixed_depth = False
depths = map(lambda d:d*1000, [1, 1.25, 1.5, 1.75, 2.0])
print '> Computing results for multiple calibration depths...'
results, results_std = [], []
for num_of_calibration_depths in xrange(1, 6): # from 1 calibration depths to 5
print '> Considering only %s calibration depth(s)...' %num_of_calibration_depths
sim.reset()
aae_2ds_aae = []
aae_2ds_phe = []
aae_3ds_aae = []
aae_3ds_phe = []
aae_3D3Ds = [] # angular error
for calibs in combinations(depths, num_of_calibration_depths):
# Now calibs is a set of depths from each of which we need calibration data
print 'Current calibration depths', calibs
calibs = list(calibs)
cp, ct = [], []
sim.reset()
sim.setCalibrationDepth(calibs)
# Perform calibration
sim.runCalibration()
cp, ct, p3d = sim.tr_pupil_locations, sim.calibration_points, sim.tr_3d_pupil_locations
# target positions are computed relative to the scene CCS
ti = map(lambda target: v(target) - v(sim.scene_camera.t), ct)
# Computing pupil pose for each gaze
ni = map(lambda p: (v(p)-v(sim.sclera_pos)).norm(), p3d) # ground truth gaze vectors
w, e, w0 = minimizeEnergy(cp, ti)
e = v(e)
# transforming pupil pose to eye camera CS
eyeR = np.array(sim.eye_camera.R[:3])
ni = map(lambda pose: eyeR.dot(np.array(pose)), ni)
R, e3d3d = minimizeEnergy(ni, ti, pose_given=True)
# R = LA.inv(R)
e3d3d = v(e3d3d)
# Now we have calibration data from multiple depths, we can test on all depths
for test_depth in depths:
sim.setTestDepth(test_depth)
aae_2d_aae, aae_2d_phe, aae_2d_std, _ = sim.runTest() # last one is PHE std
aae_2ds_aae.append((aae_2d_aae, aae_2d_std))
aae_2ds_phe.append(aae_2d_phe)
# Fetching test points
t, p, p3d = sim.test_points, sim.te_pupil_locations, sim.te_3d_pupil_locations
t = map(lambda target: v(target) - v(sim.scene_camera.t), t) # target coords in scene CCS
# 3D3D
t_3d3d = t[:]
ni = map(lambda p: v(v(p)-v(sim.sclera_pos)).norm(), p3d) # ground truth gaze vectors
# transforming pupil pose to eye camera CS
ni = map(lambda r: v(eyeR.dot(np.array(r))), ni)
# applying estimated rotation to pose vector in eye camera coordinates (Rn)
# R is estimated rotation between scene camera and eye coordinate system (not eye camera!)
# in other words, R is the rotation part of e
Rni = map(lambda n: v(R.dot(np.array(n))), ni) # now ready to compare Rn with t-e
# Intersecting gaze rays originating from the eye with the planes defined by each
# target. then we can simply compute angular error between each intersection and
# the corresponding 3D target
gis = map(lambda vec: v(vec), Rni) # gaze rays originating from eyeball
# we multiply g such that it hits t's z-plane i.e. multiply all coordinates by factor (t.z-e.z)/g.z
# then we add e to the final g so that it originates from scene camera. now both g and t are in the
# same coordinate system and originate from the same point, so we can compare them
gprimes = map(lambda tg: v(((tg[0].z - e3d3d.z)/tg[1].z)*tg[1] + e3d3d), zip(t_3d3d, gis))
AE = list(np.degrees(np.arctan((v(p[0]).cross(p[1])/(v(p[0]).dot(p[1]))).mag)) for p in zip(gprimes, t_3d3d))
N = len(t)
AAE = np.mean(AE)
STD = np.std(AE)
m, M = min(AE), max(AE)
aae_3D3Ds.append((AAE, STD))
qi = map(_q, p) # computing feature vectors from raw pupil coordinates in 2D
# computing unit gaze vectors corresponding to pupil positions
# here we use the computed mapping matrix w
gis = map(lambda q: g(q, w), qi)
# Intersecting gaze rays originating from the eye with the planes defined by each
# target. then we can simply compute angular error between each intersection and
# the corresponding 3D target
t = map(lambda vec: v(vec), t)
gis = map(lambda vec: v(vec), gis)
gprimes = map(lambda tg: v(((tg[0].z - e.z)/tg[1].z)*tg[1] + e), zip(t, gis))
AE = list(np.degrees(np.arctan((v(p[0]).cross(p[1])/(v(p[0]).dot(p[1]))).mag)) for p in zip(gprimes, t))
N = len(t)
AAE = np.mean(AE)
STD = np.std(AE)
m, M = min(AE), max(AE)
# Computing physical distance error (in meters)
PHE = list((u-v).mag/1000 for u,v in zip(t, gprimes))
N = len(t)
APHE = np.mean(PHE)
PHE_STD = np.std(PHE)
PHE_m, PHE_M = min(PHE), max(PHE)
aae_3ds_aae.append((AAE, STD))
aae_3ds_phe.append((PHE, PHE_STD))
# results only contains AAE
results.append([np.mean(np.array(aae_2ds_aae)[:,0]), np.mean(np.array(aae_3ds_aae)[:,0]), np.mean(np.array(aae_3D3Ds)[:,0])])
results_std.append([np.std(np.array(aae_2ds_aae)[:,0]), np.std(np.array(aae_3ds_aae)[:,0]), np.std(np.array(aae_3D3Ds)[:,0])])
# Old plot code
######################################################################################################
# plt.ylabel('Angular Error')
# plt.xlabel('Depth')
# fig = plt.figure(figsize=(14.0, 10.0))
# ax = fig.add_subplot(111)
# clrs = ['b', 'r', 'orange']
# _xrange = [0.5,1.5,2.5,3.5,4.5]
# ax.plot(_xrange, [res[0] for res in results], 'r', label='2D-to-2D', marker="o", linestyle='-',lw=3)
# ax.plot(_xrange, [res[1] for res in results], 'b', label='2D-to-3D', marker="o", linestyle='-',lw=3)
# ax.plot(_xrange, [res[2] for res in results], 'g', label='3D-to-3D', marker="o", linestyle='-',lw=3)
# ax.set_ylabel(r'Angular Error',fontsize=22, fontweight='bold')
# ax.set_xlabel(r'Number of Calibration Depths',fontsize=22, fontweight='bold')
# plt.legend(fontsize=20)
# # plt.legend(loc="upper left", ncol=3, title=r"$d_c$")
# # plt.legend(bbox_to_anchor=(0., 1.02, 1., .102), loc=3,
# # ncol=5, mode="expand", borderaxespad=0., fontsize=20)
# # plt.xticks(fontsize='18')
# plt.yticks(fontsize='18')
# TOPICs = [0.0,0.5,1.5,2.5,3.5,4.5,5.0]
# LABELs = ['', '1', '2', '3', '4', '5', '']
# # ax.set_xticklabels(LABELs,fontsize=18)
# plt.xticks(TOPICs, LABELs,fontsize=18)
# left = 0.1 # the left side of the subplots of the figure
# right = 0.975 # the right side of the subplots of the figure
# bottom = 0.075 # the bottom of the subplots of the figure
# top = 0.925 # the top of the subplots of the figure
# wspace = 0.2 # the amount of width reserved for blank space between subplots
# hspace = 0.4 # the amount of height reserved for white space between subplots
# plt.subplots_adjust(left=left, bottom=bottom, right=right, top=top, wspace=wspace, hspace=hspace)
# plt.show()
######################################################################################################
# New plot code based on EffectNumberofClusters.py
mean2D2D = [res[0] for res in results]
mean2D3D = [res[1] for res in results]
mean3D3D = [res[2] for res in results]
std2D2D = [res[0] for res in results_std]
std2D3D = [res[1] for res in results_std]
std3D3D = [res[2] for res in results_std]
N = 5
ind = np.asarray([0.25,1.25,2.25,3.25,4.25])
width = 0.5 # the width of the bars
# x1 = [0.4,1.4,2.4,3.4,4.4]
x2 = [0.45,1.45,2.45,3.45,4.45]
# x3 = [0.5,1.5,2.5,3.5,4.5]
x4 = [0.55,1.55,2.55,3.55,4.55]
# x5 = [0.6,1.6,2.6,3.6,4.6]
x6 = [0.50,1.50,2.50,3.50,4.50]
fig = plt.figure(figsize=(14.0, 10.0))
ax = fig.add_subplot(111)
# print mean2D2D
# print mean2D3D
# ax.axhline(linewidth=2, y = np.mean(mean2D2D),color='r')
# ax.axhline(linewidth=2, y = np.mean(mean2D3D),color='blue')
# ax.axhline(linewidth=2, y = minvaluevalue,color='black')
# ax.text(0.98, Participantmeanvalue+0.5, "Mean %.2f" % Participantmeanvalue,fontsize=12, fontweight='bold',color='r')
# ax.text(0.98, maxvaluevalue+0.5, "Maximum %.2f" % maxvaluevalue,fontsize=12, fontweight='bold',color='black')
# ax.text(0.98, minvaluevalue+0.5, "Minimum %.2f" % minvaluevalue,fontsize=12, fontweight='bold', color='black')
# rects1 = ax.bar(ind, Participantmean,width, color='r',edgecolor='black',)#, hatch='//')
rects1 = ax.errorbar(x2, mean2D2D,yerr=[std2D2D,std2D2D],fmt='o',color='red',ecolor='red',lw=3, capsize=5, capthick=2)
plt.plot(x2, mean2D2D, marker="o", linestyle='-',lw=3,color='red',label = r'2D-to-2D')
rects2 =ax.errorbar(x4, mean2D3D,yerr=[std2D3D,std2D3D],fmt='o',color='blue',ecolor='blue',lw=3, capsize=5, capthick=2)
plt.plot(x4, mean2D3D, marker="o", linestyle='-',lw=3,color='blue', label = r'2D-to-3D')
rects3 =ax.errorbar(x6, mean3D3D,yerr=[std3D3D,std3D3D],fmt='o',color='blue',ecolor='blue',lw=3, capsize=5, capthick=2)
plt.plot(x6, mean3D3D, marker="o", linestyle='-',lw=3,color='blue', label = r'3D-to-3D')
legend(fontsize=20,loc='upper right')
# rects3 = ax.errorbar(x3, meanC3,yerr=[stdC3,stdC3],fmt='o',color='black',ecolor='black',lw=3, capsize=5, capthick=2)
# plt.plot(x3, meanC3, marker="o", linestyle='-',lw=3,color='black')
#
# rects4 =ax.errorbar(x4, meanC4,yerr=[stdC4,stdC4],fmt='o',color='green',ecolor='green',lw=3, capsize=5, capthick=2)
# plt.plot(x4, meanC4, marker="o", linestyle='-',lw=3,color='green')
#
# rects5 =ax.errorbar(x5, meanC5,yerr=[stdC5,stdC5],fmt='o',color='orange',ecolor='orange',lw=3, capsize=5, capthick=2)
# plt.plot(x5, meanC5, marker="o", linestyle='-',lw=3,color='orange')
ax.set_ylabel(r'Angular Error',fontsize=22)
ax.set_xlabel(r'Number of Calibration Depths',fontsize=22)
ax.set_xticks(ind+0.25)
ax.set_xticklabels( ('D1', 'D2', 'D3','D4', 'D5') ,fontsize=18)
TOPICs = [0.0,0.5,1.5,2.5,3.5,4.5,5.0]#,110]#,120]
print TOPICs
LABELs = ["",r'1',r'2', r'3', r'4', r'5', ""]#, ""]#, ""]
# fig.canvas.set_window_title('Distance Error Correlation')
plt.xticks(TOPICs, LABELs,fontsize=18)
# legend([rects1,rects2], [r'\LARGE\textbf{2D2D}', r'\LARGE\textbf{2D3D}'], loc='lower right')
TOPICS = [0.5,1,1.5,2,2.5,3,3.5,4,4.5,5]#,110]#,120]
print TOPICS
LABELS = [r'0.5', r'1',r'1.5', r'2',r'2.5', r'3',r'3.5', r'4',r'4.5',r'5']#, ""]#, ""]
# fig.canvas.set_window_title('Accuracy - Activity Statistics')
plt.yticks(TOPICS, LABELS,fontsize=18)
def autolabel(rects):
# attach some text labels
for rect in rects:
height = rect.get_height()
ax.text(0.26+rect.get_x()+rect.get_width()/2., height +0.35, "%.2f"%float(height),
ha='center', va='bottom',fontweight='bold',fontsize=13.5)
# autolabel(rects1)
left = 0.1 # the left side of the subplots of the figure
right = 0.975 # the right side of the subplots of the figure
bottom = 0.075 # the bottom of the subplots of the figure
top = 0.925 # the top of the subplots of the figure
wspace = 0.2 # the amount of width reserved for blank space between subplots
hspace = 0.4 # the amount of height reserved for white space between subplots
plt.subplots_adjust(left=left, bottom=bottom, right=right, top=top, wspace=wspace, hspace=hspace)
plt.show()
######################################################################################################
class Parallax3Dto3DMapping(Experiment): # GT pupil pose instead of estimating the pose
'''
'''
def __run__(self):
sim = GazeSimulation(log = False)
sim.place_eyeball_on_scene_camera = False
sim.setEyeRelativeToSceneCamera(v(-65, -33, -73))
# sim.setEyeRelativeToSceneCamera(v(-65, -33, 0)) # assuming eyeball and scene camera are coplanar i.e. e = (e.x, e.y, 0)
sim.setCalibrationDepth(1 * 1000) # mm, wrt scene camera
sim.setTestDepth(1.5 * 1000)
sim.calibration_grid = True
sim.calibration_random_depth = False
sim.test_grid = True
sim.test_random_depth = False
sim.test_random_fixed_depth = False
cdepths = [1, 1.5, 2.0]
tdepths = np.linspace(1, 2, 5)
results = []
for i, cdepth in enumerate(cdepths):
sim.setCalibrationDepth(cdepth * 1000)
# Performing calibration
sim.runCalibration()
aae_2Ds = []
aae_3Ds = []
aae_3D3Ds = []
std_2Ds = []
std_3Ds = []
std_3D3Ds = []
# Fetching calibration points
t, p3d, p2d = sim.calibration_points, sim.tr_3d_pupil_locations, sim.tr_pupil_locations
# target positions are computed relative to the scene CCS
ti = map(lambda target: v(target) - v(sim.scene_camera.t), t)
# Computing pupil pose for each gaze
ni = map(lambda p: (v(p)-v(sim.sclera_pos)).norm(), p3d) # ground truth gaze vectors
eye_scene_diff = v(sim.sclera_pos) - v(sim.scene_camera.t)
e = np.array(eye_scene_diff) # eyeball coords in scene CCS
e_org = v(e[:])
# transforming pose vector to eye camera coordinates
eyeR = np.array(sim.eye_camera.R[:3]) # result is identical to cv2.Rodrigues(np.array([0., np.pi, 0.]))[0]
ni = map(lambda pose: eyeR.dot(np.array(pose)), ni)
R, e3d3d = minimizeEnergy(ni, ti, pose_given=True)
e3d3d = v(e3d3d)
# R = LA.inv(R)
w2d3d, e2d3d, w0 = minimizeEnergy(p2d, ti)
e2d3d = v(e2d3d)
print
print R
print 'e3d3d', e3d3d, 'distance =', (e3d3d-e_org).mag
print 'e2d3d', e2d3d, 'distance =', (e2d3d-e_org).mag
print 'real e', e_org
print
for j, tdepth in enumerate(tdepths):
sim.setTestDepth(tdepth * 1000)
aae_2d, _, std_2d, _ = sim.runTest()
aae_2Ds.append(aae_2d)
std_2Ds.append(std_2d)
# Fetching test points
t, p3d, p2d = sim.test_points, sim.te_3d_pupil_locations, sim.te_pupil_locations
ti = map(lambda target: v(target) - v(sim.scene_camera.t), t) # target coords in scene CCS
ni = map(lambda p: v(v(p)-v(sim.sclera_pos)).norm(), p3d) # ground truth gaze vectors
# transforming pose vector to eye camera coordinates
ni = map(lambda r: v(eyeR.dot(np.array(r))), ni)
# applying estimated rotation to pose vector in eye camera coordinates (Rn)
ni = map(lambda n: v(R.dot(np.array(n))), ni) # now ready to compare Rn with t-e
# 3D to 3D
# Intersecting gaze rays originating from the eye with the planes defined by each
# target. then we can simply compute angular error between each intersection and
# the corresponding 3D target
gis = map(lambda vec: v(vec), ni)
gprimes = map(lambda tg: v(((tg[0].z - e3d3d.z)/tg[1].z)*tg[1] + e3d3d), zip(ti, gis))
AE = list(np.degrees(np.arctan((v(p[0]).cross(p[1])/(v(p[0]).dot(p[1]))).mag)) for p in zip(gprimes, ti))
# AE = list(np.degrees(np.arccos(v(p[0]).dot(p[1])/v(p[0]).mag/v(p[1]).mag)) for p in zip(gprimes, ti))
N = len(ti)
AAE = np.mean(AE)
STD_3D3D = np.std(AE)
m, M = min(AE), max(AE)
aae_3D3Ds.append(AAE)
std_3D3Ds.append(STD_3D3D)
# 2D to 3D
qi = map(_q, p2d) # computing feature vectors from raw pupil coordinates in 2D
# computing unit gaze vectors corresponding to pupil positions
# here we use the computed mapping matrix w
gis = map(lambda q: gaze_ray(q, w2d3d), qi)
# Intersecting gaze rays originating from the eye with the planes defined by each
# target. then we can simply compute angular error between each intersection and
# the corresponding 3D target
gis = map(lambda vec: v(vec), gis)
gprimes = map(lambda tg: v(((tg[0].z - e2d3d.z)/tg[1].z)*tg[1] + e2d3d), zip(ti, gis))
AE = list(np.degrees(np.arctan((v(p[0]).cross(p[1])/(v(p[0]).dot(p[1]))).mag)) for p in zip(gprimes, ti))
# AE = list(np.degrees(np.arccos(v(p[0]).dot(p[1])/v(p[0]).mag/v(p[1]).mag)) for p in zip(gprimes, ti))
N = len(t)
AAE = np.mean(AE)
STD_2D3D = np.std(AE)
m, M = min(AE), max(AE)
# Computing physical distance error (in meters)
PHE = list((u-v).mag for u,v in zip(ti, gprimes))
N = len(ti)
APHE = np.mean(PHE)
PHE_STD = np.std(PHE)
PHE_m, PHE_M = min(PHE), max(PHE)
# aae_3Ds.append((AAE, STD, PHE, PHE_STD))
aae_3Ds.append(AAE)
std_3Ds.append(STD_2D3D)
# break
print 'depth', cdepth, 'finished.'
results.append([aae_2Ds, aae_3Ds, aae_3D3Ds, std_2Ds, std_3Ds, std_3D3Ds])
clrs = ['r', 'g', 'b', 'k', 'o']
colors = ['blue', 'orange', 'red', 'black', 'orange']
patches = []
fig = plt.figure(figsize=(14.0, 10.0))
ax = fig.add_subplot(111)
dmap = {0: '1', 1:'3', 2:'5'}
x1 = [0.375,1.375,2.375,3.375,4.375]
x2 = [0.425,1.425,2.425,3.425,4.425]
x3 = [0.475,1.475,2.475,3.475,4.475]
x4 = [0.525,1.525,2.525,3.525,4.525]
x5 = [0.575,1.575,2.575,3.575,4.575]
x6 = [0.625,1.625,2.625,3.625,4.625]
xrange_2d2d = [x1, x3, x5]
xrange_2d3d = [x2, x4, x6]
for i in [0, 1, 2]:
cdepth_results = results[i]
_xrange = xrange_2d2d[i]
aae_2d2d = cdepth_results[0]
std_2d2d = cdepth_results[3]
rects1 = ax.errorbar(_xrange, aae_2d2d,yerr=[std_2d2d, std_2d2d],fmt='o',color=colors[i],ecolor=colors[i],lw=3, capsize=5, capthick=2)
plt.plot(_xrange, aae_2d2d, marker="o", linestyle='-',lw=3,color=colors[i],label = '2D-to-2D Calibration Depth ' + dmap[i])
for i in [0, 1, 2]:
cdepth_results = results[i]
_xrange = xrange_2d3d[i]
aae_2d3d = cdepth_results[1]
std_2d3d = cdepth_results[4]
rects2 = ax.errorbar(_xrange, aae_2d3d,yerr=[std_2d3d, std_2d3d],fmt='o',color=colors[i],ecolor=colors[i],lw=3, capsize=5, capthick=2)
plt.plot(_xrange, aae_2d3d, marker="o", linestyle='--',lw=3,color=colors[i],label = '2D-to-3D Calibration Depth ' + dmap[i])
for i in [0, 1, 2]:
cdepth_results = results[i]
_xrange = xrange_2d2d[i]
aae_3d3d = cdepth_results[2]
std_3d3d = cdepth_results[5]
rects3 = ax.errorbar(_xrange, aae_3d3d,yerr=[std_3d3d, std_3d3d],fmt='o',color=colors[i],ecolor=colors[i],lw=3, capsize=5, capthick=2)
plt.plot(_xrange, aae_3d3d, marker="o", linestyle='-.',lw=3,color=colors[i],label = '3D-to-3D Calibration Depth ' + dmap[i])
ax.set_ylabel(r'\textbf{Angular Error}',fontsize=22)
ax.set_xlabel(r'\textbf{Depth}',fontsize=22)
# ax.set_ylim((0, 2.4))
TOPICS = [-0.2, 0, 0.2, 0.4,0.6,0.8,1.0,1.2,1.4,1.6,1.8,2.0,2.2,2.4]#,110]#,120]
LABELS = [r'', r'0', r'0.2',r'0.4',r'0.6', r'0.8', r'1.0', r'1.2', r'1.4', r'1.6', r'1.8', r'2.0', r'2.2', r'2.4']#, ""]#, ""]
plt.yticks(TOPICS, LABELS,fontsize=18)
plt.xlabel('Depth')
plt.ylabel('Angular Error')
plt.legend(bbox_to_anchor=(0., 1.02, 1., .102), loc=3,
ncol=3, mode="expand", borderaxespad=0., fontsize=18)
TOPICs = [0.0,0.5,1.5,2.5,3.5,4.5,5.0]
LABELs = ['', 'D1 - 1m', 'D2 - 1.25m', 'D3 - 1.5m', 'D4 - 1.75m', 'D5 - 2.0m', '']
plt.xticks(TOPICs, LABELs,fontsize=18)
left = 0.1 # the left side of the subplots of the figure
right = 0.975 # the right side of the subplots of the figure
bottom = 0.075 # the bottom of the subplots of the figure
top = 0.925 # the top of the subplots of the figure
wspace = 0.2 # the amount of width reserved for blank space between subplots
hspace = 0.4 # the amount of height reserved for white space between subplots
plt.subplots_adjust(left=left, bottom=bottom, right=right, top=top, wspace=wspace, hspace=hspace)
plt.show()
# self.sim = sim
ROOT_DATA_DIR = '/home/mmbrian/HiWi/etra2016_mohsen/code/recording/data/participants'
class Parallax2Dto2DRealData(Experiment):
'''
First it runs single calibration depth vs test depth over all combinations of depths per participant (25 cases)
Then we perform estimation with data from multiple calibration depths for each test depth
'''
def __run__(self):
sim = GazeSimulation(log = False)
camera_matrix, dist_coeffs = readCameraParams()
root_result_path = 'results/2d2d/'
if not os.path.exists(root_result_path):
os.makedirs(root_result_path)
errors = []
for d1 in os.listdir(ROOT_DATA_DIR):
if d1.startswith('p'): # every participant
if not d1 in PARTICIPANTS:
continue
participant_label = d1
participant_results = []
d2 = os.path.join(ROOT_DATA_DIR, d1) # .../pi/
d2 = os.path.join(d2, os.listdir(d2)[0]) # .../pi/../
print '> Processing participant', d1
participant_experiment = {}
for d3 in os.listdir(d2): # every recording
d4 = os.path.join(d2, d3) # .../pi/../00X/
intervals_dir = os.path.join(d4, 'gaze_intervals.npy')
if not os.path.isfile(intervals_dir):
# Participant not completely processed
print '> One or more recordings need processing...'
break
else:
p = np.load(os.path.join(d4, 'p.npy'))
break_participant = False
for point in p:
if np.isnan(point[0]):
# For this gaze position, not input with nonzero confidence exist
break_participant = True
break
if break_participant:
print '> One or more recordings miss gaze coords...'
break
# print p
t2d = np.load(os.path.join(d4, 't2d.npy'))
t3d = np.load(os.path.join(d4, 't3d.npy'))
participant_experiment[d3] = [p, t2d, t3d]
if len(participant_experiment) == 10:
print '> All recordings processed'
keys = sorted(participant_experiment.keys())
depths = zip(keys[::2], keys[1::2])
for calib_depth in depths:
cp, ct = participant_experiment[calib_depth[0]][0], participant_experiment[calib_depth[0]][1]
cdepth_value = getDepth(calib_depth)
# Perform calibration
sim.perform2D2DCalibrationOnReadData(cp, ct, (1280, 720))
for test_depth in depths:
tdepth_value = getDepth(test_depth)
tp, tt, tt3d = participant_experiment[test_depth[1]][0], participant_experiment[test_depth[1]][1], participant_experiment[test_depth[1]][2]
error = sim.run2D2DTestOnRealData(tp, tt, (1280, 720), tt3d, camera_matrix, dist_coeffs)
participant_results.append([cdepth_value, tdepth_value] + error)
print len(participant_results), 'combinations processed...'
np.save('results/2d2d/%s_2d2d_all.npy' % participant_label, np.array(participant_results))
np.savetxt('results/2d2d/%s_2d2d_all.csv' % participant_label, np.array(participant_results), delimiter=",")
print '> Computing results for multiple calibration depths...'
for num_of_calibration_depths in xrange(2, 6): # from 2 calibration depths to 5
participant_results = []
print '> Computing results for combining %s calibration depths...' % num_of_calibration_depths
for calibs in combinations(depths, num_of_calibration_depths):
# Now calibs is a set of depths, from each of those we need calibration data
cp, ct = [], []
calib_depths_label = []
for calib in calibs:
if len(cp):
cp = np.concatenate((cp, participant_experiment[calib[0]][0]), axis=0)
ct = np.concatenate((ct, participant_experiment[calib[0]][1]), axis=0)
else:
cp = participant_experiment[calib[0]][0]
ct = participant_experiment[calib[0]][1]
calib_depths_label.append(getDepth(calib))
# Perform calibration
sim.perform2D2DCalibrationOnReadData(cp, ct, (1280, 720))
# Now we have calibration data from multiple depths, we can test on all depths
for test_depth in depths:
tdepth_value = getDepth(test_depth)
tp, tt, tt3d = participant_experiment[test_depth[1]][0], participant_experiment[test_depth[1]][1], participant_experiment[test_depth[1]][2]
error = sim.run2D2DTestOnRealData(tp, tt, (1280, 720), tt3d, camera_matrix, dist_coeffs)
participant_results.append(calib_depths_label + [tdepth_value] + error)
print len(participant_results), 'combinations processed...'
result_path = 'results/2d2d/%s_calibration_depths/' % num_of_calibration_depths
if not os.path.exists(result_path):
os.makedirs(result_path)
np.save(result_path + '%s.npy' % participant_label, np.array(participant_results))
np.savetxt(result_path + '%s.csv' % participant_label, np.array(participant_results), delimiter=",")
print 'done.'
# plt.plot(tdrange, aaes)
# plt.show()
def getDepth(depth_experiments):
_map = {'000': 1, '002': 1.25, '004': 1.5, '006': 1.75, '008': 2.0}
return _map[depth_experiments[0]]
class Parallax2Dto3DRealData(Experiment):
def __run__(self):
sim = GazeSimulation(log = False)
aae_3ds = []
root_result_path = '/home/mmbrian/3D_Gaze_Tracking/work/results/2D3D/'
if not os.path.exists(root_result_path):
os.makedirs(root_result_path)
for d1 in os.listdir(ROOT_DATA_DIR):
if d1.startswith('p'): # every participant
if not d1 in PARTICIPANTS:
continue
participant_label = d1
participant_results = []
d2 = os.path.join(ROOT_DATA_DIR, d1) # .../pi/
d2 = os.path.join(d2, os.listdir(d2)[0]) # .../pi/../
print '> Processing participant', d1
participant_experiment = {}
for d3 in os.listdir(d2): # every recording
d4 = os.path.join(d2, d3) # .../pi/../00X/
intervals_dir = os.path.join(d4, 'gaze_intervals.npy')
if not os.path.isfile(intervals_dir):
# Participant not completely processed
print '> One or more recordings need processing...'
break
else:
p = np.load(os.path.join(d4, 'p.npy'))
# p = np.load(os.path.join(d4, 'p_mean.npy'))
break_participant = False
for point in p:
if np.isnan(point[0]):
# For this gaze position, not input with nonzero confidence exist
break_participant = True
break
if break_participant:
print '> One or more recordings miss gaze coords...'
break
# print p
t2d = np.load(os.path.join(d4, 't2d.npy'))
t3d = np.load(os.path.join(d4, 't3d.npy'))
# t2d = np.load(os.path.join(d4, 't2d_mean.npy'))
# t3d = np.load(os.path.join(d4, 't3d_mean.npy'))
participant_experiment[d3] = [p, t2d, t3d]
if len(participant_experiment) == 10:
print '> All recordings processed'
keys = sorted(participant_experiment.keys())
depths = zip(keys[::2], keys[1::2])
for calib_depth in depths:
cp, ct3d = participant_experiment[calib_depth[0]][0], participant_experiment[calib_depth[0]][2]
cdepth_value = getDepth(calib_depth)
# Performing calibration
w, e, w0 = minimizeEnergy(cp, ct3d)
e = v(e)
for test_depth in depths:
tdepth_value = getDepth(test_depth)
tp, tt3d = participant_experiment[test_depth[1]][0], participant_experiment[test_depth[1]][2]
t, p = tt3d, tp
qi = map(_q, p) # computing feature vectors from raw pupil coordinates in 2D
# computing unit gaze vectors corresponding to pupil positions
# here we use the computed mapping matrix w
gis = map(lambda q: g(q, w), qi)
# Intersecting gaze rays originating from the eye with the planes defined by each
# target. then we can simply compute angular error between each intersection and
# the corresponding 3D target
t = map(lambda vec: v(vec), t)
gis = map(lambda vec: v(vec), gis)
gprimes = map(lambda tg: v(((tg[0].z - e.z)/tg[1].z)*tg[1] + e), zip(t, gis))
AE = list(np.degrees(np.arctan((v(p[0]).cross(p[1])/(v(p[0]).dot(p[1]))).mag)) for p in zip(gprimes, t))
N = len(t)
AAE = sum(AE)/N
VAR = sum((ae - AAE)**2 for ae in AE)/N
STD = np.sqrt(VAR)
m, M = min(AE), max(AE)
# Computing physical distance error (in meters)
PHE = list((u-v).mag for u,v in zip(t, gprimes))
N = len(t)
APHE = sum(PHE)/N
PHE_VAR = sum((phe - APHE)**2 for phe in PHE)/N
PHE_STD = np.sqrt(VAR)
PHE_m, PHE_M = min(PHE), max(PHE)
participant_results.append([cdepth_value, tdepth_value] + [AAE, VAR, STD, m, M, APHE, PHE_VAR, PHE_STD, PHE_m, PHE_M])
print len(participant_results), 'combinations processed...'
np.save(os.path.join(root_result_path, '%s_2d3d_all.npy' % participant_label), np.array(participant_results))
np.savetxt(os.path.join(root_result_path, '%s_2d3d_all.csv' % participant_label), np.array(participant_results), delimiter=",")
print '> Computing results for multiple calibration depths...'
for num_of_calibration_depths in xrange(2, 6): # from 2 calibration depths to 5
participant_results = []
print '> Computing results for combining %s calibration depths...' % num_of_calibration_depths
for calibs in combinations(depths, num_of_calibration_depths):
# Now calibs is a set of depths, from each of those we need calibration data
cp, ct3d = [], []
calib_depths_label = []
for calib in calibs:
if len(cp):
cp = np.concatenate((cp, participant_experiment[calib[0]][0]), axis=0)
ct3d = np.concatenate((ct3d, participant_experiment[calib[0]][2]), axis=0)
else:
cp = participant_experiment[calib[0]][0]
ct3d = participant_experiment[calib[0]][2]
calib_depths_label.append(getDepth(calib))
# Performing calibration
w, e, w0 = minimizeEnergy(cp, ct3d)
e = v(e)
# Now we have calibration data from multiple depths, we can test on all depths
for test_depth in depths:
tdepth_value = getDepth(test_depth)
tp, tt3d = participant_experiment[test_depth[1]][0], participant_experiment[test_depth[1]][2]
t, p = tt3d, tp
qi = map(_q, p)
gis = map(lambda q: g(q, w), qi)
t = map(lambda vec: v(vec), t)
gis = map(lambda vec: v(vec), gis)
gprimes = map(lambda tg: v(((tg[0].z - e.z)/tg[1].z)*tg[1] + e), zip(t, gis))
AE = list(np.degrees(np.arctan((v(p[0]).cross(p[1])/(v(p[0]).dot(p[1]))).mag)) for p in zip(gprimes, t))
N = len(t)
AAE = sum(AE)/N
VAR = sum((ae - AAE)**2 for ae in AE)/N
STD = np.sqrt(VAR)
m, M = min(AE), max(AE)
# Computing physical distance error (in meters)
PHE = list((u-v).mag for u,v in zip(t, gprimes))
N = len(t)
APHE = sum(PHE)/N
PHE_VAR = sum((phe - APHE)**2 for phe in PHE)/N
PHE_STD = np.sqrt(VAR)
PHE_m, PHE_M = min(PHE), max(PHE)
participant_results.append(calib_depths_label + [tdepth_value] + [AAE, VAR, STD, m, M, APHE, PHE_VAR, PHE_STD, PHE_m, PHE_M])
print len(participant_results), 'combinations processed...'
result_path = os.path.join(root_result_path, '%s_calibration_depths/' % num_of_calibration_depths)
if not os.path.exists(result_path):
os.makedirs(result_path)
np.save(result_path + '%s.npy' % participant_label, np.array(participant_results))
np.savetxt(result_path + '%s.csv' % participant_label, np.array(participant_results), delimiter=",")
print 'done.'
class Parallax3Dto3DRealData(Experiment):
def __run__(self):
sim = GazeSimulation(log = False)
aae_3ds = []
root_result_path = '/home/mmbrian/3D_Gaze_Tracking/work/results/3D3D/'
root_pose_path = '/home/mmbrian/3D_Gaze_Tracking/work/Marker_Eye_Images/ImagesUndist/'
root_data_path = '/home/mmbrian/HiWi/etra2016_mohsen/code/recording/data/participants/'
take_only_nearest_neighbor_for_calibration = True
participants = ['p14']
if not os.path.exists(root_result_path):
os.makedirs(root_result_path)
for d1 in os.listdir(root_data_path):
if d1.startswith('p'): # every participant
if not d1 in participants:
# if not d1 in PARTICIPANTS:
continue
participant_label = d1
participant_results = []
d2 = os.path.join(root_data_path, d1) # .../pi/
d2 = os.path.join(d2, os.listdir(d2)[0]) # .../pi/../
print '> Processing participant', d1
participant_experiment = {}
for d3 in os.listdir(d2): # every recording
d4 = os.path.join(d2, d3) # .../pi/../00X/
# pose_info = np.loadtxt(open(os.path.join(root_pose_path+d1+'/'+d3, "null_pupils.csv"),"rb"),delimiter=";")
pose_info = np.loadtxt(open(os.path.join(root_pose_path+d1+'/'+d3, "simple_pupils.csv"),"rb"),delimiter=";")
frames_numbers = pose_info[:, 0]
pose_estimates = pose_info[:,4:7]
pose_info = dict(zip(frames_numbers, pose_estimates))
p_frames = np.load(os.path.join(d4, 'p_frames.npy'))
# print d4
# Fetching pose information for every target
poses = []
for target in p_frames:
pose = []
for fn in target: # all frames corresponding to this pupil
# first fn corresponds to the nearest neighbor
# for test use all correspondents from these 3 or 2 estimates
# i.e. each pose-marker creates a correspondence so 3*16=48 correspondents for test
# for calibration compare two cases, one similar to above take all the 75 correspondents
# and the other taking only the pose corresponding to nearest neighbor which results in
# the same number of correspondents as target markers
try:
pose.append(pose_info[fn])
except KeyError, err:
print err
poses.append(pose)
t2d = np.load(os.path.join(d4, 't2d.npy'))
t3d = np.load(os.path.join(d4, 't3d.npy'))
participant_experiment[d3] = [poses, t2d, t3d]
keys = sorted(participant_experiment.keys())
depths = zip(keys[::2], keys[1::2])
for calib_depth in depths:
pose_data, ct3d = participant_experiment[calib_depth[0]][0], participant_experiment[calib_depth[0]][2]
cdepth_value = getDepth(calib_depth)
if take_only_nearest_neighbor_for_calibration:
pose = np.array(list(p[0] for p in pose_data))
calib_3ds = ct3d[:]
else:
calib_3ds = []
pose = []
for i, p3d in enumerate(ct3d):
for p in pose_data[i]:
pose.append(p)
calib_3ds.append(p3d)
# Performing calibration
# First we convert gaze rays to actual pupil pose in our right hand coordinate system
# _pose = [(np.arctan(g.x/g.z), np.arctan(g.y/g.z)) for g in map(v, pose)]
_pose = map(v, pose)
print '> Running tests for calibration depth', cdepth_value
if any(g.z == 0 for g in _pose):
print 'Calibration is flawed'
# print pose
else:
print 'Calibration data is okay'
# print [g.mag for g in map(v, pose)]
# w, e, w0 = minimizeEnergy(_pose, calib_3ds, pose_given=True)
R, e = minimizeEnergy(pose, calib_3ds, pose_given=True)
# R = LA.inv(R)
print 'R', R
print 'e', e
e = v(e)
for test_depth in depths:
tdepth_value = getDepth(test_depth)
tpose_data, tt3d = participant_experiment[test_depth[1]][0], participant_experiment[test_depth[1]][2]
test_3ds = []
tpose = []
for i, p3d in enumerate(tt3d):
for p in tpose_data[i]:
tpose.append(p)
test_3ds.append(p3d)
# applying estimated rotation to bring pose vectors to scene camera coordinates
tpose = map(lambda p: v(R.dot(np.array(p))), tpose)
if any(g.z == 0 for g in map(v, tpose)):
print 'Test depth', tdepth_value, 'is flawed'
gis = map(lambda vec: v(vec), tpose)
t = map(lambda vec: v(vec), test_3ds)
gprimes = map(lambda tg: v(((tg[0].z - e.z)/tg[1].z)*tg[1] + e), zip(t, gis))
# AE = list(np.degrees(np.arctan((v(p[0]).cross(p[1])/(v(p[0]).dot(p[1]))).mag)) for p in zip(gprimes, t))
AE = list(np.degrees(np.arccos(v(p[0]).dot(p[1])/v(p[0]).mag/v(p[1]).mag)) for p in zip(gprimes, t))
AAE = np.mean(AE)
STD = np.std(AE)
m, M = min(AE), max(AE)
# Computing physical distance error (in meters)
PHE = list((u-v).mag for u,v in zip(t, gprimes))
APHE = np.mean(PHE)
PHE_STD = np.std(PHE)
PHE_m, PHE_M = min(PHE), max(PHE)
print 'Calibration', cdepth_value, 'Test', tdepth_value, AAE, 'degrees', APHE, 'meters'
participant_results.append([cdepth_value, tdepth_value] + [AAE, STD, m, M, APHE, PHE_STD, PHE_m, PHE_M])
print len(participant_results), 'combinations processed...'
np.save(os.path.join(root_result_path, '%s_3d3d_all.npy' % participant_label), np.array(participant_results))
np.savetxt(os.path.join(root_result_path, '%s_3d3d_all.csv' % participant_label), np.array(participant_results), delimiter=",")
def main():
if len(sys.argv) <= 1:
print 'Please select a mode.'
return
mode = sys.argv[1]
if mode == 'pts':
# Performs an experiment by fixing calibration depth and testing for different test depths
# to investigate parallax error in 2D to 2D mapping
ex = Parallax2Dto2DMapping()
ex.performExperiment()
if mode == '2d3d':
# This also does 3D gaze estimation and plots estimation results for both 3D and 2D estimation
ex = Parallax2Dto3DMapping()
ex.performExperiment()
# ex = Parallax2Dto3DMappingEye()
# ex.performExperiment()i
if mode == '2d2d_2d3d':
ex = Parallax3Dto3DMapping()
ex.performExperiment()
# ex = Parallax2Dto2DRealData()
# ex.performExperiment()
# ex = Parallax2Dto3DRealData()
# ex.performExperiment()
if mode == '3d3d_real':
ex = Parallax3Dto3DRealData()
ex.performExperiment()
if __name__ == '__main__':
main()

0
code/pupil/__init__.py Normal file
View file

255
code/pupil/calibrate.py Normal file
View file

@ -0,0 +1,255 @@
'''
(*)~----------------------------------------------------------------------------------
Pupil - eye tracking platform
Copyright (C) 2012-2015 Pupil Labs
Distributed under the terms of the CC BY-NC-SA License.
License details are in the file license.txt, distributed as part of this software.
----------------------------------------------------------------------------------~(*)
'''
import numpy as np
#logging
import logging
logger = logging.getLogger(__name__)
def get_map_from_cloud(cal_pt_cloud,screen_size=(2,2),threshold = 35,return_inlier_map=False,return_params=False):
"""
we do a simple two pass fitting to a pair of bi-variate polynomials
return the function to map vector
"""
# fit once using all avaiable data
model_n = 7
cx,cy,err_x,err_y = fit_poly_surface(cal_pt_cloud,model_n)
err_dist,err_mean,err_rms = fit_error_screen(err_x,err_y,screen_size)
if cal_pt_cloud[err_dist<=threshold].shape[0]: #did not disregard all points..
# fit again disregarding extreme outliers
cx,cy,new_err_x,new_err_y = fit_poly_surface(cal_pt_cloud[err_dist<=threshold],model_n)
map_fn = make_map_function(cx,cy,model_n)
new_err_dist,new_err_mean,new_err_rms = fit_error_screen(new_err_x,new_err_y,screen_size)
logger.info('first iteration. root-mean-square residuals: %s, in pixel' %err_rms)
logger.info('second iteration: ignoring outliers. root-mean-square residuals: %s in pixel',new_err_rms)
logger.info('used %i data points out of the full dataset %i: subset is %i percent' \
%(cal_pt_cloud[err_dist<=threshold].shape[0], cal_pt_cloud.shape[0], \
100*float(cal_pt_cloud[err_dist<=threshold].shape[0])/cal_pt_cloud.shape[0]))
if return_inlier_map and return_params:
return map_fn,err_dist<=threshold,(cx,cy,model_n)
if return_inlier_map and not return_params:
return map_fn,err_dist<=threshold
if return_params and not return_inlier_map:
return map_fn,(cx,cy,model_n)
return map_fn
else: # did disregard all points. The data cannot be represented by the model in a meaningful way:
map_fn = make_map_function(cx,cy,model_n)
logger.info('First iteration. root-mean-square residuals: %s in pixel, this is bad!'%err_rms)
logger.warning('The data cannot be represented by the model in a meaningfull way.')
if return_inlier_map and return_params:
return map_fn,err_dist<=threshold,(cx,cy,model_n)
if return_inlier_map and not return_params:
return map_fn,err_dist<=threshold
if return_params and not return_inlier_map:
return map_fn,(cx,cy,model_n)
return map_fn
def fit_poly_surface(cal_pt_cloud,n=7):
M = make_model(cal_pt_cloud,n)
U,w,Vt = np.linalg.svd(M[:,:n],full_matrices=0)
V = Vt.transpose()
Ut = U.transpose()
pseudINV = np.dot(V, np.dot(np.diag(1/w), Ut))
cx = np.dot(pseudINV, M[:,n])
cy = np.dot(pseudINV, M[:,n+1])
# compute model error in world screen units if screen_res specified
err_x=(np.dot(M[:,:n],cx)-M[:,n])
err_y=(np.dot(M[:,:n],cy)-M[:,n+1])
return cx,cy,err_x,err_y
def fit_error_screen(err_x,err_y,(screen_x,screen_y)):
err_x *= screen_x/2.
err_y *= screen_y/2.
err_dist=np.sqrt(err_x*err_x + err_y*err_y)
err_mean=np.sum(err_dist)/len(err_dist)
err_rms=np.sqrt(np.sum(err_dist*err_dist)/len(err_dist))
return err_dist,err_mean,err_rms
def make_model(cal_pt_cloud,n=7):
n_points = cal_pt_cloud.shape[0]
if n==3:
X=cal_pt_cloud[:,0]
Y=cal_pt_cloud[:,1]
Ones=np.ones(n_points)
ZX=cal_pt_cloud[:,2]
ZY=cal_pt_cloud[:,3]
M=np.array([X,Y,Ones,ZX,ZY]).transpose()
elif n==7:
X=cal_pt_cloud[:,0]
Y=cal_pt_cloud[:,1]
XX=X*X
YY=Y*Y
XY=X*Y
XXYY=XX*YY
Ones=np.ones(n_points)
ZX=cal_pt_cloud[:,2]
ZY=cal_pt_cloud[:,3]
M=np.array([X,Y,XX,YY,XY,XXYY,Ones,ZX,ZY]).transpose()
elif n==9:
X=cal_pt_cloud[:,0]
Y=cal_pt_cloud[:,1]
XX=X*X
YY=Y*Y
XY=X*Y
XXYY=XX*YY
XXY=XX*Y
YYX=YY*X
Ones=np.ones(n_points)
ZX=cal_pt_cloud[:,2]
ZY=cal_pt_cloud[:,3]
M=np.array([X,Y,XX,YY,XY,XXYY,XXY,YYX,Ones,ZX,ZY]).transpose()
else:
raise Exception("ERROR: Model n needs to be 3, 7 or 9")
return M
def make_map_function(cx,cy,n):
if n==3:
def fn((X,Y)):
x2 = cx[0]*X + cx[1]*Y +cx[2]
y2 = cy[0]*X + cy[1]*Y +cy[2]
return x2,y2
elif n==7:
def fn((X,Y)):
x2 = cx[0]*X + cx[1]*Y + cx[2]*X*X + cx[3]*Y*Y + cx[4]*X*Y + cx[5]*Y*Y*X*X +cx[6]
y2 = cy[0]*X + cy[1]*Y + cy[2]*X*X + cy[3]*Y*Y + cy[4]*X*Y + cy[5]*Y*Y*X*X +cy[6]
return x2,y2
elif n==9:
def fn((X,Y)):
# X Y XX YY XY XXYY XXY YYX Ones
x2 = cx[0]*X + cx[1]*Y + cx[2]*X*X + cx[3]*Y*Y + cx[4]*X*Y + cx[5]*Y*Y*X*X + cx[6]*Y*X*X + cx[7]*Y*Y*X + cx[8]
y2 = cy[0]*X + cy[1]*Y + cy[2]*X*X + cy[3]*Y*Y + cy[4]*X*Y + cy[5]*Y*Y*X*X + cy[6]*Y*X*X + cy[7]*Y*Y*X + cy[8]
return x2,y2
else:
raise Exception("ERROR: Model n needs to be 3, 7 or 9")
return fn
def preprocess_data(pupil_pts,ref_pts):
'''small utility function to deal with timestamped but uncorrelated data
input must be lists that contain dicts with at least "timestamp" and "norm_pos"
'''
cal_data = []
if len(ref_pts)<=2:
return cal_data
cur_ref_pt = ref_pts.pop(0)
next_ref_pt = ref_pts.pop(0)
while True:
matched = []
while pupil_pts:
#select all points past the half-way point between current and next ref data sample
if pupil_pts[0]['timestamp'] <=(cur_ref_pt['timestamp']+next_ref_pt['timestamp'])/2.:
matched.append(pupil_pts.pop(0))
else:
for p_pt in matched:
#only use close points
if abs(p_pt['timestamp']-cur_ref_pt['timestamp']) <= 1/15.: #assuming 30fps + slack
data_pt = p_pt["norm_pos"][0], p_pt["norm_pos"][1],cur_ref_pt['norm_pos'][0],cur_ref_pt['norm_pos'][1]
cal_data.append(data_pt)
break
if ref_pts:
cur_ref_pt = next_ref_pt
next_ref_pt = ref_pts.pop(0)
else:
break
return cal_data
# if __name__ == '__main__':
# import matplotlib.pyplot as plt
# from matplotlib import cm
# from mpl_toolkits.mplot3d import Axes3D
# cal_pt_cloud = np.load('cal_pt_cloud.npy')
# # plot input data
# # Z = cal_pt_cloud
# # ax.scatter(Z[:,0],Z[:,1],Z[:,2], c= "r")
# # ax.scatter(Z[:,0],Z[:,1],Z[:,3], c= "b")
# # fit once
# model_n = 7
# cx,cy,err_x,err_y = fit_poly_surface(cal_pt_cloud,model_n)
# map_fn = make_map_function(cx,cy,model_n)
# err_dist,err_mean,err_rms = fit_error_screen(err_x,err_y,(1280,720))
# print err_rms,"in pixel"
# threshold =15 # err_rms*2
# # fit again disregarding crass outlines
# cx,cy,new_err_x,new_err_y = fit_poly_surface(cal_pt_cloud[err_dist<=threshold],model_n)
# map_fn = make_map_function(cx,cy,model_n)
# new_err_dist,new_err_mean,new_err_rms = fit_error_screen(new_err_x,new_err_y,(1280,720))
# print new_err_rms,"in pixel"
# print "using %i datapoints out of the full dataset %i: subset is %i percent" \
# %(cal_pt_cloud[err_dist<=threshold].shape[0], cal_pt_cloud.shape[0], \
# 100*float(cal_pt_cloud[err_dist<=threshold].shape[0])/cal_pt_cloud.shape[0])
# # plot residuals
# fig_error = plt.figure()
# plt.scatter(err_x,err_y,c="y")
# plt.scatter(new_err_x,new_err_y)
# plt.title("fitting residuals full data set (y) and better subset (b)")
# # plot projection of eye and world vs observed data
# X,Y,ZX,ZY = cal_pt_cloud.transpose().copy()
# X,Y = map_fn((X,Y))
# X *= 1280/2.
# Y *= 720/2.
# ZX *= 1280/2.
# ZY *= 720/2.
# fig_projection = plt.figure()
# plt.scatter(X,Y)
# plt.scatter(ZX,ZY,c='y')
# plt.title("world space projection in pixes, mapped and observed (y)")
# # plot the fitting functions 3D plot
# fig = plt.figure()
# ax = fig.gca(projection='3d')
# outliers =cal_pt_cloud[err_dist>threshold]
# inliers = cal_pt_cloud[err_dist<=threshold]
# ax.scatter(outliers[:,0],outliers[:,1],outliers[:,2], c= "y")
# ax.scatter(outliers[:,0],outliers[:,1],outliers[:,3], c= "y")
# ax.scatter(inliers[:,0],inliers[:,1],inliers[:,2], c= "r")
# ax.scatter(inliers[:,0],inliers[:,1],inliers[:,3], c= "b")
# Z = cal_pt_cloud
# X = np.linspace(min(Z[:,0])-.2,max(Z[:,0])+.2,num=30,endpoint=True)
# Y = np.linspace(min(Z[:,1])-.2,max(Z[:,1]+.2),num=30,endpoint=True)
# X, Y = np.meshgrid(X,Y)
# ZX,ZY = map_fn((X,Y))
# ax.plot_surface(X, Y, ZX, rstride=1, cstride=1, linewidth=.1, antialiased=True,alpha=0.4,color='r')
# ax.plot_surface(X, Y, ZY, rstride=1, cstride=1, linewidth=.1, antialiased=True,alpha=0.4,color='b')
# plt.xlabel("Pupil x in Eye-Space")
# plt.ylabel("Pupil y Eye-Space")
# plt.title("Z: Gaze x (blue) Gaze y (red) World-Space, yellow=outliers")
# # X,Y,_,_ = cal_pt_cloud.transpose()
# # pts= map_fn((X,Y))
# # import cv2
# # pts = np.array(pts,dtype=np.float32).transpose()
# # print cv2.convexHull(pts)[:,0]
# plt.show()

65
code/pupil/file_methods.py Normal file
View file

@ -0,0 +1,65 @@
'''
(*)~----------------------------------------------------------------------------------
Pupil - eye tracking platform
Copyright (C) 2012-2015 Pupil Labs
Distributed under the terms of the CC BY-NC-SA License.
License details are in the file license.txt, distributed as part of this software.
----------------------------------------------------------------------------------~(*)
'''
import cPickle as pickle
import os
import logging
logger = logging.getLogger(__name__)
class Persistent_Dict(dict):
"""a dict class that uses pickle to save inself to file"""
def __init__(self, file_path):
super(Persistent_Dict, self).__init__()
self.file_path = os.path.expanduser(file_path)
try:
with open(self.file_path,'rb') as fh:
try:
self.update(pickle.load(fh))
except: #KeyError,EOFError
logger.warning("Session settings file '%s'could not be read. Will overwrite on exit."%self.file_path)
except IOError:
logger.debug("Session settings file '%s' not found. Will make new one on exit."%self.file_path)
def save(self):
d = {}
d.update(self)
try:
with open(self.file_path,'wb') as fh:
pickle.dump(d,fh,-1)
except IOError:
logger.warning("Could not save session settings to '%s'"%self.file_path)
def close(self):
self.save()
def load_object(file_path):
file_path = os.path.expanduser(file_path)
with open(file_path,'rb') as fh:
return pickle.load(fh)
def save_object(object,file_path):
file_path = os.path.expanduser(file_path)
with open(file_path,'wb') as fh:
pickle.dump(object,fh,-1)
if __name__ == '__main__':
logging.basicConfig(level=logging.DEBUG)
# settings = Persistent_Dict("~/Desktop/test")
# settings['f'] = "this is a test"
# settings['list'] = ["list 1","list2"]
# settings.close()
# save_object("string",'test')
# print load_object('test')
settings = Persistent_Dict('~/Desktop/pupil_settings/user_settings_eye')
print settings['roi']

661
code/pupil/methods.py Normal file
View file

@ -0,0 +1,661 @@
'''
(*)~----------------------------------------------------------------------------------
Pupil - eye tracking platform
Copyright (C) 2012-2015 Pupil Labs
Distributed under the terms of the CC BY-NC-SA License.
License details are in the file license.txt, distributed as part of this software.
----------------------------------------------------------------------------------~(*)
'''
import numpy as np
try:
import numexpr as ne
except:
ne = None
import cv2
import logging
logger = logging.getLogger(__name__)
class Roi(object):
"""this is a simple 2D Region of Interest class
it is applied on numpy arrays for convenient slicing
like this:
roi_array_slice = full_array[r.view]
# do something with roi_array_slice
this creates a view, no data copying done
"""
def __init__(self, array_shape):
self.array_shape = array_shape
self.lX = 0
self.lY = 0
self.uX = array_shape[1]
self.uY = array_shape[0]
self.nX = 0
self.nY = 0
@property
def view(self):
return slice(self.lY,self.uY,),slice(self.lX,self.uX)
@view.setter
def view(self, value):
raise Exception('The view field is read-only. Use the set methods instead')
def add_vector(self,(x,y)):
"""
adds the roi offset to a len2 vector
"""
return (self.lX+x,self.lY+y)
def sub_vector(self,(x,y)):
"""
subs the roi offset to a len2 vector
"""
return (x-self.lX,y-self.lY)
def set(self,vals):
if vals is not None and len(vals) is 5:
if vals[-1] == self.array_shape:
self.lX,self.lY,self.uX,self.uY,_ = vals
else:
logger.info('Image size has changed: Region of Interest has been reset')
elif vals is not None and len(vals) is 4:
self.lX,self.lY,self.uX,self.uY= vals
def get(self):
return self.lX,self.lY,self.uX,self.uY,self.array_shape
def bin_thresholding(image, image_lower=0, image_upper=256):
binary_img = cv2.inRange(image, np.asarray(image_lower),
np.asarray(image_upper))
return binary_img
def make_eye_kernel(inner_size,outer_size):
offset = (outer_size - inner_size)/2
inner_count = inner_size**2
outer_count = outer_size**2-inner_count
val_inner = -1.0 / inner_count
val_outer = -val_inner*inner_count/outer_count
inner = np.ones((inner_size,inner_size),np.float32)*val_inner
kernel = np.ones((outer_size,outer_size),np.float32)*val_outer
kernel[offset:offset+inner_size,offset:offset+inner_size]= inner
return kernel
def dif_gaus(image, lower, upper):
lower, upper = int(lower-1), int(upper-1)
lower = cv2.GaussianBlur(image,ksize=(lower,lower),sigmaX=0)
upper = cv2.GaussianBlur(image,ksize=(upper,upper),sigmaX=0)
# upper +=50
# lower +=50
dif = lower-upper
# dif *= .1
# dif = cv2.medianBlur(dif,3)
# dif = 255-dif
dif = cv2.inRange(dif, np.asarray(200),np.asarray(256))
kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (5,5))
dif = cv2.dilate(dif, kernel, iterations=2)
dif = cv2.erode(dif, kernel, iterations=1)
# dif = cv2.max(image,dif)
# dif = cv2.dilate(dif, kernel, iterations=1)
return dif
def equalize(image, image_lower=0.0, image_upper=255.0):
image_lower = int(image_lower*2)/2
image_lower +=1
image_lower = max(3,image_lower)
mean = cv2.medianBlur(image,255)
image = image - (mean-100)
# kernel = cv2.getStructuringElement(cv2.MORPH_CROSS, (3,3))
# cv2.dilate(image, kernel, image, iterations=1)
return image
def erase_specular(image,lower_threshold=0.0, upper_threshold=150.0):
"""erase_specular: removes specular reflections
within given threshold using a binary mask (hi_mask)
"""
thresh = cv2.inRange(image,
np.asarray(float(lower_threshold)),
np.asarray(256.0))
kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (7,7))
hi_mask = cv2.dilate(thresh, kernel, iterations=2)
specular = cv2.inpaint(image, hi_mask, 2, flags=cv2.INPAINT_TELEA)
# return cv2.max(hi_mask,image)
return specular
def find_hough_circles(img):
circles = cv2.HoughCircles(pupil_img,cv2.cv.CV_HOUGH_GRADIENT,1,20,
param1=50,param2=30,minRadius=0,maxRadius=80)
if circles is not None:
circles = np.uint16(np.around(circles))
for i in circles[0,:]:
# draw the outer circle
cv2.circle(img,(i[0],i[1]),i[2],(0,255,0),2)
# draw the center of the circle
cv2.circle(img,(i[0],i[1]),2,(0,0,255),3)
def chessboard(image, pattern_size=(9,5)):
status, corners = cv2.findChessboardCorners(image, pattern_size, flags=4)
if status:
mean = corners.sum(0)/corners.shape[0]
# mean is [[x,y]]
return mean[0], corners
else:
return None
def curvature(c):
try:
from vector import Vector
except:
return
c = c[:,0]
curvature = []
for i in xrange(len(c)-2):
#find the angle at i+1
frm = Vector(c[i])
at = Vector(c[i+1])
to = Vector(c[i+2])
a = frm -at
b = to -at
angle = a.angle(b)
curvature.append(angle)
return curvature
def GetAnglesPolyline(polyline,closed=False):
"""
see: http://stackoverflow.com/questions/3486172/angle-between-3-points
ported to numpy
returns n-2 signed angles
"""
points = polyline[:,0]
if closed:
a = np.roll(points,1,axis=0)
b = points
c = np.roll(points,-1,axis=0)
else:
a = points[0:-2] # all "a" points
b = points[1:-1] # b
c = points[2:] # c points
# ab = b.x - a.x, b.y - a.y
ab = b-a
# cb = b.x - c.x, b.y - c.y
cb = b-c
# float dot = (ab.x * cb.x + ab.y * cb.y); # dot product
# print 'ab:',ab
# print 'cb:',cb
# float dot = (ab.x * cb.x + ab.y * cb.y) dot product
# dot = np.dot(ab,cb.T) # this is a full matrix mulitplication we only need the diagonal \
# dot = dot.diagonal() # because all we look for are the dotproducts of corresponding vectors (ab[n] and cb[n])
dot = np.sum(ab * cb, axis=1) # or just do the dot product of the correspoing vectors in the first place!
# float cross = (ab.x * cb.y - ab.y * cb.x) cross product
cros = np.cross(ab,cb)
# float alpha = atan2(cross, dot);
alpha = np.arctan2(cros,dot)
return alpha*(180./np.pi) #degrees
# return alpha #radians
# if ne:
# def GetAnglesPolyline(polyline):
# """
# see: http://stackoverflow.com/questions/3486172/angle-between-3-points
# ported to numpy
# returns n-2 signed angles
# same as above but implemented using numexpr
# SLOWER than just numpy!
# """
# points = polyline[:,0]
# a = points[0:-2] # all "a" points
# b = points[1:-1] # b
# c = points[2:] # c points
# ax,ay = a[:,0],a[:,1]
# bx,by = b[:,0],b[:,1]
# cx,cy = c[:,0],c[:,1]
# # abx = '(bx - ax)'
# # aby = '(by - ay)'
# # cbx = '(bx - cx)'
# # cby = '(by - cy)'
# # # float dot = (ab.x * cb.x + ab.y * cb.y) dot product
# # dot = '%s * %s + %s * %s' %(abx,cbx,aby,cby)
# # # float cross = (ab.x * cb.y - ab.y * cb.x) cross product
# # cross = '(%s * %s - %s * %s)' %(abx,cby,aby,cbx)
# # # float alpha = atan2(cross, dot);
# # alpha = "arctan2(%s,%s)" %(cross,dot)
# # term = '%s*%s'%(alpha,180./np.pi)
# term = 'arctan2(((bx - ax) * (by - cy) - (by - ay) * (bx - cx)),(bx - ax) * (bx - cx) + (by - ay) * (by - cy))*57.2957795131'
# return ne.evaluate(term)
def split_at_angle(contour, curvature, angle):
"""
contour is array([[[108, 290]],[[111, 290]]], dtype=int32) shape=(number of points,1,dimension(2) )
curvature is a n-2 list
"""
segments = []
kink_index = [i for i in range(len(curvature)) if curvature[i] < angle]
for s,e in zip([0]+kink_index,kink_index+[None]): # list of slice indecies 0,i0,i1,i2,None
if e is not None:
segments.append(contour[s:e+1]) #need to include the last index
else:
segments.append(contour[s:e])
return segments
def find_kink(curvature, angle):
"""
contour is array([[[108, 290]],[[111, 290]]], dtype=int32) shape=(number of points,1,dimension(2) )
curvature is a n-2 list
"""
kinks = []
kink_index = [i for i in range(len(curvature)) if abs(curvature[i]) < angle]
return kink_index
def find_change_in_general_direction(curvature):
"""
return indecies of where the singn of curvature has flipped
"""
curv_pos = curvature > 0
split = []
currently_pos = curv_pos[0]
for c, is_pos in zip(range(curvature.shape[0]),curv_pos):
if is_pos !=currently_pos:
currently_pos = is_pos
split.append(c)
return split
def find_kink_and_dir_change(curvature,angle):
split = []
if curvature.shape[0] == 0:
return split
curv_pos = curvature > 0
currently_pos = curv_pos[0]
for idx,c, is_pos in zip(range(curvature.shape[0]),curvature,curv_pos):
if (is_pos !=currently_pos) or abs(c) < angle:
currently_pos = is_pos
split.append(idx)
return split
def find_slope_disc(curvature,angle = 15):
# this only makes sense when your polyline is longish
if len(curvature)<4:
return []
i = 2
split_idx = []
for anchor1,anchor2,candidate in zip(curvature,curvature[1:],curvature[2:]):
base_slope = anchor2-anchor1
new_slope = anchor2 - candidate
dif = abs(base_slope-new_slope)
if dif>=angle:
split_idx.add(i)
print i,dif
i +=1
return split_list
def find_slope_disc_test(curvature,angle = 15):
# this only makes sense when your polyline is longish
if len(curvature)<4:
return []
# mean = np.mean(curvature)
# print '------------------- start'
i = 2
split_idx = set()
for anchor1,anchor2,candidate in zip(curvature,curvature[1:],curvature[2:]):
base_slope = anchor2-anchor1
new_slope = anchor2 - candidate
dif = abs(base_slope-new_slope)
if dif>=angle:
split_idx.add(i)
# print i,dif
i +=1
i-= 3
for anchor1,anchor2,candidate in zip(curvature[::-1],curvature[:-1:][::-1],curvature[:-2:][::-1]):
avg = (anchor1+anchor2)/2.
dif = abs(avg-candidate)
if dif>=angle:
split_idx.add(i)
# print i,dif
i -=1
split_list = list(split_idx)
split_list.sort()
# print split_list
# print '-------end'
return split_list
def points_at_corner_index(contour,index):
"""
contour is array([[[108, 290]],[[111, 290]]], dtype=int32) shape=(number of points,1,dimension(2) )
#index n-2 because the curvature is n-2 (1st and last are not exsistent), this shifts the index (0 splits at first knot!)
"""
return [contour[i+1] for i in index]
def split_at_corner_index(contour,index):
"""
contour is array([[[108, 290]],[[111, 290]]], dtype=int32) shape=(number of points,1,dimension(2) )
#index n-2 because the curvature is n-2 (1st and last are not exsistent), this shifts the index (0 splits at first knot!)
"""
segments = []
index = [i+1 for i in index]
for s,e in zip([0]+index,index+[10000000]): # list of slice indecies 0,i0,i1,i2,
segments.append(contour[s:e+1])# +1 is for not loosing line segments
return segments
def convexity_defect(contour, curvature):
"""
contour is array([[[108, 290]],[[111, 290]]], dtype=int32) shape=(number of points,1,dimension(2) )
curvature is a n-2 list
"""
kinks = []
mean = np.mean(curvature)
if mean>0:
kink_index = [i for i in range(len(curvature)) if curvature[i] < 0]
else:
kink_index = [i for i in range(len(curvature)) if curvature[i] > 0]
for s in kink_index: # list of slice indecies 0,i0,i1,i2,None
kinks.append(contour[s+1]) # because the curvature is n-2 (1st and last are not exsistent)
return kinks,kink_index
def is_round(ellipse,ratio,tolerance=.8):
center, (axis1,axis2), angle = ellipse
if axis1 and axis2 and abs( ratio - min(axis2,axis1)/max(axis2,axis1)) < tolerance:
return True
else:
return False
def size_deviation(ellipse,target_size):
center, axis, angle = ellipse
return abs(target_size-max(axis))
def circle_grid(image, pattern_size=(4,11)):
"""Circle grid: finds an assymetric circle pattern
- circle_id: sorted from bottom left to top right (column first)
- If no circle_id is given, then the mean of circle positions is returned approx. center
- If no pattern is detected, function returns None
"""
status, centers = cv2.findCirclesGridDefault(image, pattern_size, flags=cv2.CALIB_CB_ASYMMETRIC_GRID)
if status:
return centers
else:
return None
def calibrate_camera(img_pts, obj_pts, img_size):
# generate pattern size
camera_matrix = np.zeros((3,3))
dist_coef = np.zeros(4)
rms, camera_matrix, dist_coefs, rvecs, tvecs = cv2.calibrateCamera(obj_pts, img_pts,
img_size, camera_matrix, dist_coef)
return camera_matrix, dist_coefs
def gen_pattern_grid(size=(4,11)):
pattern_grid = []
for i in xrange(size[1]):
for j in xrange(size[0]):
pattern_grid.append([(2*j)+i%2,i,0])
return np.asarray(pattern_grid, dtype='f4')
def normalize(pos, (width, height),flip_y=False):
"""
normalize return as float
"""
x = pos[0]
y = pos[1]
x /=float(width)
y /=float(height)
if flip_y:
return x,1-y
return x,y
def denormalize(pos, (width, height), flip_y=False):
"""
denormalize
"""
x = pos[0]
y = pos[1]
x *= width
if flip_y:
y = 1-y
y *= height
return x,y
def dist_pts_ellipse(((ex,ey),(dx,dy),angle),points):
"""
return unsigned euclidian distances of points to ellipse
"""
pts = np.float64(points)
rx,ry = dx/2., dy/2.
angle = (angle/180.)*np.pi
# ex,ey =ex+0.000000001,ey-0.000000001 #hack to make 0 divisions possible this is UGLY!!!
pts = pts - np.array((ex,ey)) # move pts to ellipse appears at origin , with this we copy data -deliberatly!
M_rot = np.mat([[np.cos(angle),-np.sin(angle)],[np.sin(angle),np.cos(angle)]])
pts = np.array(pts*M_rot) #rotate so that ellipse axis align with coordinate system
# print "rotated",pts
pts /= np.array((rx,ry)) #normalize such that ellipse radii=1
# print "normalize",norm_pts
norm_mag = np.sqrt((pts*pts).sum(axis=1))
norm_dist = abs(norm_mag-1) #distance of pt to ellipse in scaled space
# print 'norm_mag',norm_mag
# print 'norm_dist',norm_dist
ratio = (norm_dist)/norm_mag #scale factor to make the pts represent their dist to ellipse
# print 'ratio',ratio
scaled_error = np.transpose(pts.T*ratio) # per vector scalar multiplication: makeing sure that boradcasting is done right
# print "scaled error points", scaled_error
real_error = scaled_error*np.array((rx,ry))
# print "real point",real_error
error_mag = np.sqrt((real_error*real_error).sum(axis=1))
# print 'real_error',error_mag
# print 'result:',error_mag
return error_mag
if ne:
def dist_pts_ellipse(((ex,ey),(dx,dy),angle),points):
"""
return unsigned euclidian distances of points to ellipse
same as above but uses numexpr for 2x speedup
"""
pts = np.float64(points)
pts.shape=(-1,2)
rx,ry = dx/2., dy/2.
angle = (angle/180.)*np.pi
# ex,ey = ex+0.000000001 , ey-0.000000001 #hack to make 0 divisions possible this is UGLY!!!
x = pts[:,0]
y = pts[:,1]
# px = '((x-ex) * cos(angle) + (y-ey) * sin(angle))/rx'
# py = '(-(x-ex) * sin(angle) + (y-ey) * cos(angle))/ry'
# norm_mag = 'sqrt(('+px+')**2+('+py+')**2)'
# norm_dist = 'abs('+norm_mag+'-1)'
# ratio = norm_dist + "/" + norm_mag
# x_err = ''+px+'*'+ratio+'*rx'
# y_err = ''+py+'*'+ratio+'*ry'
# term = 'sqrt(('+x_err+')**2 + ('+y_err+')**2 )'
term = 'sqrt((((x-ex) * cos(angle) + (y-ey) * sin(angle))/rx*abs(sqrt((((x-ex) * cos(angle) + (y-ey) * sin(angle))/rx)**2+((-(x-ex) * sin(angle) + (y-ey) * cos(angle))/ry)**2)-1)/sqrt((((x-ex) * cos(angle) + (y-ey) * sin(angle))/rx)**2+((-(x-ex) * sin(angle) + (y-ey) * cos(angle))/ry)**2)*rx)**2 + ((-(x-ex) * sin(angle) + (y-ey) * cos(angle))/ry*abs(sqrt((((x-ex) * cos(angle) + (y-ey) * sin(angle))/rx)**2+((-(x-ex) * sin(angle) + (y-ey) * cos(angle))/ry)**2)-1)/sqrt((((x-ex) * cos(angle) + (y-ey) * sin(angle))/rx)**2+((-(x-ex) * sin(angle) + (y-ey) * cos(angle))/ry)**2)*ry)**2 )'
error_mag = ne.evaluate(term)
return error_mag
def metric(l):
"""
example metric for search
"""
# print 'evaluating', idecies
global evals
evals +=1
return sum(l) < 3
def pruning_quick_combine(l,fn,seed_idx=None,max_evals=1e20,max_depth=5):
"""
l is a list of object to quick_combine.
the evaluation fn should accept idecies to your list and the list
it should return a binary result on wether this set is good
this search finds all combinations but assumes:
that a bad subset can not be bettered by adding more nodes
that a good set may not always be improved by a 'passing' superset (purging subsets will revoke this)
if all items and their combinations pass the evaluation fn you get n**2 -1 solutions
which leads to (2**n - 1) calls of your evaluation fn
it needs more evaluations than finding strongly connected components in a graph because:
(1,5) and (1,6) and (5,6) may work but (1,5,6) may not pass evaluation, (n,m) being list idx's
"""
if seed_idx:
non_seed_idx = [i for i in range(len(l)) if i not in seed_idx]
else:
#start from every item
seed_idx = range(len(l))
non_seed_idx = []
mapping = seed_idx+non_seed_idx
unknown = [[node] for node in range(len(seed_idx))]
# print mapping
results = []
prune = []
while unknown and max_evals:
path = unknown.pop(0)
max_evals -= 1
# print '@idx',[mapping[i] for i in path]
# print '@content',path
if not len(path) > max_depth:
# is this combination even viable, or did a subset fail already?
if not any(m.issubset(set(path)) for m in prune):
#we have not tested this and a subset of this was sucessfull before
if fn([l[mapping[i]] for i in path]):
# yes this was good, keep as solution
results.append([mapping[i] for i in path])
# lets explore more by creating paths to each remaining node
decedents = [path+[i] for i in range(path[-1]+1,len(mapping)) ]
unknown.extend(decedents)
else:
# print "pruning",path
prune.append(set(path))
return results
# def is_subset(needle,haystack):
# """ Check if needle is ordered subset of haystack in O(n)
# taken from:
# http://stackoverflow.com/questions/1318935/python-list-filtering-remove-subsets-from-list-of-lists
# """
# if len(haystack) < len(needle): return False
# index = 0
# for element in needle:
# try:
# index = haystack.index(element, index) + 1
# except ValueError:
# return False
# else:
# return True
# def filter_subsets(lists):
# """ Given list of lists, return new list of lists without subsets
# taken from:
# http://stackoverflow.com/questions/1318935/python-list-filtering-remove-subsets-from-list-of-lists
# """
# for needle in lists:
# if not any(is_subset(needle, haystack) for haystack in lists
# if needle is not haystack):
# yield needle
def filter_subsets(l):
return [m for i, m in enumerate(l) if not any(set(m).issubset(set(n)) for n in (l[:i] + l[i+1:]))]
if __name__ == '__main__':
# tst = []
# for x in range(10):
# tst.append(gen_pattern_grid())
# tst = np.asarray(tst)
# print tst.shape
#test polyline
# *-* *
# | \ |
# * *-*
# |
# *-*
pl = np.array([[[0, 0]],[[0, 1]],[[1, 1]],[[2, 1]],[[2, 2]],[[1, 3]],[[1, 4]],[[2,4]]], dtype=np.int32)
curvature = GetAnglesPolyline(pl,closed=0)
print curvature
curvature = GetAnglesPolyline(pl,closed=1)
# print curvature
# print find_curv_disc(curvature)
# idx = find_kink_and_dir_change(curvature,60)
# print idx
# print split_at_corner_index(pl,idx)
# ellipse = ((0,0),(np.sqrt(2),np.sqrt(2)),0)
# pts = np.array([(0,1),(.5,.5),(0,-1)])
# # print pts.dtype
# print dist_pts_ellipse(ellipse,pts)
# print pts
# # print test()
# l = [1,2,1,0,1,0]
# print len(l)
# # evals = 0
# # r = quick_combine(l,metric)
# # # print r
# # print filter_subsets(r)
# # print evals
# evals = 0
# r = pruning_quick_combine(l,metric,[2])
# print r
# print filter_subsets(r)
# print evals

157
code/pupil/player_methods.py Normal file
View file

@ -0,0 +1,157 @@
'''
(*)~----------------------------------------------------------------------------------
Pupil - eye tracking platform
Copyright (C) 2012-2015 Pupil Labs
Distributed under the terms of the CC BY-NC-SA License.
License details are in the file license.txt, distributed as part of this software.
----------------------------------------------------------------------------------~(*)
'''
import os
import cv2
import numpy as np
#logging
import logging
logger = logging.getLogger(__name__)
from file_methods import save_object
def correlate_data(data,timestamps):
'''
data: dict of data :
will have at least:
timestamp: float
timestamps: timestamps list to correlate data to
this takes a data list and a timestamps list and makes a new list
with the length of the number of timestamps.
Each slot conains a list that will have 0, 1 or more assosiated data points.
Finnaly we add an index field to the data_point with the assosiated index
'''
timestamps = list(timestamps)
data_by_frame = [[] for i in timestamps]
frame_idx = 0
data_index = 0
while True:
try:
datum = data[data_index]
# we can take the midpoint between two frames in time: More appropriate for SW timestamps
ts = ( timestamps[frame_idx]+timestamps[frame_idx+1] ) / 2.
# or the time of the next frame: More appropriate for Sart Of Exposure Timestamps (HW timestamps).
# ts = timestamps[frame_idx+1]
except IndexError:
# we might loose a data point at the end but we dont care
break
if datum['timestamp'] <= ts:
datum['index'] = frame_idx
data_by_frame[frame_idx].append(datum)
data_index +=1
else:
frame_idx+=1
return data_by_frame
def update_recording_0v4_to_current(rec_dir):
logger.info("Updatig recording from v0.4x format to current version")
gaze_array = np.load(os.path.join(rec_dir,'gaze_positions.npy'))
pupil_array = np.load(os.path.join(rec_dir,'pupil_positions.npy'))
gaze_list = []
pupil_list = []
for datum in pupil_array:
ts, confidence, id, x, y, diameter = datum[:6]
pupil_list.append({'timestamp':ts,'confidence':confidence,'id':id,'norm_pos':[x,y],'diameter':diameter})
pupil_by_ts = dict([(p['timestamp'],p) for p in pupil_list])
for datum in gaze_array:
ts,confidence,x,y, = datum
gaze_list.append({'timestamp':ts,'confidence':confidence,'norm_pos':[x,y],'base':[pupil_by_ts.get(ts,None)]})
pupil_data = {'pupil_positions':pupil_list,'gaze_positions':gaze_list}
try:
save_object(pupil_data,os.path.join(rec_dir, "pupil_data"))
except IOError:
pass
def update_recording_0v3_to_current(rec_dir):
logger.info("Updatig recording from v0.3x format to current version")
pupilgaze_array = np.load(os.path.join(rec_dir,'gaze_positions.npy'))
gaze_list = []
pupil_list = []
for datum in pupilgaze_array:
gaze_x,gaze_y,pupil_x,pupil_y,ts,confidence = datum
#some bogus size and confidence as we did not save it back then
pupil_list.append({'timestamp':ts,'confidence':confidence,'id':0,'norm_pos':[pupil_x,pupil_y],'diameter':50})
gaze_list.append({'timestamp':ts,'confidence':confidence,'norm_pos':[gaze_x,gaze_y],'base':[pupil_list[-1]]})
pupil_data = {'pupil_positions':pupil_list,'gaze_positions':gaze_list}
try:
save_object(pupil_data,os.path.join(rec_dir, "pupil_data"))
except IOError:
pass
def is_pupil_rec_dir(rec_dir):
if not os.path.isdir(rec_dir):
logger.error("No valid dir supplied")
return False
meta_info_path = os.path.join(rec_dir,"info.csv")
try:
with open(meta_info_path) as info:
meta_info = dict( ((line.strip().split('\t')) for line in info.readlines() ) )
info = meta_info["Capture Software Version"]
except:
logger.error("Could not read info.csv file: Not a valid Pupil recording.")
return False
return True
def transparent_circle(img,center,radius,color,thickness):
center = tuple(map(int,center))
rgb = [255*c for c in color[:3]] # convert to 0-255 scale for OpenCV
alpha = color[-1]
radius = int(radius)
if thickness > 0:
pad = radius + 2 + thickness
else:
pad = radius + 3
roi = slice(center[1]-pad,center[1]+pad),slice(center[0]-pad,center[0]+pad)
try:
overlay = img[roi].copy()
cv2.circle(overlay,(pad,pad), radius=radius, color=rgb, thickness=thickness, lineType=cv2.cv.CV_AA)
opacity = alpha
cv2.addWeighted(overlay, opacity, img[roi], 1. - opacity, 0, img[roi])
except:
logger.debug("transparent_circle would have been partially outsize of img. Did not draw it.")
def transparent_image_overlay(pos,overlay_img,img,alpha):
"""
Overlay one image with another with alpha blending
In player this will be used to overlay the eye (as overlay_img) over the world image (img)
Arguments:
pos: (x,y) position of the top left corner in numpy row,column format from top left corner (numpy coord system)
overlay_img: image to overlay
img: destination image
alpha: 0.0-1.0
"""
roi = slice(pos[1],pos[1]+overlay_img.shape[0]),slice(pos[0],pos[0]+overlay_img.shape[1])
try:
cv2.addWeighted(overlay_img,alpha,img[roi],1.-alpha,0,img[roi])
except:
logger.debug("transparent_image_overlay was outside of the world image and was not drawn")
pass

0
code/recording/__init__.py Normal file
View file

BIN
code/recording/aruco_test Normal file
View file

Binary file not shown.

242
code/recording/aruco_test.cpp Normal file
View file

@ -0,0 +1,242 @@
/*****************************************************************************************
Copyright 2011 Rafael Muñoz Salinas. All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are
permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this list of
conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list
of conditions and the following disclaimer in the documentation and/or other materials
provided with the distribution.
THIS SOFTWARE IS PROVIDED BY Rafael Muñoz Salinas ''AS IS'' AND ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL Rafael Muñoz Salinas OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
The views and conclusions contained in the software and documentation are those of the
authors and should not be interpreted as representing official policies, either expressed
or implied, of Rafael Muñoz Salinas.
********************************************************************************************/
#include <iostream>
#include <fstream>
#include <sstream>
#include "aruco.h"
#include "cvdrawingutils.h"
#include <opencv2/highgui/highgui.hpp>
using namespace cv;
using namespace aruco;
string TheInputVideo;
string TheIntrinsicFile;
float TheMarkerSize = -1;
int ThePyrDownLevel;
MarkerDetector MDetector;
VideoCapture TheVideoCapturer;
vector< Marker > TheMarkers;
Mat TheInputImage, TheInputImageCopy;
CameraParameters TheCameraParameters;
void cvTackBarEvents(int pos, void *);
bool readCameraParameters(string TheIntrinsicFile, CameraParameters &CP, Size size);
pair< double, double > AvrgTime(0, 0); // determines the average time required for detection
double ThresParam1, ThresParam2;
int iThresParam1, iThresParam2;
int waitTime = 0;
/************************************
*
*
*
*
************************************/
bool readArguments(int argc, char **argv) {
if (argc < 2) {
cerr << "Invalid number of arguments" << endl;
cerr << "Usage: (in.avi|live[:idx_cam=0]) [intrinsics.yml] [size]" << endl;
return false;
}
TheInputVideo = argv[1];
if (argc >= 3)
TheIntrinsicFile = argv[2];
if (argc >= 4)
TheMarkerSize = atof(argv[3]);
if (argc == 3)
cerr << "NOTE: You need makersize to see 3d info!!!!" << endl;
return true;
}
int findParam(std::string param, int argc, char *argv[]) {
for (int i = 0; i < argc; i++)
if (string(argv[i]) == param)
return i;
return -1;
}
/************************************
*
*
*
*
************************************/
int main(int argc, char **argv) {
try {
if (readArguments(argc, argv) == false) {
return 0;
}
// parse arguments
// read from camera or from file
if (TheInputVideo.find("live") != string::npos) {
int vIdx = 0;
// check if the :idx is here
char cad[100];
if (TheInputVideo.find(":") != string::npos) {
std::replace(TheInputVideo.begin(), TheInputVideo.end(), ':', ' ');
sscanf(TheInputVideo.c_str(), "%s %d", cad, &vIdx);
}
cout << "Opening camera index " << vIdx << endl;
TheVideoCapturer.open(vIdx);
waitTime = 10;
} else
TheVideoCapturer.open(TheInputVideo);
// check video is open
if (!TheVideoCapturer.isOpened()) {
cerr << "Could not open video" << endl;
return -1;
}
bool isVideoFile = false;
if (TheInputVideo.find(".avi") != std::string::npos || TheInputVideo.find("live") != string::npos)
isVideoFile = true;
// read first image to get the dimensions
TheVideoCapturer >> TheInputImage;
// read camera parameters if passed
if (TheIntrinsicFile != "") {
TheCameraParameters.readFromXMLFile(TheIntrinsicFile);
TheCameraParameters.resize(TheInputImage.size());
}
// Configure other parameters
if (ThePyrDownLevel > 0)
MDetector.pyrDown(ThePyrDownLevel);
// Create gui
// cv::namedWindow("thres", 1);
// cv::namedWindow("in", 1);
MDetector.setThresholdParams(7, 7);
MDetector.setThresholdParamRange(2, 0);
// MDetector.enableLockedCornersMethod(true);
// MDetector.setCornerRefinementMethod ( MarkerDetector::SUBPIX );
MDetector.getThresholdParams(ThresParam1, ThresParam2);
iThresParam1 = ThresParam1;
iThresParam2 = ThresParam2;
// cv::createTrackbar("ThresParam1", "in", &iThresParam1, 25, cvTackBarEvents);
// cv::createTrackbar("ThresParam2", "in", &iThresParam2, 13, cvTackBarEvents);
char key = 0;
int index = 0;
// capture until press ESC or until the end of the video
TheVideoCapturer.retrieve(TheInputImage);
do {
// copy image
index++; // number of images captured
double tick = (double)getTickCount(); // for checking the speed
// Detection of markers in the image passed
MDetector.detect(TheInputImage, TheMarkers, TheCameraParameters, TheMarkerSize);
// chekc the speed by calculating the mean speed of all iterations
AvrgTime.first += ((double)getTickCount() - tick) / getTickFrequency();
AvrgTime.second++;
cout << "\rTime detection=" << 1000 * AvrgTime.first / AvrgTime.second << " milliseconds nmarkers=" << TheMarkers.size() << std::flush;
// print marker info and draw the markers in image
TheInputImage.copyTo(TheInputImageCopy);
for (unsigned int i = 0; i < TheMarkers.size(); i++) {
cout << endl << TheMarkers[i];
TheMarkers[i].draw(TheInputImageCopy, Scalar(0, 0, 255), 1);
}
if (TheMarkers.size() != 0)
cout << endl;
// print other rectangles that contains no valid markers
/** for (unsigned int i=0;i<MDetector.getCandidates().size();i++) {
aruco::Marker m( MDetector.getCandidates()[i],999);
m.draw(TheInputImageCopy,cv::Scalar(255,0,0));
}*/
// draw a 3d cube in each marker if there is 3d info
if (TheCameraParameters.isValid())
for (unsigned int i = 0; i < TheMarkers.size(); i++) {
CvDrawingUtils::draw3dCube(TheInputImageCopy, TheMarkers[i], TheCameraParameters);
CvDrawingUtils::draw3dAxis(TheInputImageCopy, TheMarkers[i], TheCameraParameters);
}
// DONE! Easy, right?
// show input with augmented information and the thresholded image
// cv::imshow("in", TheInputImageCopy);
// cv::imshow("thres", MDetector.getThresholdedImage());
// key = cv::waitKey(waitTime); // wait for key to be pressed
// key = cv::waitKey(10); // wait for key to be pressed
key = 10;
if (isVideoFile)
TheVideoCapturer.retrieve(TheInputImage);
} while (key != 27 && (TheVideoCapturer.grab() || !isVideoFile));
} catch (std::exception &ex)
{
cout << "Exception :" << ex.what() << endl;
}
}
/************************************
*
*
*
*
************************************/
void cvTackBarEvents(int pos, void *) {
if (iThresParam1 < 3)
iThresParam1 = 3;
if (iThresParam1 % 2 != 1)
iThresParam1++;
if (ThresParam2 < 1)
ThresParam2 = 1;
ThresParam1 = iThresParam1;
ThresParam2 = iThresParam2;
MDetector.setThresholdParams(ThresParam1, ThresParam2);
// recompute
MDetector.detect(TheInputImage, TheMarkers, TheCameraParameters);
TheInputImage.copyTo(TheInputImageCopy);
for (unsigned int i = 0; i < TheMarkers.size(); i++)
TheMarkers[i].draw(TheInputImageCopy, Scalar(0, 0, 255), 1);
// print other rectangles that contains no valid markers
/*for (unsigned int i=0;i<MDetector.getCandidates().size();i++) {
aruco::Marker m( MDetector.getCandidates()[i],999);
m.draw(TheInputImageCopy,cv::Scalar(255,0,0));
}*/
// draw a 3d cube in each marker if there is 3d info
if (TheCameraParameters.isValid())
for (unsigned int i = 0; i < TheMarkers.size(); i++)
CvDrawingUtils::draw3dCube(TheInputImageCopy, TheMarkers[i], TheCameraParameters);
// cv::imshow("in", TheInputImageCopy);
// cv::imshow("thres", MDetector.getThresholdedImage());
}

89
code/recording/collect_images.py Normal file
View file

@ -0,0 +1,89 @@
from __future__ import division
import os, sys, cv2
import numpy as np
import shutil
# PARTICIPANTS = ['p10', 'p16', 'p13', 'p24', 'p5', 'p14', 'p26', 'p12', 'p20', 'p7', 'p15', 'p11', 'p21', 'p25']
# PARTICIPANTS = ['p21']
# ROOT = '/BS/3D_Gaze_Tracking/archive00/participants/'
# DEST = '/BS/3D_Gaze_Tracking/work/Eye_Images/'
DEST = '/home/mmbrian/3D_Gaze_Tracking/work/Marker_Eye_Images_7/'
ROOT = '/home/mmbrian/HiWi/etra2016_mohsen/code/recording/data/participants'
def main():
# distCoeffs = np.load("/BS/3D_Gaze_Tracking/work/dist.npy")
# cameraMatrix = np.load("/BS/3D_Gaze_Tracking/work/cameraMatrix.npy")
distCoeffs = np.load("dist.npy")
cameraMatrix = np.load("cameraMatrix.npy")
PARTICIPANTS = [sys.argv[1]]
processed = 0
for p in os.listdir(ROOT):
if p in PARTICIPANTS:
print '> Collecting images for participant', p
d1 = os.path.join(ROOT, p)
d1 = os.path.join(d1, os.listdir(d1)[0]) # ../p_i/../
for d2 in os.listdir(d1):
path = os.path.join(d1, d2)
print '> Processing', path
processPath(path, p, d2, cameraMatrix, distCoeffs)
processed+=1
print '> Processed %s participants.' % processed
def processPath(path = None, participant = None, experiment = None, cameraMatrix = None, distCoeffs = None):
if not path:
path = sys.argv[1]
raw_images_dir = os.path.join(DEST, 'ImagesRaw')
raw_images_dir = os.path.join(raw_images_dir, participant)
raw_images_dir = os.path.join(raw_images_dir, experiment)
undist_images_dir = os.path.join(DEST, 'ImagesUndist')
undist_images_dir = os.path.join(undist_images_dir, participant)
undist_images_dir = os.path.join(undist_images_dir, experiment)
if not os.path.exists(raw_images_dir):
os.makedirs(raw_images_dir)
else:
print '> Already processed.'
return
if not os.path.exists(undist_images_dir):
os.makedirs(undist_images_dir)
else:
print '> Already processed.'
return
# else:
# print '> Removing old images...'
# shutil.rmtree(raw_images_dir)
# return
# os.makedirs(raw_images_dir)
p_frames = np.load(os.path.join(path, 'p_frames.npy'))
frames = sorted(reduce(lambda l1,l2: list(l1)+list(l2), p_frames))
cap = cv2.VideoCapture(os.path.join(path, 'eye0.mp4'))
fc_eye = int(cap.get(cv2.cv.CV_CAP_PROP_FRAME_COUNT))
fps_eye = int(cap.get(cv2.cv.CV_CAP_PROP_FPS))
eye_frame = 0
world_frame = 0
status, img = cap.read() # extract the first frame
while status:
try:
if eye_frame == frames[world_frame]:
save_dir = os.path.join(raw_images_dir, 'img_%s.png' %(frames[world_frame]))
cv2.imwrite(save_dir, img)
save_dir = os.path.join(undist_images_dir, 'img_%s.png' %(frames[world_frame]))
undistImg = cv2.undistort(img, cameraMatrix, distCoeffs)
cv2.imwrite(save_dir, undistImg)
world_frame+=1
except:
break
eye_frame+=1
status, img = cap.read()
cap.release()
# print '> Removing world video...'
# os.remove(world_video)
print "> Processed %d frames." % (world_frame)
if __name__ == '__main__':
main()

3
code/recording/data/.gitignore vendored Normal file
View file

@ -0,0 +1,3 @@
*.jpg
*.JPG
*.mkv

238
code/recording/data/display_setting.txt Normal file
View file

@ -0,0 +1,238 @@
name of display: :0
version number: 11.0
vendor string: The X.Org Foundation
vendor release number: 11600000
X.Org version: 1.16.0
maximum request size: 16777212 bytes
motion buffer size: 256
bitmap unit, bit order, padding: 32, LSBFirst, 32
image byte order: LSBFirst
number of supported pixmap formats: 7
supported pixmap formats:
depth 1, bits_per_pixel 1, scanline_pad 32
depth 4, bits_per_pixel 8, scanline_pad 32
depth 8, bits_per_pixel 8, scanline_pad 32
depth 15, bits_per_pixel 16, scanline_pad 32
depth 16, bits_per_pixel 16, scanline_pad 32
depth 24, bits_per_pixel 32, scanline_pad 32
depth 32, bits_per_pixel 32, scanline_pad 32
keycode range: minimum 8, maximum 255
focus: window 0x240000b, revert to Parent
number of extensions: 29
BIG-REQUESTS
Composite
DAMAGE
DOUBLE-BUFFER
DPMS
DRI2
DRI3
GLX
Generic Event Extension
MIT-SCREEN-SAVER
MIT-SHM
Present
RANDR
RECORD
RENDER
SECURITY
SGI-GLX
SHAPE
SYNC
X-Resource
XC-MISC
XFIXES
XFree86-DGA
XFree86-VidModeExtension
XINERAMA
XInputExtension
XKEYBOARD
XTEST
XVideo
default screen number: 0
number of screens: 1
screen #0:
dimensions: 3286x1080 pixels (869x286 millimeters)
resolution: 96x96 dots per inch
depths (7): 24, 1, 4, 8, 15, 16, 32
root window id: 0xbd
depth of root window: 24 planes
number of colormaps: minimum 1, maximum 1
default colormap: 0x42
default number of colormap cells: 256
preallocated pixels: black 0, white 16777215
options: backing-store WHEN MAPPED, save-unders NO
largest cursor: 256x256
current input event mask: 0xda4033
KeyPressMask KeyReleaseMask EnterWindowMask
LeaveWindowMask KeymapStateMask StructureNotifyMask
SubstructureNotifyMask SubstructureRedirectMask PropertyChangeMask
ColormapChangeMask
number of visuals: 20
default visual id: 0x40
visual:
visual id: 0x40
class: TrueColor
depth: 24 planes
available colormap entries: 256 per subfield
red, green, blue masks: 0xff0000, 0xff00, 0xff
significant bits in color specification: 8 bits
visual:
visual id: 0x41
class: DirectColor
depth: 24 planes
available colormap entries: 256 per subfield
red, green, blue masks: 0xff0000, 0xff00, 0xff
significant bits in color specification: 8 bits
visual:
visual id: 0xab
class: TrueColor
depth: 24 planes
available colormap entries: 256 per subfield
red, green, blue masks: 0xff0000, 0xff00, 0xff
significant bits in color specification: 8 bits
visual:
visual id: 0xac
class: TrueColor
depth: 24 planes
available colormap entries: 256 per subfield
red, green, blue masks: 0xff0000, 0xff00, 0xff
significant bits in color specification: 8 bits
visual:
visual id: 0xad
class: TrueColor
depth: 24 planes
available colormap entries: 256 per subfield
red, green, blue masks: 0xff0000, 0xff00, 0xff
significant bits in color specification: 8 bits
visual:
visual id: 0xae
class: TrueColor
depth: 24 planes
available colormap entries: 256 per subfield
red, green, blue masks: 0xff0000, 0xff00, 0xff
significant bits in color specification: 8 bits
visual:
visual id: 0xaf
class: TrueColor
depth: 24 planes
available colormap entries: 256 per subfield
red, green, blue masks: 0xff0000, 0xff00, 0xff
significant bits in color specification: 8 bits
visual:
visual id: 0xb0
class: TrueColor
depth: 24 planes
available colormap entries: 256 per subfield
red, green, blue masks: 0xff0000, 0xff00, 0xff
significant bits in color specification: 8 bits
visual:
visual id: 0xb1
class: TrueColor
depth: 24 planes
available colormap entries: 256 per subfield
red, green, blue masks: 0xff0000, 0xff00, 0xff
significant bits in color specification: 8 bits
visual:
visual id: 0xb2
class: TrueColor
depth: 24 planes
available colormap entries: 256 per subfield
red, green, blue masks: 0xff0000, 0xff00, 0xff
significant bits in color specification: 8 bits
visual:
visual id: 0xb3
class: DirectColor
depth: 24 planes
available colormap entries: 256 per subfield
red, green, blue masks: 0xff0000, 0xff00, 0xff
significant bits in color specification: 8 bits
visual:
visual id: 0xb4
class: DirectColor
depth: 24 planes
available colormap entries: 256 per subfield
red, green, blue masks: 0xff0000, 0xff00, 0xff
significant bits in color specification: 8 bits
visual:
visual id: 0xb5
class: DirectColor
depth: 24 planes
available colormap entries: 256 per subfield
red, green, blue masks: 0xff0000, 0xff00, 0xff
significant bits in color specification: 8 bits
visual:
visual id: 0xb6
class: DirectColor
depth: 24 planes
available colormap entries: 256 per subfield
red, green, blue masks: 0xff0000, 0xff00, 0xff
significant bits in color specification: 8 bits
visual:
visual id: 0xb7
class: DirectColor
depth: 24 planes
available colormap entries: 256 per subfield
red, green, blue masks: 0xff0000, 0xff00, 0xff
significant bits in color specification: 8 bits
visual:
visual id: 0xb8
class: DirectColor
depth: 24 planes
available colormap entries: 256 per subfield
red, green, blue masks: 0xff0000, 0xff00, 0xff
significant bits in color specification: 8 bits
visual:
visual id: 0xb9
class: DirectColor
depth: 24 planes
available colormap entries: 256 per subfield
red, green, blue masks: 0xff0000, 0xff00, 0xff
significant bits in color specification: 8 bits
visual:
visual id: 0xba
class: DirectColor
depth: 24 planes
available colormap entries: 256 per subfield
red, green, blue masks: 0xff0000, 0xff00, 0xff
significant bits in color specification: 8 bits
visual:
visual id: 0xbb
class: DirectColor
depth: 24 planes
available colormap entries: 256 per subfield
red, green, blue masks: 0xff0000, 0xff00, 0xff
significant bits in color specification: 8 bits
visual:
visual id: 0x7e
class: TrueColor
depth: 32 planes
available colormap entries: 256 per subfield
red, green, blue masks: 0xff0000, 0xff00, 0xff
significant bits in color specification: 8 bits
Screen 0: minimum 8 x 8, current 3286 x 1080, maximum 32767 x 32767
LVDS1 connected primary 1366x768+0+0 (normal left inverted right x axis y axis) 344mm x 194mm
1366x768 60.0*+
1360x768 59.8 60.0
1024x768 60.0
800x600 60.3 56.2
640x480 59.9
VGA1 connected 1920x1080+1366+0 (normal left inverted right x axis y axis) 533mm x 300mm
1920x1080 60.0*+
1600x1200 60.0
1280x1024 60.0
1152x864 75.0
1024x768 60.0
800x600 60.3
640x480 60.0 59.9
HDMI1 disconnected (normal left inverted right x axis y axis)
DP1 disconnected (normal left inverted right x axis y axis)
VIRTUAL1 disconnected (normal left inverted right x axis y axis)
256 pixels for the marker, 1920px is the width of screen
and as I measurred it is 121cm in width
==> 256./1920 * 121 ~ 16cm is the width of our marker (also measured it, seems legit)

14
code/recording/data/eye_camera/2015_10_03/000/info.csv Normal file
View file

@ -0,0 +1,14 @@
Recording Name 2015_10_03
Start Date 03.10.2015
Start Time 17:48:58
Duration Time 00:03:00
Eye Mode monocular
Duration Time 00:03:00
World Camera Frames 4309
World Camera Resolution 720x1280
Capture Software Version 0.5.7
User mmbrian
Platform Linux
Machine Brian
Release 3.16.0-49-generic
Version #65~14.04.1-Ubuntu SMP Wed Sep 9 10:03:23 UTC 2015
1 Recording Name 2015_10_03
2 Start Date 03.10.2015
3 Start Time 17:48:58
4 Duration Time 00:03:00
5 Eye Mode monocular
6 Duration Time 00:03:00
7 World Camera Frames 4309
8 World Camera Resolution 720x1280
9 Capture Software Version 0.5.7
10 User mmbrian
11 Platform Linux
12 Machine Brian
13 Release 3.16.0-49-generic
14 Version #65~14.04.1-Ubuntu SMP Wed Sep 9 10:03:23 UTC 2015

BIN
code/recording/data/eye_camera/2015_10_03/000/pupil_data Normal file
View file

Binary file not shown.

2
code/recording/data/eye_camera/2015_10_03/000/user_info.csv Normal file
View file

@ -0,0 +1,2 @@
name
additional_field change_me
1 name
2 additional_field change_me

14
code/recording/data/eye_camera/2015_10_03/001/info.csv Normal file
View file

@ -0,0 +1,14 @@
Recording Name 2015_10_03
Start Date 03.10.2015
Start Time 18:06:58
Duration Time 00:03:37
Eye Mode monocular
Duration Time 00:03:37
World Camera Frames 5215
World Camera Resolution 720x1280
Capture Software Version 0.5.7
User mmbrian
Platform Linux
Machine Brian
Release 3.16.0-49-generic
Version #65~14.04.1-Ubuntu SMP Wed Sep 9 10:03:23 UTC 2015
1 Recording Name 2015_10_03
2 Start Date 03.10.2015
3 Start Time 18:06:58
4 Duration Time 00:03:37
5 Eye Mode monocular
6 Duration Time 00:03:37
7 World Camera Frames 5215
8 World Camera Resolution 720x1280
9 Capture Software Version 0.5.7
10 User mmbrian
11 Platform Linux
12 Machine Brian
13 Release 3.16.0-49-generic
14 Version #65~14.04.1-Ubuntu SMP Wed Sep 9 10:03:23 UTC 2015

BIN
code/recording/data/eye_camera/2015_10_03/001/pupil_data Normal file
View file

Binary file not shown.

2
code/recording/data/eye_camera/2015_10_03/001/user_info.csv Normal file
View file

@ -0,0 +1,2 @@
name
additional_field change_me
1 name
2 additional_field change_me

20
code/recording/data/eye_camera/eye_camera_RMS0p39.yml Normal file
View file

@ -0,0 +1,20 @@
%YAML:1.0
calibration_time: "Sat 03 Oct 2015 18:42:34 CEST"
nframes: 30
image_width: 640
image_height: 360
board_width: 8
board_height: 6
square_size: 0.00122
flags: 0
camera_matrix: !!opencv-matrix
rows: 3
cols: 3
dt: d
data: [ 678.75284504, 0. , 301.29715044, 0. ,
667.94939515, 209.10259404, 0. , 0. , 1. ]
distortion_coefficients: !!opencv-matrix
rows: 5
cols: 1
dt: d
data: [ 0.12812696, -0.13076179, 0.00631552, -0.01349366, -2.10210424]

9
code/recording/data/frames_005/frame_300.txt Normal file
View file

@ -0,0 +1,9 @@
18=(376.883,493.993) (367.418,432.916) (429.433,424.199) (438.674,484.834) Txyz=-0.138299 0.0442204 0.497911 Rxyz=-3.10292 0.221729 -0.0411571
130=(557.896,467.496) (549.296,407.464) (609.848,398.895) (617.985,458.804) Txyz=-0.0245979 0.0282039 0.504007 Rxyz=-3.08427 0.211876 -0.115737
301=(748.172,549.641) (740.68,491.47) (799.57,482.841) (806.625,541.024) Txyz=0.0988437 0.0832783 0.516703 Rxyz=-3.05697 0.208073 -0.108889
351=(573.305,576.144) (564.956,517.414) (624.868,508.643) (632.845,566.939) Txyz=-0.0150596 0.0992645 0.509626 Rxyz=-3.04795 0.206556 -0.108844
399=(720.388,332.456) (712.982,271.43) (773.556,263.268) (780.206,324.203) Txyz=0.0793988 -0.0583554 0.504764 Rxyz=-3.05028 0.207139 -0.10664
456=(542.036,356.791) (533.264,295.4) (594.619,287.303) (602.898,348.62) Txyz=-0.0341846 -0.0425535 0.498523 Rxyz=-3.05347 0.204274 -0.0319069
608=(358.957,381.543) (348.602,319.251) (411.724,311.228) (421.632,373.302) Txyz=-0.147483 -0.0265521 0.491548 Rxyz=-3.05284 0.204884 -0.0436
659=(734.247,442.123) (726.628,382.526) (786.131,374.246) (793.406,433.659) Txyz=0.0889698 0.0122942 0.510193 Rxyz=-3.07904 0.209142 -0.116808
707=(394.177,604.361) (384.954,544.505) (446.419,535.311) (455.177,594.624) Txyz=-0.129612 0.116083 0.506384 Rxyz=-3.09125 0.216361 -0.0913303

10
code/recording/data/frames_006/frame_300.txt Normal file
View file

@ -0,0 +1,10 @@
18=(337.292,468.882) (324.79,404.901) (390.651,394.692) (402.499,457.982) Txyz=-0.154857 0.0256425 0.473357 Rxyz=-3.04908 0.244502 -0.076421
130=(527.352,437.35) (516.93,374.763) (580.082,364.783) (589.941,426.958) Txyz=-0.0420971 0.00724924 0.483541 Rxyz=-3.06316 0.24193 -0.165829
301=(725.485,517.338) (716.988,457.961) (777.014,447.674) (785.252,507.095) Txyz=0.0819062 0.0595391 0.502175 Rxyz=-3.02787 0.238118 -0.13582
351=(546.056,549.115) (536.021,488.78) (598.013,478.114) (607.443,538.074) Txyz=-0.0314374 0.0779249 0.492233 Rxyz=-3.02323 0.23531 -0.158536
399=(693.675,293.595) (685.114,229.493) (747.716,220.171) (755.538,283.927) Txyz=0.0601648 -0.081201 0.484901 Rxyz=-3.01722 0.233661 -0.14594
456=(507.989,321.903) (497.092,257.244) (561.524,247.809) (571.784,312.326) Txyz=-0.052872 -0.0630324 0.476038 Rxyz=-3.02846 0.233055 -0.0858767
608=(313.964,351.087) (300.582,284.895) (368.329,275.447) (380.611,341.093) Txyz=-0.165616 -0.0445972 0.465881 Rxyz=-3.02714 0.234292 -0.108089
659=(709.625,407.24) (701.037,345.51) (762.366,335.935) (770.285,397.398) Txyz=0.0707768 -0.0109919 0.492042 Rxyz=-3.03724 0.23253 -0.158296
707=(359.279,582.974) (347.403,521.27) (412.05,510.011) (423.197,571.361) Txyz=-0.145168 0.0967286 0.484355 Rxyz=-3.05352 0.247565 -0.117771

24
code/recording/data/frames_007/frame_300.txt Normal file
View file

@ -0,0 +1,24 @@
173=(699.455,499.98) (691.307,438.567) (752.606,429.954) (760.682,490.932) Txyz=0.0649598 0.047518 0.494322 Rxyz=-3.09118 0.21103 -0.110259
243=(558.056,453.388) (549.028,391.358) (611.729,382.452) (620.144,444.44) Txyz=-0.0231608 0.0178109 0.487122 Rxyz=-3.06591 0.212071 -0.107365
288=(509.858,595.993) (501.039,534.333) (563.524,524.831) (572.148,586.371) Txyz=-0.0533126 0.106812 0.490712 Rxyz=-3.10034 0.213607 -0.101361
325=(690.555,434.231) (682.66,372.562) (744.469,363.995) (751.952,425.615) Txyz=0.0593093 0.0062092 0.491782 Rxyz=-3.06472 0.206759 -0.107539
347=(363.361,550.855) (353.573,487.91) (418.325,478.379) (427.426,540.869) Txyz=-0.141701 0.0769299 0.483147 Rxyz=-3.10662 0.220845 -0.0927639
356=(480.526,396.484) (470.666,333.017) (534.73,324.316) (543.978,387.486) Txyz=-0.0699126 -0.0175944 0.480141 Rxyz=-3.06477 0.215313 -0.0848652
358=(432.44,540.204) (423.087,477.693) (486.636,468.411) (495.612,530.387) Txyz=-0.10047 0.0711499 0.486519 Rxyz=-3.1106 0.220745 -0.0922127
392=(442.289,606.858) (433.197,544.676) (496.376,535.089) (505.121,596.737) Txyz=-0.0950006 0.113034 0.489782 Rxyz=-3.11116 0.21787 -0.102892
399=(708.4,565.16) (700.069,504.329) (761.353,495.298) (769.126,555.978) Txyz=0.0705462 0.0888017 0.496123 Rxyz=-3.07587 0.210512 -0.114342
449=(548.411,386.917) (539.147,323.647) (602.773,315.279) (611.129,378.093) Txyz=-0.0286112 -0.0233864 0.481884 Rxyz=-3.01791 0.206092 -0.143664
528=(624.563,443.811) (616.147,381.957) (678.347,373.238) (686.266,434.894) Txyz=0.0180922 0.0120028 0.488909 Rxyz=-3.05213 0.209138 -0.118358
554=(490.591,463.222) (481.316,400.801) (544.603,391.939) (553.41,454.083) Txyz=-0.0645677 0.0236915 0.485569 Rxyz=-3.08837 0.214389 -0.108485
559=(373.466,617.896) (364.062,555.286) (428.305,545.501) (437.448,607.679) Txyz=-0.136017 0.118491 0.485214 Rxyz=-3.10462 0.217903 -0.0966003
571=(615.48,377.577) (607.189,314.515) (670.369,305.988) (677.674,368.791) Txyz=0.0125728 -0.0290879 0.482861 Rxyz=-2.98808 0.202386 -0.146367
655=(342.345,415.599) (330.378,351.444) (396.27,342.763) (407.057,406.633) Txyz=-0.151935 -0.00585876 0.474021 Rxyz=-3.0196 0.207689 -0.0499907
660=(500.258,529.687) (491.297,467.655) (554.109,458.558) (562.902,520.323) Txyz=-0.0588932 0.0651044 0.487798 Rxyz=-3.09854 0.214559 -0.0948563
664=(576.63,585.616) (568.056,524.065) (629.927,514.768) (638.399,575.89) Txyz=-0.0118773 0.100667 0.49203 Rxyz=-3.0937 0.211299 -0.126647
735=(352.832,483.454) (343.004,420.029) (407.747,411.093) (417.501,473.927) Txyz=-0.147205 0.0354299 0.480331 Rxyz=3.16704 -0.223919 0.076249
737=(633.714,509.651) (625.26,448.175) (686.896,439.242) (695.163,500.561) Txyz=0.0237445 0.0533222 0.49199 Rxyz=-3.09203 0.212088 -0.117316
782=(642.802,575.179) (634.357,514.125) (695.692,504.957) (704.056,565.739) Txyz=0.0294044 0.0947937 0.494568 Rxyz=-3.08374 0.21306 -0.111104
786=(412.026,405.993) (401.221,342.19) (465.902,333.555) (475.958,397.018) Txyz=-0.11102 -0.011715 0.477259 Rxyz=-3.03908 0.211348 -0.0405065
787=(567.301,519.555) (558.638,457.877) (620.81,448.837) (629.291,510.389) Txyz=-0.0175444 0.0592119 0.489873 Rxyz=-3.09406 0.212163 -0.109216
842=(422.538,473.23) (412.683,410.393) (476.666,401.431) (486.064,463.924) Txyz=-0.10576 0.0295403 0.482561 Rxyz=-3.08552 0.217952 -0.0668197
914=(682.094,368.247) (674.609,305.389) (737.422,296.877) (743.864,359.666) Txyz=0.0536882 -0.0348577 0.485769 Rxyz=-2.99698 0.201008 -0.128513

24
code/recording/data/frames_010/frame_300.txt Normal file
View file

@ -0,0 +1,24 @@
173=(723.927,447.07) (715.643,384.723) (778.576,375.426) (786.258,437.702) Txyz=0.0790166 0.0136108 0.484619 Rxyz=-3.04246 0.219174 -0.091972
243=(579.827,400.658) (570.222,337.215) (634.345,327.765) (643.039,391.221) Txyz=-0.00925742 -0.0150964 0.476483 Rxyz=-3.00322 0.218584 -0.10575
288=(532.679,544.107) (522.959,482.482) (585.978,472.648) (595.254,534.363) Txyz=-0.0388098 0.0736001 0.484942 Rxyz=-3.05784 0.224629 -0.0772962
325=(715.033,380.428) (706.95,317.043) (770.634,307.64) (777.861,370.973) Txyz=0.0731347 -0.0274384 0.480342 Rxyz=-3.00927 0.218851 -0.0908707
347=(384.833,499.337) (373.829,436.216) (438.669,426.422) (449.139,489.416) Txyz=-0.127388 0.0447468 0.477125 Rxyz=-3.06431 0.229277 -0.0535486
356=(500.555,342.84) (489.461,277.411) (554.996,268.146) (565.07,333.476) Txyz=-0.0562687 -0.0497284 0.468424 Rxyz=-2.98365 0.21573 -0.033857
358=(454.078,488.67) (443.545,425.754) (507.685,416.119) (517.61,478.697) Txyz=-0.0861319 0.0385554 0.479532 Rxyz=-3.06777 0.229777 -0.0678913
392=(465.08,554.994) (454.885,493.098) (518.413,483.204) (527.941,544.889) Txyz=-0.0804108 0.0801372 0.484963 Rxyz=-3.0663 0.226716 -0.0831945
399=(732.737,512.68) (724.51,451.377) (786.84,442.078) (794.647,503.35) Txyz=0.0850416 0.0548219 0.489455 Rxyz=-3.04315 0.219029 -0.0818131
449=(569.515,332.769) (559.638,267.491) (624.982,258.069) (633.689,323.424) Txyz=-0.0150215 -0.0558136 0.469471 Rxyz=-2.95079 0.213833 -0.102914
528=(647.503,390.54) (638.787,327.07) (702.535,317.713) (710.537,381.069) Txyz=0.0319074 -0.0212714 0.47804 Rxyz=-3.00741 0.218382 -0.102638
554=(511.635,410.921) (501.203,347.341) (565.709,337.896) (575.31,401.427) Txyz=-0.0505453 -0.00890131 0.475441 Rxyz=-3.02306 0.221724 -0.0744128
559=(396.416,566.132) (385.832,503.765) (449.873,493.88) (460.227,555.802) Txyz=-0.121709 0.0862522 0.482184 Rxyz=-3.06672 0.23014 -0.062468
571=(638.16,322.645) (629.505,257.342) (694.553,247.905) (701.927,313.41) Txyz=0.0261123 -0.0619344 0.470728 Rxyz=-2.95023 0.212263 -0.114807
655=(360.505,363.079) (347.015,297.287) (413.74,288.078) (426.117,353.694) Txyz=-0.138416 -0.0372608 0.46466 Rxyz=-2.97141 0.2152 -0.031309
660=(522.304,477.958) (512.311,415.374) (575.931,405.843) (585.329,468.295) Txyz=-0.0447246 0.0322643 0.480759 Rxyz=-3.0578 0.225199 -0.0738372
664=(599.766,533.608) (590.643,472.009) (653.283,462.292) (661.946,523.804) Txyz=0.00257891 0.0672197 0.485662 Rxyz=-3.04987 0.219942 -0.114197
735=(373.089,431.793) (361.365,367.453) (426.782,358.194) (437.982,422.065) Txyz=-0.132966 0.00355214 0.471777 Rxyz=-3.03753 0.22429 -0.0346352
737=(657.115,457.277) (648.23,394.901) (711.199,385.518) (719.5,447.765) Txyz=0.0378741 0.0197911 0.483178 Rxyz=-3.04552 0.22143 -0.0968667
782=(666.4,523.065) (657.721,461.573) (720.035,452.19) (728.332,513.475) Txyz=0.0438073 0.0610251 0.487713 Rxyz=-3.04853 0.219505 -0.0967276
786=(430.999,352.987) (418.778,287.389) (484.649,278.121) (495.913,343.557) Txyz=-0.0973854 -0.0435011 0.466652 Rxyz=-2.98462 0.217406 -0.0168059
787=(589.799,467.54) (580.54,405.102) (643.78,395.603) (652.536,457.897) Txyz=-0.00336534 0.0259621 0.481289 Rxyz=-3.03955 0.220231 -0.11357
842=(442.786,421.342) (431.874,357.423) (496.653,347.995) (507.008,411.669) Txyz=-0.0918841 -0.0026965 0.474376 Rxyz=-3.05417 0.227679 -0.0500495
914=(706.368,312.641) (699.076,247.258) (764.04,237.726) (770.078,303.24) Txyz=0.0671587 -0.0679941 0.472159 Rxyz=-2.93732 0.21098 -0.108571

2
code/recording/data/participants/.gitignore vendored Normal file
View file

@ -0,0 +1,2 @@
/p*
/Hosna

15
code/recording/data/participants/notes.txt Normal file
View file

@ -0,0 +1,15 @@
- for participants P1 to P11, eyetracker had to be readjusted during the experiment so that pupil could still be tracked with high confidence. eyetracker was rarely readjusted between two recordings at the same depth, mostly it was in between two depths. as a consequence, usage of different depth calibrations for a test recording would have another source of error caused by slighly repositioning the cameras.
- p12 to p26 > no readjusting the tracker
- p23 > a minor adjustment for better pupil tracking confidence from 2nd depth
- p25 is myself. my head was slightly looking down or up in between recordings so even for a fixed depth the data could result in large errors! (not so useful). it's quite hard to do better all by myself.
- for p6, 001 is splited into 001 and 002 but the movement from test point 15 to 16 is lost, so from 001 only extact data for the first 15 points, and extract point 16 from the entire 002
- for p9, depth 0, calibration video, the screen is a bit outside of the screen from left, therefore at least two marker points are lost (cannot be tracked by ArUco). it might be better not to use the data from this depth or at least not use the left edge of the grids.
- p1 to p11 + p23 had minor adjustments to eye camera (and in terms scene camera) in between recordings
- p12 to p22 + p24 and p26 didn't have any camera adjustments in between recordings
- p25 is data recorded from myself
- p6 and p9 are special cases to be handled separately
- data for Hosna is not useful as the eye camera wasn't robustly tracking the eye (less than 20% frames with nonzero confidence)

14
code/recording/data/scene_camera/2015_10_03/000/info.csv Normal file
View file

@ -0,0 +1,14 @@
Recording Name 2015_10_03
Start Date 03.10.2015
Start Time 20:15:10
Duration Time 00:00:52
Eye Mode monocular
Duration Time 00:00:52
World Camera Frames 1265
World Camera Resolution 720x1280
Capture Software Version 0.5.7
User mmbrian
Platform Linux
Machine Brian
Release 3.16.0-49-generic
Version #65~14.04.1-Ubuntu SMP Wed Sep 9 10:03:23 UTC 2015
1 Recording Name 2015_10_03
2 Start Date 03.10.2015
3 Start Time 20:15:10
4 Duration Time 00:00:52
5 Eye Mode monocular
6 Duration Time 00:00:52
7 World Camera Frames 1265
8 World Camera Resolution 720x1280
9 Capture Software Version 0.5.7
10 User mmbrian
11 Platform Linux
12 Machine Brian
13 Release 3.16.0-49-generic
14 Version #65~14.04.1-Ubuntu SMP Wed Sep 9 10:03:23 UTC 2015

BIN
code/recording/data/scene_camera/2015_10_03/000/pupil_data Normal file
View file

Binary file not shown.

2
code/recording/data/scene_camera/2015_10_03/000/user_info.csv Normal file
View file

@ -0,0 +1,2 @@
name
additional_field change_me
1 name
2 additional_field change_me

61
code/recording/data_grabber.py Normal file
View file

@ -0,0 +1,61 @@
import os
from tracker import performMarkerTracking
ROOT_DATA_DIR = '/home/mmbrian/HiWi/etra2016_mohsen/code/recording/data/participants'
SQUARE_SIZE = '0.16'
def main(force = False):
'''
Processes all the participants recordings and performs marker tracking on each video and stores marker data in a npy file near the video
'''
c = 0
for d1 in os.listdir(ROOT_DATA_DIR):
if d1.startswith('p'): # every participant
d2 = os.path.join(ROOT_DATA_DIR, d1) # .../pi/
d2 = os.path.join(d2, os.listdir(d2)[0]) # .../pi/../
for d3 in os.listdir(d2): # every recording
d4 = os.path.join(d2, d3) # .../pi/../00X/
print '> Processing', d4
frames_dir = os.path.join(d4, '_aruco_frames.npy')
if os.path.isfile(frames_dir):
if not force: # already processed this recording
print '> Recording already processed...'
continue
else: # video processed, but forced to process again
# remove old file
print '> Reprocessing file:', frames_dir
print '> Removing old marker file...'
os.remove(frames_dir)
world_path = os.path.join(d4, 'world.avi')
log_dir = os.path.join(d4, '_ArUco.log')
read_log = False
if os.path.isfile(log_dir):
read_log = True
else: # no log available > process video
if not os.path.isfile(world_path): # .avi file exists
# Creating avi
print '> AVI video does not exists, generating it from mp4 source file...'
os.popen('avconv -i ' + os.path.join(d4, 'world.mp4') + ' -c:a copy -c:v copy ' + world_path + ' -v quiet')
# Perform tracking...
performMarkerTracking(read_from_log = read_log, log = True, log_dir = log_dir,
recording_video = world_path + ' ', frames_dir = frames_dir,
square_size = SQUARE_SIZE) # in case of relocating camera.yml input the new path as cam_math
if not read_log:
# Removing avi file
print '> Removing avi file:', world_path
os.remove(world_path)
c += 1
print '> Processed %s recordings so far...' % c
# print '> Halting...'
# return
if __name__ == '__main__':
main(force=False)
print 'Finished.'

85
code/recording/eye_world_correlation.py Normal file
View file

@ -0,0 +1,85 @@
'''
(*)~----------------------------------------------------------------------------------
author:
Julian Steil
Master Thesis (2014):
Discovery of eye movement patterns in long-term human visual behaviour using topic models
----------------------------------------------------------------------------------~(*)
'''
import sys,os
import numpy as np
PARTICIPANTS = ['p10', 'p16', 'p13', 'p24', 'p5', 'p14', 'p26', 'p12', 'p20', 'p7', 'p15', 'p11', 'p21', 'p25']
ROOT = '/home/mmbrian/HiWi/etra2016_mohsen/code/recording/data/participants'
timestamps_world_path = 'world_timestamps.npy'
timestamps_eye_path = 'eye0_timestamps.npy'
def main():
for p in os.listdir(ROOT):
if p in PARTICIPANTS:
print '> Correlating eye-world images for', p
d1 = os.path.join(ROOT, p)
d1 = os.path.join(d1, os.listdir(d1)[0]) # ../p_i/../
for d2 in os.listdir(d1):
path = os.path.join(d1, d2)
print '> Processing', path
process(path)
print 'Done.'
def process(root):
timestamps_world = list(np.load(os.path.join(root, timestamps_world_path)))
timestamps_eye = list(np.load(os.path.join(root, timestamps_eye_path)))
no_frames_eye = len(timestamps_eye)
no_frames_world = len(timestamps_world)
# Detection of Synchronization-Matchings to initialize the correlation-matrix
frame_idx_world = 0
frame_idx_eye = 0
while (frame_idx_world < no_frames_world):
# if the current world_frame is before the mean of the current eye frame timestamp and the next eyeframe timestamp
if timestamps_world[frame_idx_world] <= (timestamps_eye[frame_idx_eye]+timestamps_eye[frame_idx_eye+1])/2.:
frame_idx_world+=1
else:
if frame_idx_eye >= no_frames_eye-2:
break
frame_idx_eye+=1
no_of_matched_frames = frame_idx_eye
print "no_of_matched_frames: ", no_of_matched_frames
# Synchonizing eye and world cam
print no_frames_eye, no_frames_world
correlation = []
for i in xrange(no_frames_world):
correlation.append([])
for j in xrange(1):
correlation[i].append(float(0))
frame_idx_world = 0
frame_idx_eye = 0
while (frame_idx_world < no_frames_world):
# print frame_idx_world,frame_idx_eye
# if the current world_frame is before the mean of the current eye frame timestamp and the next eye timestamp
if timestamps_world[frame_idx_world] <= (timestamps_eye[frame_idx_eye]+timestamps_eye[frame_idx_eye+1])/2.:
correlation[frame_idx_world][0] = frame_idx_eye
frame_idx_world+=1
else:
if frame_idx_eye >= no_frames_eye-2:
frame_idx_eye += 1
while (frame_idx_world < no_frames_world):
correlation[frame_idx_world][1] = frame_idx_eye
frame_idx_world+=1
break
frame_idx_eye+=1
correlation_list_path = "eye_world_correlation.npy"
correlation_list_csv_path = "eye_world_correlation.csv"
np.save(os.path.join(root, correlation_list_path),np.asarray(correlation))
np.savetxt(os.path.join(root, correlation_list_csv_path),np.asarray(correlation), delimiter=",", fmt="%f")
if __name__ == '__main__':
main()

193
code/recording/opencv/calibrate_and_save.py Normal file
View file

@ -0,0 +1,193 @@
#!/usr/bin/env python
'''
This is a modified version of opencv's calibrate.py that stores the camera
parameters in the YAML format required by ArUco. an example of this format
is given by:
%YAML:1.0
calibration_time: "Sa 08 Aug 2015 16:32:35 CEST"
nr_of_frames: 30
image_width: 752
image_height: 480
board_width: 8
board_height: 6
square_size: 24.
fix_aspect_ratio: 1.
# flags: +fix_aspect_ratio +fix_principal_point +zero_tangent_dist
flags: 14
camera_matrix: !!opencv-matrix
rows: 3
cols: 3
dt: d
data: [ 418.164617459, 0., 372.480325679, 0., 417.850564673, 229.985538918, 0., 0., 1.]
distortion_coefficients: !!opencv-matrix
rows: 5
cols: 1
dt: d
data: [ -0.3107371474, 0.1187673445, -0.0002552599, -0.0001158436, -0.0233324616]
(Reference: https://wiki.mpi-inf.mpg.de/d2/ArUco)
How to use:
- Copy and paste this in /<OPENCV_SOURCE_DIR>/samples/python2
- Run it like
python calibrate_and_save.py --save "/<PATH_TO_SAVE_CAMERA_MATRIX>/camera.yml" "/<PATH_TO_IMAGE_DIR>/*.png"
- Then you can use ArUco like
./<PATH_TO_ARUCO_SRC>/build/utils/aruco_test_gl live /<PATH_TO_SAVE_CAMERA_MATRIX>/camera.yml 0.039
More on calibration: http://www.janeriksolem.net/2014/05/how-to-calibrate-camera-with-opencv-and.html
UPDATE:
Alternatively, you can follow the steps below
- After disabling integrated webcam:
echo "0" > /sys/bus/usb/devices/2-1.6/bConfigurationValue
(need to first find which usb device corresponds to your webcam)
- Plug in pupil tracker and use the official precompiled cpp script like:
/<PATH_TO_OPENCV>/build/bin/cpp-example-calibration -w 8 -h 6 -s 0.039 -o camera.yml -op
'''
import time
import numpy as np
import cv2, cv
import os
from common import splitfn
def saveCameraParams(save_dir, nframes, w, h, bw, bh, square_size,
camera_matrix, distortion_coefficients, fix_aspect_ratio = None, flags = 0):
time_str = time.strftime('%a %d %b %Y %H:%M:%S %Z')
lines = []
lines.append('%YAML:1.0')
lines.append('calibration_time: "%s"' %time_str)
lines.append('nframes: %s' %nframes)
lines.append('image_width: %s' %w)
lines.append('image_height: %s' %h)
lines.append('board_width: %s' %bw)
lines.append('board_height: %s' %bh)
lines.append('square_size: %s' %square_size)
if fix_aspect_ratio:
lines.append('fix_aspect_ratio: %s' %fix_aspect_ratio)
lines.append('flags: %s' %flags)
lines.append('camera_matrix: !!opencv-matrix')
lines.append(' rows: 3')
lines.append(' cols: 3')
lines.append(' dt: d')
lines.append(' data: %s' %repr(camera_matrix.reshape(1,9)[0])[6:-1]) # [6:-1] removes "array(" and ")"
lines.append('distortion_coefficients: !!opencv-matrix')
lines.append(' rows: 5')
lines.append(' cols: 1')
lines.append(' dt: d')
lines.append(' data: %s' %repr(distortion_coefficients)[6:-1])
with open(save_dir, 'w') as f:
f.writelines(map(lambda l: l+'\n', lines))
def readCameraParams(cam_mat = None):
'''
Reads an openCV camera.yml file and returns camera_matrix and distortion_coefficients
'''
if not cam_mat:
cam_mat = CAMERA_MATRIX
data = ''.join(open(cam_mat.strip(), 'r').readlines()).replace('\n', '').lower()
try:
ind1 = data.index('[', data.index('camera_matrix'))
ind2 = data.index(']', ind1)
camera_matrix = eval(data[ind1:ind2+1])
camera_matrix = np.array([camera_matrix[:3],
camera_matrix[3:6],
camera_matrix[6:]])
ind1 = data.index('[', data.index('distortion_coefficients'))
ind2 = data.index(']', ind1)
dist_coeffs = np.array(eval(data[ind1:ind2+1]))
return camera_matrix, dist_coeffs
except Exception:
print 'Could not load camera parameters'
print 'Invalid camera.yml file.'
if __name__ == '__main__':
import sys, getopt
from glob import glob
args, img_mask = getopt.getopt(sys.argv[1:], '', ['save=', 'debug=', 'square_size='])
args = dict(args)
try: img_mask = img_mask[0]
except: img_mask = '../cpp/left*.jpg'
# print 'mask is', img_mask
# img_mask = img_mask.replace('10.png', '*.png')
img_names = glob(img_mask)
debug_dir = args.get('--debug')
square_size = float(args.get('--square_size', 1.0))
square_size = 0.00122
save_dir = args.get('--save')
pattern_size = (8, 6)
pattern_points = np.zeros( (np.prod(pattern_size), 3), np.float32 )
pattern_points[:,:2] = np.indices(pattern_size).T.reshape(-1, 2)
pattern_points *= square_size
obj_points = []
img_points = []
h, w = 0, 0
for fn in img_names:
print 'processing %s...' % fn,
img = cv2.imread(fn, 0)
h, w = img.shape[:2]
found, corners = cv2.findChessboardCorners(img, pattern_size)
if found:
term = ( cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_COUNT, 30, 0.1 )
cv2.cornerSubPix(img, corners, (5, 5), (-1, -1), term)
if debug_dir:
vis = cv2.cvtColor(img, cv2.COLOR_GRAY2BGR)
cv2.drawChessboardCorners(vis, pattern_size, corners, found)
path, name, ext = splitfn(fn)
cv2.imwrite('%s/%s_chess.bmp' % (debug_dir, name), vis)
if not found:
print 'chessboard not found'
continue
img_points.append(corners.reshape(-1, 2))
obj_points.append(pattern_points)
print 'ok'
# rms, camera_matrix, dist_coefs, rvecs, tvecs = cv2.calibrateCamera(obj_points, img_points, (w, h))
root_dir = '/home/mmbrian/Pictures/eye_camera_images/'
cameraMatrixguess, distCoeffsguess = readCameraParams(os.path.join(root_dir,'_camera.yml'))
print "cameraM: ", cameraMatrixguess
print "dist: ", distCoeffsguess
cameraMatrixguess[1][1] = cameraMatrixguess[0][0]
cameraMatrixguess[0][2] = 320
cameraMatrixguess[1][2] = 180
# Calibrate camera intrinsics
rms, camera_matrix, dist_coefs, rvecs, tvecs = cv2.calibrateCamera(obj_points, img_points, (w,h),
cameraMatrixguess, distCoeffsguess, None, None, flags = \
cv.CV_CALIB_USE_INTRINSIC_GUESS + cv.CV_CALIB_FIX_PRINCIPAL_POINT + cv.CV_CALIB_FIX_ASPECT_RATIO)
np.save(os.path.join(root_dir,'dist.npy'), dist_coefs)
np.save(os.path.join(root_dir,'cameraMatrix.npy'), camera_matrix)
np.savetxt(os.path.join(root_dir,'dist.csv'),np.asarray(dist_coefs), delimiter=";", fmt="%s")
np.savetxt(os.path.join(root_dir,"cameraMatrix.csv"),np.asarray(camera_matrix), delimiter=";", fmt="%s")
print "RMS:", rms
print "camera matrix:\n", camera_matrix
print "distortion coefficients: ", dist_coefs.ravel()
print 'Width:', w, 'Height:', h
print 'nframes:', len(img_names)
print 'square_size:', square_size
print 'board_width:', pattern_size[0]
print 'board_height:', pattern_size[1]
saveCameraParams(save_dir, len(img_names), w, h, pattern_size[0], pattern_size[1], square_size,
camera_matrix, dist_coefs.ravel())
print "Saved camera matrix to", save_dir
cv2.destroyAllWindows()

454
code/recording/process_recordings.py Normal file
View file

@ -0,0 +1,454 @@
from __future__ import division
'''
For each experiment, this script tracks movement of the marker in the video from the information in aruco_frames.npy
It then correlates this information with gaze data from pupil_positions.npy
finally, for every target in the video (25 targets in calibration, 16 in test), it maps 3D marker position (mean position over the duration of pause)
to the gaze position (mean position over the pause duration) and stores this info together with the projected 2D marker position in a separate npy file.
the resulting file contains the ground truth data for this experiment.
'''
import os, sys
import numpy as np
import matplotlib.pyplot as plt
from pylab import rcParams
from scipy.ndimage.filters import gaussian_filter1d as g1d
from scipy import signal
from sklearn.neighbors import NearestNeighbors as knn
# from sklearn import svm
from sklearn.cluster import AgglomerativeClustering
from tracker import readCameraParams, Marker
from util.tools import is_outlier, moving_average
sys.path.append('..') # so we can import from pupil
from pupil import player_methods
from vector import Vector as v
import pdb
ROOT_DATA_DIR = '/home/mmbrian/HiWi/etra2016_mohsen/code/recording/data/participants'
def unifiy_markers_per_frame(marker_data):
'''
Since ArUco sometimes detects a marker twice in a frame, we need to either ignore one or somehow compute their mean.
Also this method maps each final marker to its center's 3D and 2D position wrt scene camera
'''
camera_matrix, dist_coeffs = readCameraParams() # in case of relocating camera.yml input the new path as cam_math
mdata, mrdata = [], []
for fn in xrange(len(marker_data)):
if len(marker_data[fn]) > 0:
markers = map(lambda m: Marker.fromList(m), marker_data[fn])
markers = map(lambda m: np.array([np.array(m.getCenter()),
np.array(m.getCenterInImage(camera_matrix, dist_coeffs))]), markers)
marker = sum(markers)/len(markers)
marker = [marker[0][0], marker[0][1], marker[0][2], marker[1][0], marker[1][1]]
# marker_data[fn] = marker
mdata.append(marker)
mrdata.append(marker)
else: # if marker is not detected, assign last detected position to this frame
# marker_data[fn] = marker_data[fn-1]
mdata.append(mdata[fn-1])
mrdata.append([]) # this contains real marker information (all tracked positions)
# return marker_data
return np.array(mdata), mrdata
def fix_labels(labels, window = 2, elements = [0, 1], outliers = []):
labels = list(labels)
for i in xrange(window, len(labels)-window):
neighborhood = labels[i-window:i+window+1]
if outliers[i]: # removing this label from decision making
neighborhood = neighborhood[:i] + neighborhood[i+1:]
element_counts = [list(neighborhood).count(e) for e in elements]
dominant_element = elements[element_counts.index(max(element_counts))]
labels[i] = dominant_element
return labels
def find_intervals(labels, mean, marker_speed):
'''
Given the label information of frame to frame motion speed, this method returns the frame
intervals for which the marker is either "moving" or "not moving"
Notice that len(labels) equals the number of frames minus one
'''
nm_label = labels[0]
intervals = []
curr_label, start, end = -1, -1, -1
not_moving = 0
for i in xrange(len(labels)):
if curr_label < 0: # first label
curr_label = labels[i]
start = i
else:
if labels[i] != curr_label: # label changed
end = i
intervals.append([start, end, curr_label])
if curr_label == nm_label: not_moving+=1
curr_label = labels[i]
start = i+1
end = len(labels)
intervals.append([start, end, curr_label])
if curr_label == nm_label: not_moving+=1
# Now we do a post check to see if two non moving intervals are very close to each other,
# the middle interval is most likely a misclassification
# computing average interval length for moving intervals
if (len(intervals) > 49 and not_moving > 25) or (len(intervals)>31 and not_moving>16):
ret = merge_intervals(intervals, nm_label, mean, marker_speed, remove_outliers=True)
return ret, sum(1 for e in ret if e[2] == nm_label)
else:
return intervals, not_moving
def merge_intervals(intervals, nm_label, mean, marker_speed, remove_outliers=True):
mlength = np.array([seg[1] - seg[0] for seg in intervals if seg[2] != nm_label])
nmlength = np.array([seg[1] - seg[0] for seg in intervals if seg[2] == nm_label])
if remove_outliers:
mlength_outliers = mlength[is_outlier(mlength, thresh=3.5)]
avg_m_length = (sum(mlength)-sum(mlength_outliers))/(mlength.size - mlength_outliers.size)
nmlength_outliers = nmlength[is_outlier(nmlength, thresh=3.5)]
avg_nm_length = (sum(nmlength)-sum(nmlength_outliers))/(nmlength.size - nmlength_outliers.size)
else:
avg_m_length = sum(mlength)/mlength.size
avg_nm_length = sum(nmlength)/nmlength.size
thresh = 3.5 # removes a moving interval if average length is at least this time larger than its length
i = 1
ret = []
ret.append(intervals[0])
while i < len(intervals):
length = intervals[i][1] - intervals[i][0]
ratio, label = 1, intervals[i][2]
if label == nm_label:
ratio = avg_nm_length/length
else:
ratio = avg_m_length/length
if ratio>=thresh: # average length is at least 2 times larger than the length of this interval
# replace this interval by merge the two not moving intervals around it
# check if average of elements in this interval is greater than mean
if np.mean(marker_speed[intervals[i][0]:intervals[i][1]]) < mean:
last_intv = ret.pop()
ret.append([last_intv[0], intervals[i+1][1], 1-label])
print 'Merged two intervals'
i+=2
continue
else:
pass
ret.append(intervals[i])
i+=1
return ret
# def main(force=False):
# rcParams['figure.figsize'] = 15, 7
# recordings_processed = 0
# recordings_successful = 0
# for d1 in os.listdir(ROOT_DATA_DIR):
# if d1.startswith('p'): # every participant
# d2 = os.path.join(ROOT_DATA_DIR, d1) # .../pi/
# d2 = os.path.join(d2, os.listdir(d2)[0]) # .../pi/../
# for d3 in os.listdir(d2): # every recording
# d4 = os.path.join(d2, d3) # .../pi/../00X/
# print '> Processing', d4
# frames_dir = os.path.join(d4, '_aruco_frames.npy')
# if not os.path.isfile(frames_dir): # the recording is not yet processed for marker tracking
# print '> Recording does not contain marker data...'
# continue
# intervals_dir = os.path.join(d4, 'gaze_intervals.npy')
# if os.path.isfile(intervals_dir):
# print '> Recording already processed...'
# if force:
# print '> Processing again...'
# else:
# continue
# marker_data = np.load(frames_dir)
# # marker_data includes data on tracked markers per frame
# # it's a list with as many entries as the number of video frames, each entry
# # has a list of tracked markers, each marker item has marker id, marker corners, Rvec, Tvec
# wt = np.load(os.path.join(d4, 'world_timestamps.npy'))
# # Processing pupil positions
# pp = np.load(os.path.join(d4, 'pupil_positions.npy')) # timestamp confidence id pos_x pos_y diameter
# # pos_x and pos_y are normalized (Origin 0,0 at the bottom left and 1,1 at the top right)
# # converting each element to dictionary for correlation
# pp = map(lambda e: dict(zip(['timestamp', 'conf', 'id', 'x', 'y', 'diam'], e)), pp)
# pp_by_frame = player_methods.correlate_data(pp, wt)
# # Keeping only pupil positions with nonzero confidence
# pp_by_frame = map(lambda l: filter(lambda p: p['conf']>0, l), pp_by_frame)
# # Computing a single pupil position for the frame by taking mean of all detected pupil positions
# pp_by_frame = map(lambda data:
# sum(np.array([pp['x'], pp['y']]) for pp in data)/len(data) if data else np.array([-1, -1]), pp_by_frame)
# # Now each nonempty value of pp_by_frame is a tuple of (x, y) for pupil position in that frame
# # Checking if timestamps, markers per frame and pupil positions per frame are correlated
# assert len(marker_data) == len(wt) == len(pp_by_frame)
# # Good, now we need to find the frame ranges in which marker is not moving, for that we need the marker_data
# # and using the position info per frame, we can compute movement speed and detect when it is it almost zero
# marker_data, mrdata = unifiy_markers_per_frame(marker_data)
# # Smoothing x and y coords
# marker_data[:, 3] = g1d(marker_data[:, 3], sigma=2)
# marker_data[:, 4] = g1d(marker_data[:, 4], sigma=2)
# marker_speed = []
# for fn, fnp1 in ((f, f+1) for f in xrange(len(marker_data)-1)):
# if marker_data[fnp1] != [] and marker_data[fn] != []:
# # dx = marker_data[fnp1][0] - marker_data[fn][0]
# # dy = marker_data[fnp1][1] - marker_data[fn][1]
# # dz = marker_data[fnp1][2] - marker_data[fn][2]
# # speed = np.sqrt(dx**2 + dy**2 + dz**2) * 100
# # print fn, fnp1, len(marker_data), marker_data[fnp1], marker_data[fn]
# dx = marker_data[fnp1][3] - marker_data[fn][3]
# dy = marker_data[fnp1][4] - marker_data[fn][4]
# speed = np.sqrt(dx**2 + dy**2)
# # print 'marker speed:', speed
# marker_speed.append(speed)
# else:
# marker_speed.append(marker_speed[-1]) # set speed to last speed if marker could not be detected
# # Performing binary clustering on marker speed
# model = AgglomerativeClustering(n_clusters=2, linkage="ward", affinity="euclidean")
# marker_speed = np.array(marker_speed)
# # Checking for outliers based on "median absolute deviation"
# outliers = is_outlier(marker_speed, thresh=3.5)
# print sum(outliers == True), 'outliers detected'
# # removing outliers
# outlier_inds = [i for i in xrange(outliers.size) if outliers[i]]
# marker_speed = list(np.delete(marker_speed, outlier_inds))
# # replacing removed outliers by average of their neighbours
# outliers_inds = sorted(outlier_inds)
# window = 1
# for ind in outlier_inds:
# start = max(ind-window, 0)
# neighbours = marker_speed[start:ind+window]
# new_val = sum(neighbours)/len(neighbours)
# marker_speed.insert(ind, new_val)
# marker_speed = np.array(marker_speed)
# # smoothed_signal = marker_speed[:]
# smoothed_signal = signal.medfilt(marker_speed, 13)
# # smoothed_signal = g1d(marker_speed, sigma=2)
# # smoothed_signal = moving_average(smoothed_signal, 7)
# model.fit(map(lambda e: [e], smoothed_signal))
# labels = fix_labels(model.labels_, window=1, outliers = outliers)
# outliers = map(lambda e: 10 if e else 5, outliers)
# mean = np.mean(smoothed_signal)
# intervals, nm = find_intervals(labels, mean, smoothed_signal)
# print '>', len(intervals), 'Intervals found in total.', nm, 'gaze intervals.'
# interval_display = []
# for dur in intervals:
# interval_display.extend([dur[2]]*(dur[1]-dur[0]+1))
# interval_display = interval_display[:-1]
# print len(interval_display), len(marker_data)-1, intervals[-1][1]-intervals[0][0]
# # print intervals
# # print labels
# # return
# # print len(marker_data), len(marker_speed)
# plt.plot(range(len(marker_data)-1), marker_speed, 'b',
# # range(len(marker_data)-1), labels, 'r',
# range(len(marker_data)-1), smoothed_signal, 'g',
# range(len(marker_data)-1), interval_display, 'r')
# # plt.show()
# # plt.clf()
# # return
# # plt.clf()
# recordings_processed += 1
# intervals_okay = True
# if not nm in [16, 25]:
# intervals_okay = False
# pdb.set_trace()
# print '> Storing odd figure...'
# plt.savefig('./temp/%s-%s__%snm.png' % (d1, d3, str(nm)))
# # print '> Entering manual override mode...'
# # print '> Enter halt to quit.'
# # # set manual_bypass to True in case you wanna discard changes in override mode
# # cmd = raw_input(':')
# # while cmd != 'halt' and cmd != 'pass':
# # exec cmd in globals(), locals()
# # cmd = raw_input(':')
# if intervals_okay:
# print '> Intervals seem okay.'
# plt.savefig(os.path.join(d4, 'marker_motion.png'))
# recordings_successful += 1
# # Store interval information
# # Use pp_by_frame and marker_data to compute gaze and target points corresponding to this interval
# gaze_intervals = intervals[::2] # starting from the first interval, gaze, moving, gaze, moving, gaze, ...
# t2d, t3d, p = [], [], []
# for intv in gaze_intervals:
# s, e = intv[0], intv[1]
# null_gaze, null_marker = 0, 0
# gaze_point = np.array([0, 0])
# marker_3d_position = np.array([0, 0, 0])
# marker_2d_position = np.array([0, 0])
# for fn in xrange(s, e+1):
# if all(pp_by_frame[fn]==np.array([-1, -1])):
# null_gaze += 1
# else:
# gaze_point = gaze_point + pp_by_frame[fn]
# if mrdata[fn] == []:
# null_marker += 1
# else:
# marker_3d_position = marker_3d_position + np.array(mrdata[fn][:3])
# marker_2d_position = marker_2d_position + np.array(mrdata[fn][3:])
# gaze_point = gaze_point/(e-s+1-null_gaze)
# marker_3d_position = marker_3d_position/(e-s+1-null_marker)
# marker_2d_position = marker_2d_position/(e-s+1-null_marker)
# t2d.append(marker_2d_position)
# t3d.append(marker_3d_position)
# p.append(gaze_point)
# print '> Storing intervals, gaze data, and marker data...'
# np.save(intervals_dir, np.array(gaze_intervals))
# np.save(os.path.join(d4, 'p.npy'), np.array(p))
# np.save(os.path.join(d4, 't2d.npy'), np.array(t2d))
# np.save(os.path.join(d4, 't3d.npy'), np.array(t3d))
# print '>', recordings_processed, 'recordings processed.', recordings_successful, 'successful.'
# plt.clf()
PARTICIPANTS = ['p10', 'p16', 'p13', 'p24', 'p5', 'p14', 'p26', 'p12', 'p20', 'p7', 'p15', 'p11', 'p21', 'p25']
def main(force=False):
recordings_processed = 0
recordings_successful = 0
for d1 in os.listdir(ROOT_DATA_DIR):
if d1.startswith('p'): # every participant
if not d1 in PARTICIPANTS:
continue
d2 = os.path.join(ROOT_DATA_DIR, d1) # .../pi/
d2 = os.path.join(d2, os.listdir(d2)[0]) # .../pi/../
for d3 in os.listdir(d2): # every recording
d4 = os.path.join(d2, d3) # .../pi/../00X/
print '> Processing', d4
frames_dir = os.path.join(d4, '_aruco_frames.npy')
if not os.path.isfile(frames_dir): # the recording is not yet processed for marker tracking
print '> Recording does not contain marker data...'
continue
intervals_dir = os.path.join(d4, 'gaze_intervals.npy')
if os.path.isfile(intervals_dir):
print '> Recording already processed...'
if force:
print '> Processing again...'
else:
continue
marker_data = np.load(frames_dir)
# marker_data includes data on tracked markers per frame
# it's a list with as many entries as the number of video frames, each entry
# has a list of tracked markers, each marker item has marker id, marker corners, Rvec, Tvec
wt = np.load(os.path.join(d4, 'world_timestamps.npy'))
# Processing pupil positions
pp = np.load(os.path.join(d4, 'pupil_positions.npy')) # timestamp confidence id pos_x pos_y diameter
# pos_x and pos_y are normalized (Origin 0,0 at the bottom left and 1,1 at the top right)
# converting each element to dictionary for correlation
pp = map(lambda e: dict(zip(['timestamp', 'conf', 'id', 'x', 'y', 'diam'], e)), pp)
pp_by_frame = player_methods.correlate_data(pp, wt)
# Keeping only pupil positions with nonzero confidence
pp_by_frame = map(lambda l: filter(lambda p: p['conf']>0, l), pp_by_frame)
# Computing a single pupil position for the frame by taking mean of all detected pupil positions
pp_by_frame = map(lambda data:
sum(np.array([pp['x'], pp['y']]) for pp in data)/len(data) if data else np.array([-1, -1]), pp_by_frame)
# Now each nonempty value of pp_by_frame is a tuple of (x, y) for pupil position in that frame
# Checking if timestamps, markers per frame and pupil positions per frame are correlated
assert len(marker_data) == len(wt) == len(pp_by_frame)
# Good, now we need to find the frame ranges in which marker is not moving, for that we need the marker_data
# and using the position info per frame, we can compute movement speed and detect when it is it almost zero
marker_data, mrdata = unifiy_markers_per_frame(marker_data)
gaze_intervals = np.load(intervals_dir)
recordings_processed += 1
intervals_okay = True
if intervals_okay:
print '> Intervals seem okay.'
recordings_successful += 1
t2d, t3d, p = [], [], []
t2d_med, t3d_med, p_med, p_frames = [], [], [], []
for intv in gaze_intervals:
s, e = intv[0], intv[1]
null_gaze, null_marker = 0, 0
gaze_point = np.array([0, 0])
marker_3d_position = np.array([0, 0, 0])
marker_2d_position = np.array([0, 0])
gpts, m3ds, m2ds = [], [], []
valid_frames = []
for fn in xrange(s, e+1):
if all(pp_by_frame[fn]==np.array([-1, -1])) or mrdata[fn] == []:
# either pupil detection failed or marker detection
# the whole pupil-marker correspondence is invalid
# ignore this frame
pass
else:
gpts.append(pp_by_frame[fn])
marker_3d_position = marker_3d_position + np.array(mrdata[fn][:3])
marker_2d_position = marker_2d_position + np.array(mrdata[fn][3:])
m3ds.append(np.array(mrdata[fn][:3]))
m2ds.append(np.array(mrdata[fn][3:]))
valid_frames.append(fn)
if not len(valid_frames):
# this marker-pupil correspondece failed
print '> Failed to find reliable correspondece for a marker position...'
# In this case participant data should be completely ignored
# retrun
# Computing the median pupil position
final_p = np.median(gpts, axis=0)
p_med.append(final_p)
# Finding the closest pupil position to this median in the valid frames
dists = map(lambda pupil_position: (v(pupil_position)-v(final_p)).mag, gpts)
dists = zip(range(len(gpts)), dists)
closest = min(dists, key=lambda pair:pair[1])
# Getting the index for this position
ind = closest[0]
# Finding the k nearest pupil position to this one
k = 3
while True:
try:
nbrs = knn(n_neighbors=k, algorithm='ball_tree').fit(gpts)
dists, indices = nbrs.kneighbors(gpts)
break
except ValueError, err:
k-=1
nearest_ind = indices[ind]
frames_numbers = map(lambda i: valid_frames[i], nearest_ind)
p_frames.append(frames_numbers)
# Now we take eye images from these frames
# Also the pupil-marker correspondece is now final_p and m2ds[ind] m3d[ind]
t2d_med.append(m2ds[ind])
t3d_med.append(m3ds[ind])
# t2d_med.append(np.median(m2ds, axis=0))
# t3d_med.append(np.median(m3ds, axis=0))
print '> gaze and marker data...'
# np.save(intervals_dir, np.array(gaze_intervals))
np.save(os.path.join(d4, 'p_frames.npy'), np.array(p_frames))
# np.save(os.path.join(d4, 'p.npy'), np.array(p_med))
# np.save(os.path.join(d4, 't2d.npy'), np.array(t2d_med))
# np.save(os.path.join(d4, 't3d.npy'), np.array(t3d_med))
print '>', recordings_processed, 'recordings processed.', recordings_successful, 'successful.'
if __name__ == '__main__':
main(force=True)

150
code/recording/retrieve.py Normal file
View file

@ -0,0 +1,150 @@
import sys
import numpy as np
import cv2
import matplotlib.pyplot as plt
sys.path.append('..') # so we can import from pupil
from pupil import player_methods
# from tracker import processFrame
DATA_DIR = '/home/mmbrian/HiWi/pupil_clone/pupil/recordings/2015_09_10/007/'
OUTPUT_DIR = DATA_DIR + 'pp.npy'
def capture(frame_number):
cap = cv2.VideoCapture(DATA_DIR + "world.mp4")
fc = int(cap.get(cv2.cv.CV_CAP_PROP_FRAME_COUNT))
assert frame_number<fc
frame = 1
status, img = cap.read() # extract the first frame
while status:
if frame == frame_number:
save_dir = DATA_DIR + "frames/frame_%d.jpg" % frame_number
cv2.imwrite(save_dir, img)
break
frame+=1
status, img = cap.read()
def main():
p2d, t3d = [], [] # 2D-3D pupil position to target position correspondences
cap = cv2.VideoCapture(DATA_DIR + "world.mp4")
fc = int(cap.get(cv2.cv.CV_CAP_PROP_FRAME_COUNT))
wt = np.load(DATA_DIR + "world_timestamps.npy")
# time is measured in seconds (floating point) since the epoch.
# world timestamps correspond to each frame, we have to correlate timestamp information from
# pupil positions and world timestamps to find a 1-to-1 mapping between pupil positions and
# marker positions in the video
print fc, len(wt)
assert fc == len(wt)
########################################################################################################
# Processing markers
# print 'Processing markers...'
## Processing frame by frame does not work as ArUco process opens a Gtk which cannot be terminated
## automatically, therefore we better process all frames before and work with the data here
# frame = 1
# status, img = cap.read() # extract the first frame
# while status:
# # cv2.imwrite(DATA_DIR + "frames/frame-%d.jpg" % frame, img)
# save_dir = DATA_DIR + "frames/current_frame.jpg"
# cv2.imwrite(save_dir, img)
# processFrame(save_dir)
# frame+=1
# status, img = cap.read()
# print "Processed %d frames." % (frame-1)
########################################################################################################
# Processing pupil positions
print 'Processing pupil positions...'
pp = np.load(DATA_DIR + "pupil_positions.npy") # timestamp confidence id pos_x pos_y diameter
# pos_x and pos_y are normalized (Origin 0,0 at the bottom left and 1,1 at the top right)
# converting each element to dictionary for correlation
pp = map(lambda e: dict(zip(['timestamp', 'conf', 'id', 'x', 'y', 'diam'], e)), pp)
pp_by_frame = player_methods.correlate_data(pp, wt)
# Keeping only pupil positions with nonzero confidence
pp_by_frame = map(lambda l: filter(lambda p: p['conf']>0, l), pp_by_frame)
# Computing a single pupil position for the frame by taking mean of all detected pupil positions
pp_by_frame = map(lambda data:
sum(np.array([pp['x'], pp['y']]) for pp in data)/len(data) if data else np.array([-1, -1]), pp_by_frame)
# Now each nonempty value of pp_by_frame is a tuple of (x, y) for pupil position in that frame
# Next we need to associate each frame to a detected marker and by taking mean pupil point and
# mean 3D marker position over a series of frames corresponding to that marker find a 2D-3D
# mapping for calibration/test
tdiff = map(lambda e: e-wt[0], wt)
# This time correspondence to each marker was coordinated using the GazeHelper android application
# for 005 > starting from 00:56, 3 seconds gaze, 1 second for saccade
# These parameters are specific to the experiment
# 005 > 56, 3, 1
# 006 > 3, 3, 1
# 007 > 7, 3, 1 (or 8)
# 010 > 3, 3, 1
starting_point, gaze_duration, saccade_duration = 56, 3, 1 # these are specific to the experiment
# finding the starting frame
ind = 0
while tdiff[ind] < starting_point:
ind+=1
print ind
data = []
tstart = wt[ind]
for i in xrange(9):
print i
while ind<len(wt) and wt[ind] - tstart < saccade_duration:
ind+=1
if ind<len(wt):
tstart = wt[ind]
starting_ind = ind
while ind<len(wt) and wt[ind] - tstart < gaze_duration:
ind+=1
# all frames from starting_ind to ind-1 correspond to currently gazed marker
c = 0
cp = np.array([0, 0])
all_corresponding_points = []
for j in xrange(starting_ind, ind):
if pp_by_frame[j][0] >= 0:
c+=1
cp = cp + pp_by_frame[j]
all_corresponding_points.append(pp_by_frame[j])
# print c
if c>0:
ret = cp/c
else:
ret = np.array([-1, -1]) # no detected pupil for this marker
p2d.append(ret)
data.append([
np.array([starting_ind, ind-1]), # frame range
ret, # mean pupil position
all_corresponding_points]) # all pupil positions in range
if ind<len(wt):
tstart = wt[ind]
# p2d is now the list of detected pupil positions (not always 9 points)
print 'Saving data...'
np.save(OUTPUT_DIR, data)
# plt.plot([x[0] if x!=None else 0 for x in pp_by_frame], [y[1] if y!=None else 0 for y in pp_by_frame], 'ro')
# plt.show()
########################################################################################################
print len(p2d), 'gaze points'
print p2d
print len(wt), 'frames...'
if __name__ == '__main__':
main()

27
code/recording/samples/frame_1_details.txt Normal file
View file

@ -0,0 +1,27 @@
25 markers detected
[-10.2193 -4.3434 31.3079] [ 364.02288846 279.34265322]
[ 4.45719 -0.890039 30.1925 ] [ 738.90737229 366.01616286]
[ -0.241048 -0.781436 29.4198 ] [ 615.79477403 368.42821006]
[ 9.71565 2.76641 28.2099 ] [ 896.58718475 466.75202885]
[ -9.72456 -0.578541 28.3965 ] [ 351.64670069 372.91016582]
[ 3.95976 -4.62243 33.3605 ] [ 716.20539106 279.93901788]
[ 8.71796 -4.73768 34.2061 ] [ 824.90658662 279.37948968]
[-15.4225 -7.95261 33.4987 ] [ 257.11533368 200.38492012]
[-10.759 -8.10442 34.3725 ] [ 374.21786998 202.19909734]
[ -9.16547 3.18029 25.2879 ] [ 335.82956957 488.18787966]
[ 8.2166 -8.50777 37.2677 ] [ 797.87302812 207.99485459]
[ 0.251393 2.96439 26.5568 ] [ 629.74093457 476.94503839]
[ 4.95556 2.84818 27.3074 ] [ 765.81823876 471.39805837]
[ 9.2531 -0.996382 31.2532 ] [ 857.61665601 363.81620258]
[ -0.751971 -4.49805 32.4702 ] [ 604.03285282 280.22127036]
[ -4.4535 3.07115 26.0102 ] [ 487.44771208 481.98387392]
[ -1.26526 -8.25379 35.5274 ] [ 594.23607203 205.79036006]
[-14.1847 -0.48957 27.1355 ] [ 208.39215781 374.30642197]
[ -4.96301 -0.668829 28.9992 ] [ 487.46065075 371.08870379]
[-14.8024 -4.22344 30.2936 ] [ 234.96053223 278.18514623]
[-15.4051 -4.40168 31.5249 ] [ 234.93398339 278.01955418]
[ -5.99238 -8.19846 34.9627 ] [ 486.77529469 203.86450319]
[ -5.48118 -4.44441 32.0198 ] [ 487.21960875 279.85135589]
[-13.2512 3.09461 23.3448 ] [ 174.4254351 492.96000757]
[ 3.45682 -8.37202 36.4481 ] [ 697.60290572 207.58466561]

221
code/recording/tracker.py Normal file
View file

@ -0,0 +1,221 @@
import os, signal, subprocess
import cv2
import numpy as np
import json
LOG_DIR = "/home/mmbrian/HiWi/etra2016_mohsen/code/recording/data/log.log"
FRAMES_DIR = "/home/mmbrian/HiWi/etra2016_mohsen/code/recording/data/frames_012.npy"
ARUCO_EXECUTABLE = "/home/mmbrian/temp/aruco-1.3.0/build/utils/aruco_test "
ARUCO_SIMPLE_EXECUTABLE = "/home/mmbrian/temp/aruco-1.3.0/build/utils/aruco_simple "
RECORDING_VIDEO = "/home/mmbrian/HiWi/pupil_clone/pupil/recordings/2015_09_10/012/world.avi "
## convert videos to avi using:
## avconv -i world.mp4 -c:a copy -c:v copy world.avi
CAMERA_MATRIX = "/home/mmbrian/Pictures/chessboard_shots_new/camera.yml "
SQUARE_SIZE = "0.029"
class MyEncoder(json.JSONEncoder):
def default(self, o):
if '__dict__' in dir(o):
return o.__dict__
else:
return str(o)
class Marker(object):
def __init__(self, _id, p, Rvec, Tvec):
'''
p is the array of marker corners in 2D detected in target image
'''
self.id = _id
self.p = p
self.Rvec = Rvec
self.Tvec = Tvec
def getCenter(self):
'''
Returns 3D position of the marker's center in camera coordinate system
'''
ret = []
# Constructing 4x4 intrinsic matrix
R = cv2.Rodrigues(self.Rvec)[0]
for i, row in enumerate(R):
ret.append(np.concatenate((row, [self.Tvec[i]])))
ret.append([0, 0, 0, 1])
mat = np.array(ret)
# Applying the intrinsic matrix to marker center (wrt marker coordinate system)
return mat.dot(np.array([0, 0, 0, 1]))[:-1] # removing the 4th coordinate
def getCenterInImage(self, camera_matrix, dist_coeffs):
ret = cv2.projectPoints(np.array([(0.0, 0.0, 0.0)]), self.Rvec, self.Tvec, camera_matrix, dist_coeffs)
return ret[0][0][0]
@classmethod
def fromList(cls, data):
# np.array([np.array([curr_marker.id]), curr_marker.p, curr_marker.Rvec, curr_marker.Tvec])
return Marker(data[0][0], data[1], data[2], data[3])
# def processFrame(frame_path):
# cmd = ARUCO_SIMPLE_EXECUTABLE + frame_path + ' ' + CAMERA_MATRIX + SQUARE_SIZE
# p = subprocess.Popen('exec ' + cmd, shell=True, stdout=subprocess.PIPE)
# print '########################################################'
# print 'Begin Output'
# while True:
# line = p.stdout.readline()
# if line != '':
# #the real code does filtering here
# print line.rstrip()
# else:
# break
# print 'End Output'
# print '########################################################'
# # p.terminate()
# p.kill()
# # os.killpg(p.pid, signal.SIGTERM)
def readCameraParams(cam_mat = None):
'''
Reads an openCV camera.yml file and returns camera_matrix and distortion_coefficients
'''
if not cam_mat:
cam_mat = CAMERA_MATRIX
data = ''.join(open(cam_mat.strip(), 'r').readlines()).replace('\n', '').lower()
try:
ind1 = data.index('[', data.index('camera_matrix'))
ind2 = data.index(']', ind1)
camera_matrix = eval(data[ind1:ind2+1])
camera_matrix = np.array([camera_matrix[:3],
camera_matrix[3:6],
camera_matrix[6:]])
ind1 = data.index('[', data.index('distortion_coefficients'))
ind2 = data.index(']', ind1)
dist_coeffs = np.array(eval(data[ind1:ind2+1]))
return camera_matrix, dist_coeffs
except Exception: