From 3ea202956fecc3ecd6024c0adc93637386f1bb85 Mon Sep 17 00:00:00 2001 From: Xucong Zhang Date: Mon, 28 Jan 2019 13:42:07 +0100 Subject: [PATCH] Created API calls (markdown) --- API-calls.md | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100644 API-calls.md diff --git a/API-calls.md b/API-calls.md new file mode 100644 index 0000000..32d3659 --- /dev/null +++ b/API-calls.md @@ -0,0 +1,46 @@ +# Introduction +OpenGaze has the following modules: +* **opengaze**: the high-level class to call other functions, including three common use examples (see below). +* **input_handler**: a basic utility module handling the input types and data. +* **gaze_estimator**: a module to perform the gaze estimation pipeline, including face detection, head pose estimation, data normalization and predict gaze from the input face/eye image patch. +* **face_detector**: a face detection module with the third-party library to perform both face and facial landmark detection. +* **normalizer**: the data normalization module is used to extract the face/eye image patch from the input image according to 3D head pose and target center. It can compensate part of variation caused by head poses. +* **gaze_predictor**: the core module for gaze estimation. It takes an input eye/face patch and outputs the 3D gaze direction in the (normalized) camera coordinate system. +* **data**: a module that defines the stored data structure across different modules. +* **personal_calibrator**: a module to allow the user to calibrate the gaze estimation results on the 2D screen plane. + +# Get started +You can familiarize yourself with the API by exploring the main executable class OpenGaze with files `opengaze.hpp` and `opengaze.cpp`.
+You will find three examples we wrote to demonstrate how to use OpenGaze for gaze visualization, gaze estimation, and personal calibration in the "exe" directory. + +## Gaze Visualization +The main function to call for this example is `OpenGaze::runGazeEstimation()`. After initializing the input by the `InputHandler` class, a gaze estimation task can be simply run as:
+```vector output; + Mat input_image = input_handler_.getNextSample(); + Mat undist_img; + undistort(input_image, undist_img, input_handler_.camera_matrix_, input_handler_.camera_distortion_); + gaze_estimator_.estimateGaze(undist_img, output); +``` +The 3D gaze direction vector will be stored in `output[i].gaze_data.gaze3d` where `i` indicates the `ith` user inthe scene. + +## Gaze Estimation +The main function to call for this example is `OpenGaze::runGazeOnScreen()`. +You still need to estimate a 3D gaze vector first, and then simply run:
+`input_handler_.projectToDisplay(output, true);`
+to calculate the intersection of the gaze vector and screen plane. This will give you the estimated 2D location on the screen stored in `output[i].gaze_data.gaze2d`.
+Note that before this calculation, you have to provide correct calibration information. this calibration information is: +* `calibration.yml` - located in OpenGaze/content/calib/ directory, and stores the camera intrinsic parameters. These parameters can be set with the standard camera calibration procedure provided by [OpenCV](https://docs.opencv.org/3.1.0/dc/dbb/tutorial_py_calibration.html).
+* `monitor.yml` - located in OpenGaze/content/calib/ directory, and stores the rotation and translation between camera and screen. This information can be obtained with the [mirror-based camera-screen calibration method](https://dl.acm.org/citation.cfm?id=1888118). + +## Personal Calibration +The main function to call for this example is `OpenGaze::runPersonalCalibration(int num_calibration_point)`.
+The main class `PersonalCalibrator` can be initialized by:
+`PersonalCalibrator m_calibrator(input_handler_.getScreenWidth(), input_handler_.getScreenHeight());`
+It needs to know the screen size in pixels in order to show the corresponding stimuli on the screen.
+The locations of stimuli are generated randomly by specifying the number of stimulis:
+`m_calibrator.generatePoints(num_calibration_point);`
+Then each stimulus can be shown by calling:
+`m_calibrator.showNextPoint()`
+For each stimulus point, the gaze estimation module estimates the coresponding gaze point on the screen. The stimuli locations and estimated gaze points can be used to generate a personal model by:
+`m_calibrator.generateModel(prediction, ground-truth, 1)`,
+where the last parameter indicates the order of the mapping function.