Introduction

Real-time facial expression and emotion analysis, on device. The Affectiva Vision SDK detects and tracks human faces in video frames and images, and estimates facial expressions, emotions, head pose, and image quality — with all processing running locally on the CPU. No cloud connection, no GPU, and no external model files.

What the SDK provides

• Face detection and tracking — finds faces in each frame and assigns a stable identifier per face so metrics can be followed over time.
• Facial expressions — 20+ facial-action metrics (smile, brow raise, eye closure, blink and blink rate, yawn, attention, speaking, …).
• Emotions — estimates of joy, anger, sadness, surprise, fear, disgust, contempt and neutrality, along with composite measures such as valence and engagement.
• Measurements — head pose (pitch, yaw, roll, in degrees) and interocular distance (in pixels).
• Face points — pixel coordinates of key landmarks (outer eye corners, nose tip, chin tip) and a per-face bounding box.
• Image quality — a brightness metric for verifying capture conditions.

Metrics overview

Metrics are delivered per tracked face, per processed frame. The table below is a summary — the Metrics guide explains every individual metric in detail: what it represents, the facial actions behind it, its value range, and how to interpret it. Identifier spellings per platform are in the API reference.

How it works

Your application owns the camera or video source. Each captured image is wrapped in a Frame (raw pixels + timestamp) and pushed to a Detector. The detector runs the embedded neural networks and delivers results through your listener as a map of tracked Face objects.

The same five-step flow applies on every platform:

1. Construct a detector — SyncFrameDetector to process every frame on your thread, or FrameDetector for asynchronous, real-time processing that may drop frames to keep up.
2. Enable the features you need (EXPRESSIONS, EMOTIONS). Enabling only what you use saves CPU.
3. Register your listener to receive results.
4. Start the detector with your license key.
5. Process frames with strictly increasing timestamps; read metrics in your listener callback. Stop the detector when finished.

The same flow in every language

#include <SyncFrameDetector.h>
#include <FaceListener.h>
using namespace affdex::vision;
SyncFrameDetector detector;                                  // 1. construct
detector.enable({Feature::EXPRESSIONS, Feature::EMOTIONS});  // 2. enable
MyListener listener;                                         //    your FaceListener subclass
detector.setFaceListener(&listener);                         // 3. listen
detector.start("YOUR-LICENSE-KEY");                          // 4. start
Frame frame(0, 640, 480, pixels, Frame::ColorFormat::BGR);
detector.process(frame);                                     // 5. process
detector.stop();

import affvisionpy as af
detector = af.SyncFrameDetector()                            # 1. construct
detector.enable_features({af.Feature.expressions,            # 2. enable
                          af.Feature.emotions})
listener = Listener()                                        #    your af.FaceListener subclass
detector.set_face_listener(listener)                         # 3. listen
detector.start("YOUR-LICENSE-KEY")                           # 4. start
frame = af.Frame(0, 640, 480, pixels,
                 af.Frame.ColorFormat.bgr, af.Frame.Rotation.upright)
detector.process(frame)                                      # 5. process
detector.stop()

FrameDetector detector = new FrameDetector(context);         // 1. construct
detector.enable(Arrays.asList(Feature.EXPRESSIONS,           // 2. enable
                              Feature.EMOTIONS));
detector.setImageListener(resultsListener);                  // 3. listen (results on Android)
detector.start(licenseKey);                                  // 4. start
Frame frame = new Frame(640, 480, nv21,
        Frame.ColorFormat.YUV_NV21, timestampMs);
detector.process(frame);                                     // 5. process
detector.stop();
detector.dispose();                                          //    release native resources

Supported platforms

• Android packages target 64-bit ARM devices only (arm64-v8a); 32-bit devices and x86 emulator images are not supported. The native libraries are aligned for 16 KB memory pages, as required by recent Android releases.
• For platforms not listed here (for example macOS or iOS), contact your iMotions / Affectiva representative.

Hardware requirements

The SDK’s inference runs entirely on the CPU — no GPU, NPU, or special accelerator is required on any supported platform. Processing is multithreaded: by default the SDK uses number of cores − 1 worker threads, and every detector constructor accepts a maxThreads parameter to bound this.

Camera and input

• Input is raw frame data from a webcam, phone camera, or decoded video stream. The SDK does not open cameras itself; your application supplies pixel buffers.
• Accepted pixel formats: RGB, BGR, RGBA, BGRA, YUV (NV21, I420, YUY2), and 8-bit grayscale (grayscale: C++/Python only).
• Rotated inputs (90°/180°/270°) are handled by declaring the rotation on the Frame — useful for portrait-mounted cameras and phones.
• The default processing rate is 30 fps and is configurable per detector. Higher input rates are simply downsampled by the asynchronous detector.

Licensing

A valid license key is required to use the SDK. The key is checked when a detector starts, and is resolved in this order:

1. The string passed to the detector’s start(license) method, if non-empty.
2. Otherwise, the AFFECTIVA_LICENSE_KEY environment variable.

If neither is provided, start() throws an exception. To obtain a license key or discuss licensing terms, contact sales@affectiva.com.

Frequently asked questions

Get started