Metrics Guide

What every metric measures, the values it produces, and how to interpret them. Metric semantics are identical on all platforms — only the identifier spelling differs (see the per-platform API reference for names). Enum names below use the C++ spelling.

Reading metric values

Metrics are delivered per tracked face, per processed frame, through your listener. Three value shapes appear throughout the SDK:

• Continuous scores (0–100). Most expressions and emotions. The value expresses the degree of evidence that the facial action or emotion is present — 0 means absent, 100 means unmistakably present at full intensity. Values rise and fall smoothly as the expression forms and fades; there is no built-in notion of “detected” below 100, so you choose the threshold that fits your application (see best practices).
• Binary events (0 or 100). BLINK, and YAWN. The SDK has already applied a decision threshold (and, where noted, temporal filtering), so the value simply toggles between “not occurring” (0) and “occurring” (100).
• Physical measurements. Head pose in degrees, distances and coordinates in pixels. These are signed geometric quantities, not scores.

Choosing features — and what it costs

Metric families are gated by features, and the enabled feature set directly controls how much CPU each frame costs: the SDK activates only the classifier models required by the features you enable, and skips the rest entirely. Enabling less genuinely computes less.

What is enabled by default

Nothing. A newly constructed detector has no features enabled, and start() throws an exception (“at least one Feature must be enabled before calling the start method”) if you start it that way. You must enable at least one feature. The tracking, measurement, and image-quality metrics have no feature flag of their own — they activate automatically alongside the first feature you enable and are delivered for every tracked face.

The three features

Enabling and disabling

Enable features on the detector before start(); you can also toggle them later — changes take effect from the next processed frame. With SyncFrameDetector, toggle between process() calls; with the asynchronous FrameDetector, prefer configuring the feature set before start(), since processing runs on a background thread.

// Cheapest configuration: tracking only (presence, bounding box, head pose, quality).
detector.enable(Feature::FACES);

// Add analysis families only when you need them:
detector.enable(Feature::EXPRESSIONS);
detector.enable({Feature::EXPRESSIONS, Feature::EMOTIONS});   // or several at once

// Drop a family you no longer need — e.g. between videos, or when your app
// leaves the screen that uses it. Takes effect from the next frame:
detector.disable(Feature::EMOTIONS);

// Introspection:
bool haveExpressions = detector.isEnabled(Feature::EXPRESSIONS);
std::set<Feature> enabled = detector.getEnabledFeatures();

# Cheapest configuration: tracking only (presence, bounding box, head pose, quality).
detector.enable_feature(af.Feature.faces)

# Add analysis families only when you need them:
detector.enable_feature(af.Feature.expressions)
detector.enable_features({af.Feature.expressions, af.Feature.emotions})

# Drop a family you no longer need. Takes effect from the next frame:
detector.disable_feature(af.Feature.emotions)

# Introspection:
have_expressions = detector.is_enabled(af.Feature.expressions)
enabled = detector.get_enabled_features()

// Android exposes EXPRESSIONS and EMOTIONS (no FACES-only mode).
detector.enable(Feature.EXPRESSIONS);
detector.enable(Arrays.asList(Feature.EXPRESSIONS, Feature.EMOTIONS));

// Drop a family you no longer need. Takes effect from the next frame:
detector.disable(Feature.EMOTIONS);

// Introspection:
boolean haveExpressions = detector.isEnabled(Feature.EXPRESSIONS);

Does disabling features help performance?

Yes — with one structural caveat. What each configuration actually computes:

• FACES only skips every expression and emotion model. This is the configuration to use for presence/head-pose applications, and the savings are large.
• Disabling EMOTIONS while keeping EXPRESSIONS saves a little: the emotion metrics are lightweight combinations of the expression classifier outputs, which still run.
• Disabling EXPRESSIONS while keeping EMOTIONS saves little compute: the emotion metrics still require most expression classifiers internally (though the dedicated pain and behavior-signal models are skipped). It mainly changes what is reported.
• Face detection and tracking always run while the detector is started — that cost is independent of the feature set.

Individual metrics inside a family (say, a single Action Unit) cannot be toggled through the public API — the feature families are the granularity. For further performance control, the two biggest levers are elsewhere: maxNumFaces (analysis cost scales with the number of tracked faces) and, on the asynchronous FrameDetector, processFrameRate (process fewer frames per second).

Face tracking metrics

Always delivered for every tracked face, regardless of which features are enabled.

Read

Tracking metrics are delivered for every tracked face. Read them from the Face objects handed to your per-frame results callback:

// In FaceListener::onFaceResults(faces, frame):
for (const auto& entry : faces)
{
  const Face& face = entry.second;
  float confidence = face.getConfidence();   // 0-100
  auto  box        = face.getBoundingBox();  // {top-left, bottom-right}, pixels
  auto  points     = face.getFacePoints();   // std::map<FacePoint, Point>
}

# In FaceListener.results_updated(faces, frame):
for face_id, face in faces.items():
    confidence = face.get_confidence()    # 0-100
    box        = face.get_bounding_box()  # [top-left, bottom-right], pixels
    points     = face.get_face_points()   # {af.FacePoint: af.Point}

// In ImageListener.onImageResults(faces, frame):
for (Face face : faces.values()) {
    float confidence = face.getConfidence();                    // 0-100
    List<Point> box = face.getBoundingBox();                    // top-left, bottom-right
    Map<Face.FacePoint, Point> points = face.getFacePoints();
}

Expression metrics

Expression metrics measure the movement of facial muscles. Most correspond to Action Units (AUs) from the Facial Action Coding System (FACS), the standard taxonomy of facial muscle movements — each is estimated by a classifier trained on that specific action. A score of 0 means the action is absent; 100 means it is present at full intensity. Expressions are independent: several can (and usually do) score high at once.

Enabled with Feature::EXPRESSIONS.

Enable & read

Enable the feature before start(); read the values from getExpressions() in your per-frame results callback — one map entry per expression metric listed below:

detector.enable(Feature::EXPRESSIONS);   // before start()

// In FaceListener::onFaceResults(faces, frame):
auto expressions = face.getExpressions();               // std::map<Expression, float>
float smile     = expressions[Expression::SMILE];       // 0-100
float blinkRate = expressions[Expression::BLINK_RATE];  // NaN for the first 10 s

detector.enable_features({af.Feature.expressions})   # before start()

# In FaceListener.results_updated(faces, frame):
expressions = face.get_expressions()             # dict {af.Expression: float}
smile      = expressions[af.Expression.smile]    # 0-100
blink_rate = expressions[af.Expression.blink_rate]  # NaN for the first 10 s

detector.enable(Arrays.asList(Feature.EXPRESSIONS));   // before start()

// In ImageListener.onImageResults(faces, frame):
Map<Face.Expression, Float> expressions = face.getExpressions();
Float smile     = expressions.get(Face.Expression.SMILE);       // 0-100
Float blinkRate = expressions.get(Face.Expression.BLINK_RATE);  // NaN for the first 10 s

FACS Action Unit reference

Every expression metric in this guide is named after an Action Unit — a single facial muscle movement in the Facial Action Coding System.

Brow and forehead

Eyes and eyelids

Nose and upper lip

Lip corners and cheeks

Lips and jaw

Head position

Brow and forehead

Eyes and cheeks

Nose and upper lip

Mouth and jaw

Behavior signals

Emotion metrics

Emotion metrics are computed from combinations of the expression metrics above, following the EMFACS mapping between facial action units and emotional expressions. Each score expresses the degree of evidence for that emotional expression: 0 = absent, 100 = fully present. Because they share underlying facial actions, related emotions can rise together — treat them as a profile, not as mutually exclusive classes.

Enabled with Feature::EMOTIONS.

Enable & read

Enable the feature before start(); read the values from getEmotions() in your per-frame results callback — one map entry per emotion metric listed below:

detector.enable(Feature::EMOTIONS);   // before start()

// In FaceListener::onFaceResults(faces, frame):
auto emotions = face.getEmotions();          // std::map<Emotion, float>
float joy     = emotions[Emotion::JOY];      // 0-100
float valence = emotions[Emotion::VALENCE];  // -100 .. +100

detector.enable_features({af.Feature.emotions})   # before start()

# In FaceListener.results_updated(faces, frame):
emotions = face.get_emotions()            # dict {af.Emotion: float}
joy      = emotions[af.Emotion.joy]       # 0-100
valence  = emotions[af.Emotion.valence]   # -100 .. +100

detector.enable(Arrays.asList(Feature.EMOTIONS));   // before start()

// In ImageListener.onImageResults(faces, frame):
Map<Face.Emotion, Float> emotions = face.getEmotions();
Float joy     = emotions.get(Face.Emotion.JOY);      // 0-100
Float valence = emotions.get(Face.Emotion.VALENCE);  // -100 .. +100

The seven expressed emotions + neutral

Composite and contextual measures

Measurement metrics

Physical, signed quantities — delivered whenever a face is tracked (no feature flag needed). Head pose angles are in degrees, relative to a reference position of the face looking straight at the camera (pitch/yaw/roll = 0/0/0). Directions are from the person’s perspective, not the camera’s.

Read

Measurements are delivered whenever a face is tracked. Read them from getMeasurements() in your per-frame results callback:

// In FaceListener::onFaceResults(faces, frame):
auto measurements = face.getMeasurements();   // std::map<Measurement, float>
float yaw = measurements[Measurement::YAW];                   // degrees
float iod = measurements[Measurement::INTEROCULAR_DISTANCE];  // pixels

# In FaceListener.results_updated(faces, frame):
measurements = face.get_measurements()   # dict {af.Measurement: float}
yaw = measurements[af.Measurement.yaw]                    # degrees
iod = measurements[af.Measurement.interocular_distance]   # pixels

// In ImageListener.onImageResults(faces, frame):
Map<Face.Measurement, Float> measurements = face.getMeasurements();
Float yaw = measurements.get(Face.Measurement.YAW);                   // degrees
Float iod = measurements.get(Face.Measurement.INTEROCULAR_DISTANCE);  // pixels

Image quality metrics

Read

Quality metrics are delivered whenever a face is tracked. Read them from getQualities() in your per-frame results callback:

// In FaceListener::onFaceResults(faces, frame):
auto qualities = face.getQualities();                 // std::map<Quality, float>
float brightness = qualities[Quality::BRIGHTNESS];    // 0-100

# In FaceListener.results_updated(faces, frame):
brightness = face.get_qualities()[af.Quality.brightness]   # 0-100

// In ImageListener.onImageResults(faces, frame):
Float brightness = face.getQualities().get(Face.Quality.BRIGHTNESS);   // 0-100

Interpretation best practices

• Choose thresholds per use case. For continuous 0–100 scores, 50 is a sensible starting point for “present”, but the right cutoff depends on your costs: use higher thresholds (70–90) when false positives are expensive, lower ones (20–30) when you must not miss subtle expressions.
• Aggregate over time. Single-frame values are noisy by nature. For most decisions, use a rolling statistic — e.g. the mean over the last second, or “above threshold for N consecutive frames” — rather than reacting to individual frames. (The binary event metrics already do this internally.)
• Gate on tracking quality. Discard or down-weight readings when getConfidence() is low, |yaw|/|pitch| are extreme, interocular distance is small, or BRIGHTNESS is at either extreme — expression estimates degrade with tracking quality.
• Mind speech. Talking moves the mouth constantly. When subjects speak, prefer CONVERSATIONAL_VALENCE/CONVERSATIONAL_ENGAGEMENT, or gate mouth-based expressions on the SPEAKING metric.
• Read profiles, not single labels. Emotions share facial actions and are not mutually exclusive. Comparing the full emotion profile (or simply valence + engagement) is more robust than picking the single highest emotion.