
Авторы:
Издательство:
Дата издания
12.03.2026На каком языке издана
ISSN
2504-2289Область исследования
Тип исследования
Аннотация (краткое описание)
Multimodal human–AI systems generally consider facial expressions and body motions as separate input streams, leading to disjointed interpretations and diminished emotional coherence. To overcome this issue, we offer the Engagement-Safe Expressive Alignment (ESEA) paradigm and the Unified Visual Synchrony (UVS) framework as its computational implementation. UVS models the coherence between facial expressions and gestures, offering an interpretable visual synchrony signal that can function as adaptive feedback in human–AI interactions. The framework’s key component is the Consistency Index for Affective Synchrony (𝐶𝐼𝐴𝑆 ), which correlates brief visual segments with scalar synchrony scores through a common latent representation. Facial and gestural signals are processed by modality-specific projection networks into a unified latent space, and 𝐶𝐼𝐴𝑆 is derived from the similarity and short-term temporal consistency of these latent trajectories. The synchrony index is regarded as an estimation of affective visual coherence within the ESEA paradigm. We formalize the UVS/𝐶𝐼𝐴𝑆 framework and conduct a comparative experimental evaluation utilizing matched and mismatched face–gesture segments derived from rendered dialog footage. Utilizing ROC analysis, score distribution comparisons, temporal visualizations, and negative control tests, we illustrate that 𝐶𝐼𝐴𝑆 effectively captures structured face–gesture alignment that surpasses similarity-based baselines, while also delivering a persistent, time-resolved synchronization signal. These findings establish 𝐶𝐼𝐴𝑆 as a principled and interpretable feedback signal for future affect-aware, engagement-focused multimodal agents.DOI
10.3390/bdcc10030088Тип публикаций
Источник квартиля
SCImago (Scopus)Impact Factor
4.4Данный документ располагается в коллекциях
Международные публикации ученых SU
Scopus
Файлы документа
BDCC-10-00088.pdf