Try this
https://huggingface.co/PhilipC/HumanOmniV2
https://github.com/HumanMLLM/HumanOmniV2
HumanOmniV2: a model that understands the context of videos.
Alibaba Group developed HumanOmniV2, a model based on Qwen2.5-Omni-7B-thinker, which acquired the skill of understanding visual context by changing the model's thinking process itself. It was taught to follow a strict format: first describe the context, then reason, and only then give an answer.
Now, before answering a question, the model generates a detailed description of the scene in the <context> tag. At this stage, it records who is doing what, what the background is, what sounds are heard. Only after that, in the <think> tag, it builds a logical chain of reasoning, linking the question with the collected context. And only at the end, it produces the result in the <answer> tag.
To make this approach work, it was reinforced with a reward system based on RL. The model receives standard rewards for accuracy and correct format, but two new ones were introduced:
"Context reward" is given if its description is complete and relevant, and the quality of this description is assessed by another, more powerful LLM;
"Logical reward" checks that the model actually used data from video and audio in its reasoning, and did not ignore them.
To evaluate HumanOmniV2, the IntentBench benchmark was created (633 videos, 2689 questions) based on Social-IQ 2.0, EMER and MDPE.
Its trick is that questions require simultaneous analysis of: video sequence (gestures, microexpressions), dialogues (tone, meaning of lines) and social context (irony, deception, hidden intentions).
The test model outperformed open analogues on 3 benchmarks:
Daily-Omni: 58.47% (53.13% for MiniCPM-o 2.6);
WorldSense: 47.1% (45.4% for Qwen2.5-Omni);
IntentBench: 69.33% (64.20% for Qwen2.5-Omni).
Licensing: Apache 2.0 License.
