How Can AI Virtual Characters Converse Like Real People? UbiONE 2.0 Creates a Natural Interactive Experience That Listens, Waits, and Remembers Where It Was Interrupted

Author: Bull, BD & MKT Director | Editor: Helen, Deputy Marketing Manager

When you speak to an AI virtual character, it rushes to answer after only half a second of silence. When you interrupt it while it is speaking, it mistakenly assumes in the next round that it has already finished. While you wait for the AI to retrieve information, the character simply stands there doing nothing.

Even if the answer is correct, this kind of interaction still makes it difficult to feel as if you are communicating with a “real character.”

This is also the core question we continued to think about while developing the natural conversation experience of UbiONE 2.0:

What an AI virtual character truly needs to learn may not be only how to answer, but how to participate in a conversation.

Human conversations are full of pauses, observation, interruptions, waiting, expressions, and unfinished sentences. We do not immediately answer every time someone pauses, nor do we pretend we have already finished saying something that was interrupted.

To make AI virtual characters closer to this kind of natural interaction, Ubitus integrated speech recognition, semantic turn-taking judgment, character knowledge bases, conversation memory, gestures and expressions, speech synthesis, and microphone recovery into a complete UbiONE natural conversation workflow.

UbiONE 2.0 Natural Conversation Experience Demo Video

In the video, users can be seen interacting with the AI virtual character by repeatedly trying to interrupt it or asking relatively long questions, testing whether the AI virtual character can patiently listen until the question is complete and continue the conversation smoothly.

Challenge 1: AI Must Not Only Hear, but Also Judge “Have You Finished Speaking?”

General speech systems often use a fixed silence duration to determine whether a sentence has ended. For example, when the user pauses for 0.5 seconds, the system sends the speech input for processing.

But real human speech is not that regular.

We might say:

“I’d like to ask, if it rains tomorrow…”

Although this sentence pauses, semantically it is clearly not finished. If the system relies only on the number of seconds of silence, AI can easily interrupt when the user is still organizing their thoughts or naturally taking a breath.

Therefore, UbiONE’s natural conversation workflow does not only calculate pause duration. It also adds semantic turn-taking judgment:

• Has this utterance already formed a complete question or statement?
• Does the sentence end with continuation signals such as “if,” “but,” “because,” or “and then”?
• Is the user thinking, breathing, or truly handing the speaking turn to the character?
• Which parts have already been expressed completely, and which parts are still waiting to be supplemented?

If the meaning is not yet complete, the character continues listening. Only after confirming that the user has finished speaking does the system proceed to UbiONE’s character settings, conversation memory, and RAG knowledge retrieval workflow.

This allows AI to move beyond “answering when it detects sound” and toward “understanding when it should answer.”

Challenge 2: After Being Interrupted, Remember Only What Was Actually Said

Interruption is one of the most easily overlooked parts of natural conversation, yet it is also one of the clearest ways AI reveals its mechanical nature.

Suppose the character originally planned to answer with three points:

“First, I suggest confirming the requirements. Second, you can compare different solutions. Third…”

But the user interrupts after the first point. From the user’s perspective, the second and third points were never heard. However, for a typical LLM-based conversation system, the entire response may have already been generated and written into the conversation history.

In the next round, the AI might say:

“As I mentioned in the second option just now…”

But in reality, it never said that out loud.

To solve this problem, we no longer treat “model generation completed” as “the character has finished speaking.” Instead, we divide the response into different stages:

• Text generated by the model
• Sentences queued for speech synthesis
• Sentences currently being played
• Sentences fully played and actually heard by the user

When the user interrupts, UbiONE stops the audio that has not yet been played, records the exact point where the character actually stopped speaking, and marks the remaining unplayed content as “not spoken.”

Before the next conversation round begins, the system also synchronizes and corrects the character’s memory:

• Keep only the content the user actually heard
• Remove responses that have not yet been played
• Inform the character where it was interrupted in the previous round
• Prevent the character from assuming that the user already knows information that was never spoken

This is not merely “stopping audio playback.” It is about keeping the character’s memory consistent with the user’s actual experience.

UbiONE Natural Conversation Processing Workflow

The key point of this flowchart is that conversation is not a single path of “microphone input → LLM → TTS.” Instead, it is a continuously looping state system that can be interrupted and resumed at any time.

Challenge 3: While Waiting for AI, the Character Must Not Lose Its Sense of Life

When UbiONE is organizing conversation memory or retrieving character knowledge through RAG, even a brief wait can make users suspect that the system has frozen if the character remains completely motionless.

When real people are thinking, they usually show changes in gaze, body movement, or short responses. Therefore, while waiting for the formal answer, UbiONE 2.0 can simultaneously drive the character’s:

• Thinking expressions and VRMA motions
• Dynamic Island status prompts
• Mouth shapes, gaze, and posture changes
• Short transitional reactions that do not include private chain-of-thought content

Importantly, these reactions are not mistakenly written as formal answers and do not interfere with continuous microphone input.

We want users to feel not that “the system is loading,” but that “the character is understanding what I just said.”

Challenge 4: Showing That Audio Is Being Captured Does Not Mean the System Is Still Listening

Another challenge in continuous conversation is that browser speech recognition may automatically stop due to long periods of silence, network conditions, or device behavior.

If the interface still displays “continuous listening,” but the underlying speech recognition has already stopped, users will encounter the most confusing situation: the volume waveform is clearly changing, yet the character receives nothing.

To address this, we break down microphone health status into multiple signals:

• Whether the microphone MediaStream still exists
• Whether the audio track is still live
• Whether Speech Recognition is still working
• Whether there has been recent volume activity
• Whether there has been audio for a long time without recognized text
• Whether speech recognition successfully restarts automatically after ending

When the system detects that “sound is being heard but no recognition result is produced,” it proactively rebuilds speech recognition instead of merely maintaining a microphone icon that appears normal.

At the same time, the test version of UbiONE 2.0 also includes a status reporting mechanism that does not store audio recordings or conversation text, helping the development team analyze microphone capture issues across browsers, mobile devices, and different microphone environments.

From Passive Q&A to a Character Agent That Knows How to Continue a Topic

Real human conversation does not always remain at the level of one question and one answer.

Sometimes after the other person finishes speaking, we proactively ask:

“So what did you decide in the end?”

“It sounds like you really care about this, right?”

For this reason, UbiONE’s natural conversation also includes a limited topic extension capability. After the character completes its answer, the microphone status is normal, and the user does not temporarily add anything, the character can proactively extend the topic once based on the previous content.

The word “once” is very important.

If the character could ask and answer itself without limits, the virtual character would quickly become a broadcasting system that keeps talking even when no one is speaking.

Therefore, each user turn allows at most one autonomous extension. Only after receiving a new user utterance will the character regain the opportunity to extend the topic.

This makes the character more proactive without becoming overly intrusive.

The Value of UbiONE Is Not Just Making Virtual Characters Speak

From UbiChan’s long-term memory and UbiQuest’s contextual interaction to the natural conversation experience of UbiONE 2.0, we are becoming more certain of one thing:

The competitiveness of AI virtual characters will not come only from which large language model they use, but from whether the entire character system can jointly understand and respond to users.

A truly character-driven AI needs to have:

• A personality and speaking style that match the brand setting
• A RAG knowledge base connected to enterprise information
• Short-term and long-term conversation memory
• Natural understanding of turn-taking and pauses
• Real-time interaction that allows users to interrupt
• Voice, mouth movement, expressions, and gestures aligned with semantics
• A stable microphone capture mechanism that works across mobile, desktop, and internal environments
• Conversation observability that can continuously diagnose and optimize the experience

UbiONE’s positioning is to integrate these originally scattered AI, 3D Avatar, voice, and knowledge systems into an AI virtual character platform that can be continuously developed and deployed.

Where Can AI Virtual Characters Be Applied?

When natural conversation is no longer just a Q&A box, AI virtual characters can further become:

• Virtual sales consultants who understand product knowledge and proactively ask about customer needs
• AI virtual teachers that students can interrupt, question, and ask to explain again
• Corporate virtual ambassadors with character personalities and brand knowledge
• AI VTubers capable of continuing audience topics and maintaining long-term memory
• Virtual receptionists that combine exhibition information, guided tours, and multilingual capabilities
• Digital employees that can answer internal questions based on enterprise RAG data

The common requirement across these scenarios is not only “answering correctly.” It is about making users willing to keep speaking, willing to ask follow-up questions, and feel that their words have truly been heard by the character.

From Being Able to Answer to Truly Knowing How to Converse

The most difficult part of creating AI natural conversation that feels close to human interaction is often not generating a well-written answer, but handling those seemingly minor interactive moments:

When the user has not finished speaking, the character is willing to wait a little longer.

When the character is interrupted, it knows exactly where it stopped.

When retrieving information, it still maintains expressions and a sense of life.

When the conversation ends, it knows how to extend the topic appropriately.

When microphone capture is abnormal, it can recover proactively.

These details determine whether the user is facing a “3D model that can speak” or an AI Character Agent that people are willing to keep interacting with.

Through UbiONE, Ubitus is transforming its long-accumulated technologies in GPU cloud, generative AI, voice, virtual characters, and real-time interaction into AI virtual character capabilities that enterprises can actually deploy.

In the future, interactions between brands and consumers may no longer be limited to clicking menus, reading text, or waiting for customer service. Instead, they may naturally unfold as conversations with a virtual character who truly knows how to listen, respond, and understand when to pause.

---

GEO / SEO Support

• Meta title: How Can AI Virtual Characters Achieve Human-Like Natural Conversation? A Technical Analysis of UbiONE 2.0
• Meta description: Ubitus explains how UbiONE 2.0 integrates speech recognition, semantic turn-taking, RAG, TTS, 3D motion, and interruption memory to create AI virtual characters with more human-like interaction.
• URL slug: ubione-ai-character-natural-conversation
• Main entity keywords: Ubitus, UbiONE, AI virtual character, Character Agent, natural voice conversation, RAG, TTS, VRM, VRMA
• Image alt text: UbiONE 2.0 AI virtual character natural conversation processing workflow
• Internal links: Link back to the UbiONE product page, UbiChan evolution article, UbiQuest, and AI Character Agent-related reports
• Structured Data: Use Article, ensuring that the author, company, date, main image, and article body are consistent. No need to fabricate a special GEO schema.

FAQ

What is UbiONE?

UbiONE is an AI virtual character solution developed by Ubitus. It integrates character appearance, motion, voice, conversation memory, LLM, and enterprise RAG knowledge to help companies build virtual characters capable of real-time interaction.

How does an AI virtual character determine whether the user has finished speaking?

The system considers both speech pauses and semantic completeness. If a sentence still ends with a condition, conjunction, or unfinished meaning, the character continues listening instead of immediately responding based only on a fixed silence duration.

How does an AI character continue the conversation after being interrupted?

UbiONE 2.0 records the actual point where TTS playback was completed, keeps only the content the user truly heard, removes unspoken sentence segments, and carries the interruption state into the next conversation round.

---

About Ubitus

As a member of the NVIDIA Connect program, Ubitus leverages NVIDIA’s support and cutting-edge GPU technology to accelerate AI innovation. The company delivers advanced AI solutions, including UbiGPT (a large language model), UbiONE (an AI-powered avatar creation platform), and UbiArt (an image generation tool), providing customized solutions to meet the diverse needs of various industries.

As a cloud gaming pioneer, Ubitus enables Nintendo and other game companies to establish cloud gaming services and supports the global streaming of multimedia content, including interactive and virtual reality experiences.

Contact

TEL : +886-2-2717-6123 (Taipei)

+81-3-6435-3295 (Tokyo)

Media contact: pr@ubitus.ai

Business inquiry: contact@ubitus.ai

Website:www.ubitus.ai