Deviant Login Shop  Join deviantART for FREE Take the Tour
×

:icontao63: More from tao63




Details

Submitted on
September 9, 2013
Link
Thumb

Stats

Views
4,806 (2 today)
Favourites
45 (who?)
Comments
21
×
Changelog
9/10/13 - Published
9/19/13 - Fixed some errors




TABLE OF CONTENTS





=======================================================
INTRODUCTION
This tutorial is for MikuMikuDance (MMD) with versions at the time of writing. This is written as of 9/10/13. This tutorial will only discuss manual lip syncing done within MMD and no other software. There will be no tutorial on how to use pre-made VSQs or plugins to automatically input the lip syncs, though it is encouraged for others to use that in order for others not to waste too much time. There will be two parts of this tutorial, one will be for Japanese language lip syncing and English language lip syncing. Some people will call the facial expressions as "Morphs" while others will call them "Facial Sliders", both are correct. MikuMikuMoving has interpolation for morphs but considering that this tutorial is for MMD only.

Back to TOC




=======================================================
IMPORTANT NOTES

MMD MODELS
In MMD, the models are usually made by Japanese modelers so the order is usually following a, i, u, e, o for vowels instead of the English a, e, i, o, u. Just a reminder so others will not be confused as sometimes some models do not have English named bones and facial sliders.

LIMITATIONS
In MMD the models are based on manga/anime character designs. This design has many details removed and modifications to achieve attractive looks. Sometimes tongue and/or teeth are missing. Unfortunately because of that they had to use different or "cartoony" facial expressions to achieve emotions. What's more is that it completely limits the mouth movements to look like they are shouting. Because of this sometimes the mouth tends to have:


...Uncanny mouth movements. If you want to get a detailed explanation for why it looks a bit creepy, you can search on Google or Youtube about the uncanny valley. It's a short and good read.

Another limitation is that the model's facial sliders heavily depend on the modelers. Take a look at this


She's screaming. Yes she does, use your imagination. This model can only open her lips as wide as this.

DIFFERENCE BETWEEN LIP SYNCING JAPANESE AND ENGLISH LANGUAGES
Japanese Language

The Japanese and other similar languages such as Filipino are very different to English in terms of how they are pronounced. There's a very complex phonology for this subject so for simplicity sake let's just call them Hard which is derived from their words that sounds like all syllables are stressed.
It is good to have a basic idea of how the Japanese pronounce their words. Take a look at the picture below


Taken from hijapanese.blogspot.com/2008/0…

Their words are simply based on their vowels and adding a consonant before it. This makes it all of their words to sound like they are all stressed. Their words always ends with a vowel with the exception of the "nn" sound but that is a special character for them that it is completely separated to the "na" syllables. Their syllables are easily separated. The Japanese also use the "Hard" pronunciation even when they speak English words. For example when they say "easy mode" they pronounce it as "ii-zi (easy) mo-do (mode)" Also they do not have the "L" letter for them (As you can see on their alphabet above) they substitute it with "R" since most of these words that has the letter L are pretty much just borrowed words for them.

English Language
The English language and similar languages such as the Chinese language phonology is way more complex when compared to the Japanese. This is because despite their words having few syllables, the way the words are pronounced can be plenty. For example in English, the word "Safe" is a single syllable word yet it is read as "ss-ey-ff" rather than "se-ii-fu" in Japanese. The transition when pronouncing words is smooth so for simplicity sake, I'll call them as Soft words. It is possible that there are some "Hard" words mixed in these languages like the Korean language so it is up to the animator on how they will approach it.

FAQs
Q: Can I mix two sliders together? Example, mixing "a" slider with "o" slider
A: It actually depends on you. You should ask yourself first if you intend to use the animation with other models. This is because even though it looks good on the first model, other models might produce undesirable results. Usually they are deformed mouths or missing sliders. So it is recommended to do it only if you are going to use it on another model of same base or you will be using it for a single model.

Q: Do I really have to use the "a" slider for the "a" sound? Can I use something else?
A: Yes, you can use others what you think looks better, but same idea for the question above that only do it for models of same base when using a different model other than the one you animated it with.

Q: My character keeps moving! I can't do lip syncing!
A: You can do lip syncing on a steady character. Works well on a separate pmm work file and then save the lip sync animation and then load it on a model that is dancing.



Back to TOC





=======================================================
HOW TO LIP SYNC IN MMD
The reason why I separated the languages into two categories (Hard and soft) is because they are done differently, but not that much. It is recommended that you have a readable lyrics prepared like one in an opened browser or notepad or whichever software you want to put the text. This is important to make your mouth movement more accurate than doing it by ear.

Japanese words Lip syncing (Hard)
Here's a sample


Song: No Logic
Lyrics:
Kami-sama, kono uta ga kikoeru kai  Anata ga nozo'nde inakutemo
Boku wa waratteitai'ndesu  Soshite ima sakebitai'ndesu
Itsudatte saigo wa No Logic
Bokura no kono ichidou kiri no mu  Douse itsuka owari ga kuru nara
Ikiru koto o yametai toki dake tachi yamatte kangaeru kurai de, ii yo ne

"Yaritai koto dake erande, iranai mono kiri sutete"
Dare mo ga minna sonna kuu ni arukeru wake...nai yo ne



Whenever I lip sync, I usually separate a single lip movement into four parts. See below


The image should be self explanatory. I only placed the colors for convenience but they are not actually in MMD

As for lip syncing it usually goes like this:



First take a look at how I set the sliders for each part. On the blue keyframe the lip should be set to zero or left most slide. The green keyframe should be set to a certain amount of openness you prefer. It is important that the blue and green keyframes to be set BEFORE or DURING the first sound. The next one is the yellow in which the slider should be the same as the green keyframe to "pause" it or prevent any movement. Usually I place the blue keyframe for the next lip movement at the pause. Next is the red which is to close it. Usually I place the opening of the next lip movement at the close of the previous one just like how the picture above is shown.
Since the "mm" sound like for the syllable "mi" in the picture requires the "mm" movement but of course that do not exist on any model's sliders. In order to get this, I substituted the slider for the closed lip movement by putting the all lip sliders to left or zero value. Take a look at the purple circle or arrow, there is no slider for "mm" so you have to use your creativity to substitute missing sliders.

English words Lip syncing (Soft)
English lip sync sample I have done a long time ago



Half Life 2 intro G-man scene
Dialogue:
Rise and shine Mr. Freeman rise and shine
Not that I wish to imply that you have been sleeping on the job
No one is more deserving of a rest
And all the effort in the world would have gone to waste
Until...Well... Let's just say your hour has come again.
The right man in the wrong place can make all the difference in the world
So wake mr. Freeman, wake up and smell the ashes


Lip syncing English words are a bit more difficult due to the fact that you need to make smoother transitions between sounds. Take a look at the example below:



As you can see there is no pause in order to make a smoother transition. The beginning of the next sound should be set on the opening of the first sound. This is more difficult as you have to time the transition carefully. You can also add pause keyframes if required. Just a reminder that it all depends on the animator how they will tackle the movements.

For reference here's a link to the motion data of both Japanese and English lip syncs.
www.mediafire.com/folder/mgm5d…

That is all.

Back to TOC
Add a Comment:
 
:iconnaruko911:
naruko911 Featured By Owner Sep 6, 2014
would starting with the lip syncing and going back with the body animation be a good idea?
sorry if this is a dumb question or something, I've just recently tried my hand at animation.
Reply
:icontao63:
tao63 Featured By Owner Sep 6, 2014
Yes. Lip syncing can be done before or after body animation. At least for songs and dances since there is already a song to coordinate everything

However if you meant facial expressions or lip syncing for an acting then it's better to do it after body animations since it requires timing and coordinating body and lip sync animations
Reply
:iconfrostedfrozen:
FrostedFrozen Featured By Owner Mar 8, 2014  Student Digital Artist
Awesome! this helped me so much!! How do you save it with your motion data?
Reply
:icontuany-neko-daisuki:
Tuany-Neko-Daisuki Featured By Owner Dec 28, 2013
I think I understand now, but it keeps hard '-'
I'm going to practice now, maybe I'll understand better practicing ^^
Reply
:iconreishaterrin:
ReishaTerrin Featured By Owner Sep 26, 2013  Hobbyist General Artist
>.>

I've been practicing lip-syncing...  i did this today, what do you think? I think the first half is better then the second half since it was rendered in parts  >.<  i hope to get better


Reply
:icontao63:
tao63 Featured By Owner Sep 26, 2013
Well the timing is already good actually. Though you might want to add a bit more lip movement. For example when he says "dangerous" it seems you just used 3 lip movements ("a" "close lips" and "e") when in fact the dangerous word is pronounced de-in-ge-rus so that's 4 lip movements
Reply
:iconreishaterrin:
ReishaTerrin Featured By Owner Sep 26, 2013  Hobbyist General Artist
*nod nod* okay i'll remember that on my next practice~  Face and Lips program helps but i still need more practice with it 
Reply
:iconspensieratezza:
Spensieratezza Featured By Owner Sep 18, 2013
That singing Haruhi always creeps me out
Reply
:icontao63:
tao63 Featured By Owner Sep 18, 2013
The most likely explanation is the uncanny valley which I found out on Vsauce. Really disturbing stuff
Reply
:iconyuuu-chan:
Yuuu-chan Featured By Owner Sep 16, 2013
Very awesome tutorial! Thank you very much!
This would greatly help~ > u <

Just a question tho, Japanese lip syncing can be, also, applied to Filipino songs, right?
Reply
Add a Comment: