9/10/13 - Published
9/19/13 - Fixed some errors
TABLE OF CONTENTS
- Important notes (READ THIS)
- MMD Models
- Difference Between Lip syncing Japanese and English Languages
- How to lip sync in MMD
- Japanese words Lip syncing (Hard)
- English words Lip syncing (Soft)
This tutorial is for MikuMikuDance (MMD) with versions at the time of writing. This is written as of 9/10/13. This tutorial will only discuss manual lip syncing done within MMD and no other software. There will be no tutorial on how to use pre-made VSQs or plugins to automatically input the lip syncs, though it is encouraged for others to use that in order for others not to waste too much time. There will be two parts of this tutorial, one will be for Japanese language lip syncing and English language lip syncing. Some people will call the facial expressions as "Morphs" while others will call them "Facial Sliders", both are correct. MikuMikuMoving has interpolation for morphs but considering that this tutorial is for MMD only.
Back to TOC
In MMD, the models are usually made by Japanese modelers so the order is usually following a, i, u, e, o for vowels instead of the English a, e, i, o, u. Just a reminder so others will not be confused as sometimes some models do not have English named bones and facial sliders.
In MMD the models are based on manga/anime character designs. This design has many details removed and modifications to achieve attractive looks. Sometimes tongue and/or teeth are missing. Unfortunately because of that they had to use different or "cartoony" facial expressions to achieve emotions. What's more is that it completely limits the mouth movements to look like they are shouting. Because of this sometimes the mouth tends to have:
...Uncanny mouth movements. If you want to get a detailed explanation for why it looks a bit creepy, you can search on Google or Youtube about the uncanny valley. It's a short and good read.
Another limitation is that the model's facial sliders heavily depend on the modelers. Take a look at this
She's screaming. Yes she does, use your imagination. This model can only open her lips as wide as this.
DIFFERENCE BETWEEN LIP SYNCING JAPANESE AND ENGLISH LANGUAGES
The Japanese and other similar languages such as Filipino are very different to English in terms of how they are pronounced. There's a very complex phonology for this subject so for simplicity sake let's just call them Hard which is derived from their words that sounds like all syllables are stressed.
It is good to have a basic idea of how the Japanese pronounce their words. Take a look at the picture below
Taken from hijapanese.blogspot.com/2008/0…
Their words are simply based on their vowels and adding a consonant before it. This makes it all of their words to sound like they are all stressed. Their words always ends with a vowel with the exception of the "nn" sound but that is a special character for them that it is completely separated to the "na" syllables. Their syllables are easily separated. The Japanese also use the "Hard" pronunciation even when they speak English words. For example when they say "easy mode" they pronounce it as "ii-zi (easy) mo-do (mode)" Also they do not have the "L" letter for them (As you can see on their alphabet above) they substitute it with "R" since most of these words that has the letter L are pretty much just borrowed words for them.
The English language and similar languages such as the Chinese language phonology is way more complex when compared to the Japanese. This is because despite their words having few syllables, the way the words are pronounced can be plenty. For example in English, the word "Safe" is a single syllable word yet it is read as "ss-ey-ff" rather than "se-ii-fu" in Japanese. The transition when pronouncing words is smooth so for simplicity sake, I'll call them as Soft words. It is possible that there are some "Hard" words mixed in these languages like the Korean language so it is up to the animator on how they will approach it.
Q: Can I mix two sliders together? Example, mixing "a" slider with "o" slider
A: It actually depends on you. You should ask yourself first if you intend to use the animation with other models. This is because even though it looks good on the first model, other models might produce undesirable results. Usually they are deformed mouths or missing sliders. So it is recommended to do it only if you are going to use it on another model of same base or you will be using it for a single model.
Q: Do I really have to use the "a" slider for the "a" sound? Can I use something else?
A: Yes, you can use others what you think looks better, but same idea for the question above that only do it for models of same base when using a different model other than the one you animated it with.
Q: My character keeps moving! I can't do lip syncing!
A: You can do lip syncing on a steady character. Works well on a separate pmm work file and then save the lip sync animation and then load it on a model that is dancing.
Back to TOC
HOW TO LIP SYNC IN MMD
The reason why I separated the languages into two categories (Hard and soft) is because they are done differently, but not that much. It is recommended that you have a readable lyrics prepared like one in an opened browser or notepad or whichever software you want to put the text. This is important to make your mouth movement more accurate than doing it by ear.
Japanese words Lip syncing (Hard)
Here's a sample
Song: No Logic
Kami-sama, kono uta ga kikoeru kai Anata ga nozo'nde inakutemo
Boku wa waratteitai'ndesu Soshite ima sakebitai'ndesu
Itsudatte saigo wa No Logic
Bokura no kono ichidou kiri no mu Douse itsuka owari ga kuru nara
Ikiru koto o yametai toki dake tachi yamatte kangaeru kurai de, ii yo ne
"Yaritai koto dake erande, iranai mono kiri sutete"
Dare mo ga minna sonna kuu ni arukeru wake...nai yo ne
Whenever I lip sync, I usually separate a single lip movement into four parts. See below
The image should be self explanatory. I only placed the colors for convenience but they are not actually in MMD
As for lip syncing it usually goes like this:
First take a look at how I set the sliders for each part. On the blue keyframe the lip should be set to zero or left most slide. The green keyframe should be set to a certain amount of openness you prefer. It is important that the blue and green keyframes to be set BEFORE or DURING the first sound. The next one is the yellow in which the slider should be the same as the green keyframe to "pause" it or prevent any movement. Usually I place the blue keyframe for the next lip movement at the pause. Next is the red which is to close it. Usually I place the opening of the next lip movement at the close of the previous one just like how the picture above is shown.
Since the "mm" sound like for the syllable "mi" in the picture requires the "mm" movement but of course that do not exist on any model's sliders. In order to get this, I substituted the slider for the closed lip movement by putting the all lip sliders to left or zero value. Take a look at the purple circle or arrow, there is no slider for "mm" so you have to use your creativity to substitute missing sliders.
English words Lip syncing (Soft)
English lip sync sample I have done a long time ago
Half Life 2 intro G-man scene
Rise and shine Mr. Freeman rise and shine
Not that I wish to imply that you have been sleeping on the job
No one is more deserving of a rest
And all the effort in the world would have gone to waste
Until...Well... Let's just say your hour has come again.
The right man in the wrong place can make all the difference in the world
So wake mr. Freeman, wake up and smell the ashes
Lip syncing English words are a bit more difficult due to the fact that you need to make smoother transitions between sounds. Take a look at the example below:
As you can see there is no pause in order to make a smoother transition. The beginning of the next sound should be set on the opening of the first sound. This is more difficult as you have to time the transition carefully. You can also add pause keyframes if required. Just a reminder that it all depends on the animator how they will tackle the movements.
For reference here's a link to the motion data of both Japanese and English lip syncs.
That is all.
Back to TOC