Shop Mobile More Submit  Join Login
×

:icontao63: More from tao63





Details

Submitted on
September 9, 2013
Link
Thumb

Stats

Views
5,791 (4 today)
Favourites
49 (who?)
Comments
28
×
Changelog
9/10/13 - Published
9/19/13 - Fixed some errors




TABLE OF CONTENTS





=======================================================
INTRODUCTION
This tutorial is for MikuMikuDance (MMD) with versions at the time of writing. This is written as of 9/10/13. This tutorial will only discuss manual lip syncing done within MMD and no other software. There will be no tutorial on how to use pre-made VSQs or plugins to automatically input the lip syncs, though it is encouraged for others to use that in order for others not to waste too much time. There will be two parts of this tutorial, one will be for Japanese language lip syncing and English language lip syncing. Some people will call the facial expressions as "Morphs" while others will call them "Facial Sliders", both are correct. MikuMikuMoving has interpolation for morphs but considering that this tutorial is for MMD only.

Back to TOC




=======================================================
IMPORTANT NOTES

MMD MODELS
In MMD, the models are usually made by Japanese modelers so the order is usually following a, i, u, e, o for vowels instead of the English a, e, i, o, u. Just a reminder so others will not be confused as sometimes some models do not have English named bones and facial sliders.

LIMITATIONS
In MMD the models are based on manga/anime character designs. This design has many details removed and modifications to achieve attractive looks. Sometimes tongue and/or teeth are missing. Unfortunately because of that they had to use different or "cartoony" facial expressions to achieve emotions. What's more is that it completely limits the mouth movements to look like they are shouting. Because of this sometimes the mouth tends to have:


...Uncanny mouth movements. If you want to get a detailed explanation for why it looks a bit creepy, you can search on Google or Youtube about the uncanny valley. It's a short and good read.

Another limitation is that the model's facial sliders heavily depend on the modelers. Take a look at this


She's screaming. Yes she does, use your imagination. This model can only open her lips as wide as this.

DIFFERENCE BETWEEN LIP SYNCING JAPANESE AND ENGLISH LANGUAGES
Japanese Language

The Japanese and other similar languages such as Filipino are very different to English in terms of how they are pronounced. There's a very complex phonology for this subject so for simplicity sake let's just call them Hard which is derived from their words that sounds like all syllables are stressed.
It is good to have a basic idea of how the Japanese pronounce their words. Take a look at the picture below


Taken from hijapanese.blogspot.com/2008/0…

Their words are simply based on their vowels and adding a consonant before it. This makes it all of their words to sound like they are all stressed. Their words always ends with a vowel with the exception of the "nn" sound but that is a special character for them that it is completely separated to the "na" syllables. Their syllables are easily separated. The Japanese also use the "Hard" pronunciation even when they speak English words. For example when they say "easy mode" they pronounce it as "ii-zi (easy) mo-do (mode)" Also they do not have the "L" letter for them (As you can see on their alphabet above) they substitute it with "R" since most of these words that has the letter L are pretty much just borrowed words for them.

English Language
The English language and similar languages such as the Chinese language phonology is way more complex when compared to the Japanese. This is because despite their words having few syllables, the way the words are pronounced can be plenty. For example in English, the word "Safe" is a single syllable word yet it is read as "ss-ey-ff" rather than "se-ii-fu" in Japanese. The transition when pronouncing words is smooth so for simplicity sake, I'll call them as Soft words. It is possible that there are some "Hard" words mixed in these languages like the Korean language so it is up to the animator on how they will approach it.

FAQs
Q: Can I mix two sliders together? Example, mixing "a" slider with "o" slider
A: It actually depends on you. You should ask yourself first if you intend to use the animation with other models. This is because even though it looks good on the first model, other models might produce undesirable results. Usually they are deformed mouths or missing sliders. So it is recommended to do it only if you are going to use it on another model of same base or you will be using it for a single model.

Q: Do I really have to use the "a" slider for the "a" sound? Can I use something else?
A: Yes, you can use others what you think looks better, but same idea for the question above that only do it for models of same base when using a different model other than the one you animated it with.

Q: My character keeps moving! I can't do lip syncing!
A: You can do lip syncing on a steady character. Works well on a separate pmm work file and then save the lip sync animation and then load it on a model that is dancing.



Back to TOC





=======================================================
HOW TO LIP SYNC IN MMD
The reason why I separated the languages into two categories (Hard and soft) is because they are done differently, but not that much. It is recommended that you have a readable lyrics prepared like one in an opened browser or notepad or whichever software you want to put the text. This is important to make your mouth movement more accurate than doing it by ear.

Japanese words Lip syncing (Hard)
Here's a sample


Song: No Logic
Lyrics:
Kami-sama, kono uta ga kikoeru kai  Anata ga nozo'nde inakutemo
Boku wa waratteitai'ndesu  Soshite ima sakebitai'ndesu
Itsudatte saigo wa No Logic
Bokura no kono ichidou kiri no mu  Douse itsuka owari ga kuru nara
Ikiru koto o yametai toki dake tachi yamatte kangaeru kurai de, ii yo ne

"Yaritai koto dake erande, iranai mono kiri sutete"
Dare mo ga minna sonna kuu ni arukeru wake...nai yo ne



Whenever I lip sync, I usually separate a single lip movement into four parts. See below


The image should be self explanatory. I only placed the colors for convenience but they are not actually in MMD

As for lip syncing it usually goes like this:



First take a look at how I set the sliders for each part. On the blue keyframe the lip should be set to zero or left most slide. The green keyframe should be set to a certain amount of openness you prefer. It is important that the blue and green keyframes to be set BEFORE or DURING the first sound. The next one is the yellow in which the slider should be the same as the green keyframe to "pause" it or prevent any movement. Usually I place the blue keyframe for the next lip movement at the pause. Next is the red which is to close it. Usually I place the opening of the next lip movement at the close of the previous one just like how the picture above is shown.
Since the "mm" sound like for the syllable "mi" in the picture requires the "mm" movement but of course that do not exist on any model's sliders. In order to get this, I substituted the slider for the closed lip movement by putting the all lip sliders to left or zero value. Take a look at the purple circle or arrow, there is no slider for "mm" so you have to use your creativity to substitute missing sliders.

English words Lip syncing (Soft)
English lip sync sample I have done a long time ago



Half Life 2 intro G-man scene
Dialogue:
Rise and shine Mr. Freeman rise and shine
Not that I wish to imply that you have been sleeping on the job
No one is more deserving of a rest
And all the effort in the world would have gone to waste
Until...Well... Let's just say your hour has come again.
The right man in the wrong place can make all the difference in the world
So wake mr. Freeman, wake up and smell the ashes


Lip syncing English words are a bit more difficult due to the fact that you need to make smoother transitions between sounds. Take a look at the example below:



As you can see there is no pause in order to make a smoother transition. The beginning of the next sound should be set on the opening of the first sound. This is more difficult as you have to time the transition carefully. You can also add pause keyframes if required. Just a reminder that it all depends on the animator how they will tackle the movements.

For reference here's a link to the motion data of both Japanese and English lip syncs.
www.mediafire.com/folder/mgm5d…

That is all.

Back to TOC
Add a Comment:
 
:iconbattiana:
Battiana Featured By Owner Oct 27, 2014  Hobbyist Artist
Which mmd can lip sync with vsqx. files? I have a Vocaloid 3 miku, and It's not working.
Reply
:icontao63:
tao63 Featured By Owner Oct 27, 2014
I'm not familiar with importing vsqs and if vocaloid 3 uses new format... have you tried renaming to vsq? Or converting it or something? Sorry but I'm not really familiar with it so I can't help much
Reply
:iconbattiana:
Battiana Featured By Owner Nov 20, 2014  Hobbyist Artist
Ok thanks, I tried converting but I couldn't really find one I could use.
Reply
:iconamandarocks444:
Amandarocks444 Featured By Owner Oct 10, 2014  Hobbyist Digital Artist
How come when I try to make my models lip sync they like open their mouths slowly before the music starts? Also as they speak their mouths don't close when they say one word to another and then it doesn't look like they're speaking, it looks like they are making weird mouth movements. Sorry if I'm bad at explaining. but please help!!
Reply
:icontao63:
tao63 Featured By Owner Oct 12, 2014
It seems you have missed it on the post. Basically this is your problem. From the way you explained it, it sounds like you didn't put the beginning keyframe. Basically it "locks" the lips from doing anything and place it anywhere you need until you need the mouth to have any movement. After that make sure to close it before making a movement for another syllable
Reply
:iconamandarocks444:
Amandarocks444 Featured By Owner Oct 14, 2014  Hobbyist Digital Artist
I do make it close before I add a movement to another syllable though, and I register it too  :/
Reply
:icontao63:
tao63 Featured By Owner Oct 16, 2014
That's really weird. Can you post screenshot of the keyframes on one of the syllables?
Reply
:iconamandarocks444:
Amandarocks444 Featured By Owner Edited Oct 20, 2014  Hobbyist Digital Artist
I will try. I think I need to re download MMD because for some reason it hasn't been opening.
Reply
:iconnaruko911:
naruko911 Featured By Owner Sep 6, 2014
would starting with the lip syncing and going back with the body animation be a good idea?
sorry if this is a dumb question or something, I've just recently tried my hand at animation.
Reply
:icontao63:
tao63 Featured By Owner Sep 6, 2014
Yes. Lip syncing can be done before or after body animation. At least for songs and dances since there is already a song to coordinate everything

However if you meant facial expressions or lip syncing for an acting then it's better to do it after body animations since it requires timing and coordinating body and lip sync animations
Reply
Add a Comment: