person-swap 1.0.0

TODO

Person Swap

Info how to use system and how models works

Changing the photo (face)

4 models (files) are used to change the face:

1. Detector - checks if a face exists in a given photo and cuts it out if it does;
2. Recogniser - extracts the facial features from the cropped photo to be used for the change;
3. Generator - Creates a new face from the facial features of the source face to the facial expression of the target;
4. Mask detector - extracts the facial contour mask during the face change in the source photo (you can use libraries in CV2 anyway), but the model is more accurate as far as I have noticed.

Generator operation - the recognition model extracts the identity of the face in the photo (Ts for source or Tt for target).

The face change itself is performed by the generator, which consists of:

• The encoder - extracts the features of the target face;
• Identity Change Module - transfers the identity extracted from the source image to target photo features;
• Decoder - reconstructs a modified image of target photo features and source identity. 


 

 

Changing the sound (voice)

The paddlespeech library is used, which can use existing or newly developed models for voice replacement. The models used for voice replacement in the project are:

• Speech to text:

1. Transformer - text is extracted from the source audio;

• Text to speech

1. Phonetics class - converts text to phonetic data;
2. Encoder - extracts features of the target voice;
3. Acoustic model - matches the features of the source text with the features of the target sound;
4. Vocoder - text features converted to voice by the target voice features.


 

Changing the video (face and/or voice):

1. Separates the audio of the target video (if it exists);
2. If the source is a video, the first frame containing the person is taken (if the source is a photo, nothing is done);
3. Changes the face in the video (if the source is a photo or video):
1. Repeat over each frame;
2. changing the face in the frame (photo) (as in an ordinary photo);
3. the new frame (target photo with the face of the source) is saved in the list);
4. after iteration, the list of frames is converted into a video;
4. Voice in the video is changed (if the source is a video or audio file):
1. If the target or source did not have a voice - do nothing
2. If there is only the voice of the target, it is added to the created video;
3. If there is both a source and a target voice - the source's voice is replaced by the target's voice (as in voice replacement).


 

 

MIT License

Copyright (c) 2023 Segal011

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.