Testing Replacing Objects Clothes in videos - using ComfyUI and Forge (old version SD)

[negative1]

someone mentioned fake dancers in a video,

so i started out with just replacing one person first ...

i have been able to get the following

tests
====================

original blond dancing video
-> output anime style

-> output new girl with bikini

-> output new girl topless

-> output new girl without clothes

---------------------


============
tests -


ok, i have this equipment for my tests
--
i9 32G memory, nvidia 3090 with 24G Vram - running windows 10

software
---------
old version of Forge - stable diffusion, which works with replacer
segment anything

new versions of ComfyUI - with many custom nodes

Avidemux - used for creating videos from fames (blending and frame interpolation)

ffmpeg - various conversions, and creating comparison videos

===========

this wont be an easy task, will take a few hours, and you will run into several
issues most likely....

you will probably want to start with clean installs or portable installs of all the software...


test 1 - original jpop girl dancers
---------
original video -> modified video frames

1) took original video -> cut segment out, and converted to frames Using AVIDEMUX (save to .jpg image sequence)

2) using original Forge -> batch input to replacer to generate replacement frames

3) take replacement frames, and convert back to video, using blend, and change FPS, to adjust framerate,
might need to resample FPS to match original also


results : average - lots of flicker, and needs to manually check frames
duration : pretty quick, very little processing to handle 10 secs - about 20 minutes


test 2
--------

am currently trying others
------------

some of these will not match every single frame, but are usually close

dancing asian girl skirt and shirt -> without clothes, a little blurry, but can be fixed by using more frames

short animated segment -> real girl partially unclothed



-----------

upcoming tests - 2 girls dancing - change both

single topless girl - remove bottom

4 girls - change 1
5 girls - change 1


the single girl tests are pretty simple, and seem to work well.


later
| | | || |
n egative 1

 

[negative1]

tests
====================

original blond dancing video
-> output anime style



ok, it follows the motion, but of course the environment and clothes keep changing.

this was using :

comfyUI - > generate Depth and pose in control net

original video -> generate each frame

using anime or text prompt for each new frame

combine new frames to -> output video

NOTES :

DO NOT USE PEOPLE with mirrors around - you will end up with 2 objects in the pose, and have to edit each one
painstakingly out, as i found out.... same with any other people that show up in the video

later
| || || | | |
n egative 1

 

[negative1]

next test :

-> output new girl with bikini



totally new blonde girl, with a new prompt

= YES the background is changing, but the outfit is constant, and the motion is smooth
and follows the original


-> output new girl without clothes
---



=======


here is a clothesless version [each frame was run through replacer] and combined back

-> output original without clothes
-------


very short sample, and a little jumpy

later
|| | || | |
n egative 1

 

[negative1]

test 1 - original jpop girl dancers
---------
original video -> modified video frames

1) took original video -> cut segment out, and converted to frames Using AVIDEMUX (save to .jpg image sequence)

2) using original Forge -> batch input to replacer to generate replacement frames

3) take replacement frames, and convert back to video, using blend, and change FPS, to adjust framerate,
might need to resample FPS to match original also



results : average - lots of flicker, and needs to manually check frames
duration : pretty quick, very little processing to handle 10 secs - about 20 minutes

later
| || | | |
ne gative 1

 

[TMBV]

All videos are set to private.

 

[negative1]

Please, Log in or Register to view quote content!

How do you share them with people?

ah, i see, i have to add your email, and you need a youtube account ...
hold on..

ok, thats probably not going to work..

here are links to download (they are all very small):

NOTE :
===============
all models are:
---
public models
public dancers
from xfree. com - tiktok, etc
anime - not real
3d cgi - not real
ai generated - not real


-----
original jpop dancers - all nude - lots of flicker - 9 sec

Please, Log in or Register to view URLs content!

dancer clothed - nude 3 sec

Please, Log in or Register to view URLs content!

anime girl 12 sec

Please, Log in or Register to view URLs content!

blonde bikini girl to anime 15 sec

Please, Log in or Register to view URLs content!

blonde bikini girl to nude 6 sec

Please, Log in or Register to view URLs content!

blonde bikini girl to different blonde bikini girl 11 sec

Please, Log in or Register to view URLs content!

blonde bikini girl to different blonde bikini girl nude 11 sec

Please, Log in or Register to view URLs content!

if this doesnt work, let me know

||||
ne gative 1

 

[negative1]

NOTE :

most of the videos the person - model is taking up all the image area, so the flickering and changes
are very noticeable.

in the original video, the dancer is very small most of the time, so changes won't be as noticeable.

will be posting more samples soon.

later
| || |
ne gative 1

 

[TMBV]

Please, Log in or Register to view quote content!

Would be great to see one that looks as nice as the first dance one, the result after you replaced the other girls looks a lot more flickery than the initial result the original video had which was really smooth.

 

[negative1]

Ok, so it turns out this works for me:
---

1st step - Mimic Motion in ComfyUI

This will take a pictures as a reference, and conform it to a video
---

this is the start

Then you have to refine
-- face replacer
- reactor
- upscale and add detail

========

i suggest getting ComfyUI portable - with extensions,
and you will need to install the models for it too:

Please, Log in or Register to view URLs content!

this will have most of the extra nodes already installed.


then install mimic Motion, and its models:

Please, Log in or Register to view URLs content!

MimicMotion -model (3.05 GB), to ComfyUI\models\mimicmotion:

Please, Log in or Register to view URLs content!

i have made some progress on masking out a dancer..
thats next.

---
more examples in the other testing thread

later
| || | | |
ne gative 1

 

[negative1]

ok, i took this video: of 4 dancers



convert video - frames
tried to isolate the blonde dancer - used RemoveBackground on each frame of video
process each frame

and used mimic motion for a segement

Please, Log in or Register to view URLs content!

i have not composited it back into the original yet.

later
|| | | | |
ne gative 1

 

[negative1]

more tests

very stable:
---
anime girl 12 seconds, the new version is more stable than the original version, which had morphing etc.

Please, Log in or Register to view URLs content!

blonde in bikini dancing 23 seconds (took about 4 hours to render on i9- rtx 3090) - no detail or face improvement yet....

Please, Log in or Register to view URLs content!

====
to stabilize videos, export, put in a grid to create keyframes for stable videos:

--
part 1:


part 2:


he's improved on it even more: using Deforum and Temporal





====
this is much easier, but i cant get it to work
uniAnimate :

Please, Log in or Register to view URLs content!

later
| || | | |
ne gative 1

 

[negative1]

ok,

i have a decent workflow in ComfyUI now..

but it still is going to take some more refining...

2 test machines
- slow i5 nvidia 3050-4 gigs vram - takes 10 minutes to proces 1 sec video [can be done in parallel]
- fast i9 nvidia 3090 - 24 gigs vram - takes 10 minutes to process 1 minute video - mimic motion, 10x faster for segment - masking

-----

1 replacing and isolating object

Please, Log in or Register to view URLs content!

using segment anything model

- load original video

- test out first frame

- use points editor to mark object

- use segmentation masks

works best in 10 sec increments
=====
A - run through to create mask
B - create positive isolated image
C- separately run again to create masked overlay over video


2 combining masks, original video and compositing

Please, Log in or Register to view URLs content!

combine the following

1 original video
2 the masked video - convert video -> converted back to masks
3 the replacement isolated video

composite the three together..

still having an issue with the masks (white showing up)

there's probably a better way to composite in after effects, resolve, etc..

but i'm just using free programs..