Learning how to use In-Painting in Stable Diffusion to enhance existing images

over 1 year ago

Time to learn how to use in painting properly to correct images that have previously been generated. My starting image is where I left off last post, with an image of my favoured fool for my tarot card collection inspired images.

Here's the starting image.

There are many things that I want to fix within this image, I'll annoate / circle them and work through these one by one...

These are probably things that I will work on in an out of order fashion...

First, to remove the extra hand at the character's hip, number three, I simply used inpainting within the img2img tab of the Stable Diffusion web ui with a simple prompt of "hand", as a negative prompt. This made the hand disappear.

The next step was to remove the stick coming out of the character's head. That should be easy enough to do, again with a negative prompt of stick. 2 out of 5 done already.

For number four, I want the hand to be holding a white rose, as per the actual tarot card of the fool. I add a positive prompt, after painting over the area, and then this is the progress so far:

Next up is number two, which is to fix the deformed hand on the stick. Again, I paint over the impacted area, and use a negative prompt like...

extra fingers, mutated hands, ((poorly drawn hands))

Then, I've got just the one thing to fix, the slightly erroneous proportions on the torso of the character.

Here is where I find myself now... with In Painting having added more detail to the stick as a result of my prompt. That's awesome to see!

I used the negative prompt "deformed torso", and it gave me an interesting result, changing the nature of the image by changing the outfit.

This is not exactly what I wanted to see come into the image, so I might generate a few more iterations (which will be using different seeds, just like image generation) - and we'll see what happens...

The progression to the final image makes the final output probably be the most desirable yet, in terms of portraying the character, but I might try a few more quick iterations in order to see if I get any other results that are better. I'll do 5 more versions.

Looking at the enlarged version, there's a few new problems. The underarm and shoulder appears to be the wrong way round on the arm that is closest to the camera, which will need to be fixed. So, I'm iterating through some more negative prompts to try and fix this, like so:

The black part in the image below is what the negative prompt is to be applied to:

The best outcome after a series of further iterations is the following image:

There is still some deformity here, but I am realising it is not with the torso, but instead with the arm closest to the camera. So, now that I've found an outfit that I like... it is time to try and fix the "holding a rose" element of the image again. Here goes nothing, from this starting point:

And featured here, what I am doing with the Stable Diffusion In Painting and positive (top) and negative (bottom) prompts.

After several... hundred iterations, I'm getting closer to having a more realistically proportioned character, but there is still a significant amount of work to be done to have the rose being held.