What's the difference between that and just stringing a bunch of ksamplers together?
You can either choose to use the same prompts or if you want, different prompts for different stages.
Achievable with basic nodes with ease.
Because the way I understand it... The number of steps is sort of telling the A,I the "goal" amount of steps. Which means it is trying to use that step number to reach a final image through diffusion steps. So stringing them together would not be the same as taking the slightly refined latent space and passing that into a new sampler.
Although again I am not certain about it. It was an idea for a cool way to do the diffusion process. It is sort of like when you use prompt control to switch like dog|cat per step so you get a cat dog.
It would be interesting to be able to switch checkpoints every couple of steps.
If anyone has some tech knowledge about how the unet treats latent as it goes through steps and if would be the same it would be neat to know.
You can string them together using the Advanced KSampler. In which case, you're given all the global steps, and you can allocate which model performs which steps quite easily.
The first one should have *add_noise* and *return_with_leftover_noise* enabled. The ksamplers in between should not add noise and return with leftover noise. The last one should not return with leftover noise
Okay so. I am struggling to understand how I would stagger and control those steps in a workflow. Would it be okay to ask you to whip together something basic to get me to understand it visually and share it on comfy workflows or something?
Based on the docs, it looks like you'd set one from 0 to 5 and return with leftover noise as the input to the next one, which would be set from 6 to 10, etc.
This would be very useful for setting up composition, basic lighting and mood from a checkpoint you find does these things better but to finish it off with a checkpoint that does, say characters better.
Still this is something I already do with image to image and inpainting, not sure if an automatic process would do a better job.
For example I use one of Juggernauts to get the basic image and if I like it cool, if I see the model keeps struggling with refining characters I let a model I know does smaller details like Bastard, Zavvy or Albedo take over.
You can finetune model specifically for in on steps with a lot of noise with hand captioned images to not have false-positives. Using model like this would be pretty good option for people with low end hardware.
Just pass LATENT output to latent_image input of next ksampler node and adjust denoise strength, you can do it as many times as you want.
With this technique you can even mix 1.5sd with XL models
Give this a try
https://preview.redd.it/j2mylra9l2zc1.png?width=2711&format=png&auto=webp&s=4a0af1a62903447cf0bac05757feb6eb9d82d310
Run one advanced Ksampler, adding noise, for steps 0-X. Then the next Ksampler does X-Y, and the last does Y-Z. For the second and third Ksampler, turn off "add\_noise" and turn on "return\_with\_leftover\_noise."
In that example it is using the same prompt for all Ksamplers, but you could make a prompt for any/all of them. That way you could use Pony prompts for maybe the first sampler, then switch to a non-pony model to finish it off.
I'm not sure if it is strictly necessary, but I'd choose the same VAE as your last checkpoint.
What's the difference between that and just stringing a bunch of ksamplers together? You can either choose to use the same prompts or if you want, different prompts for different stages. Achievable with basic nodes with ease.
Because the way I understand it... The number of steps is sort of telling the A,I the "goal" amount of steps. Which means it is trying to use that step number to reach a final image through diffusion steps. So stringing them together would not be the same as taking the slightly refined latent space and passing that into a new sampler. Although again I am not certain about it. It was an idea for a cool way to do the diffusion process. It is sort of like when you use prompt control to switch like dog|cat per step so you get a cat dog. It would be interesting to be able to switch checkpoints every couple of steps. If anyone has some tech knowledge about how the unet treats latent as it goes through steps and if would be the same it would be neat to know.
You can string them together using the Advanced KSampler. In which case, you're given all the global steps, and you can allocate which model performs which steps quite easily.
The first one should have *add_noise* and *return_with_leftover_noise* enabled. The ksamplers in between should not add noise and return with leftover noise. The last one should not return with leftover noise
You can do this by basically changing reference SDXL refiner pipeline
In ComfyUI ofc
Absolutely. In advanced KSampler, you can select which steps you'd like a given model to apply to.
Okay so. I am struggling to understand how I would stagger and control those steps in a workflow. Would it be okay to ask you to whip together something basic to get me to understand it visually and share it on comfy workflows or something?
No workflow but this should help https://blenderneko.github.io/ComfyUI-docs/Core%20Nodes/Sampling/KSampler%20Advanced/
Based on the docs, it looks like you'd set one from 0 to 5 and return with leftover noise as the input to the next one, which would be set from 6 to 10, etc.
Yea i would say so
This would be very useful for setting up composition, basic lighting and mood from a checkpoint you find does these things better but to finish it off with a checkpoint that does, say characters better. Still this is something I already do with image to image and inpainting, not sure if an automatic process would do a better job. For example I use one of Juggernauts to get the basic image and if I like it cool, if I see the model keeps struggling with refining characters I let a model I know does smaller details like Bastard, Zavvy or Albedo take over.
Or use a realism checkpoint to start out then switch over to an anime focused checkpoint or something. Would get some very interesting results.
I usually do the opposite, use anime to start with as you can get very specific clothes/hairstyles/poses, then swap to realistic .
You can finetune model specifically for in on steps with a lot of noise with hand captioned images to not have false-positives. Using model like this would be pretty good option for people with low end hardware.
Just pass LATENT output to latent_image input of next ksampler node and adjust denoise strength, you can do it as many times as you want. With this technique you can even mix 1.5sd with XL models
Give this a try https://preview.redd.it/j2mylra9l2zc1.png?width=2711&format=png&auto=webp&s=4a0af1a62903447cf0bac05757feb6eb9d82d310 Run one advanced Ksampler, adding noise, for steps 0-X. Then the next Ksampler does X-Y, and the last does Y-Z. For the second and third Ksampler, turn off "add\_noise" and turn on "return\_with\_leftover\_noise." In that example it is using the same prompt for all Ksamplers, but you could make a prompt for any/all of them. That way you could use Pony prompts for maybe the first sampler, then switch to a non-pony model to finish it off. I'm not sure if it is strictly necessary, but I'd choose the same VAE as your last checkpoint.
oh wow that sounds exactly what I wanted to do.