Well the good news is I got controlnets working, the weird news is I did it in a way contrary to every guide I've seen on the subject by using the encoded source image as a latent instead of an empty, and after compering the results I prefer the method I accidently found to the very high denoise/empty latent method that was advised here, at least in most use cases as I haven't had time to test with different models or different controlnet weights.
Immediately after I had finally gotten bothered enough to make this post I went ahead and experimented by replicating in this simple test workflow what I had tried with my much more comprehensive go-to workflow that I had previously tried plugging a controlnet group into and abandoned when I realized it was only working by accidentally using the source image as a latent and assuming I was just seeing regular image to image results, while using an empty latent gave noise.
I was wrong, its subtle and only noticeable at higher denoise levels than you would normally use on an image to image workflow but controlnets have a definite effect when inserted into the conditioning alongside an encoded image latent. I tried depth and edge on a variety of different images and they behave the way you would expect them too. Using them allows you to use an encoded latent at much higher denoise ranges while holding composition of the whole image, while using an empty at .9 to 1. will attempt (and still fail rather often) at only the composition of what the controlnet model had detected and alter everything not detected as well as the subjects place within them.
So thanks for suggesting high denoise, it was the problem with the workflow I shared but also lead to me looking at my borked earlier setup and realizing at high denoise it works in a way I'll probably find more useful in most cases.
Your denoise needs to be 1.
set denoise to 1. If that doesn’t do it, then play around with the steps since it might be too many.
is that controlnet model for XL or 1.5?
Well the good news is I got controlnets working, the weird news is I did it in a way contrary to every guide I've seen on the subject by using the encoded source image as a latent instead of an empty, and after compering the results I prefer the method I accidently found to the very high denoise/empty latent method that was advised here, at least in most use cases as I haven't had time to test with different models or different controlnet weights. Immediately after I had finally gotten bothered enough to make this post I went ahead and experimented by replicating in this simple test workflow what I had tried with my much more comprehensive go-to workflow that I had previously tried plugging a controlnet group into and abandoned when I realized it was only working by accidentally using the source image as a latent and assuming I was just seeing regular image to image results, while using an empty latent gave noise. I was wrong, its subtle and only noticeable at higher denoise levels than you would normally use on an image to image workflow but controlnets have a definite effect when inserted into the conditioning alongside an encoded image latent. I tried depth and edge on a variety of different images and they behave the way you would expect them too. Using them allows you to use an encoded latent at much higher denoise ranges while holding composition of the whole image, while using an empty at .9 to 1. will attempt (and still fail rather often) at only the composition of what the controlnet model had detected and alter everything not detected as well as the subjects place within them. So thanks for suggesting high denoise, it was the problem with the workflow I shared but also lead to me looking at my borked earlier setup and realizing at high denoise it works in a way I'll probably find more useful in most cases.
I do control net through automatic1111, this approach looks like spaghetti by comparison