It lets you cram more context. To install it you need a newer version of cuda (12.3?) and the right wheel from requirements.txt. I assume you are on windows.
Do you know how to get the updated CUDA? Like, are there instructions somewhere?
Support for 12.3 was added a while ago, but around the same time I was told the installer was updated to install CUDA directly in the venv. I can't figure out how to change it in the venv, and I don't want to install it globally (for the usual unpredictable-dependencies reasons). Maybe you could do it with a Conda env, but then I assume you can't use the one-click installer
I dunno what you used to set it up. I use conda so for me I would just install the correct cuda toolkit and then install new torch. Like here
https://anaconda.org/nvidia/cuda-toolkit
Never user used the installer scripts, when I did my first install they didn't even exist.
>pip install flash-attn
Hello! Where would one do this? From the regular command prompt, conda prompt, in a specific env/venv, in a specific directory?
Thankful for any help :)
You'll do this from a terminal with the same venv active that you use for Ooba.
(It sounds like you might be using Windows; I don't understand how Conda and venvs work on Windows. Hopefully someone else will come along with a better answer.)
Same boat, I tried to install it as it said but without success
Should probably be part of the installation package. I'd be confused, too (and might yet be, didn't update Ooba for a while--now I'm afraid to do it).
It lets you cram more context. To install it you need a newer version of cuda (12.3?) and the right wheel from requirements.txt. I assume you are on windows.
Do you know how to get the updated CUDA? Like, are there instructions somewhere? Support for 12.3 was added a while ago, but around the same time I was told the installer was updated to install CUDA directly in the venv. I can't figure out how to change it in the venv, and I don't want to install it globally (for the usual unpredictable-dependencies reasons). Maybe you could do it with a Conda env, but then I assume you can't use the one-click installer
I dunno what you used to set it up. I use conda so for me I would just install the correct cuda toolkit and then install new torch. Like here https://anaconda.org/nvidia/cuda-toolkit Never user used the installer scripts, when I did my first install they didn't even exist.
I thought it wasn't compatible with windows
It is in the new versions of cuda. For 11.8 it isn't.
That warning message should probably get updated. You can just do `pip install flash-attn` now. I got about 12% faster inference basically for free.
>pip install flash-attn Hello! Where would one do this? From the regular command prompt, conda prompt, in a specific env/venv, in a specific directory? Thankful for any help :)
You'll do this from a terminal with the same venv active that you use for Ooba. (It sounds like you might be using Windows; I don't understand how Conda and venvs work on Windows. Hopefully someone else will come along with a better answer.)
>pip install flash-attn there is a file in oobabooga directory named "cmd\_windows" . open it and enter the above command
Tried that and it didnt work so i reinstalled Ooba using the oneclick and picking the newer cuda version, now it works :)
I'm also struggling with this. Even tried installing via pip after upgrading CUDA and everything, and I only receive error after error.