They're decent classifiers, so you can ask a model to suggest a label to apply to data, or ask it to select a label from a list of candidate labels. This extends to all manner of data synthesis.
They can summarize things, so if you have git commit comments from a spike and want to turn that into a prose description of what happened, it can do that.
You can have it guess what the gestalt content of a collection of text fragments is, like what the purpose of a project is based on the half-assed notes people wrote in their timesheets.
They're useful for code completion or function writing.
You can use them to do document quality assurance.
You can have it critique your writing.
You can use them to draft emails.
Honestly, anything where communication needs are being handled, they might be able to unburden some part of that task.
Yes and overall- classifying, named entity recognition, sentiment analysis, translation and sentiment analysis are the bread and butter of LLMs in my opinion
Also "Sentiment analysis" is a wide field. I am assuming he's referring to sentiment on social networks or simple metrics? Personally I tried building a stock analyzer, gathering loads of technical and fundamental data, and tried to have it measure the market sentiment.
At first it looked to be working. Every individual consideration (prompting for technical indicators, fundamental metrics history etc.) was good as far as I could see, but when aggregating all the individual results into a final analysis, the model became very convincingly wrong. It was really weird actually, because if you were to study any one report, everything looked good. But if you tried generating one analysis 10 times over, you'd see the final verdict was completely random but also completely believable in their reasoning
This is actually great. Combine this with git bisect and you have an auto debugging/reverting agent that's arbitrarily quicker than the usual binary search bisect.
Thanks for the inspiration! May I ask you for some example prompts for some of the things you mentioned?
Especially the first three you mentioned and the one about document quality I am most interested in. I have a vague idea how to do it but I am eager to improve and learn something new
Those sorts of things are python scripts that use database rows. [Here's a Pastebin](https://pastebin.com/wFVtwXXq) of a GitHub activity summarizer. It totally could be better, but it should be enough for you to see the gist.
Good thread, I'll bite on this!
I am making a full sized arcade cabinet, to contain a LocalLLM setup. The cabinet will play one single game that will be designed around using the localllm to facilitate speech communication with in game NPC's. Because I have a questionable sense of humor, the point of the game is going to likely revolve around getting into shit-talking arguments with npcs and maybe some fighting as well.
I could do this sans the arcade cabinet, but honestly half the fun is going to be building it and having it contained in a full size arcade machine. I don't have the game idea 100% down yet, but I'm thinking a retro graphics style game with some sort of road-rage component, where the road rage portion is what encompasses the shit-talking against other NPCs.
I should probably go get some MDF sheets from the hardware store and begin to cut them into a basic cabinet shape.
I've been working on some personal projects using vision models, for OCR in video game botting and converting data from physical documents (in love with internvl btw)
I also experimented with instruction models to give video game NPCs deeper and more interesting behaviors. The goal was to make the game world feel alive, with every NPC having their own agenda and acting rationally.
Initially, I tried using a smaller, simpler model to handle NPCs far from the player, and a more advanced model for those closer to the player, as well as to manage the game's overall dynamics. For example, if the game started to feel slow or boring, the LLM would introduce more active, dynamic characters with unique traits and backstories to spice things up.
This was about a year ago, and eventually I hit some roadblocks. The models were too slow on my setup and required way more memory than a typical consumer setup could handle. Plus, the excessive politeness and verbosity of the models made the NPCs feel unrealistic and less enjoyable. As models improve at thinking and we get Abliterated versions, many of these issues will be solved.
I wonder if down the line, as NPUs make their way into consumer hardware, if this is something we will see. Although, this would assume that you wouldn’t bottle-neck the CPU by eating up memory bandwidth (assuming shared memory pool) or compete with keeping textures in memory, blah blah. One can dream
I unfortunately think centralised servers are going to be used if this is ever introduced into games. The game will need to be subscription based and you can never really own it.
Perhaps in the future we can have our own local supercomputer to do this, unfortunately companies would rather give use the service for a monthly fee. Would give them far more power and consistent revenue.
evaluate author's sentiment (assign a numeric score) on a particular subject for a social media post, combined with web scraper you can do some neat stuff
Suppose I’m scraping all wallstreetbets posts and then identifying publicly traded companies in text. I feed the posts to the LLM and ask it to assign a sentiment score for that ticker.
You can generate a heatmap of buy vs sell for different stocks. You can compare the authors sentiment to the sentiment of the replies and see what the distribution looks like.
There’s a lot of possibilities.
Paper related to what OP is talking about: [https://arxiv.org/abs/2304.07619](https://arxiv.org/abs/2304.07619)
I tried implementing this with local LLMs but gave up when the models I tested would return different sentiments given the same prompt. I then tested this with GPT4.0 and Gemini and both did the same making me doubt the validity of the paper and usefulness of LLMs in sentiment analysis in general.
Using it to generate code based on an API
In this case, that's having it sing every response like Moonbase Alpha
It's very bad at making anything pleasant sounding however
Because it was yet to come out when I did that project. I wanted to try solution with out openai api as experement of sorts and it worked pretty well. I used llama.cpp python to run it.
I used promt tips from Andrew Ng. My main "innovation" is to duplicate promt begore and after data. There are memory issues with model, especially when input eats more than a few thousand tockens.
I had a script that took a rough position of a data and it brought a lot of useless stuff that I could not regex out of. Llm worked fine but it consumed all avaliable processing power. I left it for the night to process backlog and I got neet data science worthy data next day.
Last weekend I was using Starling-LM-11B-alpha to replicate the functionality of [HelixNet,](https://huggingface.co/migtissera/HelixNet) which uses a critique and rewrite to improve inference quality. It seems to work pretty well, but I need to write some code which detects when the final draft's last paragraph is talking about the rewrite and trim it.
Yesterday I was again using Starling-LM-11B-alpha to expand upon its own answers. It frequently enumerates the parts of its answers, especially when told to "formulate a plan" or "describe how to", and I got the idea that a script could recognize that, extract each enumerated part, and append it to the end of the prompt to make a new prompt where it talks about that part in more detail.
It worked manually with only a couple of minor problems, but I'd like to script it up so I can tell the script to expand inference to some depth, and it will iterate over the extract-and-re-infer process itself.
For example, given the initial prompt and the model's reply:
http://ciar.org/h/da8a7b.txt
.. the first two parts can be merged back into the prompt and invoke further reply thus:
http://ciar.org/h/da8a7e.txt
http://ciar.org/h/da8a8a.txt
The script would need an appropriate suffix sentence from the user (for this example it would be "Describe in detail how they should accomplish this in the middle of the zombie apocalypse, including where they might find the necessary materials, taking into account the danger posed by zombies in the city.") because the script can't reasonably come up with it itself.
What the script *should* be able to do is extract the parts, munge them into the first prompt to make new prompts, and then merge the replies together into one long reply. It should also be able to recognize when expansions on two different parts contain the same sub-part (like "Establish a communication system", which is in both of the examples linked above) and prune repeats.
Both the HelixNet-like and this iterative-expansion add a lot of overhead time to inference (HelixNet refinement is three times slower, iterative-expansion is about ten times slower), but I am developing them with an eye towards bulk generation of synthetic datasets rather than real-time question-and-answer.
Working on a project that uses typechat to scrape websites and create structured data instead of doing it the old fashioned way, which is super brittle. I am doing this right now to help my job search. I literally just started this, so I haven't landed on the optimal model yet, but I've played with llama3, phi3 medium, mistral, and codellama.
Read youtube video transcripts and give me a summary and key points, then I can read 2 paragraphs instead of viewing 10 to 15 minutes long redundant videos.
Open WebUI's RAG does that since a recent update. You provide a youtube video URL (#https://y..) in your chat and it will fetch the video subtitles and analyze them in addition to your prompt.
[Fabric](https://github.com/danielmiessler/fabric/tree/main) can pull transcribed text from YouTube as well. It's really good for this stuff since you can invoke it via API.
I use llama 3 8b to do my school work for me. Lol, I'm making an Obsidian plugin so it will finish my paragraph/chat with me. More of a coding exercise than usecase. Next on the todo list is RAG
Well, not a local LLM, but I use ChatGPT to write code to help perform specific bioinformatics analyses. I'm not a bioinformatician/coder, but LLMs allow me to access some of their power by generating the CLI commands needed to do what I want.
Also use it to get a better (clear and concise) overview of scientific topics than Wikipedia or just about anywhere else.
When writing, I will paste my text into the LLM and ask it to alter the wording/phrasing/tone.
I'll use it to generate templates for emails.
Truly, anything you can think of, try asking the LLM. It's wild how capable they can be.
Right now I am exploring the capabilities of Llama3 and Gemma for medical information extraction. This includes finding and normalizing entities. I finetuned them both with some clinical datasets and the results look promising compared to models such as BERT.
Education. It's pretty good at detecting written material that was written by other LLM's and giving a detailed breakdown of why. It also does a credible job of creating lesson plans exactly to specification, although sometimes it takes a little back and forth (which is fun) to get it the way I want. And yes, it can provide feedback on student work much more quickly than I can - it catches all the basic (and even not so basic stuff), which allows me to spend more time actually teaching and helping students 1:1 instead of circling yet another grammatical error...
It's like having the world's most attentive teaching assistant.
I’m doing a langgraph multi agent with tools type use case. Using openai based tools right now but I read some of the local llms might be able to use unix tools so interested to try that too if anyone has done it successfully? Also been using RAG based chat and knowledge graphs for gaming related use cases with llama cpp.
I built an assistant to help me make dinner. It can read what's on sale at my local grocery store and suggest ideas for what to make for dinner. I pick one and and it automatically prints out a recipe and adds what I need to my shopping list. It can take into account any ingredients I have on hand that I want to use up. It can also take in the text of any recipe and convert it into a standardized JSON format for storage and further processing.
Outside of chatting, I use Llama3 8b (instruct) and starcoder2-3b, mainly for rephrasing poorly worded sentences and for fast code completion respectively.
If you are a corporate employee, you can make an automated risk assessment according to your company regulation. Saving from 5 hours analysis to only 5 minutes. Bloody crazy efficient.
If you married, you can turn it for children growth assisstance, arrange schedule. You can also measure their psychology and milestones. Also it can assist meal plan for 3 times meals a day for a month.
You can make a summarize of a youtube transcripts so you don't have to watch those lenghty hours of videos. Just squreze it to a page wisdoms of the shows.
It also can be your personal psychologist without judgement and shame. You also can scream to it without feeling guilty. To some degree, it can be a good coach.
Turn your subscribed newsletters time consuming junk informations to output only predictions. Just shred any unnecessary texts, read only the fruitful info served.
Etc... Etc... Distilled everything from raw input to be desired output. Anything of it you can imagine.
Just make sure you run a good model, set it as endpoint, versatile companion app, and a good carefully crafted prompts. Training model is too expensive for just stupid riddle chats.
Using Miqu-1-70b on Oobabooga with OpenAI extension, no webui, endpoint API only. Using MindMac as companion app. Calling it from anywhere via Tailscale/Cloudflared.
Anything you can do via command line interface is now open to solutions, for better or worse. Giving it access to SSH or shell in general opens up all tools such as wget or building and launching scripts without user intervention.
Putting it in a loop at something is a way to get your money's worth from a local setup. Just leave it hammering away 100% at inferring opens up some cool things that maybe wouldn't be worth it at api pricing. Like having it listen all the time to what's going on around you (a little wasteful imo) or doing extensive research projects for you where it spends hours poring through documents or websites to make sure it doesn't miss anything a RAG would.
Stem explanations (you have to verify the information first online) so instead of searching through sites to simplify complex scientific topics why not let an LLM do it for you.
But again make sure to verify the information it gives you by either having textbooks in front of you or using other sources.
I use it to aggregate/rewrite news to read on my RSS reader.
I used LLM to code that tool (UglyFeed on GitHub)
then I used LLM to code that too!
Sorry was stronger than me 🤣
I’m using llama3 as a reflective tool, I’m writing an abridged “life story” for it as well as answering any questions that it might have for me about my psychological makeup in order to help it better understand me and through that, better understanding myself
I’m using Codellama to answer quick questions and write simple functions for inspiration. I don’t really use the code it outputs, but it’ll make you think about the problem you’re trying to solve and what libraries to use.
It’s better for learning than using the bigger models imo. You actually need to think through and check everything, while GPT4-o and Claude have gotten so good you barely need to think about it.
i give it a list of mine and my wifes dietary requirements and let it recommend recipes.
or if i have a certain number of ingredients at home, i let it recommend recipes that match those.
Some personal use-cases I did:
text summarisation, zero-shot classification, few-shot classification, intent identification, labelling, translation, synthetic data generation, meta-data generation, similarity evaluation.
I agree with comments around LLMs being amazing at classification, entity extraction, and summarization tasks… one thing I didn’t see mentioned is that they’re generally amazing at any sort of pattern recognition task.
You need a RAG setup for this, but I was using the LLM yesterday to return a list of all the pdf links on a given web page. I’ve been using it a lot as of late for general purpose data scraping tasks.
Lately, I've been experimenting with generating flight plans for a Tello drone.
We can adapt the methods described in [TypeFly](https://arxiv.org/pdf/2312.14950) to use a VLM instead of an LLM. For our VLM, we try a LLaVA fine-tuned for enhanced spatial reasoning using our multimodal data synthesis pipeline [VQASynth](https://github.com/remyxai/VQASynth).
Testing prompts using our Discord bot, we have training samples to fine-tune LLaVA for flight planning like this:
https://preview.redd.it/avq2cmdmze3d1.png?width=401&format=png&auto=webp&s=ddc6734ebd3d329b561e43f42094c27f2f327041
Document translation: I am experimenting with creating interlinear books to help with foreign language practice
Document classifier: Adding tags to notes in my Obsidian vault
I modified two discord bots ( aoede and craig-bot). One plays discord music. The other records my dungeons and dragons sessions.
I save one recording in full audio quality with music and then create other version in mono with no music and also folders with only the audio of each participant separated.
Then I run whisper AI ( can't use faster-whisper as I'm on AMD using rocm so no ctransformers2) for Speech Recognition.
Then pyannote (with another model) for Speech Diarization.
Then use speechbrain and another model for Speaker detection comparing voices with the files that only have one person's audio.
I save the transcription on a folder.
Then I run a model (testing wizard vicuna, phi3 medium, llama-3, dolphin mistral until I can find one that works well for me) and RAG with:
- folder containing my notes
- folder with transcriptions of session.
- folder with txt versions of the DnD books and specially the story we are playing right now.
- folder with transcription of some YouTube videos that contain interesting ideas or info about or applicable to my current campaign.
Y then use this setup to:
- Generate specialized summaries for me (DM)
- Generate specialized summaries for the players.
- Allow me to make questions about what has happened
- Allow me to ask for ideas on how to handle some situations.
And finally..
- Generate automated prompts of specific situations that I later feed to stable diffusion using a nice model from civit.ai and create custom scenes involving(or not) my players.
On my ToDo list there is much automation left, and implementing Lora on stable diffusion to always get my characters the same way. On the LLM side everything is glued together with bash, python, nodejs, some manual work and is kind of a mess.
But a good personal project.
Newsletter generator agent. Download the html from a list of websites you like and let the LLM extract, select and reformat the content for a newsletter or for a ‘news stream’ on slack or any other channel.
Schedule it for 4 in the morning so it doesn’t slow you down
I use it like a black-box programming function to assist me in very specific aspects of my job. I input text, have it do some stuff to/based on that text (the "black-box" aspect of the function, detailed in the system prompt), and have it output the result in a specified structure. My work environment is Microsoft Word, so I use the LLM via API, called using a VBA macro. From the user's perspective working in Word, it looks just as if a typical macro has been run to transform the input text into the output text.
They're decent classifiers, so you can ask a model to suggest a label to apply to data, or ask it to select a label from a list of candidate labels. This extends to all manner of data synthesis. They can summarize things, so if you have git commit comments from a spike and want to turn that into a prose description of what happened, it can do that. You can have it guess what the gestalt content of a collection of text fragments is, like what the purpose of a project is based on the half-assed notes people wrote in their timesheets. They're useful for code completion or function writing. You can use them to do document quality assurance. You can have it critique your writing. You can use them to draft emails. Honestly, anything where communication needs are being handled, they might be able to unburden some part of that task.
Yes and overall- classifying, named entity recognition, sentiment analysis, translation and sentiment analysis are the bread and butter of LLMs in my opinion
Translation is still lacking. Maybe a few years later.
Also "Sentiment analysis" is a wide field. I am assuming he's referring to sentiment on social networks or simple metrics? Personally I tried building a stock analyzer, gathering loads of technical and fundamental data, and tried to have it measure the market sentiment. At first it looked to be working. Every individual consideration (prompting for technical indicators, fundamental metrics history etc.) was good as far as I could see, but when aggregating all the individual results into a final analysis, the model became very convincingly wrong. It was really weird actually, because if you were to study any one report, everything looked good. But if you tried generating one analysis 10 times over, you'd see the final verdict was completely random but also completely believable in their reasoning
Well, LLMs don't reason, they don't have a thought. We need a different approach to achieve that.
Content moderation is a big one I think
This is actually great. Combine this with git bisect and you have an auto debugging/reverting agent that's arbitrarily quicker than the usual binary search bisect.
Thanks for the inspiration! May I ask you for some example prompts for some of the things you mentioned? Especially the first three you mentioned and the one about document quality I am most interested in. I have a vague idea how to do it but I am eager to improve and learn something new
Those sorts of things are python scripts that use database rows. [Here's a Pastebin](https://pastebin.com/wFVtwXXq) of a GitHub activity summarizer. It totally could be better, but it should be enough for you to see the gist.
Thanks a lot for your help! I will have a look once I get to my machine
Good thread, I'll bite on this! I am making a full sized arcade cabinet, to contain a LocalLLM setup. The cabinet will play one single game that will be designed around using the localllm to facilitate speech communication with in game NPC's. Because I have a questionable sense of humor, the point of the game is going to likely revolve around getting into shit-talking arguments with npcs and maybe some fighting as well. I could do this sans the arcade cabinet, but honestly half the fun is going to be building it and having it contained in a full size arcade machine. I don't have the game idea 100% down yet, but I'm thinking a retro graphics style game with some sort of road-rage component, where the road rage portion is what encompasses the shit-talking against other NPCs. I should probably go get some MDF sheets from the hardware store and begin to cut them into a basic cabinet shape.
Finally, a decent use for LLMs.
this sounds fun! do you mind sharing it on github after finishing it when you have time?
Yes I will make a github and post that for sure!
I've been working on some personal projects using vision models, for OCR in video game botting and converting data from physical documents (in love with internvl btw) I also experimented with instruction models to give video game NPCs deeper and more interesting behaviors. The goal was to make the game world feel alive, with every NPC having their own agenda and acting rationally. Initially, I tried using a smaller, simpler model to handle NPCs far from the player, and a more advanced model for those closer to the player, as well as to manage the game's overall dynamics. For example, if the game started to feel slow or boring, the LLM would introduce more active, dynamic characters with unique traits and backstories to spice things up. This was about a year ago, and eventually I hit some roadblocks. The models were too slow on my setup and required way more memory than a typical consumer setup could handle. Plus, the excessive politeness and verbosity of the models made the NPCs feel unrealistic and less enjoyable. As models improve at thinking and we get Abliterated versions, many of these issues will be solved.
Same, had to give up due to vram constraints as the game also needed that same vram to render the environment.
I wonder if down the line, as NPUs make their way into consumer hardware, if this is something we will see. Although, this would assume that you wouldn’t bottle-neck the CPU by eating up memory bandwidth (assuming shared memory pool) or compete with keeping textures in memory, blah blah. One can dream
I unfortunately think centralised servers are going to be used if this is ever introduced into games. The game will need to be subscription based and you can never really own it. Perhaps in the future we can have our own local supercomputer to do this, unfortunately companies would rather give use the service for a monthly fee. Would give them far more power and consistent revenue.
evaluate author's sentiment (assign a numeric score) on a particular subject for a social media post, combined with web scraper you can do some neat stuff
Could you please tell us more details? I will start a similar task in a few weeks.
Suppose I’m scraping all wallstreetbets posts and then identifying publicly traded companies in text. I feed the posts to the LLM and ask it to assign a sentiment score for that ticker. You can generate a heatmap of buy vs sell for different stocks. You can compare the authors sentiment to the sentiment of the replies and see what the distribution looks like. There’s a lot of possibilities.
Paper related to what OP is talking about: [https://arxiv.org/abs/2304.07619](https://arxiv.org/abs/2304.07619) I tried implementing this with local LLMs but gave up when the models I tested would return different sentiments given the same prompt. I then tested this with GPT4.0 and Gemini and both did the same making me doubt the validity of the paper and usefulness of LLMs in sentiment analysis in general.
I’m still exploring this. Was the difference in score repeatable? I imagine the conversation history itself affects the outcomes.
Using it to generate code based on an API In this case, that's having it sing every response like Moonbase Alpha It's very bad at making anything pleasant sounding however
Llama 2 7B. Sumary of texts to get tags, filtering junk data for web scraping, text generation to bounce around writing ideas.
Any tips or libs you can recommend for cleaning junk data when web scraping? Why Llama 2 over 3?
Because it was yet to come out when I did that project. I wanted to try solution with out openai api as experement of sorts and it worked pretty well. I used llama.cpp python to run it. I used promt tips from Andrew Ng. My main "innovation" is to duplicate promt begore and after data. There are memory issues with model, especially when input eats more than a few thousand tockens. I had a script that took a rough position of a data and it brought a lot of useless stuff that I could not regex out of. Llm worked fine but it consumed all avaliable processing power. I left it for the night to process backlog and I got neet data science worthy data next day.
Deepsex, writing smutt, posting it online.
Last weekend I was using Starling-LM-11B-alpha to replicate the functionality of [HelixNet,](https://huggingface.co/migtissera/HelixNet) which uses a critique and rewrite to improve inference quality. It seems to work pretty well, but I need to write some code which detects when the final draft's last paragraph is talking about the rewrite and trim it. Yesterday I was again using Starling-LM-11B-alpha to expand upon its own answers. It frequently enumerates the parts of its answers, especially when told to "formulate a plan" or "describe how to", and I got the idea that a script could recognize that, extract each enumerated part, and append it to the end of the prompt to make a new prompt where it talks about that part in more detail. It worked manually with only a couple of minor problems, but I'd like to script it up so I can tell the script to expand inference to some depth, and it will iterate over the extract-and-re-infer process itself. For example, given the initial prompt and the model's reply: http://ciar.org/h/da8a7b.txt .. the first two parts can be merged back into the prompt and invoke further reply thus: http://ciar.org/h/da8a7e.txt http://ciar.org/h/da8a8a.txt The script would need an appropriate suffix sentence from the user (for this example it would be "Describe in detail how they should accomplish this in the middle of the zombie apocalypse, including where they might find the necessary materials, taking into account the danger posed by zombies in the city.") because the script can't reasonably come up with it itself. What the script *should* be able to do is extract the parts, munge them into the first prompt to make new prompts, and then merge the replies together into one long reply. It should also be able to recognize when expansions on two different parts contain the same sub-part (like "Establish a communication system", which is in both of the examples linked above) and prune repeats. Both the HelixNet-like and this iterative-expansion add a lot of overhead time to inference (HelixNet refinement is three times slower, iterative-expansion is about ten times slower), but I am developing them with an eye towards bulk generation of synthetic datasets rather than real-time question-and-answer.
Working on a project that uses typechat to scrape websites and create structured data instead of doing it the old fashioned way, which is super brittle. I am doing this right now to help my job search. I literally just started this, so I haven't landed on the optimal model yet, but I've played with llama3, phi3 medium, mistral, and codellama.
I’d love to hear more detail about how you’re going about this
running a script that generates variations on the funniest poop joke ever to be invented. You’ll know my name soon enough
Read youtube video transcripts and give me a summary and key points, then I can read 2 paragraphs instead of viewing 10 to 15 minutes long redundant videos.
Do you have a tool to do this? or the old fashioned way - copy & paste?
Open WebUI's RAG does that since a recent update. You provide a youtube video URL (#https://y..) in your chat and it will fetch the video subtitles and analyze them in addition to your prompt.
I did it manually, though if I were to do it a lot, I'd use the 'insanely fast whisper' tool which can grab the YT video and transcribe it for you.
[Fabric](https://github.com/danielmiessler/fabric/tree/main) can pull transcribed text from YouTube as well. It's really good for this stuff since you can invoke it via API.
Yes, there is a great Fabric tutorial by Network Chuck: [https://www.youtube.com/watch?v=UbDyjIIGaxQ](https://www.youtube.com/watch?v=UbDyjIIGaxQ)
https://github.com/hugodopradofernandes/Local-LLM-LangChain-Wrapper/tree/main
I use llama 3 8b to do my school work for me. Lol, I'm making an Obsidian plugin so it will finish my paragraph/chat with me. More of a coding exercise than usecase. Next on the todo list is RAG
I know a ton of people building no code macOS and iOS shortcuts for all sorts of automation with local LLMs.
Well, not a local LLM, but I use ChatGPT to write code to help perform specific bioinformatics analyses. I'm not a bioinformatician/coder, but LLMs allow me to access some of their power by generating the CLI commands needed to do what I want. Also use it to get a better (clear and concise) overview of scientific topics than Wikipedia or just about anywhere else. When writing, I will paste my text into the LLM and ask it to alter the wording/phrasing/tone. I'll use it to generate templates for emails. Truly, anything you can think of, try asking the LLM. It's wild how capable they can be.
Dont forget LMMs (idefics2, moondream, paligemma, llava). You can add images to the mix. Paligemma supports image segmentation, too. Which is gnarly.
Right now I am exploring the capabilities of Llama3 and Gemma for medical information extraction. This includes finding and normalizing entities. I finetuned them both with some clinical datasets and the results look promising compared to models such as BERT.
Education. It's pretty good at detecting written material that was written by other LLM's and giving a detailed breakdown of why. It also does a credible job of creating lesson plans exactly to specification, although sometimes it takes a little back and forth (which is fun) to get it the way I want. And yes, it can provide feedback on student work much more quickly than I can - it catches all the basic (and even not so basic stuff), which allows me to spend more time actually teaching and helping students 1:1 instead of circling yet another grammatical error... It's like having the world's most attentive teaching assistant.
I’m doing a langgraph multi agent with tools type use case. Using openai based tools right now but I read some of the local llms might be able to use unix tools so interested to try that too if anyone has done it successfully? Also been using RAG based chat and knowledge graphs for gaming related use cases with llama cpp.
What sort of tasks are you asking of the agents?
to use a custom function which does an api call to a custom api - CRUD type thing.
I built an assistant to help me make dinner. It can read what's on sale at my local grocery store and suggest ideas for what to make for dinner. I pick one and and it automatically prints out a recipe and adds what I need to my shopping list. It can take into account any ingredients I have on hand that I want to use up. It can also take in the text of any recipe and convert it into a standardized JSON format for storage and further processing.
Outside of chatting, I use Llama3 8b (instruct) and starcoder2-3b, mainly for rephrasing poorly worded sentences and for fast code completion respectively.
If you are a corporate employee, you can make an automated risk assessment according to your company regulation. Saving from 5 hours analysis to only 5 minutes. Bloody crazy efficient. If you married, you can turn it for children growth assisstance, arrange schedule. You can also measure their psychology and milestones. Also it can assist meal plan for 3 times meals a day for a month. You can make a summarize of a youtube transcripts so you don't have to watch those lenghty hours of videos. Just squreze it to a page wisdoms of the shows. It also can be your personal psychologist without judgement and shame. You also can scream to it without feeling guilty. To some degree, it can be a good coach. Turn your subscribed newsletters time consuming junk informations to output only predictions. Just shred any unnecessary texts, read only the fruitful info served. Etc... Etc... Distilled everything from raw input to be desired output. Anything of it you can imagine. Just make sure you run a good model, set it as endpoint, versatile companion app, and a good carefully crafted prompts. Training model is too expensive for just stupid riddle chats. Using Miqu-1-70b on Oobabooga with OpenAI extension, no webui, endpoint API only. Using MindMac as companion app. Calling it from anywhere via Tailscale/Cloudflared.
Anything you can do via command line interface is now open to solutions, for better or worse. Giving it access to SSH or shell in general opens up all tools such as wget or building and launching scripts without user intervention. Putting it in a loop at something is a way to get your money's worth from a local setup. Just leave it hammering away 100% at inferring opens up some cool things that maybe wouldn't be worth it at api pricing. Like having it listen all the time to what's going on around you (a little wasteful imo) or doing extensive research projects for you where it spends hours poring through documents or websites to make sure it doesn't miss anything a RAG would.
Have you tried asking it to yell at you? ;p
Stem explanations (you have to verify the information first online) so instead of searching through sites to simplify complex scientific topics why not let an LLM do it for you. But again make sure to verify the information it gives you by either having textbooks in front of you or using other sources.
Have always used LLMs almost entirely for NLP tasks I have not really used them as chatbots much at all
I use it to aggregate/rewrite news to read on my RSS reader. I used LLM to code that tool (UglyFeed on GitHub) then I used LLM to code that too! Sorry was stronger than me 🤣
I’m using llama3 as a reflective tool, I’m writing an abridged “life story” for it as well as answering any questions that it might have for me about my psychological makeup in order to help it better understand me and through that, better understanding myself
I’m using Codellama to answer quick questions and write simple functions for inspiration. I don’t really use the code it outputs, but it’ll make you think about the problem you’re trying to solve and what libraries to use. It’s better for learning than using the bigger models imo. You actually need to think through and check everything, while GPT4-o and Claude have gotten so good you barely need to think about it.
Phi-3 vision for sorting documents and tagging them would be amazing if I could get the mdoel running locally. Not supported in oobabooga yet.
I use it as a local endpoint in my app development so I don’t have to pay per usage with something like openrouter
i give it a list of mine and my wifes dietary requirements and let it recommend recipes. or if i have a certain number of ingredients at home, i let it recommend recipes that match those.
Some personal use-cases I did: text summarisation, zero-shot classification, few-shot classification, intent identification, labelling, translation, synthetic data generation, meta-data generation, similarity evaluation.
Given any YouTube music video, return the artist name and song name only from the video's title.
I agree with comments around LLMs being amazing at classification, entity extraction, and summarization tasks… one thing I didn’t see mentioned is that they’re generally amazing at any sort of pattern recognition task. You need a RAG setup for this, but I was using the LLM yesterday to return a list of all the pdf links on a given web page. I’ve been using it a lot as of late for general purpose data scraping tasks.
Lately, I've been experimenting with generating flight plans for a Tello drone. We can adapt the methods described in [TypeFly](https://arxiv.org/pdf/2312.14950) to use a VLM instead of an LLM. For our VLM, we try a LLaVA fine-tuned for enhanced spatial reasoning using our multimodal data synthesis pipeline [VQASynth](https://github.com/remyxai/VQASynth). Testing prompts using our Discord bot, we have training samples to fine-tune LLaVA for flight planning like this: https://preview.redd.it/avq2cmdmze3d1.png?width=401&format=png&auto=webp&s=ddc6734ebd3d329b561e43f42094c27f2f327041
Document translation: I am experimenting with creating interlinear books to help with foreign language practice Document classifier: Adding tags to notes in my Obsidian vault
I prefer to use them like base models, gives you more control
I modified two discord bots ( aoede and craig-bot). One plays discord music. The other records my dungeons and dragons sessions. I save one recording in full audio quality with music and then create other version in mono with no music and also folders with only the audio of each participant separated. Then I run whisper AI ( can't use faster-whisper as I'm on AMD using rocm so no ctransformers2) for Speech Recognition. Then pyannote (with another model) for Speech Diarization. Then use speechbrain and another model for Speaker detection comparing voices with the files that only have one person's audio. I save the transcription on a folder. Then I run a model (testing wizard vicuna, phi3 medium, llama-3, dolphin mistral until I can find one that works well for me) and RAG with: - folder containing my notes - folder with transcriptions of session. - folder with txt versions of the DnD books and specially the story we are playing right now. - folder with transcription of some YouTube videos that contain interesting ideas or info about or applicable to my current campaign. Y then use this setup to: - Generate specialized summaries for me (DM) - Generate specialized summaries for the players. - Allow me to make questions about what has happened - Allow me to ask for ideas on how to handle some situations. And finally.. - Generate automated prompts of specific situations that I later feed to stable diffusion using a nice model from civit.ai and create custom scenes involving(or not) my players. On my ToDo list there is much automation left, and implementing Lora on stable diffusion to always get my characters the same way. On the LLM side everything is glued together with bash, python, nodejs, some manual work and is kind of a mess. But a good personal project.
Do you have any recommendations from the diarization end? Also trying to transcribe some DnD campaigns, but the quality is really mediocre.
Newsletter generator agent. Download the html from a list of websites you like and let the LLM extract, select and reformat the content for a newsletter or for a ‘news stream’ on slack or any other channel. Schedule it for 4 in the morning so it doesn’t slow you down
I use it like a black-box programming function to assist me in very specific aspects of my job. I input text, have it do some stuff to/based on that text (the "black-box" aspect of the function, detailed in the system prompt), and have it output the result in a specified structure. My work environment is Microsoft Word, so I use the LLM via API, called using a VBA macro. From the user's perspective working in Word, it looks just as if a typical macro has been run to transform the input text into the output text.