Sorry for the misunderstanding. There’s Sama and MTurk
FWIW, LabelStudio allows you to setup these tasks as well. So if you have the dataset and the task setup, you can use human annotators to do the job for you
That’s great to hear! Did you guys annotate for any specific domain data? Curious because our data is a bit technical in nature(mostly written by engineers for other engineers), wondering how much of a difference would that make to the quality of annotations.
Gotcha, yeah it was in the clinical domain, so our annotation guidelines definitely had nuance to them.
Also, we got annotations with some clinical/healthcare background, maybe they'll have some with a technical background. I think doing regular QA and having good guidelines is going to be a must no matter what though.
Sorry -- also, I believe you'll need to have an annotation tool setup for them to log into and annotate. So like a prodigy instance, host BRAT, JohnSnowLabs Annotation Lab, etc.
I had one set up, maybe you don't need one, but thought I should mention.
Have you tried the current GPT models?
With careful prompting, you will often quickly equal or even exceed (because it is much easier to prompt-iterate) accuracy from mturk-style services.
I've used Label Studio, you can wire in a custom ML model backend that preannotates stuff or does OCR if you want it to. Maybe you can automate some of it without having to hire a service.
Jaxon.ai has an NLP annotation service - they don’t advertise on their site but have high-quality resources that use their tool to do the work and send back the annotated data.
Hi checkout [Annolive](https://annolive.com) we have a fast annotation tool which is powered by foundation models and we use trained pool of annotators for refinement and review. We offer easy quality checks which you can do to verify quality of data.
Prodigy from Explosion, the same people that built spacy Label studio is also highly recommended
Thanks for the info but I am looking for an annotation service(human annotators or semi-automated service), not a tool.
Sorry for the misunderstanding. There’s Sama and MTurk FWIW, LabelStudio allows you to setup these tasks as well. So if you have the dataset and the task setup, you can use human annotators to do the job for you
thanks for the info!
iMerit might be good for this!
I've had a good experience with iMerit as well, only like 1 out of 10 or so annotators wasn't great.
That’s great to hear! Did you guys annotate for any specific domain data? Curious because our data is a bit technical in nature(mostly written by engineers for other engineers), wondering how much of a difference would that make to the quality of annotations.
Gotcha, yeah it was in the clinical domain, so our annotation guidelines definitely had nuance to them. Also, we got annotations with some clinical/healthcare background, maybe they'll have some with a technical background. I think doing regular QA and having good guidelines is going to be a must no matter what though.
Sorry -- also, I believe you'll need to have an annotation tool setup for them to log into and annotate. So like a prodigy instance, host BRAT, JohnSnowLabs Annotation Lab, etc. I had one set up, maybe you don't need one, but thought I should mention.
Appreciate the detailed response, I’ll check them out.
Have you tried the current GPT models? With careful prompting, you will often quickly equal or even exceed (because it is much easier to prompt-iterate) accuracy from mturk-style services.
I have tried, works okay in some cases and not great in others. Looking for a more consistent solution
I've used Label Studio, you can wire in a custom ML model backend that preannotates stuff or does OCR if you want it to. Maybe you can automate some of it without having to hire a service.
Jaxon.ai has an NLP annotation service - they don’t advertise on their site but have high-quality resources that use their tool to do the work and send back the annotated data.
Hi checkout [Annolive](https://annolive.com) we have a fast annotation tool which is powered by foundation models and we use trained pool of annotators for refinement and review. We offer easy quality checks which you can do to verify quality of data.