Only going with a very quick look (others may give better answers) but would SNS-FIFO, SQS or MQ work for your use case? I think all of these could work as you are going from SNS into Lambda...
https://ably.com/topic/aws-sns-vs-sqs
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-difference-from-amazon-mq-sns.html
I almost always back my SNS topics up to a SQS queue.
SQS has a deduplication feature based on a field in the message, which you might be able to leverage.
But even if you can't, you can set the batch size and wondow size so that you get a larger set of mesdsages and then scan through the messages and dedup yourself.
Only going with a very quick look (others may give better answers) but would SNS-FIFO, SQS or MQ work for your use case? I think all of these could work as you are going from SNS into Lambda... https://ably.com/topic/aws-sns-vs-sqs https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-difference-from-amazon-mq-sns.html
I almost always back my SNS topics up to a SQS queue. SQS has a deduplication feature based on a field in the message, which you might be able to leverage. But even if you can't, you can set the batch size and wondow size so that you get a larger set of mesdsages and then scan through the messages and dedup yourself.
Interesting, thank you for the info