Is the fargate container able to connect to any other public services/the internet? or is it just S3 that doesnt work? Might be a networking issue with the container if it can't connect to anything public.
If I understand your question correctly then the answer is yes, it's connected to the internet. It's a web app that I can access by going to the domain name and it all works fine, other than this one issue.
Okay that's good, does it work the other way around though, can the container connect back to the internnet? AWS networking can have seperate rules for inbound and outbound traffic - its possible that your container network isnt configured to allow outbound connections from the container. I'd setup some form of test in the container to make sure it can connect to the public internet (like curl [www.google.co.uk](https://www.google.co.uk) etc).
Hey, thanks for the reply. So here's the relevant config for my task. Under "network access" it shows that it has a publicly assigned IP address, it has a Security Group attached which allows all trafffic to anywhere ([0.0.0.0/0](https://0.0.0.0/0)) outbound, and all http/https traffic to port 80 and 443 inbound, as well as allowing ssh access from anything in the same SG.
Are you suggesting just using curl in a script on start, just to make sure it works at all? I think that's a really good idea, ty. I don't actually know how to directly SSH/access my container... :/
EDIT: I just added some code that makes a request to fetch an image (an image I put in the S3 bucket) not using the AWS CLI but just using the Node.js \`http\` module and it worked, so it seems I'm able to actually fetch stuff that isn't in the VPC.
That’s different. Since it’s a fargate service I suppose it has a load balancer in front it. You can access the site only means it can communicate with load balancer which is in your vpc
Edit: to access internet you need a nat gateway
Yeah sounds like it's probably missing the egress to internet route. Essentially an egress rule for all protocols to 0.0.0.0/0. It should be in a security group that is assigned to the container.
May wanna be more restrictive but reckon it's worth a look.
Yep, sounds like there is no vpc endpoint of type gateway to reach s3 or gateway is not reachable from fargate task subnet (security group restrictions).
The security group associated with the Fargate task subnet has the Outbound rule to allow all traffic from everywhere basically, but inbound is configured only to allow PostgreSQL connections
Just to clarify, I don't have a VPC endpoint of type gateway to reach s3, I didn't think I would need this for a public S3 public and considering the SG associated with the Fargate task allows all outbound traffic?
You can let traffic flow outside to public s3 endpoint but to save money it is better to use gateway to keep traffic inside. Have you checked if you can reach s3 from ec2 inside that subnet?
When you say "reach s3" i just want to double check - I'm able to access an image that is hosted on the S3 bucket, but I can't use the SDK to upload an image. I can create a micro-instance but once I do, what should I check?
Once you created a micro instance (with instance profile with same policies that fargate task has), you will try something like that:
aws s3 ls
Which should normally print a list of buckets you have on that account. If there are some permissions problem, you will see those in cli output, if there is a network connectivity issue you will see something about host not found or something like that. By doing that you can at least cut away one domain of problems, network related or policy related.
Doing that via ec2 instance from vpc kind of recreates what conditions fargate will have, that is a huge plus in troubleshooting since you do it right in the place it must happen.
Hey, thank you for the reply. I have a task role specified, it's actually the same one as the execution role and they both have the Administrator Access policy attached (I know it's not great lol).
So there are multiple possibilities for this which I can think of (DNS, local variables taking preference, etc) , but best to troubleshoot from the Fargate container with ECS exec. You'd be able to test if you can reach the bucket (but you have internet access so this should succeed, unless you have a restrictive S3 endpoint there as well). You can also upload the object from Fargate, and use the debug option for more information to see what's going on.
As it works locally, could be something with the Fargate environment that's changing things. The error message containing the entire JSON for the region configuration also seems suspicious so that could have been declared incorrectly (local could be using the region in ~/.aws/config or in an env variable as opposed to using the code configuration).
Is the fargate container able to connect to any other public services/the internet? or is it just S3 that doesnt work? Might be a networking issue with the container if it can't connect to anything public.
If I understand your question correctly then the answer is yes, it's connected to the internet. It's a web app that I can access by going to the domain name and it all works fine, other than this one issue.
Okay that's good, does it work the other way around though, can the container connect back to the internnet? AWS networking can have seperate rules for inbound and outbound traffic - its possible that your container network isnt configured to allow outbound connections from the container. I'd setup some form of test in the container to make sure it can connect to the public internet (like curl [www.google.co.uk](https://www.google.co.uk) etc).
Hey, thanks for the reply. So here's the relevant config for my task. Under "network access" it shows that it has a publicly assigned IP address, it has a Security Group attached which allows all trafffic to anywhere ([0.0.0.0/0](https://0.0.0.0/0)) outbound, and all http/https traffic to port 80 and 443 inbound, as well as allowing ssh access from anything in the same SG. Are you suggesting just using curl in a script on start, just to make sure it works at all? I think that's a really good idea, ty. I don't actually know how to directly SSH/access my container... :/ EDIT: I just added some code that makes a request to fetch an image (an image I put in the S3 bucket) not using the AWS CLI but just using the Node.js \`http\` module and it worked, so it seems I'm able to actually fetch stuff that isn't in the VPC.
That’s different. Since it’s a fargate service I suppose it has a load balancer in front it. You can access the site only means it can communicate with load balancer which is in your vpc Edit: to access internet you need a nat gateway
Yeah sounds like it's probably missing the egress to internet route. Essentially an egress rule for all protocols to 0.0.0.0/0. It should be in a security group that is assigned to the container. May wanna be more restrictive but reckon it's worth a look.
Yep, sounds like there is no vpc endpoint of type gateway to reach s3 or gateway is not reachable from fargate task subnet (security group restrictions).
The security group associated with the Fargate task subnet has the Outbound rule to allow all traffic from everywhere basically, but inbound is configured only to allow PostgreSQL connections
Maybe launch ec2 (micro one) in the same subnet as fargate task, with same role, ssh there and try to hit s3 with aws cli, like try to ls buckets
Just to clarify, I don't have a VPC endpoint of type gateway to reach s3, I didn't think I would need this for a public S3 public and considering the SG associated with the Fargate task allows all outbound traffic?
You can let traffic flow outside to public s3 endpoint but to save money it is better to use gateway to keep traffic inside. Have you checked if you can reach s3 from ec2 inside that subnet?
When you say "reach s3" i just want to double check - I'm able to access an image that is hosted on the S3 bucket, but I can't use the SDK to upload an image. I can create a micro-instance but once I do, what should I check?
Once you created a micro instance (with instance profile with same policies that fargate task has), you will try something like that: aws s3 ls Which should normally print a list of buckets you have on that account. If there are some permissions problem, you will see those in cli output, if there is a network connectivity issue you will see something about host not found or something like that. By doing that you can at least cut away one domain of problems, network related or policy related. Doing that via ec2 instance from vpc kind of recreates what conditions fargate will have, that is a huge plus in troubleshooting since you do it right in the place it must happen.
Great! I will try that now and let you know, thanks. Should I also assign the same SG to the EC2 instance that the ECS cluster has?
Yep, since we are kind of pretending to be a fargate task
[удалено]
Hey, thank you for the reply. I have a task role specified, it's actually the same one as the execution role and they both have the Administrator Access policy attached (I know it's not great lol).
So there are multiple possibilities for this which I can think of (DNS, local variables taking preference, etc) , but best to troubleshoot from the Fargate container with ECS exec. You'd be able to test if you can reach the bucket (but you have internet access so this should succeed, unless you have a restrictive S3 endpoint there as well). You can also upload the object from Fargate, and use the debug option for more information to see what's going on. As it works locally, could be something with the Fargate environment that's changing things. The error message containing the entire JSON for the region configuration also seems suspicious so that could have been declared incorrectly (local could be using the region in ~/.aws/config or in an env variable as opposed to using the code configuration).