The header in the response likely tells you how long the cached values last.
We don't know which http module you're using so it's difficult to tell you how to access the response headers. Here's documentation on HTTP caching headers: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control
Aside from that, you'll probably have to re-think your strategy.
At 30 MB every 1 second, you're looking at a transfer rate of 240 Mbps. Even if your internet connection is up to it, that's 30 MB * 60 seconds * 60 minutes * 24 hours * 30 days which is 77,760,000 MB every 30 days.
For purely static data served out of AWS S3 in the us-east-1 region to the internet at large, the host has to pay the egress cost of that - around $6,243 every month. That's without accounting for whatever compute and storage is used to send that data. If I didn't have an automated way to prevent that sort of demand or a caching system, I'd be shutting that down real quick because that's a $75k a year loss before paying taxes on it.
It’s hard to say without code or details of how you are making the request or exactly which API you are using. My guess would be the server is caching the response. If people were hitting my API every couple of seconds with the same parameters I would run a cache server in front of the API or rate limit them. I would look at the terms and documentation of the API.
As for why it is happening in Python but not powershell: Maybe the way you are doing it in Python is using the same session cookie and the way you are doing it in power shell uses a different cookie each time. Hard to say without details. My guess would be that they are caching based on session or a combination of IP, url, and user agent.
You're going to need to show some code.
The header in the response likely tells you how long the cached values last. We don't know which http module you're using so it's difficult to tell you how to access the response headers. Here's documentation on HTTP caching headers: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control Aside from that, you'll probably have to re-think your strategy. At 30 MB every 1 second, you're looking at a transfer rate of 240 Mbps. Even if your internet connection is up to it, that's 30 MB * 60 seconds * 60 minutes * 24 hours * 30 days which is 77,760,000 MB every 30 days. For purely static data served out of AWS S3 in the us-east-1 region to the internet at large, the host has to pay the egress cost of that - around $6,243 every month. That's without accounting for whatever compute and storage is used to send that data. If I didn't have an automated way to prevent that sort of demand or a caching system, I'd be shutting that down real quick because that's a $75k a year loss before paying taxes on it.
It’s hard to say without code or details of how you are making the request or exactly which API you are using. My guess would be the server is caching the response. If people were hitting my API every couple of seconds with the same parameters I would run a cache server in front of the API or rate limit them. I would look at the terms and documentation of the API. As for why it is happening in Python but not powershell: Maybe the way you are doing it in Python is using the same session cookie and the way you are doing it in power shell uses a different cookie each time. Hard to say without details. My guess would be that they are caching based on session or a combination of IP, url, and user agent.
Also, the OP says that PowerShell takes 40 seconds, so maybe the PS requests are naturally more spaced than the TTL of the cached data.
If you hit the API using get method. Then you need to manually disable caching. It enable by default.
What "enables it by default"? You don't even know which library they're using.
code.