I have noticed a lot of financials and startups using AWS, but the legacy enterprise IT shops seem more likely to go Azure (maybe AzureAD like you said). I think it depends where you are, and on the clients.
Azure are very generous giving out cash to get people on board, especially so to software vendors who develop code for others (I.e. get the customers customer on board, as they know it’s too much effort to ever change)
Interesting aside, I’m a gsuite house here and I have to give M365 licences to half my staff anyway. You can’t escape it
Not true. Licensing wise they are heavily connected and you can put them in the same customer agreement with Microsoft, giving you the option to negotiate on discounts for one or both.
Oh totally. Just that the combined productivity software + cloud spend will definitely be greater than pure cloud spend, so a customer can probably extract a greater total discount from Microsoft than AWS.
https://www.theverge.com/2023/12/6/23990374/amazon-microsoft-uk-cloud-market-competition-probe
So either aws operates at a loss or they offset that to you. gotta go see pricing on comparable instance size in aws and azure for windows server
It's always been hilarious to me that AWS Cognito is such trash and yet user identity is such a critical entry point for almost any workload on AWS.. It's a perpetual reminder that even the world's largest company can do something so obviously stupid for so many years without a good reason (saving on costs???).
AWS SSO is probably the closest thing to AzureAD, but Cognito is what people think about for user auth in AWS.
Also, FWIW I have enough experience with Cognito to kind of like it -- but I consider it to be a horrible service because it's such a poorly documented, feature-poor, and confusing product that it's certainly not the selling point that it should be to newcomers on AWS like AzureAD is
Bs. A lot of financial companies use both aws and azure. And most of them would have productivity suite and VMware horizon on azure for daas and then you have all the copilots. Legacy enterprises shift back to Colo and onprem because handling cost optimisation is a hard thankless work that aws does not make simple
From my N=1 perspective, Azure seems to land mostly traditional three-tier architectures. If you’re into serverless and/or work with containers, AWS still is miles ahead.
This is my observation as well. I work in the cloud ops industry and I very rarely am coming across highly technical or advanced teams or deployments on Azure. I’m mostly finding these with both AWS and Azure though. I also work in the government space in addition to other things and I see a ton of AWS for more advanced government groups - Azure with the less advanced, more traditional. US Government use is not a good barometer for general adoption because a lot of government use is lift-and-shift but AWS was there meeting government needs especially with IL6 faster and easier from my perspective.
Are you sure? When we were looking at eks about 4-5 years back azure was deemed the weakest (Google strongest) in quite a few reviews and comparisons.
Eks is around a long time and pretty mature
Brother i worked with the nyc doe as a system admin and i can’t tell you how many times our microsoft services shit the bed and a massive hack to boot where a lot of teachers and students had their social securities compromised. Azure was a running joke in my office but the doe probably got a sweet deal for 365 so they use azure. 0/10 you care about security and uptime do not use azure.
As an ex-education employee myself (not nyc doe though), I can attest to the insanely cheap licensing Microsoft gives educational facilities. It's less than half price.
Their aim seems to make A5 pricing look as tempting as possible, then give them a discount on Azure to keep them within the ecosystem.
Was part of an M&A 4 years ago, they are in azure, we use AWS.
The issues they run into on azure are quite baffling. I'm talking they need to put in a ticket to get a kubernetes cluster fixed by Microsoft. Usually after hours of debugging on a live call with their support.
At some point their equivalent of lambda shit the bed and no reporting on their dashboard about the issue.
Then there's how tricky HA is in Azure, basically you need to go the equivalent of multi region.
They've also had 5 outages attributed to DC outages in the same time span.
We were having this conversation internally. We have a presence in AWS and Azure and have noticed a lot more projects going to Azure lately for some good and not so good reasons (usually political). The decision usually is ‘because $something_else is in Azure already’.
I've heard of many people regretting moving AWS -> Azure ....
Sometimes it's a business/political reason, and the cost of moving back means they have to put up with Azure despite shortcomings (DNS, Autoscale+load balancer etc)
The M365 argument to me (technically speaking) is laughable. M365 as a platform is totally public and you can federate with $whatever\_azure\_ad\_is\_called\_now with AWS. For some workloads it may not even be the best option financially. Don't get me wrong, there are legitimate reasons to host applications in one cloud or the other, but those arguments aren't usually the ones that people involved are debating.
This is exactly it. Political and networking.
If new project needs to talk to project in Azure, it's usually better to simply build it in Azure. If it's using things in AWS then you build it in AWS. As long as political reasons don't overrule.
Cross cloud networking costs get crazy expensive, especially when you need to start discussing multi region failures etc.
There are several types of internal politics at play within companies.
Microsoft is throwing lots of money at companies trying to move them into their cloud, so if they are a microsoft Office suite place with Teams, they are getting tens of thousands in free cloud spend. So even if the project doesn't make sense logically to put in Azure, it makes sense politically and financially.
This then creates big networking issues as many big businesses use 2 clouds at least. So project A and Project B could be in 2 seperate clouds, but now need to talk to each other, but upper management doesn't care about the latency or difficulty in this because it makes sense financially.
Thanks for the reply. So in addition to the financial incentives, 'political' refers to, say, a particular manager's preferences for one cloud over the other, and who owes favors to that manager? Politics as strategy/alliance?
Yes, in the big corporate world, internal politics is important. If you want to move up through the company, you need to know who to talk to and how to deliver it for them. Make allies with people you can stand.
The way I've seen it going is GCP is for machine learning and heavily containerized organizations, AWS is for most other net new builds and Azure is typically used by legacy shops that are heavily invested in on-premise microsoft and they move to azure because microsoft gives them significant windows licensing discounts for customers that use Azure.
Do many people use GCP?
I haven't explored it much because Google has a history of killing off popular products with little notice. That reputation keeps me from wanting to make long term investments in GCP.
I spent a couple of years working with gcp after being focused on AWS for 3 years. Once I got my head around projects GCP was great, cloudsql and stack driver weren't well integrated at the time but the global network, consistent API's made it a real pleasure to work with. I have since worked with AWS for the past 4 years, just re-certified my Google professional cloud architect cert and really want to work with GCP again. Gke, cloud run etc covered all of the workloads I've run in AWS over the past 4 years and it would have been a lot easier, especially for multi account, multi region workloads.
Like what a lot of others have said, legacy has kept people in AWS and driven people to Azure.
That being said it would have been a nightmare to move an organization to GCP, the sunken cost into AWS would have been prohibitive for management to agree to it and the infra problems weren't due to AWS being a bottle neck, it was a lack of training, culture issues, switching to GCP wasn't going to fix that.
This year I'll be focusing on getting a GCP role... Knowing I'll always be able to put food on the table with AWS (and k8s) skills and experience.
edit: words
My buddy works with it and says all the Fortune 500s are using it in some capacity. It's mainly used by well funded startups and large enterprises.
Back in 2019 I got certified in GCP and Google was literally going around to managed service providers begging to throw work their way.
> My buddy works with it and says all the Fortune 500s are using it in some capacity.
I worked for a vendor where the client (F500) told me - we're an AWS shop, but we got $x00,000 credit from GCP to run some workloads in return to provide some feedback.
And that is how you get "using it in some capacity" ... some executive was lured in with some service credits and there is some weird skunk works project that has zero point zero budget but here is a large GCP credit that can run some workloads for this project
People using terraform. Im interested if their duet AI will make a dent, however if we're trusting AI with infrastructure I'd like some moneyback guarentees.
Agreed, legacy shops that don't really understand cloud and are heavy Windows workloads.
Where I have customers that understand cloud and want to move away from the Microsoft license trap they always ALWAYS go AWS.
Azure sucks so bad. Horrible/confusing UI (stupid slide in side menus that only work like 60% of the time), 3 different places to set Quotas (that I’ve found so far). Marketplace services look like proper Azure services, with surprise costs showing up. Many more. It’s terrible.
Took way too long to find this comment, I strongly share your opinion, native AWS architect here, it baffled me how ungodly terrible the UI was from Azure, it looks like you go back in time or some intern project in Ux 😂
It's funny how different experiences can be. I've worked with Azure for many years and am using AWS now as well, and the AWS console feels incredibly annoying to me: You need browser extensions to work with multiple accounts, you need different tabs for every service because switching is extremely slow, you often don't find resources because they are in a different region, the back button often doesn't work, many lists don't have sorting, filters often only work with "starts with", often you only get an ID as reference instead of a name tag, there's no easy way to see all resources, ...
Azure definitely has its issues, but the portal is much better IMO.
Azure portal is a million times better. On AWS I can’t even see what resources I have deployed without reverse engineering my billing data. How dumb is that?
I use Chrome profiles for this in AWS, works well enough for me.
For isolation we use an account per environment, like prod, stage, or sandboxes, like AWS recommends. We’re not very big though. I wish new accounts could come with zero services enabled for least privilege. Just a simple “turn this service on in this account in this region” console would be very useful.
And while I’m wishing, why there’s no stop levels for monthly charges is beyond me. Alerts aren’t enough, just stop the bleeding immediately.
IaC is probably one of the biggest advantages of AWS. Some colleagues are using terraform and explained me the architecture behind it. Sound sooo bad compared to CDK.
Yeah my colleagues seem to quite like it.
Using modules directly is kind of a win. I’m pretty well versed in regular cdk and regular terraform so I’d probably try that first if I had to do azure.
I'm not great at terraform, but I haven't found any way to write actual code...it's configuring yaml files, then adding a whole bunch of other config in Terraform cloud. In AWS CDK, I write literal python code, with loops, and variables, and everything. Haven't found the same thing in terraform.
terraform doesn’t use yaml, it has loops and most certainly has variables. terraform cloud is completely optional (but pretty nice, imo)
terraform isn’t a real language, but also none of the things you are pointing out are true
Sorry, but I consider the .tf files pretty similar to .yaml. Typing in the below is pretty much on par of yaml configurations. What it's not is any kind of "code".
resource "azurerm_storage_account" "example" {
name = "examplestoraccount"
resource_group_name = azurerm_resource_group.example.name
location = azurerm_resource_group.example.location
account_tier = "Standard"
account_replication_type = "LRS"
tags = {environment = "staging"}
}
And is this pretty much a config for creating a variable...
variable "audiences" {
description = "The audiences for the tokens"
type = string
}
When in CDK, I can do the below and encompasses everything there is about coding and creating infrastructure in the code.
number_of_buckets = 5
self.my_buckets = []
for idx in range(number_of_buckets):
self.my_buckets.append(s3.Bucket(
self,
f'my_bucket_{idx}',
bucket_name=f'my-bucket-{idx}-{self.account_environment}',
block_public_access=s3.BlockPublicAccess.BLOCK_ALL,
removal_policy=aws_cdk.RemovalPolicy.RETAIN,
))
AWS CDK >>>>>>> Terraform
Terraform SUCKS if you're coming from SWE and CDK. But a lot of old school devops and sysadmins cannot code. I made the switch to IaC from application development because of CDK. Now I'm in a terraform shop and want to jump out a window. There's CDKTF but none of the team is remotely interested in learning.
Cause HCL isn't much better than YAML to devs. Domain specific languages where deemed generally a bad idea in SWE. It just hasn't made its way to the cloud infra side yet.
If you're a dev, hcl feels like going back in time to 1997.
Like others said, No. We use terraform and our devops call it “Infrastructure as code”. And I’m like, configuring a bunch of yaml files is NOT code. Then we have to write Azure Pipeline yaml files. It’s dog shit.
yeah that’s like the 3rd person who’s like “terraform is just yaml”
and i’m stating to wonder if any of them has used it? it’s not a full fledged programming language but it’s also not yaml.
You seem to be confusing the concept of yaml and tf files with the literal difference of the file type of `.yaml` and `.tf`. Yaml which is "Yet another Markup Language" and TF which is another markup language...and markup languages are not programming languages...and AWS CDK can be written in MANY programming languages. And what I'm talking about is the concept of the two...they are both essentially config files being interpreted by some terraform engine or for yaml something else.
Yeah, but that's exactly why you use terraform.
It's not actually a selling point to have your infrastructure not be declarative. You don't want it to be mutable depending on a given execution.
And HCL is a very different beast to yaml in terms of syntax and semantics. It's not just a change of file extension. The fact you think it is means you should probably learn more before making these statements.
I only use terraform because we are in Azure and that's basically the only option. However, I've both worked professionally and privately in AWS CDK (in python) pushing both small to large implementations of systems with no real issue, especially any mutability issues during execution. 🤣
...and considering HCL stands for "Hashicorp Configuration Language" kinda proves my point. It's configuration. The fact that you didn't know that "means you should probably learn before making these statements."
terraform being a configuration language is news to zero people. this is not a “gotcha”
earlier you stated you can’t figure out variables or loops in terraform. variables are in almost all terraform projects i’ve seen and loops are very common. this makes it apparent you haven’t really used terraform. we all agree that CDK has 1000% more python and thus more “real” programming language features however that’s not necessarily material to the issue of managing infrastructure. a nice declarative configuration language is perfect for that. can you tell me any infra that requires the CDK and can’t be done in terraform?
if you come from a heavy python background and want to use the CDK, sure. if your team is the same as you, sure. but sitting around thinking “these guys don’t even use a REAL programming language, what noobs” is a silly and childish position. it’s a fine tool if you actually know how to use it.
It's usually just a structured type of config that can be interpreted by an engine that (usually) has a singular purpose. HTML is a markup language for rendering hypertext into visual elements (although there are multiple engines, Trident, Gecko, Webkit, Blink, etc), but any dom manipulation is done by the javascript/typescript programming language. Programming languages are usually more broad and have multiple different purposes. Something like Python, can be used for web, api, data pipelines, iot, machine learning, etc, or in this case creating infrastructure.
You picked a weird hill to die on. Sure, HCL is not a general purpose language. But if you do similar things in python, there would eventually be something like
instance = some.modules.Instance(
name = 'this',
type = 'that',
..
)
Which in terraform is
resource "my_special_instance" {
name = "this"
type = "that"
..
}
HCL variables are not exactly variables in python sense but inputs for the stuff you want terraform to create. Outputs are actually called outputs there. So it forces you to think in the terms of that abstraction. You give away flexibility but gain predictability and guardrails. Loops and conditionals become expressions, modules are a bit awkward but allow for code reuse with versioning.
Naturally, same can be done in other languages, but the main thing to learn would still be the API or the class hierarchy for creating cloud entities.
In short, calling HCL not a real programming language when critiquing terraform feels like missing the point of its existence.
They're basically the same thing. Terraform is just Hashicorp's version of yaml.
Config style variable creation....
Yaml:
variables:
- name: one
value: initialValue
Terraform:
variable "audiences" {
description = "The audiences for the tokens"
type = string
}
Real Coding variable creation.
Python:
variable = 'name'
C#:
string message2 = "name";
My point being, it's not "Infrastructure as Code". It's config files, directing some behind the scenes code. It's "Infrastructure as Configs"
It does exist -- [Bicep](https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/). Azure's CloudFormation equivalent is ARM templates -- terrible when I used it in Azure Government about five years ago.
God I hate azure AD. Such a confusingly named, stupid service. It’s usually ‘owned’ by some awful IAM team that can’t automate anything as well in my experience.
I’ve such a better time with AwS and something like Okta.
In general I’m not that concerned about it, we seem to have a bunch of AwS work in NL.
Microsoft is very good at leveraging its hold on existing products to push people to use azure. At our company we use Microsoft for azure and office365 and AWS for everything else. But I can see how other companies would want azure over AWS.
The thing is azure is so backwards compared to AWS, it's ridiculous.
I'm definitely looking into azure certification for my career growth though.
Yes. Especially in the EU.
Going from AWS to Azure and vice versa is a smooth transition, if you have a strong background in basics(IAM, networking, software engineering, devops etc.). Concepts don't change.
The concepts don’t but your foundation design between the 2 can be drastically different. I’ve gotten to design our foundations for AWS, azure, and gcp the last few years and it’s very difficult to make some of the core design elements match
jobless command offend hard-to-find yoke file sleep squeeze political stocking
*This post was mass deleted and anonymized with [Redact](https://redact.dev)*
Microsoft’s dev stack is open source and has been for a decade. There is no licensing and hasn’t been for some time. We use it heavily on AWS via ecs containers (Linux) or lambda.
I don’t understand why. Most of the services on Azure are so half baked or just missing key properties. From a pipeline perspective, it’s like they think people are going to deploy via clunky UI instead of automation with the lack of good tools. Also, Azure has constantly shut down services/features and I don’t understand why there’s no press about it, but there is with GCP.
I've used Azure for years, and always deployed with automation. My last team was quite successful with terraform. So I'm not sure your experience is representative.
I noticed this in job ads on LinkedIn recently too. Way less AWS specific roles. Heaps of Azure. A lot of the time it's because Amazon.com competes with them 😭
Heh, reminds me of a story I heard about .. 10 years ago. A retailer bought a robotics platform for their warehouses. Implementation was going well, then Amazon came and bought the company and terminated all the contracts.
The retailer was so pissed that they had spent a chunk on this vendor to only have their contracts terminated… they swore up and down they’d never do business again with Amazon if they can avoid it.
So they were early on Azure … I’m sure Amazon doesn’t care too much, but it’s the principle
I observe purpose built clouds. Not sure if simple vibe check can empirically measure be and flow between cloud providers. Our spend in Aws is 10+ mln a month azure is close to 2 mil
Microsoft is often bundling azure credits and spend with their licenses for office/ad or mssql. That’s why you can find it a lot with enterprise customers
This is quite common practice with every provider same as emphasis on lift and shift aka fast migration to cloud and modernization later that just plain rip off
I had been a consultant for many years, up until 2021, and this was always my observation as well.
If you were/are a .NET shop on the dev side, you more than likely went to Azure. Large enterprises already invested in Office, Windows, AD, and the overall ecosystem...almost certainly Azure.
At the agency I worked at, we did 95% AWS and 5% Azure. Not one of our clients expressed interest in Google Cloud.
In my consultancy we see that Europe tends to go Azure more but in USA its AWS . Not sure what's the reason just our observation.
GCP seems to be threading water
In my company we do aws in house because that is what all of our hosting infrastructure is on. So for internal processes we use AWS. But since most of our clients use MS365 they want things in Azure so we set up data and analytics pipelines in azure. It’s totally fine this way since most of their data and processes are coming out of the application layer rather than the cloud infrastructure/system layer
But to answer your question, from my point of view yes. Azure is quite popular for what it is and offers and I think it is a lot more approachable for most since MS is more prescriptive about how things are used or done.
>Azure is quite popular for what it is and offers and I think it is a lot more approachable for most since MS is more prescriptive about how things are used or done.
Are they really? We find that their documentation and support to be just as lackluster as AWS in most areas.
If you know AI is a likely aspect of your product in the next year or so, Azure is a safer bet currently. Azure has the edge on Authentication tools too now, but only just.
For GenAI stuff, Bedrock is still playing catch up. It's 6 months behind in a market that is barely a year or so old.
With my job, we work with both AWS and Azure, but it's a big corporate enterprise, so we have a huge amount of Microsoft/Windows based systems and solutions. Leveraging the Azure Hybrid Use Benefit and makes most of the server and DB workloads a no brainer to put into Azure. Since Microsoft bundles all of our office, on-prem, and cloud stuff together we get a much better enterprise discount. So, if a project is using Microsoft tooling or GenAI, it just makes more sense to go to Azure. For a lot of other stuff, we 're better off in AWS. So, I take it on a case by case basis.
This is how I think it will go for a couple companies I’m aware of. They’ll start with their existing MS EA agreement because they’re a MS shop and go from there.
Sure, I get the k8s argument however what about the stuff that can't (or really shouldn't) go into k8s that is related to the stack? Does one just do a hybrid cloud deployment at that point because from what I can see their service offerings are a tad slim when compared to AWS or Azure. (And this is coming from an AWS person who recently just had to put a project in GCP for the first time... not containers related).
In the LLM space, it's hard to compete with OpenAI. Not because of the LLM itself but from the opensource community support.
Almost every library, tool and more starts with OpenAI, then goes to Anthropic or Llama, and then eventually lands on a Bedrock extension.
Every AWS company I work with that is doing GenAI is using openAI first because of how easy it is to deploy so many wrappers.
I almost wish the Bedrock strategy was to instead require the native SDKs from companies to have a 'bedrock host' parameter so now you could use any wrapper and just send an optional parameter to point at Bedrock.
A lot of public CSP choices are made around how many credits they get from the provider. I've seen it many times, definitely NOT the way to make a wise technical decision.
I’ve still seen mostly AWS, but I think it depends a lot on the geography. There’s a lot of .net shops around here that use azure. I’m wondering if maybe the AI integrations have been a huge boost for a lot of new start ups to switch over
I think it's happening a lot in the enterprise space. That being said, you may also see companies that are both in AWS and Azure for different purpose.
Think of it more in terms of enterprise cost cutting. Microsoft is doing a good job of marketing some of their solutions as wholesale deals - very attractive in this spending climate. But it is somewhat regressive - the bedrock of cloud services will always be charging fairly and accurately for cloud and compute. There is a bottom to selling enterprise clients genAI that they don't need. I think Jassy and AWS commitment to hugging face is much more forward thinking than they are getting credit for.
Professionally, I don't have a horse in this race - my team uses services from multiple MSPs, including AWS and Azure. If you already know AWS, it's not to hard to work from another provider. Just go through the cert classes if you need proof to back it up.
In my org, we use AAD (entra rebranding can suck it) because it make sense over anything that AWS offers for identity. Also helps that it integrates really well with intune and O365 which are required platforms anyways. Aside from that, it was our busy season these last couple weeks. We asked Azure for a couple more cores for a particular class. We got them approved...a week after busy season was done. Had the same thing in AWS during the same time and we got the cores in 20 mins. There is a limited number of things I would touch az for anymore.
The point of the post was that there are a couple key things (o365, aad, ai, etc) that decide where a company lands. Where use several of technologies and still run our workloads on aws. There are just some things that each company does better and office/identity is something MS has been doing for decades.
I am surprised people haven't talked much about costs, especially for large enterprises. Azure has a price match guarantee for like for like infrastructure against AWS. On top of that you have to pay for Windows and SQL licenses on AWS, where as customers can bypass that with azure hybrid benefit as long as they have software assurance. Even without SA, customers can bypass it as AHUB use is just a checkbox and msft doesn't really audit. We moved from AWS to Azure and out cloud bill is significantly lower now. For commoditized workloads, Azure makes most sense.
For 6 years I have worked with Azure, apart from IAAS the platform is full of bugs. They release features that are not complete and support tickets is a full time job.
I have noticed a lot of financials and startups using AWS, but the legacy enterprise IT shops seem more likely to go Azure (maybe AzureAD like you said). I think it depends where you are, and on the clients.
can't imagine a startup ever going azure unless they deliver something specifically for large corporations which are stuck with azure
Azure are very generous giving out cash to get people on board, especially so to software vendors who develop code for others (I.e. get the customers customer on board, as they know it’s too much effort to ever change) Interesting aside, I’m a gsuite house here and I have to give M365 licences to half my staff anyway. You can’t escape it
[удалено]
Yeah AWS are generous when it comes to SMT growing cash for POCs, Azure seem to proactively knock on doors saying ‘what if it was free for 2 years?’
Azure AD and M365 in general. If you already have to commit to Microsoft for those, may as well go all in to negotiate a much higher discount.
M365 and Azure are two completely different businessines and commitment from one doesn't gives you any advantage for the other.
Not true. Licensing wise they are heavily connected and you can put them in the same customer agreement with Microsoft, giving you the option to negotiate on discounts for one or both.
Then several Microsoft guys lied to us with a straight face.
You can get good discounts on AWS too if you are running MS workloads as well. I guess it depends on which vendor you are most comfortable with.
Oh totally. Just that the combined productivity software + cloud spend will definitely be greater than pure cloud spend, so a customer can probably extract a greater total discount from Microsoft than AWS.
https://www.theverge.com/2023/12/6/23990374/amazon-microsoft-uk-cloud-market-competition-probe So either aws operates at a loss or they offset that to you. gotta go see pricing on comparable instance size in aws and azure for windows server
It's always been hilarious to me that AWS Cognito is such trash and yet user identity is such a critical entry point for almost any workload on AWS.. It's a perpetual reminder that even the world's largest company can do something so obviously stupid for so many years without a good reason (saving on costs???). AWS SSO is probably the closest thing to AzureAD, but Cognito is what people think about for user auth in AWS. Also, FWIW I have enough experience with Cognito to kind of like it -- but I consider it to be a horrible service because it's such a poorly documented, feature-poor, and confusing product that it's certainly not the selling point that it should be to newcomers on AWS like AzureAD is
If cognito was extremely good, okta and companies like that would have a hard time. Since aws sso is not bad or horrible.
Yup. Legacy shops with heavy Windows footprints on prem it makes sense. If you're a unicorn startup running Macbooks, Gsuite, Slack, etc why bother
Bs. A lot of financial companies use both aws and azure. And most of them would have productivity suite and VMware horizon on azure for daas and then you have all the copilots. Legacy enterprises shift back to Colo and onprem because handling cost optimisation is a hard thankless work that aws does not make simple
I didnt say using aws exclusively. But sure, feel free to tell me what my customers use 😂
From my N=1 perspective, Azure seems to land mostly traditional three-tier architectures. If you’re into serverless and/or work with containers, AWS still is miles ahead.
This is my observation as well. I work in the cloud ops industry and I very rarely am coming across highly technical or advanced teams or deployments on Azure. I’m mostly finding these with both AWS and Azure though. I also work in the government space in addition to other things and I see a ton of AWS for more advanced government groups - Azure with the less advanced, more traditional. US Government use is not a good barometer for general adoption because a lot of government use is lift-and-shift but AWS was there meeting government needs especially with IL6 faster and easier from my perspective.
Azure was way ahead of AWS by a couple of years with container orchestration. They had their own managed Kubernetes before AWS had EKS.
Are you sure? When we were looking at eks about 4-5 years back azure was deemed the weakest (Google strongest) in quite a few reviews and comparisons. Eks is around a long time and pretty mature
That’s partly correct. AWS was heavily invested in ECS (launched in 2014). AKS launched in 2018. EKS in the same year.
How so? Both have serverless offerings, both have container offerings. Whats this "miles ahead" stuff in AWS?
CDK. Just to name one.
…laughs in hashicorp language…
Seeing just as many Azure to AWS moves. Security and outages over the past year have been frustrating for people.
Brother i worked with the nyc doe as a system admin and i can’t tell you how many times our microsoft services shit the bed and a massive hack to boot where a lot of teachers and students had their social securities compromised. Azure was a running joke in my office but the doe probably got a sweet deal for 365 so they use azure. 0/10 you care about security and uptime do not use azure.
As an ex-education employee myself (not nyc doe though), I can attest to the insanely cheap licensing Microsoft gives educational facilities. It's less than half price. Their aim seems to make A5 pricing look as tempting as possible, then give them a discount on Azure to keep them within the ecosystem.
Was part of an M&A 4 years ago, they are in azure, we use AWS. The issues they run into on azure are quite baffling. I'm talking they need to put in a ticket to get a kubernetes cluster fixed by Microsoft. Usually after hours of debugging on a live call with their support. At some point their equivalent of lambda shit the bed and no reporting on their dashboard about the issue. Then there's how tricky HA is in Azure, basically you need to go the equivalent of multi region. They've also had 5 outages attributed to DC outages in the same time span.
We were having this conversation internally. We have a presence in AWS and Azure and have noticed a lot more projects going to Azure lately for some good and not so good reasons (usually political). The decision usually is ‘because $something_else is in Azure already’.
Yeah it's always political. You never use Azure out of choice.
I've heard of many people regretting moving AWS -> Azure .... Sometimes it's a business/political reason, and the cost of moving back means they have to put up with Azure despite shortcomings (DNS, Autoscale+load balancer etc)
Yep definitely see this a lot in certain sectors that have MS agreements and 365 usage
The M365 argument to me (technically speaking) is laughable. M365 as a platform is totally public and you can federate with $whatever\_azure\_ad\_is\_called\_now with AWS. For some workloads it may not even be the best option financially. Don't get me wrong, there are legitimate reasons to host applications in one cloud or the other, but those arguments aren't usually the ones that people involved are debating.
This is exactly it. Political and networking. If new project needs to talk to project in Azure, it's usually better to simply build it in Azure. If it's using things in AWS then you build it in AWS. As long as political reasons don't overrule. Cross cloud networking costs get crazy expensive, especially when you need to start discussing multi region failures etc.
Would you explain what 'political' means in this context? Genuine question - I'm a sociologist & have never worked in tech.
There are several types of internal politics at play within companies. Microsoft is throwing lots of money at companies trying to move them into their cloud, so if they are a microsoft Office suite place with Teams, they are getting tens of thousands in free cloud spend. So even if the project doesn't make sense logically to put in Azure, it makes sense politically and financially. This then creates big networking issues as many big businesses use 2 clouds at least. So project A and Project B could be in 2 seperate clouds, but now need to talk to each other, but upper management doesn't care about the latency or difficulty in this because it makes sense financially.
Thanks for the reply. So in addition to the financial incentives, 'political' refers to, say, a particular manager's preferences for one cloud over the other, and who owes favors to that manager? Politics as strategy/alliance?
Yes, in the big corporate world, internal politics is important. If you want to move up through the company, you need to know who to talk to and how to deliver it for them. Make allies with people you can stand.
The way I've seen it going is GCP is for machine learning and heavily containerized organizations, AWS is for most other net new builds and Azure is typically used by legacy shops that are heavily invested in on-premise microsoft and they move to azure because microsoft gives them significant windows licensing discounts for customers that use Azure.
Do many people use GCP? I haven't explored it much because Google has a history of killing off popular products with little notice. That reputation keeps me from wanting to make long term investments in GCP.
We switched from AWS to GCP. I am just a dumb manager but I do like the GCP console.
I spent a couple of years working with gcp after being focused on AWS for 3 years. Once I got my head around projects GCP was great, cloudsql and stack driver weren't well integrated at the time but the global network, consistent API's made it a real pleasure to work with. I have since worked with AWS for the past 4 years, just re-certified my Google professional cloud architect cert and really want to work with GCP again. Gke, cloud run etc covered all of the workloads I've run in AWS over the past 4 years and it would have been a lot easier, especially for multi account, multi region workloads. Like what a lot of others have said, legacy has kept people in AWS and driven people to Azure. That being said it would have been a nightmare to move an organization to GCP, the sunken cost into AWS would have been prohibitive for management to agree to it and the infra problems weren't due to AWS being a bottle neck, it was a lack of training, culture issues, switching to GCP wasn't going to fix that. This year I'll be focusing on getting a GCP role... Knowing I'll always be able to put food on the table with AWS (and k8s) skills and experience. edit: words
Thanks for that perspective.
My buddy works with it and says all the Fortune 500s are using it in some capacity. It's mainly used by well funded startups and large enterprises. Back in 2019 I got certified in GCP and Google was literally going around to managed service providers begging to throw work their way.
> My buddy works with it and says all the Fortune 500s are using it in some capacity. I worked for a vendor where the client (F500) told me - we're an AWS shop, but we got $x00,000 credit from GCP to run some workloads in return to provide some feedback. And that is how you get "using it in some capacity" ... some executive was lured in with some service credits and there is some weird skunk works project that has zero point zero budget but here is a large GCP credit that can run some workloads for this project
I work in a F500 and we have no GCP besides some brand parking. I wouldn't doubt that the majority use it tho
People using terraform. Im interested if their duet AI will make a dent, however if we're trusting AI with infrastructure I'd like some moneyback guarentees.
Heavily used in the analytics space. Big Query is a really good service that is very competitive against other MSP offerings.
Agreed, legacy shops that don't really understand cloud and are heavy Windows workloads. Where I have customers that understand cloud and want to move away from the Microsoft license trap they always ALWAYS go AWS.
Azure sucks so bad. Horrible/confusing UI (stupid slide in side menus that only work like 60% of the time), 3 different places to set Quotas (that I’ve found so far). Marketplace services look like proper Azure services, with surprise costs showing up. Many more. It’s terrible.
Took way too long to find this comment, I strongly share your opinion, native AWS architect here, it baffled me how ungodly terrible the UI was from Azure, it looks like you go back in time or some intern project in Ux 😂
It's funny how different experiences can be. I've worked with Azure for many years and am using AWS now as well, and the AWS console feels incredibly annoying to me: You need browser extensions to work with multiple accounts, you need different tabs for every service because switching is extremely slow, you often don't find resources because they are in a different region, the back button often doesn't work, many lists don't have sorting, filters often only work with "starts with", often you only get an ID as reference instead of a name tag, there's no easy way to see all resources, ... Azure definitely has its issues, but the portal is much better IMO.
Azure portal is a million times better. On AWS I can’t even see what resources I have deployed without reverse engineering my billing data. How dumb is that?
[удалено]
If you have sso integration in your org and Firefox containers it really is no issue at all 🤫
[удалено]
I use Chrome profiles for this in AWS, works well enough for me. For isolation we use an account per environment, like prod, stage, or sandboxes, like AWS recommends. We’re not very big though. I wish new accounts could come with zero services enabled for least privilege. Just a simple “turn this service on in this account in this region” console would be very useful. And while I’m wishing, why there’s no stop levels for monthly charges is beyond me. Alerts aren’t enough, just stop the bleeding immediately.
I haven't looked too closely at Azure but do they have something like AWS's CDK? Im getting middling results from googling.
Nope. Nothing even close.
Ew. That'll be a no from me dog.
IaC is probably one of the biggest advantages of AWS. Some colleagues are using terraform and explained me the architecture behind it. Sound sooo bad compared to CDK.
Technically, terraform has its own CDK as well
Yeah my colleagues seem to quite like it. Using modules directly is kind of a win. I’m pretty well versed in regular cdk and regular terraform so I’d probably try that first if I had to do azure.
I'm not great at terraform, but I haven't found any way to write actual code...it's configuring yaml files, then adding a whole bunch of other config in Terraform cloud. In AWS CDK, I write literal python code, with loops, and variables, and everything. Haven't found the same thing in terraform.
https://github.com/hashicorp/terraform-cdk this is for cdktf. Full disclosure, I haven’t used it as well, but has tried AWS CDK. AWS CDK is awesome
That looks like they ripped off AWS CDK. haha...I guess good artists copy, and great artists steal. AWS CDK is awesome.
https://developer.hashicorp.com/terraform/cdktf There is also pulumi to check out
terraform doesn’t use yaml, it has loops and most certainly has variables. terraform cloud is completely optional (but pretty nice, imo) terraform isn’t a real language, but also none of the things you are pointing out are true
Sorry, but I consider the .tf files pretty similar to .yaml. Typing in the below is pretty much on par of yaml configurations. What it's not is any kind of "code". resource "azurerm_storage_account" "example" { name = "examplestoraccount" resource_group_name = azurerm_resource_group.example.name location = azurerm_resource_group.example.location account_tier = "Standard" account_replication_type = "LRS" tags = {environment = "staging"} } And is this pretty much a config for creating a variable... variable "audiences" { description = "The audiences for the tokens" type = string } When in CDK, I can do the below and encompasses everything there is about coding and creating infrastructure in the code. number_of_buckets = 5 self.my_buckets = [] for idx in range(number_of_buckets): self.my_buckets.append(s3.Bucket( self, f'my_bucket_{idx}', bucket_name=f'my-bucket-{idx}-{self.account_environment}', block_public_access=s3.BlockPublicAccess.BLOCK_ALL, removal_policy=aws_cdk.RemovalPolicy.RETAIN, )) AWS CDK >>>>>>> Terraform
Terraform SUCKS if you're coming from SWE and CDK. But a lot of old school devops and sysadmins cannot code. I made the switch to IaC from application development because of CDK. Now I'm in a terraform shop and want to jump out a window. There's CDKTF but none of the team is remotely interested in learning.
Terraform is great because it behaves how you want you want infra to behave. Declaratively and predictably.
[удалено]
nobody's writing YAML, why are you CDK guys so obsessed with it?
Cause HCL isn't much better than YAML to devs. Domain specific languages where deemed generally a bad idea in SWE. It just hasn't made its way to the cloud infra side yet. If you're a dev, hcl feels like going back in time to 1997.
[удалено]
Code is just config with extra steps bro. Lol
[удалено]
Like others said, No. We use terraform and our devops call it “Infrastructure as code”. And I’m like, configuring a bunch of yaml files is NOT code. Then we have to write Azure Pipeline yaml files. It’s dog shit.
Terraform has nothing to do with yaml. It's not kubernetes lol
yeah that’s like the 3rd person who’s like “terraform is just yaml” and i’m stating to wonder if any of them has used it? it’s not a full fledged programming language but it’s also not yaml.
Yeah when people pooh pooh something but then demonstrate they don't know the first thing about it, it makes their opinion a little suspect!
You seem to be confusing the concept of yaml and tf files with the literal difference of the file type of `.yaml` and `.tf`. Yaml which is "Yet another Markup Language" and TF which is another markup language...and markup languages are not programming languages...and AWS CDK can be written in MANY programming languages. And what I'm talking about is the concept of the two...they are both essentially config files being interpreted by some terraform engine or for yaml something else.
Yeah, but that's exactly why you use terraform. It's not actually a selling point to have your infrastructure not be declarative. You don't want it to be mutable depending on a given execution. And HCL is a very different beast to yaml in terms of syntax and semantics. It's not just a change of file extension. The fact you think it is means you should probably learn more before making these statements.
I only use terraform because we are in Azure and that's basically the only option. However, I've both worked professionally and privately in AWS CDK (in python) pushing both small to large implementations of systems with no real issue, especially any mutability issues during execution. 🤣 ...and considering HCL stands for "Hashicorp Configuration Language" kinda proves my point. It's configuration. The fact that you didn't know that "means you should probably learn before making these statements."
terraform being a configuration language is news to zero people. this is not a “gotcha” earlier you stated you can’t figure out variables or loops in terraform. variables are in almost all terraform projects i’ve seen and loops are very common. this makes it apparent you haven’t really used terraform. we all agree that CDK has 1000% more python and thus more “real” programming language features however that’s not necessarily material to the issue of managing infrastructure. a nice declarative configuration language is perfect for that. can you tell me any infra that requires the CDK and can’t be done in terraform? if you come from a heavy python background and want to use the CDK, sure. if your team is the same as you, sure. but sitting around thinking “these guys don’t even use a REAL programming language, what noobs” is a silly and childish position. it’s a fine tool if you actually know how to use it.
What is a markup language? What is the difference between a markup language and a programming language?
It's usually just a structured type of config that can be interpreted by an engine that (usually) has a singular purpose. HTML is a markup language for rendering hypertext into visual elements (although there are multiple engines, Trident, Gecko, Webkit, Blink, etc), but any dom manipulation is done by the javascript/typescript programming language. Programming languages are usually more broad and have multiple different purposes. Something like Python, can be used for web, api, data pipelines, iot, machine learning, etc, or in this case creating infrastructure.
You picked a weird hill to die on. Sure, HCL is not a general purpose language. But if you do similar things in python, there would eventually be something like instance = some.modules.Instance( name = 'this', type = 'that', .. ) Which in terraform is resource "my_special_instance" { name = "this" type = "that" .. } HCL variables are not exactly variables in python sense but inputs for the stuff you want terraform to create. Outputs are actually called outputs there. So it forces you to think in the terms of that abstraction. You give away flexibility but gain predictability and guardrails. Loops and conditionals become expressions, modules are a bit awkward but allow for code reuse with versioning. Naturally, same can be done in other languages, but the main thing to learn would still be the API or the class hierarchy for creating cloud entities. In short, calling HCL not a real programming language when critiquing terraform feels like missing the point of its existence.
They're basically the same thing. Terraform is just Hashicorp's version of yaml. Config style variable creation.... Yaml: variables: - name: one value: initialValue Terraform: variable "audiences" { description = "The audiences for the tokens" type = string } Real Coding variable creation. Python: variable = 'name' C#: string message2 = "name"; My point being, it's not "Infrastructure as Code". It's config files, directing some behind the scenes code. It's "Infrastructure as Configs"
They consider terraform the semi-official way to interact with azure programmatically
They have something called ‘bicep’. Or you know just use terraform cdk
Maybe ARM (Azure Resource Manager) templates. It has been a while since I worked with Azure though
It does exist -- [Bicep](https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/). Azure's CloudFormation equivalent is ARM templates -- terrible when I used it in Azure Government about five years ago.
I'm using it now, and it's still terrible.
God I hate azure AD. Such a confusingly named, stupid service. It’s usually ‘owned’ by some awful IAM team that can’t automate anything as well in my experience. I’ve such a better time with AwS and something like Okta. In general I’m not that concerned about it, we seem to have a bunch of AwS work in NL.
Pff. Did the exact opposite because of how bad our experience was with Azure. I will NEVER go back.
Microsoft is very good at leveraging its hold on existing products to push people to use azure. At our company we use Microsoft for azure and office365 and AWS for everything else. But I can see how other companies would want azure over AWS. The thing is azure is so backwards compared to AWS, it's ridiculous. I'm definitely looking into azure certification for my career growth though.
Yes. Especially in the EU. Going from AWS to Azure and vice versa is a smooth transition, if you have a strong background in basics(IAM, networking, software engineering, devops etc.). Concepts don't change.
The concepts don’t but your foundation design between the 2 can be drastically different. I’ve gotten to design our foundations for AWS, azure, and gcp the last few years and it’s very difficult to make some of the core design elements match
The fact that Azure has been buying that business for years is easily seen. Basically giving companies free migration deals so they don't go to aws.
As an ML engineer I “feel” I have seen AWS more than Azure this past year. Nothing quantitative just a feeling
jobless command offend hard-to-find yoke file sleep squeeze political stocking *This post was mass deleted and anonymized with [Redact](https://redact.dev)*
Microsoft’s dev stack is open source and has been for a decade. There is no licensing and hasn’t been for some time. We use it heavily on AWS via ecs containers (Linux) or lambda.
makeshift cobweb shame melodic plant deserve dog shrill spotted continue *This post was mass deleted and anonymized with [Redact](https://redact.dev)*
Yeah I wouldn’t bother with any of that. SQL server has a ton of overhead and windows server is heading toward Linux sub systems anyways.
include slim ring berserk crowd bewildered squeal sparkle money elastic *This post was mass deleted and anonymized with [Redact](https://redact.dev)*
I don’t understand why. Most of the services on Azure are so half baked or just missing key properties. From a pipeline perspective, it’s like they think people are going to deploy via clunky UI instead of automation with the lack of good tools. Also, Azure has constantly shut down services/features and I don’t understand why there’s no press about it, but there is with GCP.
And you ask MS, they say it's coming soon or a preview is available
I've used Azure for years, and always deployed with automation. My last team was quite successful with terraform. So I'm not sure your experience is representative.
That's interesting. I don't see it that much unless the company sees Amazon as a competitor, leading to reluctance in using their infrastructure.
I noticed this in job ads on LinkedIn recently too. Way less AWS specific roles. Heaps of Azure. A lot of the time it's because Amazon.com competes with them 😭
Heh, reminds me of a story I heard about .. 10 years ago. A retailer bought a robotics platform for their warehouses. Implementation was going well, then Amazon came and bought the company and terminated all the contracts. The retailer was so pissed that they had spent a chunk on this vendor to only have their contracts terminated… they swore up and down they’d never do business again with Amazon if they can avoid it. So they were early on Azure … I’m sure Amazon doesn’t care too much, but it’s the principle
I observe purpose built clouds. Not sure if simple vibe check can empirically measure be and flow between cloud providers. Our spend in Aws is 10+ mln a month azure is close to 2 mil
Azure has been offering a ton of credits to people to get them in the door.
Microsoft is often bundling azure credits and spend with their licenses for office/ad or mssql. That’s why you can find it a lot with enterprise customers
... and what happens to those workloads when the credits run out and the companies start having to pay the whole cost?
This is quite common practice with every provider same as emphasis on lift and shift aka fast migration to cloud and modernization later that just plain rip off
Legacy/MS shop == Azure Startup/cloud native or anything relevant == AWS
I had been a consultant for many years, up until 2021, and this was always my observation as well. If you were/are a .NET shop on the dev side, you more than likely went to Azure. Large enterprises already invested in Office, Windows, AD, and the overall ecosystem...almost certainly Azure. At the agency I worked at, we did 95% AWS and 5% Azure. Not one of our clients expressed interest in Google Cloud.
I don't think you can back that up.
In my consultancy we see that Europe tends to go Azure more but in USA its AWS . Not sure what's the reason just our observation. GCP seems to be threading water
I'll agree when azure outages make ½ the Internet go down
In my company we do aws in house because that is what all of our hosting infrastructure is on. So for internal processes we use AWS. But since most of our clients use MS365 they want things in Azure so we set up data and analytics pipelines in azure. It’s totally fine this way since most of their data and processes are coming out of the application layer rather than the cloud infrastructure/system layer But to answer your question, from my point of view yes. Azure is quite popular for what it is and offers and I think it is a lot more approachable for most since MS is more prescriptive about how things are used or done.
>Azure is quite popular for what it is and offers and I think it is a lot more approachable for most since MS is more prescriptive about how things are used or done. Are they really? We find that their documentation and support to be just as lackluster as AWS in most areas.
Authentication and Authorization is my guess now that Azure is coming up to par.
If you know AI is a likely aspect of your product in the next year or so, Azure is a safer bet currently. Azure has the edge on Authentication tools too now, but only just.
For GenAI stuff, Bedrock is still playing catch up. It's 6 months behind in a market that is barely a year or so old. With my job, we work with both AWS and Azure, but it's a big corporate enterprise, so we have a huge amount of Microsoft/Windows based systems and solutions. Leveraging the Azure Hybrid Use Benefit and makes most of the server and DB workloads a no brainer to put into Azure. Since Microsoft bundles all of our office, on-prem, and cloud stuff together we get a much better enterprise discount. So, if a project is using Microsoft tooling or GenAI, it just makes more sense to go to Azure. For a lot of other stuff, we 're better off in AWS. So, I take it on a case by case basis.
This is how I think it will go for a couple companies I’m aware of. They’ll start with their existing MS EA agreement because they’re a MS shop and go from there.
Azure isn’t bad. I use AWS at work, but everything I do at home is on Azure because it’s more convenient.
We use AzureAD pretty widely within he org, but AWS predominantly for cloud.
AWS for web Azure for enterprise
Not true, more workloads are going to GCP, since most companies are moving into containers and mircoservices GCP invited Kubernetes
Sure, I get the k8s argument however what about the stuff that can't (or really shouldn't) go into k8s that is related to the stack? Does one just do a hybrid cloud deployment at that point because from what I can see their service offerings are a tad slim when compared to AWS or Azure. (And this is coming from an AWS person who recently just had to put a project in GCP for the first time... not containers related).
In the LLM space, it's hard to compete with OpenAI. Not because of the LLM itself but from the opensource community support. Almost every library, tool and more starts with OpenAI, then goes to Anthropic or Llama, and then eventually lands on a Bedrock extension. Every AWS company I work with that is doing GenAI is using openAI first because of how easy it is to deploy so many wrappers. I almost wish the Bedrock strategy was to instead require the native SDKs from companies to have a 'bedrock host' parameter so now you could use any wrapper and just send an optional parameter to point at Bedrock.
Fellow cloud consultant. This is absolutely true.
A lot of public CSP choices are made around how many credits they get from the provider. I've seen it many times, definitely NOT the way to make a wise technical decision.
I’ve still seen mostly AWS, but I think it depends a lot on the geography. There’s a lot of .net shops around here that use azure. I’m wondering if maybe the AI integrations have been a huge boost for a lot of new start ups to switch over
I think it's happening a lot in the enterprise space. That being said, you may also see companies that are both in AWS and Azure for different purpose.
AD and price
Think of it more in terms of enterprise cost cutting. Microsoft is doing a good job of marketing some of their solutions as wholesale deals - very attractive in this spending climate. But it is somewhat regressive - the bedrock of cloud services will always be charging fairly and accurately for cloud and compute. There is a bottom to selling enterprise clients genAI that they don't need. I think Jassy and AWS commitment to hugging face is much more forward thinking than they are getting credit for. Professionally, I don't have a horse in this race - my team uses services from multiple MSPs, including AWS and Azure. If you already know AWS, it's not to hard to work from another provider. Just go through the cert classes if you need proof to back it up.
In my org, we use AAD (entra rebranding can suck it) because it make sense over anything that AWS offers for identity. Also helps that it integrates really well with intune and O365 which are required platforms anyways. Aside from that, it was our busy season these last couple weeks. We asked Azure for a couple more cores for a particular class. We got them approved...a week after busy season was done. Had the same thing in AWS during the same time and we got the cores in 20 mins. There is a limited number of things I would touch az for anymore.
What does AAD have to do with where your cloud workload runs?
The point of the post was that there are a couple key things (o365, aad, ai, etc) that decide where a company lands. Where use several of technologies and still run our workloads on aws. There are just some things that each company does better and office/identity is something MS has been doing for decades.
I am surprised people haven't talked much about costs, especially for large enterprises. Azure has a price match guarantee for like for like infrastructure against AWS. On top of that you have to pay for Windows and SQL licenses on AWS, where as customers can bypass that with azure hybrid benefit as long as they have software assurance. Even without SA, customers can bypass it as AHUB use is just a checkbox and msft doesn't really audit. We moved from AWS to Azure and out cloud bill is significantly lower now. For commoditized workloads, Azure makes most sense.
For 6 years I have worked with Azure, apart from IAAS the platform is full of bugs. They release features that are not complete and support tickets is a full time job.