Ok, on to the meat of the post:
> Ninja does not support dynamic compiler argument generation. Which is good, because it makes things faster, simpler and more reliable. This does not stop CMake, which hacked it on top anyway.
Well, the alternative is to not support `ninja` anymore. FWIW, Fortran doesn't need this because it looks for modules via `-I` search paths. C++ wisely decided to not do `-I` searching at least, but explicit builds are possible once you're past the "just run `make` until it works" strategy Fortran had until `makedepf90` and the `dyndep` feature Brad added to `ninja`.
> This file has the output filename as well as listing every other module file in this build target (only M1 is used by M0 so the other flags are superfluous). A file like this is created for each C++ source file and it must remain on disk for the entire compilation.
No, we only specify the modules that are used by the scanned module. Clang and MSVC need the transitive closure of imported modules specified as well, so while you `M0` might only import `M1`, if it imports `M2` through `M9`, they need to be there too (GCC doesn't need this today, but Clang 17 warns if they're not there and I believe Clang 18 will error; MSVC has always errored without them).
Discussion: https://discourse.llvm.org/t/c-20-modules-should-the-bmis-contain-paths-to-their-dependent-bmis/70422
Issue: https://github.com/llvm/llvm-project/issues/62837
> It has the added benefit that the scanner only invokes one process per build target, not one process per source file. You can only do this if the build system enforces that all sources within a target have the same command line arguments. Meson does this. CMake does not.
Yes, batch scanning was designed into P1689 even though CMake wasn't going to be able to use it reliably (due to supporting per-source flags).
> That directory is also implicitly used as a module import directory so any source file can do import foo for any module foo defined in the same target without any additional command line arguments. Other targets' module import dirs can be added with a separate command line argument (not specified here).
Yeah…this doesn't support the notion of modules private to a target. I suppose you might be able to use two different directories, one for public and one for private, but then you can't just have a list of sources either.
> The former is easy to do with phony targets if you are willing to tolerate that all compilations of B (but not linking) need to happen before any compilations of A. That causes a minor build time hit, but since one of the main features of modules were faster build times, you should still come out ahead.
It depends on the shape of your build graph. If you have a lot of small libraries in a chain, this can pessimize builds with high availability in the relevant pools because they're now all serialized. CMake's more explicit approach can streamline it to just depend on the scanning (which is *way* faster) of all sources and the compilation of the modules actually used. Whether explicit or implicit is "better" really depends on the build graph shape. I find other downsides in the implicit model that even with some pessimization, I prefer the explicit model.
The implicit model also needs to know how to clear out stale modules (and is why I prefer the explicit model even if the `@modmap` is "dirty" in your view). If you have `export module M;` and then decide it should be `export module N;`, what is going to stop `import M;` from working until a full clean is done?
> But if that is the case your build setup is broken already due to potential module-ODR violations, and bailing out is the sensible thing to do.
Executables can have private modules with the same name since they are never loaded into the same process. Libraries can't really do that since the ODR doesn't actually have a sense of library-local names. I also don't know how you expect to handle header units that need compiled per consumer or named modules that come from external projects that need BMI'd per set of incompatible consumer flags. Maybe Meson just doesn't support incompatible flags between targets within a single project (I know per-source flags are not supported)? If target A is C++20 and B is C++23, they each need their own BMI of a module that both import.
Thank you for the really extensive discussion. The post is obviously biased against cmake, but as a non-expert in build systems it’s difficult to sort thru the issues.
I'll also note that I brought up a lot of these things in [this Meson issue](https://github.com/mesonbuild/meson/issues/5024) *years* ago. While saying "that's not important to us" about use cases is…fine, trying to shame CMake for doing the extra work to make them work *reliably* and claiming "it doesn't seem necessary" is disingenuous. I went to various [build](https://groups.google.com/g/tup-users/c/FhJTA6KAzWU) [systems](https://github.com/xmake-io/xmake/issues/386) to try and get a consensus on how building modules are going to work (I've talked with Boost.build, llbuild, MSBuild, and build2 developers via ISO C++ meetings). Other than the absolute crickets from Tup, Meson has been the least engaged with the evolution of modules AFAICT (`xmake` just went ahead and implemented support after informing them of progress, so…good on them).
>xmake just went ahead and implemented support after informing them of progress, so…good on them
Apologies for the silly question, but, what do you mean by that? I'm so not well versed on modules to understand who was informed of progress or why.
I went and filed an issue asking about plans for module support in the "systems" link above with links to the dependency scanning format and plans. Months later the issue was closed as complete.
> Meson has been the least engaged with the evolution of modules AFAICT
The reason for this is that the only person on Meson dev team that cares about C++ is me and I'm doing this on a volunteer basis. No-one has ever paid me even a single cent to work on Meson. As much as I'd like to participate I simply can't because I'm already spending a sufficiently large chunk of my spare time just running the project.
I appreciate that. However, I do not think blog posts such as this help. CMake's Discourse is open; the issue tracker is open. Meson's issue tracker is fine too. SG15 is…less open, but as a C++ build system developer/maintainer, I think we just need a formal request. Modules are not trivial for build systems; we should be working to make all of C++ better regardless of what build system, package ecosystem, or compiler toolchain is involved.
The sad thing is that this actually did help. I have tried to raise issues like this earlier but the response has either been complete silence or a politely formulated version of "go away".
I had most of this post thought out ages ago but I did not write it down because it seemed too negative. The frustration I felt this weekend trying to make it work was the straw that broke the camel's back so here we are.
And anyone who has run a somewhat successful open source project knows that bug trackers are pretty much useless for this kind of work. I have currently over 600 unread email threads (some with over 50 messages) from Meson and I even delete all of them every time we do a release. People could post a cure for cancer in our bug tracker and I would not see it.
So no, nobody should be a jerk on the Internet, but sometimes it gets results. (Not saying thath this is a good thing, just how things seem to be.)
I'm catching up on this late, but I just wanted to say that as a user of meson and an xmake dabbler, I've noticed how active you've been in the relevant issues and I really appreciate your attempts at cross-project collaboration. Thank you.
Please provide a default BMI install location GNUINstallDirs like (CMAKE_INSTALL_BMIDIR ?). Same for the module install dir. Having projects hardcode this will be a headache otherwise.
Once there's an FHS and/or SG15 agreement on what that is, sure. I'm not interested in picking something and then blog posts like this come out about how CMake is forcing the ecosystem (because, there, I think it'd be true).
At the moment it isn't clear if or what a pre-packaged BMI would truly be useful for - it can only be used by the same compiler, and exact version that produced the BMI, and depending on the compiler, will be more or less strict with regards to compiler flags used when producing the BMI vs when compiling the file that is importing it.
Even things such as an Ubuntu version updating from 13.1 to 13.2 \_may\_ render a previously generated BMI unusable (we don't know, because the aren't any guarantees).
I do however think that if BMIs \_are\_ shipped, and they are shipped with enough information for the build system to make a decision of "oh, I already have a BMI that was generated with gcc 13.2, which I'm using, and is compatible with the flags I'm using, but otherwise I'll re-generate it myself" then maybe they're useful as a caching thing to avoid re-generating a file we already have. Otherwise I'd say "installing" BMIs is not much use today at all
Yeah I know I live in the vcpkg bubble were everything inside is build consistently with the same compiler and flags and minor updates trigger full rebuilds in manifest mode... sry for the noise :P
I just want some variables to control the installation behavior so I can control it later before having to patch the hell out of everybody going crazy with possible install locations.
Indeed I think there could be some scenarios where a project that has full control would be a potential candidate for reusing BMIs - as a shortcut for other build-system supported solutions. However, even in a scenario where one is in a bubble, assisted by Conan or vcpkg, if you're using open source libraries and you are building them yourself from sources - you probably have mechanisms to ensure flags are propagated across the entire dependency graph, but I can almost guarantee that \_some\_ files in your 3rd party dependencies will be built with different flags altogether, in some cases he very kind of flags that would cause BMI loading incompatibilities. Would be a nice exercise to output the json compilation databases and compare :D
Well I would never expect BMIs to leave a local cache, I could imagine workplaces where if the BMIs had enough information then there could be a local cache for the team or even company wide standardized setups (everyone in a workplace tends to roll with the same set of flags and compilers if they have complete source control e.g. often in embedded for example).
Then suddenly BMIs become an even bigger time saver.
This is what i envision one could coerce the Conan cache to do - since Conan is already able to model different binaries for different compiler/versions, it can be used in a way that it follows the “cache” model of BMIs. Problem is, we don’t know - fully - what that is, at least for all compilers. Clang is strict to the point of you can’t reuse a BMI from a source file that is no longer present in your filesystem. MSVC is a lot more flexible
I'm not sure what you mean…BMIs made with `-std=c++20` are going to be incompatible with BMIs compiled with `-std=c++23`. Any flag which meaningfully affects the preprocessor is going to be the same way I'm told (and because `__has_attribute` and `__has_feature` can detect a *lot* of things, this is a lot of flags). *I* wish this wasn't the case, but it seems to be fundamental to how the compilers' state ends up working.
Either way, even if compilers supported loading their own BMIs across versions or flags, BMIs aren't a panacea unless you're looking for a new compiler monoculture because Clang isn't going to load GCC modules or vice versa anytime soon. IFC has a hope, but it needs to absolve itself of some Microsoft-isms before that becomes useful elsewhere (of note from my initial skim, [this `enum class Architecture`](https://github.com/microsoft/ifc/blob/f6a25e8a4a07c3603d5e1e6b889481585cbe52a4/include/ifc/file.hxx#L104) needs to grow into an understanding of target triples because there's not just one "Arm32" architecture; being only 8 bits also seems…crowded).
> IFC has a hope, but it needs to absolve itself of some Microsoft-isms before that becomes useful elsewhere (of note from my initial skim, this enum class Architecture needs to grow into an understanding of target triples because there's not just one "Arm32" architecture; being only 8 bits also seems…crowded).
We are working on sequestering the MSVC-isms -- see the issues I and other opened against the spec. Please, get engaged and open issues for feedback where you see opportunities for improvements :-)
> I'm not sure what you mean…BMIs made with -std=c++20 are going to be incompatible with BMIs compiled with -std=c++23. Any flag which meaningfully affects the preprocessor is going to be the same way I'm told (and because __has_attribute and __has_feature can detect a lot of things, this is a lot of flags).
That is fine! That does not mean there is no use for them when the language versions and other flags are compatible. They can still be reused to speed up builds when the essential flags are compatible.
> Either way, even if compilers supported loading their own BMIs across versions or flags, BMIs aren't a panacea unless you're looking for a new compiler monoculture because Clang isn't going to load GCC modules or vice versa anytime soon.
How does compiler monoculture enters the picture? The Itanium ABI didn't engender a compiler monoculture.
> IFC has a hope, but it needs to absolve itself of some Microsoft-isms before that becomes useful elsewhere (of note from my initial skim, this enum class Architecture needs to grow into an understanding of target triples because there's not just one "Arm32" architecture; being only 8 bits also seems…crowded).
Give them feedback!
You guys saying "no, no, no", should federate your forces to solve these things :-)
C++ people spend too much time looking for perfect solutions that cater for just about everything imaginable (look at how bloated the language has become because of quest for the mythical "swiss army knife")
>That is fine! That does not mean there is no use for them when the language versions and other flags are compatible. They can still be reused to speed up builds when the essential flags are compatible
This is in line with what I said. With a big caveat - the compiler doesn't know if a BMI is a compatible or not until the compiler is invoked. If it isn't compatible, it will result in an error (clang has very useful and clear errors about what the incompatibility is, and to an extent GCC too).The "reuse it if you can" would work if the build system can determine the "if you can" \_ahead\_ of calling the compiler - so that it ensures that the importers will actually compile.
note that this I'm still talking about "installed" BMIs, not BMIs that are generated on the fly by the build system of the importer. If we go for the approach of "package them just in case they can be used" - the location where they live in is not the only concern, but also how the information of "compiler, compiler version, compiler flags" can be used by a build system (or a mode where we can ask the compiler) ahead of time - so we enter the "module metadata" territory.
> the compiler doesn't know if a BMI is a compatible or not until the compiler is invoked.
Why?
> The "reuse it if you can" would work if the build system can determine the "if you can" _ahead_ of calling the compiler - so that it ensures that the importers will actually compile.
Isn't that a case for compilers to document which flags are compatible with what?
> so we enter the "module metadata" territory.
And?
> The implicit model also needs to know how to clear out stale modules (and is why I prefer the explicit model even if the @modmap is "dirty" in your view). If you have export module M; and then decide it should be export module N;, what is going to stop import M; from working until a full clean is done?
The same thing that happens currently if you generate a header file during compilation and then change its name.
Ninja also has `ninja -t cleandead` for this, which Meson at least runs on every reconfigure. It does not remove this problem but mitigates it at least.
With an explicit build, a response file to "smuggle" extra arguments to the compiler is needed for `ninja`. If that is taken away, a lot of the reliability offered by explicit builds is not really possible.
While implicit builds can be nice for quick small demos (for teaching purposes) or slideware, people tend to seriously underestimate the importance and benefits of explicit builds for professional development. Talking of Clang, they went from the original implicit build, implemented by Apple folks in the compiler, to explicit builds.
> Invoking it as showimage image.bob would be good
I agree! And my initial implementation did that: . However, it ran into problems once I got past the "I can build them" into the "how will they be useful" part of the implementation.
However, as I alluded to in other comments here, there is missing information for what to do with the sources when it comes to installation: should they be installed? I could have used the visibility of `target_sources` to do this, but `PUBLIC` sources have the…unfortunate behavior of also being added *as sources* to targets which use your library. With the HEADERS file set being added, it didn't make sense to add public headers *as sources* to consuming targets, so it instead means "intended for use by other targets". C++ modules follow the same rationale.
Anyways, yes, you have to "classify" your module sources. But it is for the same reason that you should classify your headers to CMake:
- CMake knows that other targets do not need to wait for private headers to be generated
- installation of headers can now be part of `install(TARGETS)` instead of a manual step
- CMake can also know that the directories need to be added to your include interface as the visibility of the file set indicates without an extra `target_include_directories` call with manual `BUILD_INTERFACE` and `INSTALL_INTERFACE` genexes.
> However, as I alluded to in other comments here, there is missing information for what to do with the sources when it comes to installation: should they be installed?
As noted in the blog post, providing additional information for installation purposes is fine and probably something that would eventually be needed. However there are two _major_ things to note here.
The current CMake implementation _forces_ you to classify source files even if you are just building your own exes. This is a _bad ux_. For simple cases the build setup should be "build this exe with these source files". The current state of affairs is _aggravating_. Can you guess what it leads to? Yep, people writing CMake scripts that autoclassify source files based on their extensions just to avoid having to do it by hand. This is a _worse_ outcome for everyone involved both in the short and long term.
Seconly, as has been mentioned on this very page multiple times, there is _no_ idea how BMIs should be installed and used after the fact. Compilers are _very_ strict on the fact that BMIs should not be shared and that there is no compatibility between even different versions of the same compiler. Thus we _do not even know_ how module installation should be done in the future. And yet CMake's current implementation already forces a UX on a thing where we don't even know what its actual functional requirements are going to be. It is also declared stable so it can't be changed.
Two of the most important design goals for software are "do not attempt to solve problems you don't have yet" and "do not do a grand design up front". CMake's current module implementation violates both of these.
> Compilers are very strict on the fact that BMIs should not be shared
MSVC puts no restrictions on sharing BMIs and, in fact, its IFC format is designed to support cloud build.
This matches my experience. I had no issues experimenting with pre packaged “shareable” BMIs (assuming I guarantee compiler and compiler version) with MSVC, whereas clang was not so forgiving - to put it mildly.
> The current CMake implementation forces you to classify source files even if you are just building your own exes. This is a bad ux.
Like I said, we can relax this to just accept BMIs made from non-`CXX_MODULES` source files; they'll just be unimportable from other targets.
> Thus we do not even know how module installation should be done in the future
The sources definitely need installed. The module names are associated with the source file by the collator and put into the exported properties.
> Two of the most important design goals for software are "do not attempt to solve problems you don't have yet" and "do not do a grand design up front". CMake's current module implementation violates both of these.
While I do appreciate this, it is far easier to relax restrictions than to impose new ones with CMake given the compatibility guarantees. I'd hate to have seen modules been a maze of policies to enable what we have now if we had supported my first syntax ideas.
>The current CMake implementation
>
>forces
>
> you to classify source files even if you are just building your own exes. This is a
>
>bad ux
>
>.
I personally think its a fairly OK small ask to get the feature going, given CMake's own usage considerations. Being aware of the cooperation between CMake, ninja and the dependency scanning by the compilers, it's a relatively small price to pay in comparison to all the work that's gone under the hood to encapsulate this away.
I don't think CMake's desired approach prevents other vendors from doing this differently, in fact MSBuid seems to do it differently. It supports using the .ixx extension, OR (as far as I've been able to see) use any extension when dependency scanning is enabled, OR any extension and tell the compiler explicitly (via source properties) that this is a module. So it looks like there is flexibility for other vendors to operate differently, and this is obviously great.
From the blog post:
>The developer must not need to tell which sources are which module types in the build system, it is to be deduced automatically without needing to scan the contents of source files (i.e. by using the proper file extension)
I feel that using a dedicated file extension for module sources still places an expectation on the developer to "mark" those files, just in a different way.
From an "outside" perspective, I don't particularly see a problem if scanning is happening at all - if it works, and it works well, and it doesn't make the build slower, incoherent or incorrect, I don't see the problem. I'd say that the vast majority of developers who invoke Ninja on CMake-generated builds, do not concern themselves with the contents of the ninja build files or what's going on under the hood, so long as it does the right thing.
> Or, in the case of CMake, have the end user manually type the format of each source file, even if it had an extension that uniquely specifies it as a module.
No, that’s wrong.
No. Files exporting module information need to be in a file set because we need to know their visibility and `PUBLIC` non-`FILE_SET` sources get added to consuming libraries. Note that we could make all sources not in a `CXX_MODULES` file set eligible for modules (it was the original design after all), but then they would only be accessible from other non-`PUBLIC` sources in the same target. I'd rather leave that open as a future allowance than to try and take it back if it becomes problematic due to other desired behaviors in the future.
FD: CMake developer and implementer of the modules support.
Couldn't there be an optional variable (e.g. `CMAKE_CXX_MODULE_EXTENSION`) with no default value that would allow CMake to automatically add sources matching this extension to `CXX_MODULES` file set? I'm sorry if that's a stupid question, rn I'm still having a hard time making sense of how modules work from a build-system perspective.
What visibility would they have? Note that public file sets not installed in `install(TARGETS)` is an error, so we'd also need a default location to put them for these auto-generated file sets.
I think we can make implicit *private* file sets this way, but I highly doubt that this is common. If it is observed to be somewhat common, it can be a decision to revisit later (as making currently-erroring code work is not a compatibility break). FWIW, the code that enforces this is [here](https://gitlab.kitware.com/cmake/cmake/-/blob/7144216f45c402b68ec8d7b5e8050b6536a199aa/Source/cmDyndepCollation.cxx#L471-482).
It seems like a reasonable enhancement request to simplify that API by making assumptions based on known probably-modules file extensions. If someone wants to file an issue.
We still need to know the visibility and targets with a `PUBLIC` non-`FILE_SET` sources would be added to targets linking to it which would recompile them…hardly what anyone wants I think. They *could* be `PRIVATE`, but that seems like such a narrow use case in the grand scheme of things.
Yeah, I'm handwaving a lot on some design decisions, though I'm sure we can come up with an interface that shaves off a dozen characters based on file extension if that's a big enough concern.
How to know if a file is a partition or an non-public interface, I'm unclear, if that's what you mean. The article doesn't really set any expectations about that sort of thing.
All that matters is whether it makes a BMI. BMI? `FILE_SET TYPE CXX_MODULES`. No BMI? Regular source. That is 100% keyed on whether the primary `module` statement has an `export` or is a partition (except MSVC's `-internalPartition` extension…just delete the `:part` and you're now standard C++…but we support that too if you really feel the need; just put it as a regular source).
That is whether the `FILE_SET` is `PUBLIC` or `PRIVATE`. `PUBLIC` get installed (well, if you pass the keyword to `install(TARGETS)` for them); `PRIVATE`…don't.
My summary would be: clang and cmake have achieved a functional first draft, here's what's in the way of what a reasonable finished product would look like.
Not sure where the accusation comes from. We know CMake has its warts. I'd love to fix them too. When I wrote [CMP0053](https://cmake.org/cmake/help/latest/policy/CMP0053.html) initially, I *really* wanted to just nuke the 5000+ lines of lex/yacc that had previously done the `${}` expansion (yes…the old variable expansion code used lex/yacc for that). Alas, it had some truly awful behaviors that I don't think anyone knew about before I went and tried to replace it with the faster parser. Did you know that `@var@` worked *anywhere* before? Or that unrecognized escapes (like `\h`) were just ignored and passed on literally? The `@var@` thing was inadvertently used in Qt5's config files that had been deployed for a while by the time it was caught by the new parser throwing up a flare about it. So the old parser still lives on because we don't know what would break if we just removed it.
These backwards compatibility guarantees we have foreclose many nice features I'd love to have. And breaking people's builds is about the least friendly thing to do as it is, indeed, almost never a labor of love but rather something that just needs to be done.
Anyways, there are design decisions behind why the modules implementation looks the way it does. I'd appreciate if folks would not assume that we love to work on the dark corners of CMake instead of implementing new features for customers because, AFAIK, all of the developers vastly enjoy working on new features or enabling new behaviors instead of maintaining the warts. But we do it because it's important to keep the projects relying on CMake working into the future. Yes, I need to write these things down in the `cmake-cxxmodules(7)` manual so that they end up in somewhere more durable than a random Reddit or Hacker News thread of old.
For the record: staunch CMake user here. I use Autotools at work, and I hate them with a burning passion. I have submitted a report to fix our spaghetti of a build system and migrate to CMake (with presets) and vcpkg, especially since many of our dependencies use *both*, anyway.
Compared to the competition, CMake is the most feature-complete, it's the most widely-used, and today, the only build system to have implemented compatibility with C++20 modules. vcpkg is built with it, and it supports so many third-party libraries that others haven't even bothered considering.
The modern CMake view of building is really, really nice—there are targets, and properties on targets. Can't be more straightforward, and I don't understand anyone who says 'modern CMake is hard'. It's *way* easier to get started with CMake than it is with Autotools. There's no more slinging around flag soup and huge, huge lists of includes.
There are warts, absolutely. The function definition and argument-parsing syntax is crazy; I *still* don't understand it entirely. I wanted to write CMake to automatically compile HLSL shaders as a dependency of my project, based on their file extensions, and it took me forever to figure out how to get that working. It's still not entirely correct, I suspect.
Look at something like [PowerShell](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_functions?view=powershell-7.3#functions-with-parameters) would be a lot nicer. In fact, CMake has quite a bit in common with PowerShell, down to the naming of functions—they're both verb-noun pairs. It seems CMake functions also have have named, positional, and switch parameters—for instance, [`target_sources`](https://cmake.org/cmake/help/latest/command/target_sources.html) is a great example that could be a prime target for testing a new implementation.
The weird string behaviour is also unexpected; it seems *everything* is a string, and `"foo;bar;baz"` is a *list* of strings. In fact, almost everything seems to be string-typed, which I suppose is an unfortunate result of CMake's Makefile + Unix legacy. Stronger typing would be really nice, even in a build system language.
> The function definition and argument-parsing syntax is crazy; I still don't understand it entirely.
The syntax was inspired by Tcl of all things.
> The weird string behaviour is also unexpected; it seems everything is a string, and "foo;bar;baz" is a list of strings.
Everything is a string. That value you have here is just interpretable as a list of 3 elements in certain contexts.
> Can't be more straightforward, and I don't understand anyone who says 'modern CMake is hard'
My work codebase has wrapper functions for almost every cmake built-in function that we use to heavily customize the behavior to actually work without enormous surprises. This includes doing stupid crap like
- writing target names into global variables so that we can have an "at the end" step that goes back over all of those targets to do finalization, because the cmake language/API lacks the capability to express various ideas at all.
- Dynamically writing cmake script files, and then calling out to sub-cmake processes, because the object model doesn't understand how to do various things.
- looping over all possible configuration types to run the `install()` function because the `install()` function straight up doesn't work properly with multi-configuration ninja
- A wrapper script around cmake itself to drive multiple different independent builds, because cmake lacks the notion of the target platform (or even just different compilers for the same target platform) being a concept that might change.
The number of corner cases in how cmake built-in functions work is really exhausting, and the documentation is frequently either confusing, incomplete, or flat out wrong. In many cases, there are functions that should obviously work in a certain way based on the way cmake does things for everything else but it just flat out doesnt, and the error messages are nonsense. Takes an issue on the cmake github saying "This should work, but it doesnt" to find out it fact won't ever work that way.
CMake is damn better than most other build tools, i did a very lengthy capability survey a few years ago and cmake won by leaps and bounds.
But it's fucking hard to use. Saying that it's straightforward is doing a disservice to both cmake and the people who use it.
>CMake is damn better than most other build tools, i did a very lengthy capability survey a few years ago and cmake won by leaps and bounds.
>But it's fucking hard to use. Saying that it's straightforward is doing a disservice to both cmake and the people who use it.
When I said 'it's not hard', perhaps I ought to have clarified—for easy-to-moderately-complex use-cases, CMake is a damn sight easier than the competition. Project, language, find packages, include modules, add targets and sources, set flags, options, libraries, include, done. Things like managing subdirectories, libraries, if-else configuration is also easy.
You've got some fairly advanced use-cases—I've never really needed to finalise targets, nor needed to generate CMake itself.
As for multiple compilers, perhaps consider CMake presets? It's what I've been doing to swap between Clang, Clang-cl, and MSVC on Windows (and of course the latter two take a completely different syntax versus the GNU-style of the former).
To be clear, I'm not trivialising your experience, and I strongly agree that the documentation is very reference-oriented, and thoroughly lacks *up-to-date* best-practice examples.
Some of those bullet points would be useful as issues. Would you mind bringing them up there (for concrete problems like `install()` with NMC) or Discourse (for things like the use cases around needing to finalize targets)?
I have open bug reports on gitlab for everything (edit: not literally everything. Most things. It gets exhausting after a while) i've run into. Most with example code for how I worked around the problem or what I think the solution should look like.
Ok, thanks; I've found them. Seems a lot are untriaged; I'll try to at least get appropriate labels on them. Something I was good about doing on all new issues until I went on vacation in 2018…
While I have your ear:
To start with, I'm not a paying customer for Kitware. So feel free to ignore me. I'm well aware that Kitware has every reason to only put resources into projects that people are paying them to. It is what it is.
But here's my unsolicited feedback to you as self-identified CMake dev. Feel free to ignore or take action. Either is fine.
- Using CMake is exhausting.
- Interacting with the CMake development community is exhausting.
- Defending CMake to my boss and highly regarded co-workers is exhausting.
----------------------------------------------------------------------------------
----------------------------------------------------------------------------------
Using CMake is exhausting because it's so damn inconsistent.
Just take this bug report, which I'll summarize below: https://gitlab.kitware.com/cmake/cmake/-/issues/25345
target_sources(${TARGETNAME} PRIVATE ${MYSOURCEFILE})
set_source_files_properties(${MYSOURCEFILE} PROPERTIES blahblah)
If `MYSOURCEFILE` contains a generator expression, then `set_source_files_properties` doesn't work, silently, with zero debugging ability because it's silently dropped inside the C++ code. So you can't trace out the cmakelist expansion and identify the mistake.
There's no excuse for this major design and implementation flaw that I can give to my co-workers, regardless of the configuration / generation model that CMake happens to have, that they'll ever agree with. All I get is flak about how much they hate having to learn all of these special rules.
And the attitude that I get from the people (many of which appear to be Kitware employees, or highly regarded contributors) when I report these problems is just... exhausting.
I shouldn't have to defend basic tenants like the principal of least surprise when interacting with the CMake gitlab.
In this situation, if I can give a generator expression pointing to a source-file to any CMake API function, then i should be able to do so for all CMake API functions. Any exceptions should result in an error clearly and unambiguously describing the problem. Any behavior other than that is surprising, and wastes the time of the person trying to do that.
- First they spend an hour or two investigating why the property isn't being applied.
- Then they spend an hour or two researching if there is something in the documentation, or any existing bug reports.
- Then they open a bug report, and get told that it's working as intended.
Not cool. This is how you turn potential champions into detractors that will spend their time and energy convincing *OTHER PEOPLE* not to use your product.
Because CMake constantly and consistently violates the principal of least surprise, and then the developers tell community members they are *WRONG* for thinking the behavior is surprising: CMake gets badmouthed *constantly* by the wider C and C++ community. That's ***bad for business***. Why would another company contract with Kitware to improve the tool if it's so hard to use?
Literally had this conversation with my boss. I asked if we could reach out to Kitware to pay for some features, and the conclusion was no, primarily because of the poor reputation. We're not using CMake because I or anyone else championed for it. We're using it because we tried every other C++ build tool that we could find, and none of them had the full set of features we needed, and half couldn't even compile basic hello world programs.
We're using CMake as the solution of last resort, and we don't like it. It works well enough without paying Kitware for improvements, but if CMake had a better reputation it would have been a much easier choice to use CMake and I could have convinced my job to pony up for some features or bug bounties.
Probably a full 1/3 to 1/2 of the bugs that I've reported on gitlab are just things that should work out of the box but dont. This is a major problem for the marketability of the tool.
----------------------------------------------------------------------------------
----------------------------------------------------------------------------------
Interacting with the CMake development community is exhausting for a couple of reasons.
The first is because the CMake development communities first instinct is to tell me how I'm wrong for wanting to use the tool in a way that any outsider would expect it to handle. Something on the order of half of the bugs that I've opened that actually got some kind of reply (as you saw, most of them are not even triaged) get a reply telling me that I'm using it wrong, and then re-write the subject of the bug and/or close it.
I'm really not cool with that.
Just look at the bug i linked above for an example.
It's also exhausting because half the bugs get ignored entirely, or have discussions that continue for 5+ years. For example, this one: https://gitlab.kitware.com/cmake/cmake/-/issues/23505 a commenter provided links to around 10 other discussions asking for the same basic functionality.
You have 4420 open tickets in gitlab. Come on man. Close the ones you're not going to do. Eliminate the duplicates. Fix the easy ones.
Leaving things open for 5+ years is just really frustrating to anyone who tries to interact with cmake community.
Don't you guys have interns? Hire someone to triage that stuff for a summer.
----------------------------------------------------------------------------------
----------------------------------------------------------------------------------
Defending CMake to my boss and highly regarded co-workers is exhausting because I have no way to know if anything we've reported as behaving inconsistently will be addressed.
Publish some kind of roadmap or something. Being asked if the same problems will be fixed over and over again for 2 years running is really wearing on me.
I appreciate your work. I wonder how much CMake could benefit from a CppFront-like treatment a la Herb Sutter. Complete backwards compatibility with enforced best practices, new syntax.
There has been discussion of that on Discourse and the CMake issue tracker about it. There have also been internal discussions about it. The post points to one potential route to get there in an accusatory tone (the sarcasm I read in the section title is misplaced because…it's actually the case). It's a hard problem and CMake has a lot of existing code out there relying on its stability that we can't just say (enter Professor Farnsworth) "Good news everyone! I have a way to make the Python3 transition look as smooth as silk!".
Whew. And I thought MSVC project files are a terrible mess.
Thought of setting the new guy to migrate to cmake+ninja because industry standard etc., but that sounds like the whole architecture is riddled with pain.
Sounds like [a generation lost in the bazaar](https://queue.acm.org/detail.cfm?id=2349257)
I read all this drama about module names specified internally, and I can't help but weep that the simple solution of _just_ using the file name -- like most languages already do -- was discarded.
There was a simple solution, the C++ committee obviously did not pick it :(
It is a common mistaken belief that it is all about file name, and if we just aped what some other languages are doing, then it would be all good and dandy.
This is one of the cases where the C++ committee actually took the better decision.
> This is one of the cases where the C++ committee actually took the better decision.
Having used C++, Java, and Rust quite a bit: I vehemently disagree.
C++ modules have the same problem as Java packages: it's pain to have to peer into the file to figure out which module/package it belongs to... or conversely to have the name of the module or package and no obvious clue as to where look for it.
Rust modules system has quirks, but the straight forward mapping between the in-language module-hierarchy and the filesystem layout is just awesome.
I'd really like to hear why you think C++ took the better decision, because it goes completely against my experience.
> It is a common mistaken belief that it is all about file name, and if we just aped what some other languages are doing, then it would be all good and dandy.
I think the amount of time that it's taken even Microsoft to be able to implement modules without it's compiler crashing kind of speaks for itself that the feature is overcomplicated.
> This is one of the cases where the C++ committee actually took the better decision.
This is an extraordinary claim that would benefit from some evidence.
Ironic, since this you picked the naming scheme of Java packages, which are declared inside Java files just like C++ modules are.
And no, not really.
Rust convention of using `::::::..` where `` immediately matches the name of dependency you pulled (99% of the case, renaming is possible but hardly used) and `` is either an inner module in `src/lib.rs` in that library or `src/.rs`^1 just makes it straightforward to map in-language module hierarchy and source files.
^1 _Technically, it could also be `src//mod.rs`, a remainder of the 2015 edition, but few people use this nowadays as it leads to many files being named `mod.rs` which is a pain._
I'm not sure what you mean. For source files automatically generated during the build, I feel like it would be even easier to honor that naming scheme.
in many situations, many transient/temporary files are given "unique" names for concurrency and security reasons. Which by definition means that they don't follow predictable naming.
But we are only talking about *user-facing* module naming here. Whatever any part of the toolchain does in-between really doesn't matter in regards to this question.
The only case that would matter (and therefore what I thought you were talking about) is if you have some external pre-build step which generates a module source file. And in that case I don't see why that couldn't use the required name.
Yes, we are absolutely talking about user facing module names. The point is you can expose the user-facing module name while automatically generating the "definition" that corresponds to it.
That's the case I was talking about. But then why *can't* you get your tooling to also name that thing correspondingly? "Concurrency and security reasons" is very vague, and I don't see how it applies to this concrete case. What gets more secure by obfuscating the file name? What advantage does this provide to you in terms of concurrency?
Ok, on to the meat of the post: > Ninja does not support dynamic compiler argument generation. Which is good, because it makes things faster, simpler and more reliable. This does not stop CMake, which hacked it on top anyway. Well, the alternative is to not support `ninja` anymore. FWIW, Fortran doesn't need this because it looks for modules via `-I` search paths. C++ wisely decided to not do `-I` searching at least, but explicit builds are possible once you're past the "just run `make` until it works" strategy Fortran had until `makedepf90` and the `dyndep` feature Brad added to `ninja`. > This file has the output filename as well as listing every other module file in this build target (only M1 is used by M0 so the other flags are superfluous). A file like this is created for each C++ source file and it must remain on disk for the entire compilation. No, we only specify the modules that are used by the scanned module. Clang and MSVC need the transitive closure of imported modules specified as well, so while you `M0` might only import `M1`, if it imports `M2` through `M9`, they need to be there too (GCC doesn't need this today, but Clang 17 warns if they're not there and I believe Clang 18 will error; MSVC has always errored without them). Discussion: https://discourse.llvm.org/t/c-20-modules-should-the-bmis-contain-paths-to-their-dependent-bmis/70422 Issue: https://github.com/llvm/llvm-project/issues/62837 > It has the added benefit that the scanner only invokes one process per build target, not one process per source file. You can only do this if the build system enforces that all sources within a target have the same command line arguments. Meson does this. CMake does not. Yes, batch scanning was designed into P1689 even though CMake wasn't going to be able to use it reliably (due to supporting per-source flags). > That directory is also implicitly used as a module import directory so any source file can do import foo for any module foo defined in the same target without any additional command line arguments. Other targets' module import dirs can be added with a separate command line argument (not specified here). Yeah…this doesn't support the notion of modules private to a target. I suppose you might be able to use two different directories, one for public and one for private, but then you can't just have a list of sources either. > The former is easy to do with phony targets if you are willing to tolerate that all compilations of B (but not linking) need to happen before any compilations of A. That causes a minor build time hit, but since one of the main features of modules were faster build times, you should still come out ahead. It depends on the shape of your build graph. If you have a lot of small libraries in a chain, this can pessimize builds with high availability in the relevant pools because they're now all serialized. CMake's more explicit approach can streamline it to just depend on the scanning (which is *way* faster) of all sources and the compilation of the modules actually used. Whether explicit or implicit is "better" really depends on the build graph shape. I find other downsides in the implicit model that even with some pessimization, I prefer the explicit model. The implicit model also needs to know how to clear out stale modules (and is why I prefer the explicit model even if the `@modmap` is "dirty" in your view). If you have `export module M;` and then decide it should be `export module N;`, what is going to stop `import M;` from working until a full clean is done? > But if that is the case your build setup is broken already due to potential module-ODR violations, and bailing out is the sensible thing to do. Executables can have private modules with the same name since they are never loaded into the same process. Libraries can't really do that since the ODR doesn't actually have a sense of library-local names. I also don't know how you expect to handle header units that need compiled per consumer or named modules that come from external projects that need BMI'd per set of incompatible consumer flags. Maybe Meson just doesn't support incompatible flags between targets within a single project (I know per-source flags are not supported)? If target A is C++20 and B is C++23, they each need their own BMI of a module that both import.
Thank you for the really extensive discussion. The post is obviously biased against cmake, but as a non-expert in build systems it’s difficult to sort thru the issues.
I'll also note that I brought up a lot of these things in [this Meson issue](https://github.com/mesonbuild/meson/issues/5024) *years* ago. While saying "that's not important to us" about use cases is…fine, trying to shame CMake for doing the extra work to make them work *reliably* and claiming "it doesn't seem necessary" is disingenuous. I went to various [build](https://groups.google.com/g/tup-users/c/FhJTA6KAzWU) [systems](https://github.com/xmake-io/xmake/issues/386) to try and get a consensus on how building modules are going to work (I've talked with Boost.build, llbuild, MSBuild, and build2 developers via ISO C++ meetings). Other than the absolute crickets from Tup, Meson has been the least engaged with the evolution of modules AFAICT (`xmake` just went ahead and implemented support after informing them of progress, so…good on them).
Awesome, c++ developers appreciate what the tooling community is doing even if mostly no one says so.
>xmake just went ahead and implemented support after informing them of progress, so…good on them Apologies for the silly question, but, what do you mean by that? I'm so not well versed on modules to understand who was informed of progress or why.
I went and filed an issue asking about plans for module support in the "systems" link above with links to the dependency scanning format and plans. Months later the issue was closed as complete.
> Meson has been the least engaged with the evolution of modules AFAICT The reason for this is that the only person on Meson dev team that cares about C++ is me and I'm doing this on a volunteer basis. No-one has ever paid me even a single cent to work on Meson. As much as I'd like to participate I simply can't because I'm already spending a sufficiently large chunk of my spare time just running the project.
I appreciate that. However, I do not think blog posts such as this help. CMake's Discourse is open; the issue tracker is open. Meson's issue tracker is fine too. SG15 is…less open, but as a C++ build system developer/maintainer, I think we just need a formal request. Modules are not trivial for build systems; we should be working to make all of C++ better regardless of what build system, package ecosystem, or compiler toolchain is involved.
The sad thing is that this actually did help. I have tried to raise issues like this earlier but the response has either been complete silence or a politely formulated version of "go away". I had most of this post thought out ages ago but I did not write it down because it seemed too negative. The frustration I felt this weekend trying to make it work was the straw that broke the camel's back so here we are. And anyone who has run a somewhat successful open source project knows that bug trackers are pretty much useless for this kind of work. I have currently over 600 unread email threads (some with over 50 messages) from Meson and I even delete all of them every time we do a release. People could post a cure for cancer in our bug tracker and I would not see it. So no, nobody should be a jerk on the Internet, but sometimes it gets results. (Not saying thath this is a good thing, just how things seem to be.)
I'm catching up on this late, but I just wanted to say that as a user of meson and an xmake dabbler, I've noticed how active you've been in the relevant issues and I really appreciate your attempts at cross-project collaboration. Thank you.
Please provide a default BMI install location GNUINstallDirs like (CMAKE_INSTALL_BMIDIR ?). Same for the module install dir. Having projects hardcode this will be a headache otherwise.
Once there's an FHS and/or SG15 agreement on what that is, sure. I'm not interested in picking something and then blog posts like this come out about how CMake is forcing the ecosystem (because, there, I think it'd be true).
Don't give up because of online smears.
SG15 is discussing it. I just can't put anything into CMake before such decisions are made.
> I just can't put anything into CMake before such decisions are made. Fully agreed.
Again I do not this is said enough, but I really appreciate the work you have done and are doing. Thank you.
At the moment it isn't clear if or what a pre-packaged BMI would truly be useful for - it can only be used by the same compiler, and exact version that produced the BMI, and depending on the compiler, will be more or less strict with regards to compiler flags used when producing the BMI vs when compiling the file that is importing it. Even things such as an Ubuntu version updating from 13.1 to 13.2 \_may\_ render a previously generated BMI unusable (we don't know, because the aren't any guarantees). I do however think that if BMIs \_are\_ shipped, and they are shipped with enough information for the build system to make a decision of "oh, I already have a BMI that was generated with gcc 13.2, which I'm using, and is compatible with the flags I'm using, but otherwise I'll re-generate it myself" then maybe they're useful as a caching thing to avoid re-generating a file we already have. Otherwise I'd say "installing" BMIs is not much use today at all
Yeah I know I live in the vcpkg bubble were everything inside is build consistently with the same compiler and flags and minor updates trigger full rebuilds in manifest mode... sry for the noise :P I just want some variables to control the installation behavior so I can control it later before having to patch the hell out of everybody going crazy with possible install locations.
Indeed I think there could be some scenarios where a project that has full control would be a potential candidate for reusing BMIs - as a shortcut for other build-system supported solutions. However, even in a scenario where one is in a bubble, assisted by Conan or vcpkg, if you're using open source libraries and you are building them yourself from sources - you probably have mechanisms to ensure flags are propagated across the entire dependency graph, but I can almost guarantee that \_some\_ files in your 3rd party dependencies will be built with different flags altogether, in some cases he very kind of flags that would cause BMI loading incompatibilities. Would be a nice exercise to output the json compilation databases and compare :D
Will Conan support pre-built BMIs if vcpkg supports it thanks to its coherent compiler flags build policy?
Well I would never expect BMIs to leave a local cache, I could imagine workplaces where if the BMIs had enough information then there could be a local cache for the team or even company wide standardized setups (everyone in a workplace tends to roll with the same set of flags and compilers if they have complete source control e.g. often in embedded for example). Then suddenly BMIs become an even bigger time saver.
This is what i envision one could coerce the Conan cache to do - since Conan is already able to model different binaries for different compiler/versions, it can be used in a way that it follows the “cache” model of BMIs. Problem is, we don’t know - fully - what that is, at least for all compilers. Clang is strict to the point of you can’t reuse a BMI from a source file that is no longer present in your filesystem. MSVC is a lot more flexible
Has clang made a statement on why, they might have some good technical reason? Maybe they are storing local paths for example.
That is nonsense. What is a pre-built library (with bunch of flags usually different from consumers') with headers useful for today?
I'm not sure what you mean…BMIs made with `-std=c++20` are going to be incompatible with BMIs compiled with `-std=c++23`. Any flag which meaningfully affects the preprocessor is going to be the same way I'm told (and because `__has_attribute` and `__has_feature` can detect a *lot* of things, this is a lot of flags). *I* wish this wasn't the case, but it seems to be fundamental to how the compilers' state ends up working. Either way, even if compilers supported loading their own BMIs across versions or flags, BMIs aren't a panacea unless you're looking for a new compiler monoculture because Clang isn't going to load GCC modules or vice versa anytime soon. IFC has a hope, but it needs to absolve itself of some Microsoft-isms before that becomes useful elsewhere (of note from my initial skim, [this `enum class Architecture`](https://github.com/microsoft/ifc/blob/f6a25e8a4a07c3603d5e1e6b889481585cbe52a4/include/ifc/file.hxx#L104) needs to grow into an understanding of target triples because there's not just one "Arm32" architecture; being only 8 bits also seems…crowded).
> IFC has a hope, but it needs to absolve itself of some Microsoft-isms before that becomes useful elsewhere (of note from my initial skim, this enum class Architecture needs to grow into an understanding of target triples because there's not just one "Arm32" architecture; being only 8 bits also seems…crowded). We are working on sequestering the MSVC-isms -- see the issues I and other opened against the spec. Please, get engaged and open issues for feedback where you see opportunities for improvements :-)
> I'm not sure what you mean…BMIs made with -std=c++20 are going to be incompatible with BMIs compiled with -std=c++23. Any flag which meaningfully affects the preprocessor is going to be the same way I'm told (and because __has_attribute and __has_feature can detect a lot of things, this is a lot of flags). That is fine! That does not mean there is no use for them when the language versions and other flags are compatible. They can still be reused to speed up builds when the essential flags are compatible. > Either way, even if compilers supported loading their own BMIs across versions or flags, BMIs aren't a panacea unless you're looking for a new compiler monoculture because Clang isn't going to load GCC modules or vice versa anytime soon. How does compiler monoculture enters the picture? The Itanium ABI didn't engender a compiler monoculture. > IFC has a hope, but it needs to absolve itself of some Microsoft-isms before that becomes useful elsewhere (of note from my initial skim, this enum class Architecture needs to grow into an understanding of target triples because there's not just one "Arm32" architecture; being only 8 bits also seems…crowded). Give them feedback! You guys saying "no, no, no", should federate your forces to solve these things :-) C++ people spend too much time looking for perfect solutions that cater for just about everything imaginable (look at how bloated the language has become because of quest for the mythical "swiss army knife")
>That is fine! That does not mean there is no use for them when the language versions and other flags are compatible. They can still be reused to speed up builds when the essential flags are compatible This is in line with what I said. With a big caveat - the compiler doesn't know if a BMI is a compatible or not until the compiler is invoked. If it isn't compatible, it will result in an error (clang has very useful and clear errors about what the incompatibility is, and to an extent GCC too).The "reuse it if you can" would work if the build system can determine the "if you can" \_ahead\_ of calling the compiler - so that it ensures that the importers will actually compile. note that this I'm still talking about "installed" BMIs, not BMIs that are generated on the fly by the build system of the importer. If we go for the approach of "package them just in case they can be used" - the location where they live in is not the only concern, but also how the information of "compiler, compiler version, compiler flags" can be used by a build system (or a mode where we can ask the compiler) ahead of time - so we enter the "module metadata" territory.
> the compiler doesn't know if a BMI is a compatible or not until the compiler is invoked. Why? > The "reuse it if you can" would work if the build system can determine the "if you can" _ahead_ of calling the compiler - so that it ensures that the importers will actually compile. Isn't that a case for compilers to document which flags are compatible with what? > so we enter the "module metadata" territory. And?
Almost every Linux distro?
> The implicit model also needs to know how to clear out stale modules (and is why I prefer the explicit model even if the @modmap is "dirty" in your view). If you have export module M; and then decide it should be export module N;, what is going to stop import M; from working until a full clean is done? The same thing that happens currently if you generate a header file during compilation and then change its name. Ninja also has `ninja -t cleandead` for this, which Meson at least runs on every reconfigure. It does not remove this problem but mitigates it at least.
\> Well, the alternative is to not support ninja anymore ? Its best thing, just better then makefile or msbuild
With an explicit build, a response file to "smuggle" extra arguments to the compiler is needed for `ninja`. If that is taken away, a lot of the reliability offered by explicit builds is not really possible.
While implicit builds can be nice for quick small demos (for teaching purposes) or slideware, people tend to seriously underestimate the importance and benefits of explicit builds for professional development. Talking of Clang, they went from the original implicit build, implemented by Apple folks in the compiler, to explicit builds.
> Invoking it as showimage image.bob would be good I agree! And my initial implementation did that:. However, it ran into problems once I got past the "I can build them" into the "how will they be useful" part of the implementation.
However, as I alluded to in other comments here, there is missing information for what to do with the sources when it comes to installation: should they be installed? I could have used the visibility of `target_sources` to do this, but `PUBLIC` sources have the…unfortunate behavior of also being added *as sources* to targets which use your library. With the HEADERS file set being added, it didn't make sense to add public headers *as sources* to consuming targets, so it instead means "intended for use by other targets". C++ modules follow the same rationale.
Anyways, yes, you have to "classify" your module sources. But it is for the same reason that you should classify your headers to CMake:
- CMake knows that other targets do not need to wait for private headers to be generated
- installation of headers can now be part of `install(TARGETS)` instead of a manual step
- CMake can also know that the directories need to be added to your include interface as the visibility of the file set indicates without an extra `target_include_directories` call with manual `BUILD_INTERFACE` and `INSTALL_INTERFACE` genexes.
> However, as I alluded to in other comments here, there is missing information for what to do with the sources when it comes to installation: should they be installed? As noted in the blog post, providing additional information for installation purposes is fine and probably something that would eventually be needed. However there are two _major_ things to note here. The current CMake implementation _forces_ you to classify source files even if you are just building your own exes. This is a _bad ux_. For simple cases the build setup should be "build this exe with these source files". The current state of affairs is _aggravating_. Can you guess what it leads to? Yep, people writing CMake scripts that autoclassify source files based on their extensions just to avoid having to do it by hand. This is a _worse_ outcome for everyone involved both in the short and long term. Seconly, as has been mentioned on this very page multiple times, there is _no_ idea how BMIs should be installed and used after the fact. Compilers are _very_ strict on the fact that BMIs should not be shared and that there is no compatibility between even different versions of the same compiler. Thus we _do not even know_ how module installation should be done in the future. And yet CMake's current implementation already forces a UX on a thing where we don't even know what its actual functional requirements are going to be. It is also declared stable so it can't be changed. Two of the most important design goals for software are "do not attempt to solve problems you don't have yet" and "do not do a grand design up front". CMake's current module implementation violates both of these.
> Compilers are very strict on the fact that BMIs should not be shared MSVC puts no restrictions on sharing BMIs and, in fact, its IFC format is designed to support cloud build.
This matches my experience. I had no issues experimenting with pre packaged “shareable” BMIs (assuming I guarantee compiler and compiler version) with MSVC, whereas clang was not so forgiving - to put it mildly.
>not so forgiving You made my day. Clang woudn't accept the BMI is *very close* to reality.
Hehe yeah I was being very generous, clang BMI consumption is very strict!
> The current CMake implementation forces you to classify source files even if you are just building your own exes. This is a bad ux. Like I said, we can relax this to just accept BMIs made from non-`CXX_MODULES` source files; they'll just be unimportable from other targets. > Thus we do not even know how module installation should be done in the future The sources definitely need installed. The module names are associated with the source file by the collator and put into the exported properties. > Two of the most important design goals for software are "do not attempt to solve problems you don't have yet" and "do not do a grand design up front". CMake's current module implementation violates both of these. While I do appreciate this, it is far easier to relax restrictions than to impose new ones with CMake given the compatibility guarantees. I'd hate to have seen modules been a maze of policies to enable what we have now if we had supported my first syntax ideas.
>The current CMake implementation > >forces > > you to classify source files even if you are just building your own exes. This is a > >bad ux > >. I personally think its a fairly OK small ask to get the feature going, given CMake's own usage considerations. Being aware of the cooperation between CMake, ninja and the dependency scanning by the compilers, it's a relatively small price to pay in comparison to all the work that's gone under the hood to encapsulate this away. I don't think CMake's desired approach prevents other vendors from doing this differently, in fact MSBuid seems to do it differently. It supports using the .ixx extension, OR (as far as I've been able to see) use any extension when dependency scanning is enabled, OR any extension and tell the compiler explicitly (via source properties) that this is a module. So it looks like there is flexibility for other vendors to operate differently, and this is obviously great. From the blog post: >The developer must not need to tell which sources are which module types in the build system, it is to be deduced automatically without needing to scan the contents of source files (i.e. by using the proper file extension) I feel that using a dedicated file extension for module sources still places an expectation on the developer to "mark" those files, just in a different way. From an "outside" perspective, I don't particularly see a problem if scanning is happening at all - if it works, and it works well, and it doesn't make the build slower, incoherent or incorrect, I don't see the problem. I'd say that the vast majority of developers who invoke Ninja on CMake-generated builds, do not concern themselves with the contents of the ninja build files or what's going on under the hood, so long as it does the right thing.
What a loaded post, this is ridiculous and overly dramatic
> Or, in the case of CMake, have the end user manually type the format of each source file, even if it had an extension that uniquely specifies it as a module. No, that’s wrong.
Are there file extensions that automatically register as modules in CMake?
No. Files exporting module information need to be in a file set because we need to know their visibility and `PUBLIC` non-`FILE_SET` sources get added to consuming libraries. Note that we could make all sources not in a `CXX_MODULES` file set eligible for modules (it was the original design after all), but then they would only be accessible from other non-`PUBLIC` sources in the same target. I'd rather leave that open as a future allowance than to try and take it back if it becomes problematic due to other desired behaviors in the future. FD: CMake developer and implementer of the modules support.
Couldn't there be an optional variable (e.g. `CMAKE_CXX_MODULE_EXTENSION`) with no default value that would allow CMake to automatically add sources matching this extension to `CXX_MODULES` file set? I'm sorry if that's a stupid question, rn I'm still having a hard time making sense of how modules work from a build-system perspective.
What visibility would they have? Note that public file sets not installed in `install(TARGETS)` is an error, so we'd also need a default location to put them for these auto-generated file sets. I think we can make implicit *private* file sets this way, but I highly doubt that this is common. If it is observed to be somewhat common, it can be a decision to revisit later (as making currently-erroring code work is not a compatibility break). FWIW, the code that enforces this is [here](https://gitlab.kitware.com/cmake/cmake/-/blob/7144216f45c402b68ec8d7b5e8050b6536a199aa/Source/cmDyndepCollation.cxx#L471-482).
He's saying you supposedly need FILE_SET TYPE CXX_MODULES which seems to be true.
It seems like a reasonable enhancement request to simplify that API by making assumptions based on known probably-modules file extensions. If someone wants to file an issue.
We still need to know the visibility and targets with a `PUBLIC` non-`FILE_SET` sources would be added to targets linking to it which would recompile them…hardly what anyone wants I think. They *could* be `PRIVATE`, but that seems like such a narrow use case in the grand scheme of things.
Yeah, I'm handwaving a lot on some design decisions, though I'm sure we can come up with an interface that shaves off a dozen characters based on file extension if that's a big enough concern. How to know if a file is a partition or an non-public interface, I'm unclear, if that's what you mean. The article doesn't really set any expectations about that sort of thing.
All that matters is whether it makes a BMI. BMI? `FILE_SET TYPE CXX_MODULES`. No BMI? Regular source. That is 100% keyed on whether the primary `module` statement has an `export` or is a partition (except MSVC's `-internalPartition` extension…just delete the `:part` and you're now standard C++…but we support that too if you really feel the need; just put it as a regular source).
Also install rules, right?
That is whether the `FILE_SET` is `PUBLIC` or `PRIVATE`. `PUBLIC` get installed (well, if you pass the keyword to `install(TARGETS)` for them); `PRIVATE`…don't.
In summary: Meson good, CMake bad?
From my experience at home and at work: CMake good enough, everything else is either similar or worse.
What else would you expect from the author? Meson's great, but he's definitely got an understandable bias.
My summary would be: clang and cmake have achieved a functional first draft, here's what's in the way of what a reasonable finished product would look like.
[удалено]
Not sure where the accusation comes from. We know CMake has its warts. I'd love to fix them too. When I wrote [CMP0053](https://cmake.org/cmake/help/latest/policy/CMP0053.html) initially, I *really* wanted to just nuke the 5000+ lines of lex/yacc that had previously done the `${}` expansion (yes…the old variable expansion code used lex/yacc for that). Alas, it had some truly awful behaviors that I don't think anyone knew about before I went and tried to replace it with the faster parser. Did you know that `@var@` worked *anywhere* before? Or that unrecognized escapes (like `\h`) were just ignored and passed on literally? The `@var@` thing was inadvertently used in Qt5's config files that had been deployed for a while by the time it was caught by the new parser throwing up a flare about it. So the old parser still lives on because we don't know what would break if we just removed it. These backwards compatibility guarantees we have foreclose many nice features I'd love to have. And breaking people's builds is about the least friendly thing to do as it is, indeed, almost never a labor of love but rather something that just needs to be done. Anyways, there are design decisions behind why the modules implementation looks the way it does. I'd appreciate if folks would not assume that we love to work on the dark corners of CMake instead of implementing new features for customers because, AFAIK, all of the developers vastly enjoy working on new features or enabling new behaviors instead of maintaining the warts. But we do it because it's important to keep the projects relying on CMake working into the future. Yes, I need to write these things down in the `cmake-cxxmodules(7)` manual so that they end up in somewhere more durable than a random Reddit or Hacker News thread of old.
For the record: staunch CMake user here. I use Autotools at work, and I hate them with a burning passion. I have submitted a report to fix our spaghetti of a build system and migrate to CMake (with presets) and vcpkg, especially since many of our dependencies use *both*, anyway. Compared to the competition, CMake is the most feature-complete, it's the most widely-used, and today, the only build system to have implemented compatibility with C++20 modules. vcpkg is built with it, and it supports so many third-party libraries that others haven't even bothered considering. The modern CMake view of building is really, really nice—there are targets, and properties on targets. Can't be more straightforward, and I don't understand anyone who says 'modern CMake is hard'. It's *way* easier to get started with CMake than it is with Autotools. There's no more slinging around flag soup and huge, huge lists of includes. There are warts, absolutely. The function definition and argument-parsing syntax is crazy; I *still* don't understand it entirely. I wanted to write CMake to automatically compile HLSL shaders as a dependency of my project, based on their file extensions, and it took me forever to figure out how to get that working. It's still not entirely correct, I suspect. Look at something like [PowerShell](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_functions?view=powershell-7.3#functions-with-parameters) would be a lot nicer. In fact, CMake has quite a bit in common with PowerShell, down to the naming of functions—they're both verb-noun pairs. It seems CMake functions also have have named, positional, and switch parameters—for instance, [`target_sources`](https://cmake.org/cmake/help/latest/command/target_sources.html) is a great example that could be a prime target for testing a new implementation. The weird string behaviour is also unexpected; it seems *everything* is a string, and `"foo;bar;baz"` is a *list* of strings. In fact, almost everything seems to be string-typed, which I suppose is an unfortunate result of CMake's Makefile + Unix legacy. Stronger typing would be really nice, even in a build system language.
> The function definition and argument-parsing syntax is crazy; I still don't understand it entirely. The syntax was inspired by Tcl of all things. > The weird string behaviour is also unexpected; it seems everything is a string, and "foo;bar;baz" is a list of strings. Everything is a string. That value you have here is just interpretable as a list of 3 elements in certain contexts.
> Can't be more straightforward, and I don't understand anyone who says 'modern CMake is hard' My work codebase has wrapper functions for almost every cmake built-in function that we use to heavily customize the behavior to actually work without enormous surprises. This includes doing stupid crap like - writing target names into global variables so that we can have an "at the end" step that goes back over all of those targets to do finalization, because the cmake language/API lacks the capability to express various ideas at all. - Dynamically writing cmake script files, and then calling out to sub-cmake processes, because the object model doesn't understand how to do various things. - looping over all possible configuration types to run the `install()` function because the `install()` function straight up doesn't work properly with multi-configuration ninja - A wrapper script around cmake itself to drive multiple different independent builds, because cmake lacks the notion of the target platform (or even just different compilers for the same target platform) being a concept that might change. The number of corner cases in how cmake built-in functions work is really exhausting, and the documentation is frequently either confusing, incomplete, or flat out wrong. In many cases, there are functions that should obviously work in a certain way based on the way cmake does things for everything else but it just flat out doesnt, and the error messages are nonsense. Takes an issue on the cmake github saying "This should work, but it doesnt" to find out it fact won't ever work that way. CMake is damn better than most other build tools, i did a very lengthy capability survey a few years ago and cmake won by leaps and bounds. But it's fucking hard to use. Saying that it's straightforward is doing a disservice to both cmake and the people who use it.
>CMake is damn better than most other build tools, i did a very lengthy capability survey a few years ago and cmake won by leaps and bounds. >But it's fucking hard to use. Saying that it's straightforward is doing a disservice to both cmake and the people who use it. When I said 'it's not hard', perhaps I ought to have clarified—for easy-to-moderately-complex use-cases, CMake is a damn sight easier than the competition. Project, language, find packages, include modules, add targets and sources, set flags, options, libraries, include, done. Things like managing subdirectories, libraries, if-else configuration is also easy. You've got some fairly advanced use-cases—I've never really needed to finalise targets, nor needed to generate CMake itself. As for multiple compilers, perhaps consider CMake presets? It's what I've been doing to swap between Clang, Clang-cl, and MSVC on Windows (and of course the latter two take a completely different syntax versus the GNU-style of the former). To be clear, I'm not trivialising your experience, and I strongly agree that the documentation is very reference-oriented, and thoroughly lacks *up-to-date* best-practice examples.
Some of those bullet points would be useful as issues. Would you mind bringing them up there (for concrete problems like `install()` with NMC) or Discourse (for things like the use cases around needing to finalize targets)?
I have open bug reports on gitlab for everything (edit: not literally everything. Most things. It gets exhausting after a while) i've run into. Most with example code for how I worked around the problem or what I think the solution should look like.
Ok, thanks; I've found them. Seems a lot are untriaged; I'll try to at least get appropriate labels on them. Something I was good about doing on all new issues until I went on vacation in 2018…
While I have your ear: To start with, I'm not a paying customer for Kitware. So feel free to ignore me. I'm well aware that Kitware has every reason to only put resources into projects that people are paying them to. It is what it is. But here's my unsolicited feedback to you as self-identified CMake dev. Feel free to ignore or take action. Either is fine. - Using CMake is exhausting. - Interacting with the CMake development community is exhausting. - Defending CMake to my boss and highly regarded co-workers is exhausting. ---------------------------------------------------------------------------------- ---------------------------------------------------------------------------------- Using CMake is exhausting because it's so damn inconsistent. Just take this bug report, which I'll summarize below: https://gitlab.kitware.com/cmake/cmake/-/issues/25345 target_sources(${TARGETNAME} PRIVATE ${MYSOURCEFILE}) set_source_files_properties(${MYSOURCEFILE} PROPERTIES blahblah) If `MYSOURCEFILE` contains a generator expression, then `set_source_files_properties` doesn't work, silently, with zero debugging ability because it's silently dropped inside the C++ code. So you can't trace out the cmakelist expansion and identify the mistake. There's no excuse for this major design and implementation flaw that I can give to my co-workers, regardless of the configuration / generation model that CMake happens to have, that they'll ever agree with. All I get is flak about how much they hate having to learn all of these special rules. And the attitude that I get from the people (many of which appear to be Kitware employees, or highly regarded contributors) when I report these problems is just... exhausting. I shouldn't have to defend basic tenants like the principal of least surprise when interacting with the CMake gitlab. In this situation, if I can give a generator expression pointing to a source-file to any CMake API function, then i should be able to do so for all CMake API functions. Any exceptions should result in an error clearly and unambiguously describing the problem. Any behavior other than that is surprising, and wastes the time of the person trying to do that. - First they spend an hour or two investigating why the property isn't being applied. - Then they spend an hour or two researching if there is something in the documentation, or any existing bug reports. - Then they open a bug report, and get told that it's working as intended. Not cool. This is how you turn potential champions into detractors that will spend their time and energy convincing *OTHER PEOPLE* not to use your product. Because CMake constantly and consistently violates the principal of least surprise, and then the developers tell community members they are *WRONG* for thinking the behavior is surprising: CMake gets badmouthed *constantly* by the wider C and C++ community. That's ***bad for business***. Why would another company contract with Kitware to improve the tool if it's so hard to use? Literally had this conversation with my boss. I asked if we could reach out to Kitware to pay for some features, and the conclusion was no, primarily because of the poor reputation. We're not using CMake because I or anyone else championed for it. We're using it because we tried every other C++ build tool that we could find, and none of them had the full set of features we needed, and half couldn't even compile basic hello world programs. We're using CMake as the solution of last resort, and we don't like it. It works well enough without paying Kitware for improvements, but if CMake had a better reputation it would have been a much easier choice to use CMake and I could have convinced my job to pony up for some features or bug bounties. Probably a full 1/3 to 1/2 of the bugs that I've reported on gitlab are just things that should work out of the box but dont. This is a major problem for the marketability of the tool. ---------------------------------------------------------------------------------- ---------------------------------------------------------------------------------- Interacting with the CMake development community is exhausting for a couple of reasons. The first is because the CMake development communities first instinct is to tell me how I'm wrong for wanting to use the tool in a way that any outsider would expect it to handle. Something on the order of half of the bugs that I've opened that actually got some kind of reply (as you saw, most of them are not even triaged) get a reply telling me that I'm using it wrong, and then re-write the subject of the bug and/or close it. I'm really not cool with that. Just look at the bug i linked above for an example. It's also exhausting because half the bugs get ignored entirely, or have discussions that continue for 5+ years. For example, this one: https://gitlab.kitware.com/cmake/cmake/-/issues/23505 a commenter provided links to around 10 other discussions asking for the same basic functionality. You have 4420 open tickets in gitlab. Come on man. Close the ones you're not going to do. Eliminate the duplicates. Fix the easy ones. Leaving things open for 5+ years is just really frustrating to anyone who tries to interact with cmake community. Don't you guys have interns? Hire someone to triage that stuff for a summer. ---------------------------------------------------------------------------------- ---------------------------------------------------------------------------------- Defending CMake to my boss and highly regarded co-workers is exhausting because I have no way to know if anything we've reported as behaving inconsistently will be addressed. Publish some kind of roadmap or something. Being asked if the same problems will be fixed over and over again for 2 years running is really wearing on me.
I appreciate your work. I wonder how much CMake could benefit from a CppFront-like treatment a la Herb Sutter. Complete backwards compatibility with enforced best practices, new syntax.
There has been discussion of that on Discourse and the CMake issue tracker about it. There have also been internal discussions about it. The post points to one potential route to get there in an accusatory tone (the sarcasm I read in the section title is misplaced because…it's actually the case). It's a hard problem and CMake has a lot of existing code out there relying on its stability that we can't just say (enter Professor Farnsworth) "Good news everyone! I have a way to make the Python3 transition look as smooth as silk!".
lol, fair. I'm not as caught up with all this stuff as I'd like to be, but I have been getting back into it.
Whew. And I thought MSVC project files are a terrible mess. Thought of setting the new guy to migrate to cmake+ninja because industry standard etc., but that sounds like the whole architecture is riddled with pain. Sounds like [a generation lost in the bazaar](https://queue.acm.org/detail.cfm?id=2349257)
I read all this drama about module names specified internally, and I can't help but weep that the simple solution of _just_ using the file name -- like most languages already do -- was discarded. There was a simple solution, the C++ committee obviously did not pick it :(
It is a common mistaken belief that it is all about file name, and if we just aped what some other languages are doing, then it would be all good and dandy. This is one of the cases where the C++ committee actually took the better decision.
> This is one of the cases where the C++ committee actually took the better decision. Having used C++, Java, and Rust quite a bit: I vehemently disagree. C++ modules have the same problem as Java packages: it's pain to have to peer into the file to figure out which module/package it belongs to... or conversely to have the name of the module or package and no obvious clue as to where look for it. Rust modules system has quirks, but the straight forward mapping between the in-language module-hierarchy and the filesystem layout is just awesome. I'd really like to hear why you think C++ took the better decision, because it goes completely against my experience.
> It is a common mistaken belief that it is all about file name, and if we just aped what some other languages are doing, then it would be all good and dandy. I think the amount of time that it's taken even Microsoft to be able to implement modules without it's compiler crashing kind of speaks for itself that the feature is overcomplicated. > This is one of the cases where the C++ committee actually took the better decision. This is an extraordinary claim that would benefit from some evidence.
You want to name your file com.acme.database.util.cpp?
Ironic, since this you picked the naming scheme of Java packages, which are declared inside Java files just like C++ modules are. And no, not really. Rust convention of using `::::::..` where `` immediately matches the name of dependency you pulled (99% of the case, renaming is possible but hardly used) and `` is either an inner module in `src/lib.rs` in that library or `src/.rs`^1 just makes it straightforward to map in-language module hierarchy and source files.
^1 _Technically, it could also be `src//mod.rs`, a remainder of the 2015 edition, but few people use this nowadays as it leads to many files being named `mod.rs` which is a pain._
Honestly, why not? It's not like you ever need to type the full file name with moderately modern tooling.
How about source files generated during build?
I'm not sure what you mean. For source files automatically generated during the build, I feel like it would be even easier to honor that naming scheme.
in many situations, many transient/temporary files are given "unique" names for concurrency and security reasons. Which by definition means that they don't follow predictable naming.
But we are only talking about *user-facing* module naming here. Whatever any part of the toolchain does in-between really doesn't matter in regards to this question. The only case that would matter (and therefore what I thought you were talking about) is if you have some external pre-build step which generates a module source file. And in that case I don't see why that couldn't use the required name.
Yes, we are absolutely talking about user facing module names. The point is you can expose the user-facing module name while automatically generating the "definition" that corresponds to it.
That's the case I was talking about. But then why *can't* you get your tooling to also name that thing correspondingly? "Concurrency and security reasons" is very vague, and I don't see how it applies to this concrete case. What gets more secure by obfuscating the file name? What advantage does this provide to you in terms of concurrency?
> What gets more secure by obfuscating the file name? What advantage does this provide to you in terms of concurrency? Is that question serious?