+ +

Driver Design & Internals

+ +

Introduction
Features and Goals +
- GCC Compatibility
- Flexible
- Low Overhead
- Simple
+
Design +
+

+ + + +

Introduction

+ + +

This document describes the Clang driver. The purpose of this + document is to describe both the motivation and design goals + for the driver, as well as details of the internal + implementation.

+ + +

Features and Goals

+ + +

The Clang driver is intended to be a production quality + compiler driver providing access to the Clang compiler and + tools, with a command line interface which is compatible with + the gcc driver.

+ +

Although the driver is part of and driven by the Clang + project, it is logically a separate tool which shares many of + the same goals as Clang:

+ +

Features:

GCC Compatibility
Flexible
Low Overhead
Simple

+ + +

GCC Compatibility

+ + +

The number one goal of the driver is to ease the adoption of + Clang by allowing users to drop Clang into a build system + which was designed to call GCC. Although this makes the driver + much more complicated than might otherwise be necessary, we + decided that being very compatible with the gcc command line + interface was worth it in order to allow users to quickly test + clang on their projects.

+ + +

Flexible

+ + +

The driver was designed to be flexible and easily accommodate + new uses as we grow the clang and LLVM infrastructure. As one + example, the driver can easily support the introduction of + tools which have an integrated assembler; something we hope to + add to LLVM in the future.

+ +

Similarly, most of the driver functionality is kept in a + library which can be used to build other tools which want to + implement or accept a gcc like interface.

+ + +

Low Overhead

+ + +

The driver should have as little overhead as possible. In + practice, we found that the gcc driver by itself incurred a + small but meaningful overhead when compiling many small + files. The driver doesn't do much work compared to a + compilation, but we have tried to keep it as efficient as + possible by following a few simple principles:

Avoid memory allocation and string copying when + possible.
Don't parse arguments more than once.
Provide a few simple interfaces for efficiently searching + arguments.

+ + +

Simple

+ + +

Finally, the driver was designed to be "as simple as + possible", given the other goals. Notably, trying to be + completely compatible with the gcc driver adds a significant + amount of complexity. However, the design of the driver + attempts to mitigate this complexity by dividing the process + into a number of independent stages instead of a single + monolithic task.

+ + +

Internal Design and Implementation

+ + +

Internals Introduction
Design Overview
Additional Notes
Relation to GCC Driver Concepts

+ + +

Internals Introduction

+ + +

In order to satisfy the stated goals, the driver was designed + to completely subsume the functionality of the gcc executable; + that is, the driver should not need to delegate to gcc to + perform subtasks. On Darwin, this implies that the Clang + driver also subsumes the gcc driver-driver, which is used to + implement support for building universal images (binaries and + object files). This also implies that the driver should be + able to call the language specific compilers (e.g. cc1) + directly, which means that it must have enough information to + forward command line arguments to child processes + correctly.

+ + +

Design Overview

+ + +

The diagram below shows the significant components of the + driver architecture and how they relate to one another. The + orange components represent concrete data structures built by + the driver, the green components indicate conceptually + distinct stages which manipulate these data structures, and + the blue components are important helper classes.

+ +

+ +

+ + +

Driver Stages

+ + +

The driver functionality is conceptually divided into five stages:

+ +

+ Parse: Option Parsing + +
The command line argument strings are decomposed into + arguments (Arg instances). The driver expects to + understand all available options, although there is some + facility for just passing certain classes of options + through (like -Wl,).
+ +
Each argument corresponds to exactly one + abstract Option definition, which describes how + the option is parsed along with some additional + metadata. The Arg instances themselves are lightweight and + merely contain enough information for clients to determine + which option they correspond to and their values (if they + have additional parameters).
+ +
For example, a command line like "-Ifoo -I foo" would + parse to two Arg instances (a JoinedArg and a SeparateArg + instance), but each would refer to the same Option.
+ +
Options are lazily created in order to avoid populating + all Option classes when the driver is loaded. Most of the + driver code only needs to deal with options by their + unique ID (e.g., options::OPT_I),
+ +
Arg instances themselves do not generally store the + values of parameters. In many cases, this would + simply result in creating unnecessary string + copies. Instead, Arg instances are always embedded inside + an ArgList structure, which contains the original vector + of argument strings. Each Arg itself only needs to contain + an index into this vector instead of storing its values + directly.
+ +
The clang driver can dump the results of this + stage using the -ccc-print-options flag (which + must precede any actual command line arguments). For + example:
+
```
+            $ clang -ccc-print-options -Xarch_i386 -fomit-frame-pointer -Wa,-fast -Ifoo -I foo t.c
+            Option 0 - Name: "-Xarch_", Values: {"i386", "-fomit-frame-pointer"}
+            Option 1 - Name: "-Wa,", Values: {"-fast"}
+            Option 2 - Name: "-I", Values: {"foo"}
+            Option 3 - Name: "-I", Values: {"foo"}
+            Option 4 - Name: "<input>", Values: {"t.c"}
+          
```
+ +
After this stage is complete the command line should be + broken down into well defined option objects with their + appropriate parameters. Subsequent stages should rarely, + if ever, need to do any string processing.
+
+ Pipeline: Compilation Job Construction + +
Once the arguments are parsed, the tree of subprocess + jobs needed for the desired compilation sequence are + constructed. This involves determining the input files and + their types, what work is to be done on them (preprocess, + compile, assemble, link, etc.), and constructing a list of + Action instances for each task. The result is a list of + one or more top-level actions, each of which generally + corresponds to a single output (for example, an object or + linked executable).
+ +
The majority of Actions correspond to actual tasks, + however there are two special Actions. The first is + InputAction, which simply serves to adapt an input + argument for use as an input to other Actions. The second + is BindArchAction, which conceptually alters the + architecture to be used for all of its input Actions.
+ +
The clang driver can dump the results of this + stage using the -ccc-print-phases flag. For + example:
+
```
+            $ clang -ccc-print-phases -x c t.c -x assembler t.s
+            0: input, "t.c", c
+            1: preprocessor, {0}, cpp-output
+            2: compiler, {1}, assembler
+            3: assembler, {2}, object
+            4: input, "t.s", assembler
+            5: assembler, {4}, object
+            6: linker, {3, 5}, image
+          
```
+
Here the driver is constructing seven distinct actions, + four to compile the "t.c" input into an object file, two to + assemble the "t.s" input, and one to link them together.
+ +
A rather different compilation pipeline is shown here; in + this example there are two top level actions to compile + the input files into two separate object files, where each + object file is built using lipo to merge results + built for two separate architectures.
+
```
+            $ clang -ccc-print-phases -c -arch i386 -arch x86_64 t0.c t1.c
+            0: input, "t0.c", c
+            1: preprocessor, {0}, cpp-output
+            2: compiler, {1}, assembler
+            3: assembler, {2}, object
+            4: bind-arch, "i386", {3}, object
+            5: bind-arch, "x86_64", {3}, object
+            6: lipo, {4, 5}, object
+            7: input, "t1.c", c
+            8: preprocessor, {7}, cpp-output
+            9: compiler, {8}, assembler
+            10: assembler, {9}, object
+            11: bind-arch, "i386", {10}, object
+            12: bind-arch, "x86_64", {10}, object
+            13: lipo, {11, 12}, object
+          
```
+ +
After this stage is complete the compilation process is + divided into a simple set of actions which need to be + performed to produce intermediate or final outputs (in + some cases, like -fsyntax-only, there is no + "real" final output). Phases are well known compilation + steps, such as "preprocess", "compile", "assemble", + "link", etc.
+
+ Bind: Tool & Filename Selection + +
This stage (in conjunction with the Translate stage) + turns the tree of Actions into a list of actual subprocess + to run. Conceptually, the driver performs a top down + matching to assign Action(s) to Tools. The ToolChain is + responsible for selecting the tool to perform a particular + action; once selected the driver interacts with the tool + to see if it can match additional actions (for example, by + having an integrated preprocessor). + +
Once Tools have been selected for all actions, the driver + determines how the tools should be connected (for example, + using an inprocess module, pipes, temporary files, or user + provided filenames). If an output file is required, the + driver also computes the appropriate file name (the suffix + and file location depend on the input types and options + such as -save-temps). + +
The driver interacts with a ToolChain to perform the Tool + bindings. Each ToolChain contains information about all + the tools needed for compilation for a particular + architecture, platform, and operating system. A single + driver invocation may query multiple ToolChains during one + compilation in order to interact with tools for separate + architectures.
+ +
The results of this stage are not computed directly, but + the driver can print the results via + the -ccc-print-bindings option. For example:
+
```
+            $ clang -ccc-print-bindings -arch i386 -arch ppc t0.c
+            # "i386-apple-darwin9" - "clang", inputs: ["t0.c"], output: "/tmp/cc-Sn4RKF.s"
+            # "i386-apple-darwin9" - "darwin::Assemble", inputs: ["/tmp/cc-Sn4RKF.s"], output: "/tmp/cc-gvSnbS.o"
+            # "i386-apple-darwin9" - "darwin::Link", inputs: ["/tmp/cc-gvSnbS.o"], output: "/tmp/cc-jgHQxi.out"
+            # "ppc-apple-darwin9" - "gcc::Compile", inputs: ["t0.c"], output: "/tmp/cc-Q0bTox.s"
+            # "ppc-apple-darwin9" - "gcc::Assemble", inputs: ["/tmp/cc-Q0bTox.s"], output: "/tmp/cc-WCdicw.o"
+            # "ppc-apple-darwin9" - "gcc::Link", inputs: ["/tmp/cc-WCdicw.o"], output: "/tmp/cc-HHBEBh.out"
+            # "i386-apple-darwin9" - "darwin::Lipo", inputs: ["/tmp/cc-jgHQxi.out", "/tmp/cc-HHBEBh.out"], output: "a.out"
+          
```
+ +
This shows the tool chain, tool, inputs and outputs which + have been bound for this compilation sequence. Here clang + is being used to compile t0.c on the i386 architecture and + darwin specific versions of the tools are being used to + assemble and link the result, but generic gcc versions of + the tools are being used on PowerPC.
+
+ Translate: Tool Specific Argument Translation + +
Once a Tool has been selected to perform a particular + Action, the Tool must construct concrete Jobs which will be + executed during compilation. The main work is in translating + from the gcc style command line options to whatever options + the subprocess expects.
+ +
Some tools, such as the assembler, only interact with a + handful of arguments and just determine the path of the + executable to call and pass on their input and output + arguments. Others, like the compiler or the linker, may + translate a large number of arguments in addition.
+ +
The ArgList class provides a number of simple helper + methods to assist with translating arguments; for example, + to pass on only the last of arguments corresponding to some + option, or all arguments for an option.
+ +
The result of this stage is a list of Jobs (executable + paths and argument strings) to execute.
+
+ Execute +
Finally, the compilation pipeline is executed. This is + mostly straightforward, although there is some interaction + with options + like -pipe, -pass-exit-codes + and -time.
+

+ + +

Additional Notes

+ + +

The Compilation Object

+ +

The driver constructs a Compilation object for each set of + command line arguments. The Driver itself is intended to be + invariant during construction of a Compilation; an IDE should be + able to construct a single long lived driver instance to use + for an entire build, for example.

+ +

The Compilation object holds information that is particular + to each compilation sequence. For example, the list of used + temporary files (which must be removed once compilation is + finished) and result files (which should be removed if + compilation fails).

+ +

Unified Parsing & Pipelining

+ +

Parsing and pipelining both occur without reference to a + Compilation instance. This is by design; the driver expects that + both of these phases are platform neutral, with a few very well + defined exceptions such as whether the platform uses a driver + driver.

+ +

ToolChain Argument Translation

+ +

In order to match gcc very closely, the clang driver + currently allows tool chains to perform their own translation of + the argument list (into a new ArgList data structure). Although + this allows the clang driver to match gcc easily, it also makes + the driver operation much harder to understand (since the Tools + stop seeing some arguments the user provided, and see new ones + instead).

+ +

For example, on Darwin -gfull gets translated into two + separate arguments, -g + and -fno-eliminate-unused-debug-symbols. Trying to write Tool + logic to do something with -gfull will not work, because Tool + argument translation is done after the arguments have been + translated.

+ +

A long term goal is to remove this tool chain specific + translation, and instead force each tool to change its own logic + to do the right thing on the untranslated original arguments.

+ +

Unused Argument Warnings

The driver operates by parsing all arguments but giving Tools + the opportunity to choose which arguments to pass on. One + downside of this infrastructure is that if the user misspells + some option, or is confused about which options to use, some + command line arguments the user really cared about may go + unused. This problem is particularly important when using + clang as a compiler, since the clang compiler does not support + anywhere near all the options that gcc does, and we want to make + sure users know which ones are being used.

+ +

To support this, the driver maintains a bit associated with + each argument of whether it has been used (at all) during the + compilation. This bit usually doesn't need to be set by hand, + as the key ArgList accessors will set it automatically.

+ +

When a compilation is successful (there are no errors), the + driver checks the bit and emits an "unused argument" warning for + any arguments which were never accessed. This is conservative + (the argument may not have been used to do what the user wanted) + but still catches the most obvious cases.

+ + +

Relation to GCC Driver Concepts

+ + +

For those familiar with the gcc driver, this section provides + a brief overview of how things from the gcc driver map to the + clang driver.

+ +

+ Driver Driver +
The driver driver is fully integrated into the clang + driver. The driver simply constructs additional Actions to + bind the architecture during the Pipeline + phase. The tool chain specific argument translation is + responsible for handling -Xarch_.
+ +
The one caveat is that this approach + requires -Xarch_ not be used to alter the + compilation itself (for example, one cannot + provide -S as an -Xarch_ argument). The + driver attempts to reject such invocations, and overall + there isn't a good reason to abuse -Xarch_ to + that end in practice.
+ +
The upside is that the clang driver is more efficient and + does little extra work to support universal builds. It also + provides better error reporting and UI consistency.
+
+ Specs +
The clang driver has no direct correspondent for + "specs". The majority of the functionality that is + embedded in specs is in the Tool specific argument + translation routines. The parts of specs which control the + compilation pipeline are generally part of + the Pipeline stage.
+
+ Toolchains +
The gcc driver has no direct understanding of tool + chains. Each gcc binary roughly corresponds to the + information which is embedded inside a single + ToolChain.
+ +
The clang driver is intended to be portable and support + complex compilation environments. All platform and tool + chain specific code should be protected behind either + abstract or well defined interfaces (such as whether the + platform supports use as a driver driver).
+