+ +

Precompiled Headers

+ +

This document describes the design and implementation of Clang's + precompiled headers (PCH). If you are interested in the end-user + view, please see the User's Manual.

+ +

Table of Contents

Using Precompiled Headers with + clang
Design Philosophy
Precompiled Header Contents +
+
Precompiled Header Integration + Points

+ +

Using Precompiled Headers with `clang`

+ +

The Clang compiler frontend, clang -cc1, supports two command line +options for generating and using PCH files.

+ +

To generate PCH files using clang -cc1, use the option +-emit-pch: + +

 $ clang -cc1 test.h -emit-pch -o test.h.pch

+ +

This option is transparently used by clang when generating +PCH files. The resulting PCH file contains the serialized form of the +compiler's internal representation after it has completed parsing and +semantic analysis. The PCH file can then be used as a prefix header +with the -include-pch option:

+ +

+  $ clang -cc1 -include-pch test.h.pch test.c -o test.s
+

+ +

Design Philosophy

+ +

Precompiled headers are meant to improve overall compile times for + projects, so the design of precompiled headers is entirely driven by + performance concerns. The use case for precompiled headers is + relatively simple: when there is a common set of headers that is + included in nearly every source file in the project, we + precompile that bundle of headers into a single precompiled + header (PCH file). Then, when compiling the source files in the + project, we load the PCH file first (as a prefix header), which acts + as a stand-in for that bundle of headers.

+ +

A precompiled header implementation improves performance when:

Loading the PCH file is significantly faster than re-parsing the + bundle of headers stored within the PCH file. Thus, a precompiled + header design attempts to minimize the cost of reading the PCH + file. Ideally, this cost should not vary with the size of the + precompiled header file.
The cost of generating the PCH file initially is not so large + that it counters the per-source-file performance improvement due to + eliminating the need to parse the bundled headers in the first + place. This is particularly important on multi-core systems, because + PCH file generation serializes the build when all compilations + require the PCH file to be up-to-date.

+ +

Clang's precompiled headers are designed with a compact on-disk +representation, which minimizes both PCH creation time and the time +required to initially load the PCH file. The PCH file itself contains +a serialized representation of Clang's abstract syntax trees and +supporting data structures, stored using the same compressed bitstream +as LLVM's bitcode +file format.

+ +

Clang's precompiled headers are loaded "lazily" from disk. When a +PCH file is initially loaded, Clang reads only a small amount of data +from the PCH file to establish where certain important data structures +are stored. The amount of data read in this initial load is +independent of the size of the PCH file, such that a larger PCH file +does not lead to longer PCH load times. The actual header data in the +PCH file--macros, functions, variables, types, etc.--is loaded only +when it is referenced from the user's code, at which point only that +entity (and those entities it depends on) are deserialized from the +PCH file. With this approach, the cost of using a precompiled header +for a translation unit is proportional to the amount of code actually +used from the header, rather than being proportional to the size of +the header itself.

+ +

When given the -print-stats option, Clang produces +statistics describing how much of the precompiled header was actually +loaded from disk. For a simple "Hello, World!" program that includes +the Apple Cocoa.h header (which is built as a precompiled +header), this option illustrates how little of the actual precompiled +header is required:

+ +

+*** PCH Statistics:
+  933 stat cache hits
+  4 stat cache misses
+  895/39981 source location entries read (2.238563%)
+  19/15315 types read (0.124061%)
+  20/82685 declarations read (0.024188%)
+  154/58070 identifiers read (0.265197%)
+  0/7260 selectors read (0.000000%)
+  0/30842 statements read (0.000000%)
+  4/8400 macros read (0.047619%)
+  1/4995 lexical declcontexts read (0.020020%)
+  0/4413 visible declcontexts read (0.000000%)
+  0/7230 method pool entries read (0.000000%)
+  0 method pool misses
+

+ +

For this small program, only a tiny fraction of the source +locations, types, declarations, identifiers, and macros were actually +deserialized from the precompiled header. These statistics can be +useful to determine whether the precompiled header implementation can +be improved by making more of the implementation lazy.

+ +

Precompiled headers can be chained. When you create a PCH while +including an existing PCH, Clang can create the new PCH by referencing +the original file and only writing the new data to the new file. For +example, you could create a PCH out of all the headers that are very +commonly used throughout your project, and then create a PCH for every +single source file in the project that includes the code that is +specific to that file, so that recompiling the file itself is very fast, +without duplicating the data from the common headers for every file.

+ +

Precompiled Header Contents

+ +

+ +

Clang's precompiled headers are organized into several different +blocks, each of which contains the serialized representation of a part +of Clang's internal representation. Each of the blocks corresponds to +either a block or a record within LLVM's bitstream +format. The contents of each of these logical blocks are described +below.

+ +

For a given precompiled header, the llvm-bcanalyzer +utility can be used to examine the actual structure of the bitstream +for the precompiled header. This information can be used both to help +understand the structure of the precompiled header and to isolate +areas where precompiled headers can still be optimized, e.g., through +the introduction of abbreviations.

+ +

Metadata Block

+ +

The metadata block contains several records that provide +information about how the precompiled header was built. This metadata +is primarily used to validate the use of a precompiled header. For +example, a precompiled header built for a 32-bit x86 target cannot be used +when compiling for a 64-bit x86 target. The metadata block contains +information about:

+ +

Language options: Describes the particular language dialect used to compile the +PCH file, including major options (e.g., Objective-C support) and more +minor options (e.g., support for "//" comments). The contents of this +record correspond to the LangOptions class.
Target architecture: The target triple that describes the architecture, platform, and +ABI for which the PCH file was generated, e.g., +i386-apple-darwin9.
PCH version: The major and minor version numbers of the precompiled header +format. Changes in the minor version number should not affect backward +compatibility, while changes in the major version number imply that a +newer compiler cannot read an older precompiled header (and +vice-versa).
Original file name: The full path of the header that was used to generate the +precompiled header.
Predefines buffer: Although not explicitly stored as part of the metadata, the +predefines buffer is used in the validation of the precompiled header. +The predefines buffer itself contains code generated by the compiler +to initialize the preprocessor state according to the current target, +platform, and command-line options. For example, the predefines buffer +will contain "#define __STDC__ 1" when we are compiling C +without Microsoft extensions. The predefines buffer itself is stored +within the source manager block, but its +contents are verified along with the rest of the metadata.

+ +

A chained PCH file (that is, one that references another PCH) has +a slightly different metadata block, which contains the following +information:

+ +

Referenced file: The name of the referenced PCH file. It is looked up like a file +specified using -include-pch.
PCH version: This is the same as in normal PCH files.
Original file name: The full path of the header that was used to generate this +precompiled header.

+ +

The language options, target architecture and predefines buffer data +is taken from the end of the chain, since they have to match anyway.

+ +

Source Manager Block

+ +

The source manager block contains the serialized representation of +Clang's SourceManager class, +which handles the mapping from source locations (as represented in +Clang's abstract syntax tree) into actual column/line positions within +a source file or macro instantiation. The precompiled header's +representation of the source manager also includes information about +all of the headers that were (transitively) included when building the +precompiled header.

+ +

The bulk of the source manager block is dedicated to information +about the various files, buffers, and macro instantiations into which +a source location can refer. Each of these is referenced by a numeric +"file ID", which is a unique number (allocated starting at 1) stored +in the source location. Clang serializes the information for each kind +of file ID, along with an index that maps file IDs to the position +within the PCH file where the information about that file ID is +stored. The data associated with a file ID is loaded only when +required by the front end, e.g., to emit a diagnostic that includes a +macro instantiation history inside the header itself.

+ +

The source manager block also contains information about all of the +headers that were included when building the precompiled header. This +includes information about the controlling macro for the header (e.g., +when the preprocessor identified that the contents of the header +dependent on a macro like LLVM_CLANG_SOURCEMANAGER_H) +along with a cached version of the results of the stat() +system calls performed when building the precompiled header. The +latter is particularly useful in reducing system time when searching +for include files.

+ +

Preprocessor Block

+ +

The preprocessor block contains the serialized representation of +the preprocessor. Specifically, it contains all of the macros that +have been defined by the end of the header used to build the +precompiled header, along with the token sequences that comprise each +macro. The macro definitions are only read from the PCH file when the +name of the macro first occurs in the program. This lazy loading of +macro definitions is triggered by lookups into the identifier table.

+ +

Types Block

+ +

The types block contains the serialized representation of all of +the types referenced in the translation unit. Each Clang type node +(PointerType, FunctionProtoType, etc.) has a +corresponding record type in the PCH file. When types are deserialized +from the precompiled header, the data within the record is used to +reconstruct the appropriate type node using the AST context.

+ +

Each type has a unique type ID, which is an integer that uniquely +identifies that type. Type ID 0 represents the NULL type, type IDs +less than NUM_PREDEF_TYPE_IDS represent predefined types +(void, float, etc.), while other +"user-defined" type IDs are assigned consecutively from +NUM_PREDEF_TYPE_IDS upward as the types are encountered. +The PCH file has an associated mapping from the user-defined types +block to the location within the types block where the serialized +representation of that type resides, enabling lazy deserialization of +types. When a type is referenced from within the PCH file, that +reference is encoded using the type ID shifted left by 3 bits. The +lower three bits are used to represent the const, +volatile, and restrict qualifiers, as in +Clang's QualType +class.

+ +

Declarations Block

+ +

The declarations block contains the serialized representation of +all of the declarations referenced in the translation unit. Each Clang +declaration node (VarDecl, FunctionDecl, +etc.) has a corresponding record type in the PCH file. When +declarations are deserialized from the precompiled header, the data +within the record is used to build and populate a new instance of the +corresponding Decl node. As with types, each declaration +node has a numeric ID that is used to refer to that declaration within +the PCH file. In addition, a lookup table provides a mapping from that +numeric ID to the offset within the precompiled header where that +declaration is described.

+ +

Declarations in Clang's abstract syntax trees are stored +hierarchically. At the top of the hierarchy is the translation unit +(TranslationUnitDecl), which contains all of the +declarations in the translation unit. These declarations (such as +functions or struct types) may also contain other declarations inside +them, and so on. Within Clang, each declaration is stored within a declaration +context, as represented by the DeclContext class. +Declaration contexts provide the mechanism to perform name lookup +within a given declaration (e.g., find the member named x +in a structure) and iterate over the declarations stored within a +context (e.g., iterate over all of the fields of a structure for +structure layout).

+ +

In Clang's precompiled header format, deserializing a declaration +that is a DeclContext is a separate operation from +deserializing all of the declarations stored within that declaration +context. Therefore, Clang will deserialize the translation unit +declaration without deserializing the declarations within that +translation unit. When required, the declarations stored within a +declaration context will be deserialized. There are two representations +of the declarations within a declaration context, which correspond to +the name-lookup and iteration behavior described above:

+ +

When the front end performs name lookup to find a name + x within a given declaration context (for example, + during semantic analysis of the expression p->x, + where p's type is defined in the precompiled header), + Clang deserializes a hash table mapping from the names within that + declaration context to the declaration IDs that represent each + visible declaration with that name. The entire hash table is + deserialized at this point (into the llvm::DenseMap + stored within each DeclContext object), but the actual + declarations are not yet deserialized. In a second step, those + declarations with the name x will be deserialized and + will be used as the result of name lookup.
When the front end performs iteration over all of the + declarations within a declaration context, all of those declarations + are immediately de-serialized. For large declaration contexts (e.g., + the translation unit), this operation is expensive; however, large + declaration contexts are not traversed in normal compilation, since + such a traversal is unnecessary. However, it is common for the code + generator and semantic analysis to traverse declaration contexts for + structs, classes, unions, and enumerations, although those contexts + contain relatively few declarations in the common case.

+ +

Statements and Expressions

+ +

Statements and expressions are stored in the precompiled header in +both the types and the declarations blocks, because every statement or +expression will be associated with either a type or declaration. The +actual statement and expression records are stored immediately +following the declaration or type that owns the statement or +expression. For example, the statement representing the body of a +function will be stored directly following the declaration of the +function.

+ +

As with types and declarations, each statement and expression kind +in Clang's abstract syntax tree (ForStmt, +CallExpr, etc.) has a corresponding record type in the +precompiled header, which contains the serialized representation of +that statement or expression. Each substatement or subexpression +within an expression is stored as a separate record (which keeps most +records to a fixed size). Within the precompiled header, the +subexpressions of an expression are stored, in reverse order, prior to the expression +that owns those expression, using a form of Reverse +Polish Notation. For example, an expression 3 - 4 + 5 +would be represented as follows:

+ + + + + + + + +

IntegerLiteral(5)

IntegerLiteral(4)

IntegerLiteral(3)

BinaryOperator(-)

BinaryOperator(+)

STOP

+ +

When reading this representation, Clang evaluates each expression +record it encounters, builds the appropriate abstract syntax tree node, +and then pushes that expression on to a stack. When a record contains N +subexpressions--BinaryOperator has two of them--those +expressions are popped from the top of the stack. The special STOP +code indicates that we have reached the end of a serialized expression +or statement; other expression or statement records may follow, but +they are part of a different expression.

+ +

Identifier Table Block

+ +

The identifier table block contains an on-disk hash table that maps +each identifier mentioned within the precompiled header to the +serialized representation of the identifier's information (e.g, the +IdentifierInfo structure). The serialized representation +contains:

+ +

The actual identifier string.
Flags that describe whether this identifier is the name of a + built-in, a poisoned identifier, an extension token, or a + macro.
If the identifier names a macro, the offset of the macro + definition within the preprocessor + block.
If the identifier names one or more declarations visible from + translation unit scope, the declaration IDs of these + declarations.

+ +

When a precompiled header is loaded, the precompiled header +mechanism introduces itself into the identifier table as an external +lookup source. Thus, when the user program refers to an identifier +that has not yet been seen, Clang will perform a lookup into the +identifier table. If an identifier is found, its contents (macro +definitions, flags, top-level declarations, etc.) will be deserialized, at which point the corresponding IdentifierInfo structure will have the same contents it would have after parsing the headers in the precompiled header.

+ +

Within the PCH file, the identifiers used to name declarations are represented with an integral value. A separate table provides a mapping from this integral value (the identifier ID) to the location within the on-disk +hash table where that identifier is stored. This mapping is used when +deserializing the name of a declaration, the identifier of a token, or +any other construct in the PCH file that refers to a name.

+ +

Method Pool Block

+ +

The method pool block is represented as an on-disk hash table that +serves two purposes: it provides a mapping from the names of +Objective-C selectors to the set of Objective-C instance and class +methods that have that particular selector (which is required for +semantic analysis in Objective-C) and also stores all of the selectors +used by entities within the precompiled header. The design of the +method pool is similar to that of the identifier +table: the first time a particular selector is formed during the +compilation of the program, Clang will search in the on-disk hash +table of selectors; if found, Clang will read the Objective-C methods +associated with that selector into the appropriate front-end data +structure (Sema::InstanceMethodPool and +Sema::FactoryMethodPool for instance and class methods, +respectively).

+ +

As with identifiers, selectors are represented by numeric values +within the PCH file. A separate index maps these numeric selector +values to the offset of the selector within the on-disk hash table, +and will be used when de-serializing an Objective-C method declaration +(or other Objective-C construct) that refers to the selector.

+ +

Precompiled Header Integration Points

+ +

The "lazy" deserialization behavior of precompiled headers requires +their integration into several completely different submodules of +Clang. For example, lazily deserializing the declarations during name +lookup requires that the name-lookup routines be able to query the +precompiled header to find entities within the PCH file.

+ +

For each Clang data structure that requires direct interaction with +the precompiled header logic, there is an abstract class that provides +the interface between the two modules. The PCHReader +class, which handles the loading of a precompiled header, inherits +from all of these abstract classes to provide lazy deserialization of +Clang's data structures. PCHReader implements the +following abstract classes:

+ +

StatSysCallCache: This abstract interface is associated with the + FileManager class, and is used whenever the file + manager is going to perform a stat() system call.
ExternalSLocEntrySource: This abstract interface is associated with the + SourceManager class, and is used whenever the + source manager needs to load the details + of a file, buffer, or macro instantiation.
IdentifierInfoLookup: This abstract interface is associated with the + IdentifierTable class, and is used whenever the + program source refers to an identifier that has not yet been seen. + In this case, the precompiled header implementation searches for + this identifier within its identifier table + to load any top-level declarations or macros associated with that + identifier.
ExternalASTSource: This abstract interface is associated with the + ASTContext class, and is used whenever the abstract + syntax tree nodes need to loaded from the precompiled header. It + provides the ability to de-serialize declarations and types + identified by their numeric values, read the bodies of functions + when required, and read the declarations stored within a + declaration context (either for iteration or for name lookup).
ExternalSemaSource: This abstract interface is associated with the Sema + class, and is used whenever semantic analysis needs to read + information from the global method + pool.

+ +