Precompiled Headers

This document describes the design and implementation of Clang's precompiled headers (PCH). If you are interested in the end-user view, please see the User's Manual.

Table of Contents

• Using Precompiled Headers with clang
• Design Philosophy
• Precompiled Header Contents
  • Metadata Block
  • Source Manager Block
  • Preprocessor Block
  • Types Block
  • Declarations Block
  • Statements and Expressions
  • Identifier Table Block
  • Method Pool Block
• Precompiled Header Integration Points

Using Precompiled Headers with clang

The Clang compiler frontend, clang -cc1, supports two command line options for generating and using PCH files.

To generate PCH files using clang -cc1, use the option -emit-pch:

 $ clang -cc1 test.h -emit-pch -o test.h.pch 

This option is transparently used by clang when generating PCH files. The resulting PCH file contains the serialized form of the compiler's internal representation after it has completed parsing and semantic analysis. The PCH file can then be used as a prefix header with the -include-pch option:

  $ clang -cc1 -include-pch test.h.pch test.c -o test.s

Design Philosophy

Precompiled headers are meant to improve overall compile times for projects, so the design of precompiled headers is entirely driven by performance concerns. The use case for precompiled headers is relatively simple: when there is a common set of headers that is included in nearly every source file in the project, we precompile that bundle of headers into a single precompiled header (PCH file). Then, when compiling the source files in the project, we load the PCH file first (as a prefix header), which acts as a stand-in for that bundle of headers.

A precompiled header implementation improves performance when:

• Loading the PCH file is significantly faster than re-parsing the bundle of headers stored within the PCH file. Thus, a precompiled header design attempts to minimize the cost of reading the PCH file. Ideally, this cost should not vary with the size of the precompiled header file.
• The cost of generating the PCH file initially is not so large that it counters the per-source-file performance improvement due to eliminating the need to parse the bundled headers in the first place. This is particularly important on multi-core systems, because PCH file generation serializes the build when all compilations require the PCH file to be up-to-date.

Clang's precompiled headers are designed with a compact on-disk representation, which minimizes both PCH creation time and the time required to initially load the PCH file. The PCH file itself contains a serialized representation of Clang's abstract syntax trees and supporting data structures, stored using the same compressed bitstream as LLVM's bitcode file format.

Clang's precompiled headers are loaded "lazily" from disk. When a PCH file is initially loaded, Clang reads only a small amount of data from the PCH file to establish where certain important data structures are stored. The amount of data read in this initial load is independent of the size of the PCH file, such that a larger PCH file does not lead to longer PCH load times. The actual header data in the PCH file--macros, functions, variables, types, etc.--is loaded only when it is referenced from the user's code, at which point only that entity (and those entities it depends on) are deserialized from the PCH file. With this approach, the cost of using a precompiled header for a translation unit is proportional to the amount of code actually used from the header, rather than being proportional to the size of the header itself.

When given the -print-stats option, Clang produces statistics describing how much of the precompiled header was actually loaded from disk. For a simple "Hello, World!" program that includes the Apple Cocoa.h header (which is built as a precompiled header), this option illustrates how little of the actual precompiled header is required:

*** PCH Statistics:
  933 stat cache hits
  4 stat cache misses
  895/39981 source location entries read (2.238563%)
  19/15315 types read (0.124061%)
  20/82685 declarations read (0.024188%)
  154/58070 identifiers read (0.265197%)
  0/7260 selectors read (0.000000%)
  0/30842 statements read (0.000000%)
  4/8400 macros read (0.047619%)
  1/4995 lexical declcontexts read (0.020020%)
  0/4413 visible declcontexts read (0.000000%)
  0/7230 method pool entries read (0.000000%)
  0 method pool misses

For this small program, only a tiny fraction of the source locations, types, declarations, identifiers, and macros were actually deserialized from the precompiled header. These statistics can be useful to determine whether the precompiled header implementation can be improved by making more of the implementation lazy.

Precompiled headers can be chained. When you create a PCH while including an existing PCH, Clang can create the new PCH by referencing the original file and only writing the new data to the new file. For example, you could create a PCH out of all the headers that are very commonly used throughout your project, and then create a PCH for every single source file in the project that includes the code that is specific to that file, so that recompiling the file itself is very fast, without duplicating the data from the common headers for every file.

Precompiled Header Contents

Clang's precompiled headers are organized into several different blocks, each of which contains the serialized representation of a part of Clang's internal representation. Each of the blocks corresponds to either a block or a record within LLVM's bitstream format. The contents of each of these logical blocks are described below.

For a given precompiled header, the llvm-bcanalyzer utility can be used to examine the actual structure of the bitstream for the precompiled header. This information can be used both to help understand the structure of the precompiled header and to isolate areas where precompiled headers can still be optimized, e.g., through the introduction of abbreviations.

Metadata Block

The metadata block contains several records that provide information about how the precompiled header was built. This metadata is primarily used to validate the use of a precompiled header. For example, a precompiled header built for a 32-bit x86 target cannot be used when compiling for a 64-bit x86 target. The metadata block contains information about:

Language options

Describes the particular language dialect used to compile the PCH file, including major options (e.g., Objective-C support) and more minor options (e.g., support for "//" comments). The contents of this record correspond to the LangOptions class.

Target architecture

The target triple that describes the architecture, platform, and ABI for which the PCH file was generated, e.g., i386-apple-darwin9.

PCH version

The major and minor version numbers of the precompiled header format. Changes in the minor version number should not affect backward compatibility, while changes in the major version number imply that a newer compiler cannot read an older precompiled header (and vice-versa).

Original file name

The full path of the header that was used to generate the precompiled header.

Predefines buffer

Although not explicitly stored as part of the metadata, the predefines buffer is used in the validation of the precompiled header. The predefines buffer itself contains code generated by the compiler to initialize the preprocessor state according to the current target, platform, and command-line options. For example, the predefines buffer will contain "#define __STDC__ 1" when we are compiling C without Microsoft extensions. The predefines buffer itself is stored within the source manager block, but its contents are verified along with the rest of the metadata.

A chained PCH file (that is, one that references another PCH) has a slightly different metadata block, which contains the following information:

Referenced file

The name of the referenced PCH file. It is looked up like a file specified using -include-pch.

PCH version

This is the same as in normal PCH files.

Original file name

The full path of the header that was used to generate this precompiled header.

The language options, target architecture and predefines buffer data is taken from the end of the chain, since they have to match anyway.

Source Manager Block

The source manager block contains the serialized representation of Clang's SourceManager class, which handles the mapping from source locations (as represented in Clang's abstract syntax tree) into actual column/line positions within a source file or macro instantiation. The precompiled header's representation of the source manager also includes information about all of the headers that were (transitively) included when building the precompiled header.

The bulk of the source manager block is dedicated to information about the various files, buffers, and macro instantiations into which a source location can refer. Each of these is referenced by a numeric "file ID", which is a unique number (allocated starting at 1) stored in the source location. Clang serializes the information for each kind of file ID, along with an index that maps file IDs to the position within the PCH file where the information about that file ID is stored. The data associated with a file ID is loaded only when required by the front end, e.g., to emit a diagnostic that includes a macro instantiation history inside the header itself.

The source manager block also contains information about all of the headers that were included when building the precompiled header. This includes information about the controlling macro for the header (e.g., when the preprocessor identified that the contents of the header dependent on a macro like LLVM_CLANG_SOURCEMANAGER_H) along with a cached version of the results of the stat() system calls performed when building the precompiled header. The latter is particularly useful in reducing system time when searching for include files.

Preprocessor Block

The preprocessor block contains the serialized representation of the preprocessor. Specifically, it contains all of the macros that have been defined by the end of the header used to build the precompiled header, along with the token sequences that comprise each macro. The macro definitions are only read from the PCH file when the name of the macro first occurs in the program. This lazy loading of macro definitions is triggered by lookups into the identifier table.

Types Block

The types block contains the serialized representation of all of the types referenced in the translation unit. Each Clang type node (PointerType, FunctionProtoType, etc.) has a corresponding record type in the PCH file. When types are deserialized from the precompiled header, the data within the record is used to reconstruct the appropriate type node using the AST context.

Each type has a unique type ID, which is an integer that uniquely identifies that type. Type ID 0 represents the NULL type, type IDs less than NUM_PREDEF_TYPE_IDS represent predefined types (void, float, etc.), while other "user-defined" type IDs are assigned consecutively from NUM_PREDEF_TYPE_IDS upward as the types are encountered. The PCH file has an associated mapping from the user-defined types block to the location within the types block where the serialized representation of that type resides, enabling lazy deserialization of types. When a type is referenced from within the PCH file, that reference is encoded using the type ID shifted left by 3 bits. The lower three bits are used to represent the const, volatile, and restrict qualifiers, as in Clang's QualType class.

Declarations Block

The declarations block contains the serialized representation of all of the declarations referenced in the translation unit. Each Clang declaration node (VarDecl, FunctionDecl, etc.) has a corresponding record type in the PCH file. When declarations are deserialized from the precompiled header, the data within the record is used to build and populate a new instance of the corresponding Decl node. As with types, each declaration node has a numeric ID that is used to refer to that declaration within the PCH file. In addition, a lookup table provides a mapping from that numeric ID to the offset within the precompiled header where that declaration is described.

Declarations in Clang's abstract syntax trees are stored hierarchically. At the top of the hierarchy is the translation unit (TranslationUnitDecl), which contains all of the declarations in the translation unit. These declarations (such as functions or struct types) may also contain other declarations inside them, and so on. Within Clang, each declaration is stored within a declaration context, as represented by the DeclContext class. Declaration contexts provide the mechanism to perform name lookup within a given declaration (e.g., find the member named x in a structure) and iterate over the declarations stored within a context (e.g., iterate over all of the fields of a structure for structure layout).

In Clang's precompiled header format, deserializing a declaration that is a DeclContext is a separate operation from deserializing all of the declarations stored within that declaration context. Therefore, Clang will deserialize the translation unit declaration without deserializing the declarations within that translation unit. When required, the declarations stored within a declaration context will be deserialized. There are two representations of the declarations within a declaration context, which correspond to the name-lookup and iteration behavior described above:

• When the front end performs name lookup to find a name x within a given declaration context (for example, during semantic analysis of the expression p->x, where p's type is defined in the precompiled header), Clang deserializes a hash table mapping from the names within that declaration context to the declaration IDs that represent each visible declaration with that name. The entire hash table is deserialized at this point (into the llvm::DenseMap stored within each DeclContext object), but the actual declarations are not yet deserialized. In a second step, those declarations with the name x will be deserialized and will be used as the result of name lookup.
• When the front end performs iteration over all of the declarations within a declaration context, all of those declarations are immediately de-serialized. For large declaration contexts (e.g., the translation unit), this operation is expensive; however, large declaration contexts are not traversed in normal compilation, since such a traversal is unnecessary. However, it is common for the code generator and semantic analysis to traverse declaration contexts for structs, classes, unions, and enumerations, although those contexts contain relatively few declarations in the common case.

Statements and Expressions

Statements and expressions are stored in the precompiled header in both the types and the declarations blocks, because every statement or expression will be associated with either a type or declaration. The actual statement and expression records are stored immediately following the declaration or type that owns the statement or expression. For example, the statement representing the body of a function will be stored directly following the declaration of the function.

As with types and declarations, each statement and expression kind in Clang's abstract syntax tree (ForStmt, CallExpr, etc.) has a corresponding record type in the precompiled header, which contains the serialized representation of that statement or expression. Each substatement or subexpression within an expression is stored as a separate record (which keeps most records to a fixed size). Within the precompiled header, the subexpressions of an expression are stored, in reverse order, prior to the expression that owns those expression, using a form of Reverse Polish Notation. For example, an expression 3 - 4 + 5 would be represented as follows:

IntegerLiteral(5)

IntegerLiteral(4)

IntegerLiteral(3)

BinaryOperator(-)

BinaryOperator(+)

STOP

When reading this representation, Clang evaluates each expression record it encounters, builds the appropriate abstract syntax tree node, and then pushes that expression on to a stack. When a record contains N subexpressions--BinaryOperator has two of them--those expressions are popped from the top of the stack. The special STOP code indicates that we have reached the end of a serialized expression or statement; other expression or statement records may follow, but they are part of a different expression.

Identifier Table Block

The identifier table block contains an on-disk hash table that maps each identifier mentioned within the precompiled header to the serialized representation of the identifier's information (e.g, the IdentifierInfo structure). The serialized representation contains:

• The actual identifier string.
• Flags that describe whether this identifier is the name of a built-in, a poisoned identifier, an extension token, or a macro.
• If the identifier names a macro, the offset of the macro definition within the preprocessor block.
• If the identifier names one or more declarations visible from translation unit scope, the declaration IDs of these declarations.

When a precompiled header is loaded, the precompiled header mechanism introduces itself into the identifier table as an external lookup source. Thus, when the user program refers to an identifier that has not yet been seen, Clang will perform a lookup into the identifier table. If an identifier is found, its contents (macro definitions, flags, top-level declarations, etc.) will be deserialized, at which point the corresponding IdentifierInfo structure will have the same contents it would have after parsing the headers in the precompiled header.

Within the PCH file, the identifiers used to name declarations are represented with an integral value. A separate table provides a mapping from this integral value (the identifier ID) to the location within the on-disk hash table where that identifier is stored. This mapping is used when deserializing the name of a declaration, the identifier of a token, or any other construct in the PCH file that refers to a name.

Method Pool Block

The method pool block is represented as an on-disk hash table that serves two purposes: it provides a mapping from the names of Objective-C selectors to the set of Objective-C instance and class methods that have that particular selector (which is required for semantic analysis in Objective-C) and also stores all of the selectors used by entities within the precompiled header. The design of the method pool is similar to that of the identifier table: the first time a particular selector is formed during the compilation of the program, Clang will search in the on-disk hash table of selectors; if found, Clang will read the Objective-C methods associated with that selector into the appropriate front-end data structure (Sema::InstanceMethodPool and Sema::FactoryMethodPool for instance and class methods, respectively).

As with identifiers, selectors are represented by numeric values within the PCH file. A separate index maps these numeric selector values to the offset of the selector within the on-disk hash table, and will be used when de-serializing an Objective-C method declaration (or other Objective-C construct) that refers to the selector.

Precompiled Header Integration Points

The "lazy" deserialization behavior of precompiled headers requires their integration into several completely different submodules of Clang. For example, lazily deserializing the declarations during name lookup requires that the name-lookup routines be able to query the precompiled header to find entities within the PCH file.

For each Clang data structure that requires direct interaction with the precompiled header logic, there is an abstract class that provides the interface between the two modules. The PCHReader class, which handles the loading of a precompiled header, inherits from all of these abstract classes to provide lazy deserialization of Clang's data structures. PCHReader implements the following abstract classes:

StatSysCallCache

This abstract interface is associated with the FileManager class, and is used whenever the file manager is going to perform a stat() system call.

ExternalSLocEntrySource

This abstract interface is associated with the SourceManager class, and is used whenever the source manager needs to load the details of a file, buffer, or macro instantiation.

IdentifierInfoLookup

This abstract interface is associated with the IdentifierTable class, and is used whenever the program source refers to an identifier that has not yet been seen. In this case, the precompiled header implementation searches for this identifier within its identifier table to load any top-level declarations or macros associated with that identifier.

ExternalASTSource

This abstract interface is associated with the ASTContext class, and is used whenever the abstract syntax tree nodes need to loaded from the precompiled header. It provides the ability to de-serialize declarations and types identified by their numeric values, read the bodies of functions when required, and read the declarations stored within a declaration context (either for iteration or for name lookup).

ExternalSemaSource

This abstract interface is associated with the Sema class, and is used whenever semantic analysis needs to read information from the global method pool.