Add the clang library to the repo (with some of my changes, too).

author: Zancanaro; Carlo <czan8762@plang3.cs.usyd.edu.au> 2012-09-24 09:58:17 +1000
committer: Zancanaro; Carlo <czan8762@plang3.cs.usyd.edu.au> 2012-09-24 09:58:17 +1000
commit: 222e2a7620e6520ffaf4fc4e69d79c18da31542e (patch)
tree: 7bfbc05bfa3b41c8f9d2e56d53a0bc3e310df239 /clang/docs/PCHInternals.html
parent: 3d206f03985b50beacae843d880bccdc91a9f424 (diff)
1 files changed, 532 insertions, 0 deletions
diff --git a/clang/docs/PCHInternals.html b/clang/docs/PCHInternals.html
new file mode 100644
index 0000000..28ce1ce
--- /dev/null
+++ b/clang/docs/PCHInternals.html
@@ -0,0 +1,532 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
+          "http://www.w3.org/TR/html4/strict.dtd">
+<html>
+<head>
+  <title>Precompiled Headers (PCH)</title>
+  <link type="text/css" rel="stylesheet" href="../menu.css">
+  <link type="text/css" rel="stylesheet" href="../content.css">
+  <style type="text/css">
+    td {
+    vertical-align: top;
+    }
+  </style>
+</head>
+
+<body>
+
+<!--#include virtual="../menu.html.incl"-->
+
+<div id="content">
+
+<h1>Precompiled Headers</h1>
+
+  <p>This document describes the design and implementation of Clang's
+  precompiled headers (PCH). If you are interested in the end-user
+  view, please see the <a
+   href="UsersManual.html#precompiledheaders">User's Manual</a>.</p>
+
+  <p><b>Table of Contents</b></p>
+  <ul>
+    <li><a href="#usage">Using Precompiled Headers with
+    <tt>clang</tt></a></li>
+    <li><a href="#philosophy">Design Philosophy</a></li>
+    <li><a href="#contents">Precompiled Header Contents</a>
+      <ul>
+        <li><a href="#metadata">Metadata Block</a></li>
+        <li><a href="#sourcemgr">Source Manager Block</a></li>
+        <li><a href="#preprocessor">Preprocessor Block</a></li>
+        <li><a href="#types">Types Block</a></li>
+        <li><a href="#decls">Declarations Block</a></li>
+        <li><a href="#stmt">Statements and Expressions</a></li>
+        <li><a href="#idtable">Identifier Table Block</a></li>
+        <li><a href="#method-pool">Method Pool Block</a></li>
+      </ul>
+    </li>
+    <li><a href="#tendrils">Precompiled Header Integration
+    Points</a></li>
+</ul>
+    
+<h2 id="usage">Using Precompiled Headers with <tt>clang</tt></h2>
+
+<p>The Clang compiler frontend, <tt>clang -cc1</tt>, supports two command line
+options for generating and using PCH files.<p>
+
+<p>To generate PCH files using <tt>clang -cc1</tt>, use the option
+<b><tt>-emit-pch</tt></b>:
+
+<pre> $ clang -cc1 test.h -emit-pch -o test.h.pch </pre>
+
+<p>This option is transparently used by <tt>clang</tt> when generating
+PCH files. The resulting PCH file contains the serialized form of the
+compiler's internal representation after it has completed parsing and
+semantic analysis. The PCH file can then be used as a prefix header
+with the <b><tt>-include-pch</tt></b> option:</p>
+
+<pre>
+  $ clang -cc1 -include-pch test.h.pch test.c -o test.s
+</pre>
+
+<h2 id="philosophy">Design Philosophy</h2>
+  
+<p>Precompiled headers are meant to improve overall compile times for
+  projects, so the design of precompiled headers is entirely driven by
+  performance concerns. The use case for precompiled headers is
+  relatively simple: when there is a common set of headers that is
+  included in nearly every source file in the project, we
+  <i>precompile</i> that bundle of headers into a single precompiled
+  header (PCH file). Then, when compiling the source files in the
+  project, we load the PCH file first (as a prefix header), which acts
+  as a stand-in for that bundle of headers.</p>
+
+<p>A precompiled header implementation improves performance when:</p>
+<ul>
+  <li>Loading the PCH file is significantly faster than re-parsing the
+  bundle of headers stored within the PCH file. Thus, a precompiled
+  header design attempts to minimize the cost of reading the PCH
+  file. Ideally, this cost should not vary with the size of the
+  precompiled header file.</li>
+  
+  <li>The cost of generating the PCH file initially is not so large
+  that it counters the per-source-file performance improvement due to
+  eliminating the need to parse the bundled headers in the first
+  place. This is particularly important on multi-core systems, because
+  PCH file generation serializes the build when all compilations
+  require the PCH file to be up-to-date.</li>
+</ul>
+
+<p>Clang's precompiled headers are designed with a compact on-disk
+representation, which minimizes both PCH creation time and the time
+required to initially load the PCH file. The PCH file itself contains
+a serialized representation of Clang's abstract syntax trees and
+supporting data structures, stored using the same compressed bitstream
+as <a href="http://llvm.org/docs/BitCodeFormat.html">LLVM's bitcode
+file format</a>.</p>
+
+<p>Clang's precompiled headers are loaded "lazily" from disk. When a
+PCH file is initially loaded, Clang reads only a small amount of data
+from the PCH file to establish where certain important data structures
+are stored. The amount of data read in this initial load is
+independent of the size of the PCH file, such that a larger PCH file
+does not lead to longer PCH load times. The actual header data in the
+PCH file--macros, functions, variables, types, etc.--is loaded only
+when it is referenced from the user's code, at which point only that
+entity (and those entities it depends on) are deserialized from the
+PCH file. With this approach, the cost of using a precompiled header
+for a translation unit is proportional to the amount of code actually
+used from the header, rather than being proportional to the size of
+the header itself.</p> 
+
+<p>When given the <code>-print-stats</code> option, Clang produces
+statistics describing how much of the precompiled header was actually
+loaded from disk. For a simple "Hello, World!" program that includes
+the Apple <code>Cocoa.h</code> header (which is built as a precompiled
+header), this option illustrates how little of the actual precompiled
+header is required:</p>
+
+<pre>
+*** PCH Statistics:
+  933 stat cache hits
+  4 stat cache misses
+  895/39981 source location entries read (2.238563%)
+  19/15315 types read (0.124061%)
+  20/82685 declarations read (0.024188%)
+  154/58070 identifiers read (0.265197%)
+  0/7260 selectors read (0.000000%)
+  0/30842 statements read (0.000000%)
+  4/8400 macros read (0.047619%)
+  1/4995 lexical declcontexts read (0.020020%)
+  0/4413 visible declcontexts read (0.000000%)
+  0/7230 method pool entries read (0.000000%)
+  0 method pool misses
+</pre>
+
+<p>For this small program, only a tiny fraction of the source
+locations, types, declarations, identifiers, and macros were actually
+deserialized from the precompiled header. These statistics can be
+useful to determine whether the precompiled header implementation can
+be improved by making more of the implementation lazy.</p>
+
+<p>Precompiled headers can be chained. When you create a PCH while
+including an existing PCH, Clang can create the new PCH by referencing
+the original file and only writing the new data to the new file. For
+example, you could create a PCH out of all the headers that are very
+commonly used throughout your project, and then create a PCH for every
+single source file in the project that includes the code that is
+specific to that file, so that recompiling the file itself is very fast,
+without duplicating the data from the common headers for every file.</p>
+
+<h2 id="contents">Precompiled Header Contents</h2>
+
+<img src="PCHLayout.png" style="float:right" alt="Precompiled header layout">
+
+<p>Clang's precompiled headers are organized into several different
+blocks, each of which contains the serialized representation of a part
+of Clang's internal representation. Each of the blocks corresponds to
+either a block or a record within <a
+ href="http://llvm.org/docs/BitCodeFormat.html">LLVM's bitstream
+format</a>. The contents of each of these logical blocks are described
+below.</p>
+
+<p>For a given precompiled header, the <a
+href="http://llvm.org/cmds/llvm-bcanalyzer.html"><code>llvm-bcanalyzer</code></a>
+utility can be used to examine the actual structure of the bitstream
+for the precompiled header. This information can be used both to help
+understand the structure of the precompiled header and to isolate
+areas where precompiled headers can still be optimized, e.g., through
+the introduction of abbreviations.</p>
+
+<h3 id="metadata">Metadata Block</h3>
+
+<p>The metadata block contains several records that provide
+information about how the precompiled header was built. This metadata
+is primarily used to validate the use of a precompiled header. For
+example, a precompiled header built for a 32-bit x86 target cannot be used
+when compiling for a 64-bit x86 target. The metadata block contains
+information about:</p>
+
+<dl>
+  <dt>Language options</dt>
+  <dd>Describes the particular language dialect used to compile the
+PCH file, including major options (e.g., Objective-C support) and more
+minor options (e.g., support for "//" comments). The contents of this
+record correspond to the <code>LangOptions</code> class.</dd>
+  
+  <dt>Target architecture</dt>
+  <dd>The target triple that describes the architecture, platform, and
+ABI for which the PCH file was generated, e.g.,
+<code>i386-apple-darwin9</code>.</dd>
+  
+  <dt>PCH version</dt>
+  <dd>The major and minor version numbers of the precompiled header
+format. Changes in the minor version number should not affect backward
+compatibility, while changes in the major version number imply that a
+newer compiler cannot read an older precompiled header (and
+vice-versa).</dd>
+
+  <dt>Original file name</dt>
+  <dd>The full path of the header that was used to generate the
+precompiled header.</dd>
+
+  <dt>Predefines buffer</dt>
+  <dd>Although not explicitly stored as part of the metadata, the
+predefines buffer is used in the validation of the precompiled header.
+The predefines buffer itself contains code generated by the compiler
+to initialize the preprocessor state according to the current target,
+platform, and command-line options. For example, the predefines buffer
+will contain "<code>#define __STDC__ 1</code>" when we are compiling C
+without Microsoft extensions. The predefines buffer itself is stored
+within the <a href="#sourcemgr">source manager block</a>, but its
+contents are verified along with the rest of the metadata.</dd>
+
+</dl>
+
+<p>A chained PCH file (that is, one that references another PCH) has
+a slightly different metadata block, which contains the following
+information:</p>
+
+<dl>
+  <dt>Referenced file</dt>
+  <dd>The name of the referenced PCH file. It is looked up like a file
+specified using -include-pch.</dd>
+
+  <dt>PCH version</dt>
+  <dd>This is the same as in normal PCH files.</dd>
+
+  <dt>Original file name</dt>
+  <dd>The full path of the header that was used to generate this
+precompiled header.</dd>
+
+</dl>
+
+<p>The language options, target architecture and predefines buffer data
+is taken from the end of the chain, since they have to match anyway.</p>
+
+<h3 id="sourcemgr">Source Manager Block</h3>
+
+<p>The source manager block contains the serialized representation of
+Clang's <a
+ href="InternalsManual.html#SourceLocation">SourceManager</a> class,
+which handles the mapping from source locations (as represented in
+Clang's abstract syntax tree) into actual column/line positions within
+a source file or macro instantiation. The precompiled header's
+representation of the source manager also includes information about
+all of the headers that were (transitively) included when building the
+precompiled header.</p>
+
+<p>The bulk of the source manager block is dedicated to information
+about the various files, buffers, and macro instantiations into which
+a source location can refer. Each of these is referenced by a numeric
+"file ID", which is a unique number (allocated starting at 1) stored
+in the source location. Clang serializes the information for each kind
+of file ID, along with an index that maps file IDs to the position
+within the PCH file where the information about that file ID is
+stored. The data associated with a file ID is loaded only when
+required by the front end, e.g., to emit a diagnostic that includes a
+macro instantiation history inside the header itself.</p>
+
+<p>The source manager block also contains information about all of the
+headers that were included when building the precompiled header. This
+includes information about the controlling macro for the header (e.g.,
+when the preprocessor identified that the contents of the header
+dependent on a macro like <code>LLVM_CLANG_SOURCEMANAGER_H</code>)
+along with a cached version of the results of the <code>stat()</code>
+system calls performed when building the precompiled header. The
+latter is particularly useful in reducing system time when searching
+for include files.</p>
+
+<h3 id="preprocessor">Preprocessor Block</h3>
+
+<p>The preprocessor block contains the serialized representation of
+the preprocessor. Specifically, it contains all of the macros that
+have been defined by the end of the header used to build the
+precompiled header, along with the token sequences that comprise each
+macro. The macro definitions are only read from the PCH file when the
+name of the macro first occurs in the program. This lazy loading of
+macro definitions is triggered by lookups into the <a
+ href="#idtable">identifier table</a>.</p>
+
+<h3 id="types">Types Block</h3>
+
+<p>The types block contains the serialized representation of all of
+the types referenced in the translation unit. Each Clang type node
+(<code>PointerType</code>, <code>FunctionProtoType</code>, etc.) has a
+corresponding record type in the PCH file. When types are deserialized
+from the precompiled header, the data within the record is used to
+reconstruct the appropriate type node using the AST context.</p>
+
+<p>Each type has a unique type ID, which is an integer that uniquely
+identifies that type. Type ID 0 represents the NULL type, type IDs
+less than <code>NUM_PREDEF_TYPE_IDS</code> represent predefined types
+(<code>void</code>, <code>float</code>, etc.), while other
+"user-defined" type IDs are assigned consecutively from
+<code>NUM_PREDEF_TYPE_IDS</code> upward as the types are encountered.
+The PCH file has an associated mapping from the user-defined types
+block to the location within the types block where the serialized
+representation of that type resides, enabling lazy deserialization of
+types. When a type is referenced from within the PCH file, that
+reference is encoded using the type ID shifted left by 3 bits. The
+lower three bits are used to represent the <code>const</code>,
+<code>volatile</code>, and <code>restrict</code> qualifiers, as in
+Clang's <a
+ href="http://clang.llvm.org/docs/InternalsManual.html#Type">QualType</a>
+class.</p>
+
+<h3 id="decls">Declarations Block</h3>
+
+<p>The declarations block contains the serialized representation of
+all of the declarations referenced in the translation unit. Each Clang
+declaration node (<code>VarDecl</code>, <code>FunctionDecl</code>,
+etc.) has a corresponding record type in the PCH file. When
+declarations are deserialized from the precompiled header, the data
+within the record is used to build and populate a new instance of the
+corresponding <code>Decl</code> node. As with types, each declaration
+node has a numeric ID that is used to refer to that declaration within
+the PCH file. In addition, a lookup table provides a mapping from that
+numeric ID to the offset within the precompiled header where that
+declaration is described.</p>
+
+<p>Declarations in Clang's abstract syntax trees are stored
+hierarchically. At the top of the hierarchy is the translation unit
+(<code>TranslationUnitDecl</code>), which contains all of the
+declarations in the translation unit. These declarations (such as
+functions or struct types) may also contain other declarations inside
+them, and so on. Within Clang, each declaration is stored within a <a
+href="http://clang.llvm.org/docs/InternalsManual.html#DeclContext">declaration
+context</a>, as represented by the <code>DeclContext</code> class.
+Declaration contexts provide the mechanism to perform name lookup
+within a given declaration (e.g., find the member named <code>x</code>
+in a structure) and iterate over the declarations stored within a
+context (e.g., iterate over all of the fields of a structure for
+structure layout).</p>
+
+<p>In Clang's precompiled header format, deserializing a declaration
+that is a <code>DeclContext</code> is a separate operation from
+deserializing all of the declarations stored within that declaration
+context. Therefore, Clang will deserialize the translation unit
+declaration without deserializing the declarations within that
+translation unit. When required, the declarations stored within a
+declaration context will be deserialized. There are two representations
+of the declarations within a declaration context, which correspond to
+the name-lookup and iteration behavior described above:</p>
+
+<ul>
+  <li>When the front end performs name lookup to find a name
+  <code>x</code> within a given declaration context (for example,
+  during semantic analysis of the expression <code>p-&gt;x</code>,
+  where <code>p</code>'s type is defined in the precompiled header),
+  Clang deserializes a hash table mapping from the names within that
+  declaration context to the declaration IDs that represent each
+  visible declaration with that name. The entire hash table is
+  deserialized at this point (into the <code>llvm::DenseMap</code>
+  stored within each <code>DeclContext</code> object), but the actual
+  declarations are not yet deserialized. In a second step, those
+  declarations with the name <code>x</code> will be deserialized and
+  will be used as the result of name lookup.</li>
+
+  <li>When the front end performs iteration over all of the
+  declarations within a declaration context, all of those declarations
+  are immediately de-serialized. For large declaration contexts (e.g.,
+  the translation unit), this operation is expensive; however, large
+  declaration contexts are not traversed in normal compilation, since
+  such a traversal is unnecessary. However, it is common for the code
+  generator and semantic analysis to traverse declaration contexts for
+  structs, classes, unions, and enumerations, although those contexts
+  contain relatively few declarations in the common case.</li>
+</ul>
+
+<h3 id="stmt">Statements and Expressions</h3>
+
+<p>Statements and expressions are stored in the precompiled header in
+both the <a href="#types">types</a> and the <a
+ href="#decls">declarations</a> blocks, because every statement or
+expression will be associated with either a type or declaration. The
+actual statement and expression records are stored immediately
+following the declaration or type that owns the statement or
+expression. For example, the statement representing the body of a
+function will be stored directly following the declaration of the
+function.</p>
+
+<p>As with types and declarations, each statement and expression kind
+in Clang's abstract syntax tree (<code>ForStmt</code>,
+<code>CallExpr</code>, etc.) has a corresponding record type in the
+precompiled header, which contains the serialized representation of
+that statement or expression. Each substatement or subexpression
+within an expression is stored as a separate record (which keeps most
+records to a fixed size). Within the precompiled header, the
+subexpressions of an expression are stored, in reverse order, prior to the expression
+that owns those expression, using a form of <a
+href="http://en.wikipedia.org/wiki/Reverse_Polish_notation">Reverse
+Polish Notation</a>. For example, an expression <code>3 - 4 + 5</code>
+would be represented as follows:</p>
+
+<table border="1">
+  <tr><td><code>IntegerLiteral(5)</code></td></tr>
+  <tr><td><code>IntegerLiteral(4)</code></td></tr>
+  <tr><td><code>IntegerLiteral(3)</code></td></tr>
+  <tr><td><code>BinaryOperator(-)</code></td></tr>
+  <tr><td><code>BinaryOperator(+)</code></td></tr>
+  <tr><td>STOP</td></tr>
+</table>
+
+<p>When reading this representation, Clang evaluates each expression
+record it encounters, builds the appropriate abstract syntax tree node,
+and then pushes that expression on to a stack. When a record contains <i>N</i>
+subexpressions--<code>BinaryOperator</code> has two of them--those
+expressions are popped from the top of the stack. The special STOP
+code indicates that we have reached the end of a serialized expression
+or statement; other expression or statement records may follow, but
+they are part of a different expression.</p>
+
+<h3 id="idtable">Identifier Table Block</h3>
+
+<p>The identifier table block contains an on-disk hash table that maps
+each identifier mentioned within the precompiled header to the
+serialized representation of the identifier's information (e.g, the
+<code>IdentifierInfo</code> structure). The serialized representation
+contains:</p>
+
+<ul>
+  <li>The actual identifier string.</li>
+  <li>Flags that describe whether this identifier is the name of a
+  built-in, a poisoned identifier, an extension token, or a
+  macro.</li>
+  <li>If the identifier names a macro, the offset of the macro
+  definition within the <a href="#preprocessor">preprocessor
+  block</a>.</li>
+  <li>If the identifier names one or more declarations visible from
+  translation unit scope, the <a href="#decls">declaration IDs</a> of these
+  declarations.</li>
+</ul>
+
+<p>When a precompiled header is loaded, the precompiled header
+mechanism introduces itself into the identifier table as an external
+lookup source. Thus, when the user program refers to an identifier
+that has not yet been seen, Clang will perform a lookup into the
+identifier table. If an identifier is found, its contents (macro 
+definitions, flags, top-level declarations, etc.) will be deserialized, at which point the corresponding <code>IdentifierInfo</code> structure will have the same contents it would have after parsing the headers in the precompiled header.</p>
+
+<p>Within the PCH file, the identifiers used to name declarations are represented with an integral value. A separate table provides a mapping from this integral value (the identifier ID) to the location within the on-disk
+hash table where that identifier is stored. This mapping is used when
+deserializing the name of a declaration, the identifier of a token, or
+any other construct in the PCH file that refers to a name.</p>
+
+<h3 id="method-pool">Method Pool Block</h3>
+
+<p>The method pool block is represented as an on-disk hash table that
+serves two purposes: it provides a mapping from the names of
+Objective-C selectors to the set of Objective-C instance and class
+methods that have that particular selector (which is required for
+semantic analysis in Objective-C) and also stores all of the selectors
+used by entities within the precompiled header. The design of the
+method pool is similar to that of the <a href="#idtable">identifier
+table</a>: the first time a particular selector is formed during the
+compilation of the program, Clang will search in the on-disk hash
+table of selectors; if found, Clang will read the Objective-C methods
+associated with that selector into the appropriate front-end data
+structure (<code>Sema::InstanceMethodPool</code> and
+<code>Sema::FactoryMethodPool</code> for instance and class methods,
+respectively).</p>
+
+<p>As with identifiers, selectors are represented by numeric values
+within the PCH file. A separate index maps these numeric selector
+values to the offset of the selector within the on-disk hash table,
+and will be used when de-serializing an Objective-C method declaration
+(or other Objective-C construct) that refers to the selector.</p>
+
+<h2 id="tendrils">Precompiled Header Integration Points</h2>
+
+<p>The "lazy" deserialization behavior of precompiled headers requires
+their integration into several completely different submodules of
+Clang. For example, lazily deserializing the declarations during name
+lookup requires that the name-lookup routines be able to query the
+precompiled header to find entities within the PCH file.</p>
+
+<p>For each Clang data structure that requires direct interaction with
+the precompiled header logic, there is an abstract class that provides
+the interface between the two modules. The <code>PCHReader</code>
+class, which handles the loading of a precompiled header, inherits
+from all of these abstract classes to provide lazy deserialization of
+Clang's data structures. <code>PCHReader</code> implements the
+following abstract classes:</p>
+
+<dl>
+  <dt><code>StatSysCallCache</code></dt>
+  <dd>This abstract interface is associated with the
+    <code>FileManager</code> class, and is used whenever the file
+    manager is going to perform a <code>stat()</code> system call.</dd>
+    
+  <dt><code>ExternalSLocEntrySource</code></dt>
+  <dd>This abstract interface is associated with the
+    <code>SourceManager</code> class, and is used whenever the
+    <a href="#sourcemgr">source manager</a> needs to load the details
+    of a file, buffer, or macro instantiation.</dd>
+
+  <dt><code>IdentifierInfoLookup</code></dt>
+  <dd>This abstract interface is associated with the
+    <code>IdentifierTable</code> class, and is used whenever the
+    program source refers to an identifier that has not yet been seen.
+    In this case, the precompiled header implementation searches for
+    this identifier within its <a href="#idtable">identifier table</a>
+    to load any top-level declarations or macros associated with that
+    identifier.</dd>
+
+  <dt><code>ExternalASTSource</code></dt>
+  <dd>This abstract interface is associated with the
+    <code>ASTContext</code> class, and is used whenever the abstract
+    syntax tree nodes need to loaded from the precompiled header. It
+    provides the ability to de-serialize declarations and types
+    identified by their numeric values, read the bodies of functions
+    when required, and read the declarations stored within a
+    declaration context (either for iteration or for name lookup).</dd>
+    
+  <dt><code>ExternalSemaSource</code></dt>
+  <dd>This abstract interface is associated with the <code>Sema</code>
+    class, and is used whenever semantic analysis needs to read
+    information from the <a href="#methodpool">global method
+    pool</a>.</dd>
+</dl>
+
+</div>
+
+</body>
+</html>
author	Zancanaro; Carlo <czan8762@plang3.cs.usyd.edu.au>	2012-09-24 09:58:17 +1000
committer	Zancanaro; Carlo <czan8762@plang3.cs.usyd.edu.au>	2012-09-24 09:58:17 +1000
commit	222e2a7620e6520ffaf4fc4e69d79c18da31542e (patch)
tree	7bfbc05bfa3b41c8f9d2e56d53a0bc3e310df239 /clang/docs/PCHInternals.html
parent	3d206f03985b50beacae843d880bccdc91a9f424 (diff)