From 222e2a7620e6520ffaf4fc4e69d79c18da31542e Mon Sep 17 00:00:00 2001 From: "Zancanaro; Carlo" Date: Mon, 24 Sep 2012 09:58:17 +1000 Subject: Add the clang library to the repo (with some of my changes, too). --- clang/docs/PTHInternals.html | 179 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 179 insertions(+) create mode 100644 clang/docs/PTHInternals.html (limited to 'clang/docs/PTHInternals.html') diff --git a/clang/docs/PTHInternals.html b/clang/docs/PTHInternals.html new file mode 100644 index 0000000..b15f681 --- /dev/null +++ b/clang/docs/PTHInternals.html @@ -0,0 +1,179 @@ + + + + Pretokenized Headers (PTH) + + + + + + + + +
+ +

Pretokenized Headers (PTH)

+ +

This document first describes the low-level +interface for using PTH and then briefly elaborates on its design and +implementation. If you are interested in the end-user view, please see the +User's Manual.

+ + +

Using Pretokenized Headers with clang (Low-level Interface)

+ +

The Clang compiler frontend, clang -cc1, supports three command line +options for generating and using PTH files.

+ +

To generate PTH files using clang -cc1, use the option +-emit-pth: + +

 $ clang -cc1 test.h -emit-pth -o test.h.pth
+ +

This option is transparently used by clang when generating PTH +files. Similarly, PTH files can be used as prefix headers using the +-include-pth option:

+ +
+  $ clang -cc1 -include-pth test.h.pth test.c -o test.s
+
+ +

Alternatively, Clang's PTH files can be used as a raw "token-cache" +(or "content" cache) of the source included by the original header +file. This means that the contents of the PTH file are searched as substitutes +for any source files that are used by clang -cc1 to process a +source file. This is done by specifying the -token-cache +option:

+ +
+  $ cat test.h
+  #include <stdio.h>
+  $ clang -cc1 -emit-pth test.h -o test.h.pth
+  $ cat test.c
+  #include "test.h"
+  $ clang -cc1 test.c -o test -token-cache test.h.pth
+
+ +

In this example the contents of stdio.h (and the files it includes) +will be retrieved from test.h.pth, as the PTH file is being used in +this case as a raw cache of the contents of test.h. This is a low-level +interface used to both implement the high-level PTH interface as well as to +provide alternative means to use PTH-style caching.

+ +

PTH Design and Implementation

+ +

Unlike GCC's precompiled headers, which cache the full ASTs and preprocessor +state of a header file, Clang's pretokenized header files mainly cache the raw +lexer tokens that are needed to segment the stream of characters in a +source file into keywords, identifiers, and operators. Consequently, PTH serves +to mainly directly speed up the lexing and preprocessing of a source file, while +parsing and type-checking must be completely redone every time a PTH file is +used.

+ +

Basic Design Tradeoffs

+ +

In the long term there are plans to provide an alternate PCH implementation +for Clang that also caches the work for parsing and type checking the contents +of header files. The current implementation of PCH in Clang as pretokenized +header files was motivated by the following factors:

+ +

+ +

Further, compared to GCC's PCH implementation (which is the dominate +precompiled header file implementation that Clang can be directly compared +against) the PTH design in Clang yields several attractive features:

+ + + +

Despite these strengths, PTH's simple design suffers some algorithmic +handicaps compared to other PCH strategies such as those used by GCC. While PTH +can greatly speed up the processing time of a header file, the amount of work +required to process a header file is still roughly linear in the size of the +header file. In contrast, the amount of work done by GCC to process a +precompiled header is (theoretically) constant (the ASTs for the header are +literally memory mapped into the compiler). This means that only the pieces of +the header file that are referenced by the source file including the header are +the only ones the compiler needs to process during actual compilation. While +GCC's particular implementation of PCH mitigates some of these algorithmic +strengths via the use of copy-on-write pages, the approach itself can +fundamentally dominate at an algorithmic level, especially when one considers +header files of arbitrary size.

+ +

There are plans to potentially implement an complementary PCH implementation +for Clang based on the lazy deserialization of ASTs. This approach would +theoretically have the same constant-time algorithmic advantages just mentioned +but would also retain some of the strengths of PTH such as reduced memory +pressure (ideal for multi-core builds).

+ +

Internal PTH Optimizations

+ +

While the main optimization employed by PTH is to reduce lexing time of +header files by caching pre-lexed tokens, PTH also employs several other +optimizations to speed up the processing of header files:

+ + + +
+ + -- cgit v1.2.3