The Fossil Build Process
Not logged in
HomeTimelineBranchesTagsTicketsWikiLogin

1.0 Introduction

The build process for Fossil is tricky in that the source code needs to be processed by three different preprocessor programs before it is compiled. Most users will download a precompiled binary so this is of no consequence to them, and even those who want to compile the code themselves can use one of the existing makefiles. So must people do not need to be concerned with the build complexities of Fossil. But hard-core developers who desire a deep understanding of how Fossil is put together can benefit from reviewing this article.

2.0 Source Code Tour

The source code for Fossil is found in the src/ subdirectory of the source tree. The src/ subdirectory contains all code, including the code for the separate preprocessor programs.

Each preprocessor program is a separate C program implemented in a single file of C source code. The three preprocessor programs are:

  1. mkindex.c
  2. translate.c
  3. makeheaders.c

Fossil makes use of SQLite for on-disk storage. The SQLite implementation is contained in three source code files that do not participate in the preprocessing steps. These three files that implement SQLite are:

  1. sqlite3.c
  2. sqlite3.h
  3. shell.c

The sqlite3.c and sqlite3.h source files are byte-for-byte copies of a standard amalgamation. The shell.c source file is code for the SQLite command-line shell that is used to help implement the fossil sql command. The shell.c source file has been modified slightly from the standard shell.c file in the SQLite release. Search for "Fossil Patch" in the shell.c source file of Fossil to see the changes.

The TH1 script engine is implemented using files:

  1. th.c
  2. th.h

These two files are imports like the SQLite source files, and so are not preprocessed.

The VERSION.h header file is generated from other information sources using a small program called:

  1. mkversion.c

The src/ subdirectory also contains documentation about the makeheaders preprocessor program:

  1. makeheaders.html

Click on the link to read this documentation. In addition there is a Tcl script used to build the various makefiles:

  1. makemake.tcl

Running this Tcl script will automatically regenerate all makefiles. In order to add a new source file to the Fossil implementation, simply edit makemake.tcl to add the new filename, then rerun the script, and all of the makefiles for all targets will be rebuild.

Finally, there is one of the makefiles generated by makemake.tcl:

  1. main.mk

The main.mk makefile is invoked from the Makefile in the top-level directory. The main.mk is generated by makemake.tcl and should not be hand edited. Other makefiles generated by makemake.tcl are in other subdirectories (currently all in the win/ subdirectory).

All the other files in the src/ subdirectory (79 files at the time of this writing) are C source code files that are subject to the preprocessing steps described below. In the sequel, we will call these other files "src.c" in order to have a convenient name. The reader should understand that whenever "src.c" or "src.h" is used in the text that follows, we really mean all (79) other source files other than the exceptions described above.

3.0 Automatically generated files

The "VERSION.h" header file contains some C preprocessor macros that identify the version of Fossil that is to be build. The VERSION.h file is generated automatically from information extracted from the "manifest", "manifest.uuid", and "VERSION" source files in the root directory of the source tree. (The "manifest" and "manifest.uuid" files are automatically generated and updated by Fossil itself. See the fossil set manifest command for additional information.)

The VERSION.h header file is generated by a C program: src/mkversion.c. To run the VERSION.h generator, first compile the src/mkversion.c source file into a command-line program (named "mkversion.exe") than run: 

mkversion.exe manifest.uuid manifest VERSION >VERSION.h

The pathnames in the above command might need to be adjusted to get the directories right. The point is that the manifest.uuid, manifest, and VERSION files in the root of the source tree are the three arguments and the generated VERSION.h file appears on standard output.

4.0 Preprocessing

There are three preprocessors for the Fossil sources. The mkindex and translate preprocessors can be run in any order. The makeheaders preprocessor must be run after translate.

4.1 The mkindex preprocessor

The mkindex program scans the "src.c" source files looking for special comments that identify routines that implement various Fossil commands, web interface methods, and help text comments. The mkindex program generates some C code that Fossil uses in order to dispatch commands and HTTP requests and to show on-line help. Compile the mkindex program from the mkindex.c source file. Then run: 

./mkindex src.c >page_index.h

Note that "src.c" in the above is a stand-in for the (79) regular source files of Fossil - all source files except for the exceptions described in section 2.0 above.

The output of the mkindex program is a header file that is #include-ed by the main.c source file during the final compilation step.

4.2 The translate preprocessor

The translate preprocessor looks for lines of source code that begin with "@" and converts those lines into string constants or (depending on context) into special "printf" operations for generating the output of an HTTP request. The translate preprocessor is a simple C program whose sources are in the translate.c source file. The translate preprocess is run on each of the other ordinary source files separately, like this: 

./translate src.c >src_.c

In this case, the "src.c" file represents any single source file from the set of ordinary source files as described in section 2.0 above. Note that each source file is translated separately. By convention, the names of the translated source files are the names of the input sources with a single "_" character at the end. But a new makefile can use any naming convention it wants - the "_" is not critical to the build process.

After being translated, the output files (the "src_.c" files) should be used for all subsequent preprocessing and compilation steps.

4.3 The makeheaders preprocessor

For each C source module "src.c", there is an automatically generated header module "src.h" that contains all of the datatype and procedure declarations needed by the source module. These header files are generated automatically by the makeheaders program. The sources to makeheaders are contained in a single file "makeheaders.c". Additional documentation on makeheaders can be found in src/makeheaders.html.

The makeheaders program is run once. It scans all inputs source files and generates header files for each one. Note that the sqlite3.c and shell.c source files are not scanned by makeheaders. Makeheaders only runs over "ordinary" source files, not the exceptional source files. However, makeheaders also uses some extra header files as input. The general format is like this: 

makeheaders src_.c:src.h sqlite3.h th.h VERSION.h

In the example above the "src_.c" and "src.h" names represent all of the (79) ordinary C source files, each as a separate argument.

5.0 Compilation

After all generated files have been created and all ordinary source files have been preprocessed, the generated and preprocessed files can be combined into a single executable using a C compiler. This can be done all at once, or each preprocessed source file can be compiled into a separate object code file and the resulting object code files linked together in a final step.

Some files require special C-preprocessor macro definitions. When compiling sqlite.c, the following macros are recommended:

The first and second symbol definitions above are required; the others are merely recommended. The "localtime()" library function in SQLite must be redefined to invoke fossil_localtime() instead. The fossil_localtime() routine will invoke either gmtime() or localtime() depending on the "Use UTC" setting for the fossil repository. Extension loading is omitted as a security measure. Fossil is single-threaded so mutexing is disabled in SQLite as a performance enhancement.

When compiling the shell.c source file, these macros are required:

The "main()" routine in the shell must be changed into sqlite3_main() to prevent it from colliding with the real main() in Fossil, and to give Fossil an entry point to jump to when the fossil sql command is invoked.

All the other source code files can be compiled without any special options.

6.0 Linkage

Fossil needs to be linked against zlib. If the HTTPS option is enabled, then it will also need to link against the appropriate SSL implementation. And, of course, Fossil needs to link against the standard C library. No other libraries or external dependences are used.

Fossil version [4df919803b] 2020-05-25 23:23:49