path: root/site/posts/
diff options
authorThomas Letan <>2020-04-02 20:16:53 +0200
committerThomas Letan <>2020-04-02 20:16:53 +0200
commit46b2e7a3be44312f93b3d638f9bbd944f5c063a1 (patch)
treeb1eff65fa11c462e114204aadf7e6645d70d6b31 /site/posts/
parent9329e9883a52eb95c0803a46560c396d149ef2c6 (diff)
Refactor the build process to use cleopatra the Second
Diffstat (limited to 'site/posts/')
1 files changed, 320 insertions, 0 deletions
diff --git a/site/posts/ b/site/posts/
new file mode 100644
index 0000000..349e237
--- /dev/null
+++ b/site/posts/
@@ -0,0 +1,320 @@
+<h1><strong><code>cleopatra</code></strong>: Bootstrapping an Extensible Toolchain</h1>
+You are about to read the first version of *~cleopatra~*, the toolchain
+initially implemented to build the website you are reading. Since then,
+*~cleopatra~* has been completely rewritten as a
+[[][independant, more generic command-line
+program]]. That being said, the build process described in this write-up remains
+the one implemented in *~cleopatra~* the Second.
+A literate program is a particular type of software program where code is not
+directly written in source files, but rather in text document as code
+snippets. In some sense, literate programming allows for writing in the same
+place both the software program and its technical documentation.
+That being said, *~cleopatra~* is a toolchain to build a website before being a
+literate program, and one of its objective is to be /part of this very website
+it is used to generate/. To acheive this, *~cleopatra~* has been written as a
+collection of org files which can be either “tangled” using
+[[][Babel]] or “exported” as a HTML
+document. Tangling here refers to extracted marked code blocks into files.
+The page you are currently reading is *~cleopatra~* entry point. Its primilarly
+purpose is to introduce two Makefiles: ~Makefile~ and
+#+TOC: headlines 2
+~Makefile~ serves two purposes: it initiates a few global variables, and it
+provides a rule to generate At this point, some readers may
+wonder /why/ we need ~Makefile~ in this context, and the motivation behind this
+choice is really reminescent of a boot sequence. The rationale is that we need a
+“starting point” for *~cleopatra~*. The toolchain cannot live solely inside
+org-files, otherwise there would not have any code to execute the first time we
+tried to generate the website. We need an initial Makefile, one that has little
+chance to change, so that we can almost consider it read-only. Contrary to the
+other Makefiles that we will generate, this one will not be deleted by ~make
+This is similar to your computer: it requires a firmware to boot, whose purpose
+—in a nutshell— is to find and load an operating system.
+Modifying the content of ~Makefile~ in this document /will/ modify
+~Makefile~. This means one can easily put *~cleopatra~* into an inconsistent
+state, which would prevent further generation. This is why the generated
+~Makefile~ should be versioned, so that you can restore it using ~git~ if you
+made a mistake when you modified it.
+For readers interested in using *~cleopatra~* for their own websites, this
+documents tries to highlight the potential modifications they would have to
+* Global Constants and Variables
+First, ~Makefile~ defines several global “constants” (although as far as I know
+~make~ does not support true constant values, it is expected further generation
+process will not modify them).
+In a nutshell,
+- ~ROOT~ ::
+ Tell Emacs where the root of your website sources is, so that tangled output
+ filenames can be given relative to it rather than the org files. So for
+ instance, the ~BLOCK_SRC~ tangle parameter for ~Makefile~ looks like ~:tangle
+ Makefile~, instead of ~:tangle ../../Makefile~.
+- ~CLEODIR~ ::
+ Tell *~cleopatra~* where its sources live. If you place it inside the ~site/~
+ directory (as it is intended), and you enable the use of ~org~ files to author
+ your contents, then *~cleopatra~* documents will be part of your website. If
+ you don’t want that, just move the directory outside the ~site/~ directory,
+ and update the ~CLEODIR~ variable accordingly.
+For this website, these constants are defined as follows.
+#+BEGIN_SRC makefile :tangle Makefile :noweb no-export
+ROOT := $(shell pwd)
+CLEODIR := site/cleopatra
+We then introduce two variables to list the output of the generation processes,
+with two purposes in mind: keeping the ~.gitignore~ up-to-date automatically,
+and providing rules to remove them.
+ Short-term artifacts which can be removed frequently without too much
+ hassle. They will be removed by ~make clean~.
+ Long-term artifacts whose generation can be time consuming. They will only be
+ removed by ~make cleanall~.
+#+BEGIN_SRC makefile :tangle Makefile
+ARTIFACTS := build.log
+Generation processes shall declare new build outputs using the ~+=~ assignement
+operators. Using another operator will likely provent an underisable result.
+* Easy Tangling of Org Documents
+*~cleopatra~* is a literate program implemented with Org mode, an Emacs major
+editing mode. We provide the necessary bits to easily tangle Org documents.
+The configuration of Babel is done using an emacs lisp script called
+~tangle-org.el~ whose status is similar to ~Makefile~. It is part of the
+bootstrap process, and therefore lives “outside” of *~cleopatra~* (it is not
+deleted with ~make clean~ for instance). However, it is overwritten. If you try
+to modify it and find that *~cleopatra~* does not work properly, you should
+restore it using ~git~.
+#+BEGIN_SRC emacs-lisp :tangle scripts/tangle-org.el
+(require 'org)
+(cd (getenv "ROOT"))
+(setq org-confirm-babel-evaluate nil)
+(setq org-src-preserve-indentation t)
+(add-to-list 'org-babel-default-header-args
+ '(:mkdirp . "yes"))
+ 'org-babel-load-languages
+ '((shell . t)))
+We define variables that ensure that the ~ROOT~ environment variable is set and
+~tangle-org.el~ is loaded when using Emacs.
+#+BEGIN_SRC makefile :tangle Makefile
+EMACSBIN := emacs
+TANGLE := --batch \
+ --load="${ROOT}/scripts/tangle-org.el" \
+ 2>> build.log
+Finally, we introduce a
+recipe]] to seamlessly tangle a given file.
+#+BEGIN_SRC makefile :tangle Makefile
+define emacs-tangle =
+echo " tangle $<"
+${EMACS} $< ${TANGLE}
+* Bootstrapping
+The core purpose of ~Makefile~ remains to bootstrap the chain of generation
+processes. This chain is divided into three stages: ~prebuild~, ~build~, and
+This translates as follows in ~Makefile~.
+#+BEGIN_SRC makefile :tangle Makefile
+default : postbuild ignore
+init :
+ @rm -f build.log
+prebuild : init
+build : prebuild
+postbuild : build
+.PHONY : init prebuild build postbuild ignore
+A *generation process* in *~cleopatra~* is a Makefile which provides rules for
+these three stages, along with the utilities used by these rules. More
+precisely, a generation process ~proc~ is defined in The rules of for each stage are expected to be prefixed by ~proc-~, /e.g./,
+~proc-prebuild~ for the ~prebuild~ stage.
+Eventually, the following dependencies are expected between within the chain of
+generation processes.
+#+BEGIN_SRC makefile
+prebuild : proc-prebuild
+build : proc-build
+postbuild : proc-postbuild
+proc-build : proc-prebuild
+proc-postbuild : proc build
+Because *~cleopatra~* is a literate program, generation processes are defined in
+Org documents –which may contains additional utilities like scripts or
+templates—, and therefore need to be tangled prior to be effectively
+useful. *~cleopatra~ relies on a particular behavior of ~make~ regarding the
+~include~ directive. If there exists a rule to generate a Makefile used as an
+operand of ~include~, ~make~ will use this rule to update (if necessary) said
+Makefile before actually including it.
+Therefore, rules of the following form achieve our ambition of extensibility.
+#+BEGIN_SRC makefile :noweb yes
+<<extends(PROC="${PROC}", IN="${IN}", AUX="${AUX}")>>
+- ~${IN}~ is the Org document which contains the generation process code
+- ~${PROC}~ is the name of the generation process
+- ~${AUX}~ lists the utilities of the generation process tangled from ~${IN}~
+ with ~${PROC}.mk~
+We use ~&:~ is used in place of ~:~ to separate the target from its dependencies
+in the “tangle rule.” This tells ~make~ that the recipe of this rule generates
+all these files.
+Writing these rules manually —has yours truly had to do in the early days of his
+website— has proven to be error-prone.
+One desirable feature for *~cleopatra~* would be to generate them automatically,
+by looking for relevant ~:tangle~ directives inside the input Org document. The
+challenge lies in the “relevant” part: the risk exists that we have false
+posivite. However and as a first steps towards a fully automated solution, we
+can leverage the evaluation features of Babel here.
+Here is a bash script which, given the proper variables, would generate the
+expected Makefile rule.
+#+NAME: extends
+#+BEGIN_SRC bash :var PROC="" :var AUX="" :var IN="" :results output
+cat <<EOF
+include ${PROC}.mk
+prebuild : ${PROC}-prebuild
+build : ${PROC}-build
+postbuild : ${PROC}-postbuild
+${PROC}-prebuild : ${PROC}.mk ${AUX}
+${PROC}-build : ${PROC}-prebuild
+${PROC}-postbuild : ${PROC}-build
+${PROC}.mk ${AUX} &:\\
+ \${CLEODIR}/${IN}
+ @\$(emacs-tangle)
+.PHONY : ${PROC}-prebuild \\
+ ${PROC}-build \\
+ ${PROC}-postbuild
+The previous source block is given a name (=extends=), and an explicit lists of
+variables (~IN~, ~PROC~, and ~AUX~). Thanks to the
+[[][noweb syntax of
+Babel]], we can insert the result of the evaluation of =extends= inside another
+source block when the latter is tangled.
+We derive the rule to tangle using =extends=, which gives us the
+following Makefile snippet.
+#+BEGIN_SRC makefile :tangle Makefile :noweb yes
+<<extends(IN="", PROC="bootstrap", AUX="scripts/")>>
+Beware that, as a consequence, modifying code block of =extends= is as
+“dangerous” as modifying ~Makefile~ itself. Keep that in mind if you start
+hacking *~cleopatra~*!
+Additional customizations of *~cleopatra~* will be parth, rather
+than ~Makefile~.
+* Generation Processes
+Using the =extends= noweb reference, *~cleopatra~* is easily extensible. In
+this section, we first detail the structure of a typical generation process.
+Then, we construct by enumerating the generation processes that
+are currently used to generate the website you are reading.
+Each generation process shall
+1. Define ~proc-prebuild~, ~proc-build~, and ~proc-postbuild~
+2. Declare dependencies between stages of generation processes
+3. Declare build outputs (see ~ARTIFACTS~ and ~CONFIGURE~)
+* Wrapping-up
+#+BEGIN_SRC bash :tangle scripts/ :shebang "#+/bin/bash"
+BEGIN_MARKER="# begin generated files"
+END_MARKER="# begin generated files"
+# remove the previous list of generated files to ignore
+sed -i -e "/${BEGIN_MARKER}/,/${END_MARKER}/d" .gitignore
+# remove trailing empty lines
+sed -i -e :a -e '/^\n*$/{$d;N;};/\n$/ba' .gitignore
+# output the list of files to ignore
+echo "" >> .gitignore
+echo ${BEGIN_MARKER} >> .gitignore
+for f in $@; do
+ echo "${f}" >> .gitignore
+echo ${END_MARKER} >> .gitignore
+#+BEGIN_SRC makefile :tangle
+ignore :
+ @echo " update gitignore"
+ @scripts/ \
+clean :
+ @rm -rf ${ARTIFACTS}
+cleanall : clean
+ @rm -rf ${CONFIGURE}
+# Local Variables:
+# org-src-preserve-indentation: t
+# End: