Code Walkthrough with Example¶
The key primitives of literate programming are mangling and tangling.
Mangle - generate source code
Tangle - generate documentation
In wheel, both mangle are tangle are done with the help of cog and shell scripts. Cog is a generic preprocessor written in python. I want to make general preprocessing, source pipes, source filters the main focus. This is where cog comes in.
How does cog work ?¶
Cog is a file based preprocessor that executes python and substitutes the output inside the file. For string templating you can use python’s % character but to get the maximum benefit you can use a template engine. Mako was used for the intial prototype but mako doesn’t allow you to define custom delimiters, so it was dropped in favour of jinja templates.
Cog goes through the main source file supplied via the command line (the source file for the current document). That file contains all the commands and utilities to generate docs inline and code via writing files. This is the main file you edit in while writing code.
The code within cog blocks is replaced with the output, typically the code shown in the documentation so cog code is surrounded by sphinx code fences.
The code chunks are stored inside a global variable
_.Additional side effects - write the code to a file. You can write the code to the file much later as well, as the code is available from the
_stash. The write_file utility is used to generate the source code in this case, for the python interpreter. Jinja is used to substitute code chunks between@[ @]tags in the final code chunk that is written to disk.
Syntax markers
@<,@>and@@are cog block markers@{@}as start of Jinja blocks@[amd@]as start of Jinja variable substitutions
https://zerowidthspace.me was used for escaping cog commands here.
```
@<
import cog
import os
from jinja2 import Environment, BaseLoader
Template = Environment(loader=BaseLoader, block_start_string="%%{", block_end_string="%%}",variable_start_string="%%", variable_end_string="%%")
def write_file(name, fn):
global _
code = Template.from_string(_[name]).render(_=_)
os.makedirs(os.path.dirname(fn), exist_ok=True)
os.system("safe-rm " + fn)
f = open(fn, "w")
f.write(code)
f.close()
_ = {} # globally available
_["Stuff"] = """
## Title
```{code-block} <lang>
---
force: true
---
Docs
```
"""
cog.out(_["Stuff"])
write_file("Stuff", "file.py")
@>
@@
```
You need to use force:true for sphinx to ignore cog syntax exceptions.
If you have multiple section you should use triple quotes to define and store source code chunks in _. If you just have one code section you can use cog.previous.
wheel.sh¶
This is main script that generates the docs and source code. Its written in fish shell. By using entr, you get file watching which means whenever you are done editing the code, docs and working code are automatically generated.
Running the script,
ls source/*.cog | entr -r fish wheel.sh --build
The script,
set DEV "TRUE"
# https://stackoverflow.com/questions/30615783/fish-shell-is-it-possible-to-conveniently-strip-extensions
function build
for f in source/*.cog
set rootname (echo $f | sed 's/\.[^.]*$//')
safe-rm $rootname.md
./env/bin/cog --markers="@< @> @@" -D DEV=$DEV -d $f > $rootname.md
end
safe-rm -rf build
make html
end
function final
set DEV "FALSE"
build
end
function clean
safe-rm source/*.md
safe-rm source/conf.py
safe-rm source/index.rst
safe-rm issue.html
safe-rm issue.html
safe-rm fizz_buzz.py
safe-rm test.py
safe-rm source/sphinx.cog
safe-rm source/move.cog
safe-rm source/code.cog
safe-rm source/_static/meme.png
safe-rm source/_static/analytics.js
cp sample/conf.py source/conf.py
cp sample/index.rst source/index.rst
cp sample/code-walkthrough.cog source/code-walkthrough.cog
cp sample/setup.md source/setup.md
touch source/about.md
touch source/user-guide.md
touch source/contributing-source-code-license-support-and-credits.md
touch source/changelog.md
touch source/bookmarks.md
end
function pack
set array --exclude "*.pyc" \
--exclude "*.tgz" \
--exclude "_sources" \
--exclude "searchindex.js" \
--exclude ".DS_Store" \
--exclude "*.inv" \
--exclude "*.tcl" \
--exclude "draft" \
--exclude "doctrees" \
--exclude "__pycache__" \
--exclude "*env*" \
--exclude "tags.*" \
--exclude "tags" \
--exclude "__init__.py" \
--exclude ".git"
gtar -czv $array -f wheel.tgz .
end
switch $argv[1]
case "--build"
build
case "--prod"
final
case "--zip"
pack
case "--clean-all"
clean
end
Note the usage of --markers="@< @> @@" and the -D DEV=$DEV. This is useful to pass information to cog that you can use to affect the preprocessing. The DEV flag is useful for hiding some sections. The script offers two additional functions to pack and build the source code for production deployment.
Simple Example (Extend this)¶
For your own projects, you should extend code from this section.
Main¶
This code is substituted in the below block! Its not written to disk. Its stored inside the global _ stash.
if __name__=="__main__":
print(fizzBuzz(3))
Test¶
Write tests first and also present tests first in the documentation.
import unittest
from fizz_buzz import fizzBuzz
class TestSum(unittest.TestCase):
def test_fiz(self):
self.assertEqual(fizzBuzz(3), ['1', '2', 'Fizz'], "Should be Fizz")
if __name__ == '__main__':
unittest.main()
Fizz Buzz¶
This is the main file written to disk.
def fizzBuzz(n):
"""
:type n: int
:rtype: List[str]
"""
result = []
for i in range(1,n+1):
if i% 3== 0 and i%5==0:
result.append("FizzBuzz")
elif i %3==0:
result.append("Fizz")
elif i% 5 == 0:
result.append("Buzz")
else:
result.append(str(i))
return result
# Code is substituted from the global _ to this by write_file before writing the final file to disk. Mangling.
@[ _["Main"] @]
Execute Code¶
python fizz_buzz.py
python test.py