hp2pretty | hp2ps | both | toggle

I recently wrote about graphing heap profiles, and now it's easier to try out my work-in-progress hp2ps rewrite as I uploaded hp2pretty version 0.1 to Hackage. Currently it has no options and works as a filter from heap profiles to scalable vector graphics, for example:

hp2pretty <in.hp >out.svg

Some rough benchmarks on a huge (37MB) .hp file give almost identical timings for hp2pretty and hp2ps, though hp2pretty does use more memory. I haven't yet benchmarked the more common case of small .hp files.

The code is available at hp2pretty/hp2pretty on Gitorious, or you can download hp2pretty-0.1.tar.gz, or you can get it with cabal install hp2pretty.

EDIT (2010-07-29): this post is somewhat misleading regarding #2 below. See GHC ticket #4225 for some additional clarification.

hp2ps is a tool for generating graphs (in the PostScript file format) from heap profile logs dumped by the GHC RTS for Haskell programs. These graphs are invaluable for debugging errant programs and reducing memory consumption, but there are some problems:

1. hp2ps is written in C that no one seems willing to maintain
2. heap profile logs have time stamps that include the overhead of profiling
3. the output graphs aren't as beautiful as they could be

In particular, #2 can give very misleading graphs, where periods of high heap usage appear to dominate the running time. This warping is because (fairly obviously) it takes longer to generate a heap profile sample when the heap is larger. With the profiling overheads taken out, you can compare graphs with different heap sample intervals more easily, because the shape will be the same (only with more or less detail).

I've been working on a replacement for hp2ps, written in Haskell (which solves #1). I added an option to graph against sample index instead of sample time (which solves #2, provided that the heap sample interval is constant, which is currently the case). And hopefully the picture shows that I'm making progress on #3 (some more pictures)

Currently I'm just building up SVG strings, but I should switch to using a library such as wumpus-core to get both PostScript and SVG output (the other alternatives for vector graphics output available on Hackage seem to have heavy external library dependencies such as Cairo or OpenGL).

The code is available at hp2pretty/hp2pretty on Gitorious.

Video download options:

• DVD (589MB)
• Ogg (44MB)
• MP4 (41MB)

A new release of mandulia (a zooming visualization of the Mandelbrot Set as many Julia Sets, with Lua scripting support) is installable from Hackage:

cabal install mandulia

or, if your CPU doesn't support SSE4 instructions:

cabal install mandulia -f-SSE4

After you have installed it, you can try it out by running:

mandulia main

and hit the 'a' key to enter attract mode or use the key controls listed in the README for interactive navigation.

mandulia-0.5 runs in asymptotic constant space, aesthetics tweaks give more colourful 'neon' output, and the randomized attract mode picks more interesting places to explore.

O(1) space usage was achieved by recomputing the current path down from the top of the substitution tiling tree each time, instead of navigating around a shared tree that could grow indefinitely; and by using greater strictness in some mutable references (such as the statistics logger and the scripting interface).

There is one small bug in this release that I noticed too late (and isn't quite big enough to warrant an entirely new upload): the program help text states that configurations 'interactive' and 'random' are included, but they have been combined into one configuration called 'main'. A feature of this new 'main' configuration allows you to specify additional personal configuration in the user settings directory, for example (on Linux) ~/.mandulia/config.lua.

Source code is GPLv3+ licensed, Git repository at maximus/mandulia on Gitorious, or you can download mandulia-0.5.tar.gz.

My project mandulia (a zooming visualization of the Mandelbrot Set as many Julia sets, with Lua scripting support) is reaching adolescence, and is ready to take its first steps alone into the outside world. It's now available from Hackage, the Haskell world's package repository, which makes installing it as simple as:

cabal install mandulia

or, if your CPU doesn't support SSE4 instructions:

cabal install mandulia -f-SSE4

After you have installed it, you can try it out by running:

mandulia random

for a randomized attract mode, or:

mandulia interactive

for keyboard controlled navigation.

A couple of short excerpts from the documentation follow, first the command line help:

$ mandulia --help
Usage: mandulia [OPTION]... CONFIGURATION [ARGUMENT]...

CONFIGURATION is considered (in this order):
  as a file to load directly;
  as a module to load from the current directory;
  as a module to load from the user settings directory;
  as a module to load from the global settings directory.

Configurations available in this package may include:
  interactive
  random

The ARGUMENT list is supplied to the selected configuration.

Options:
  +RTS [OPTION].. -RTS   options for the run time system
                         (Try `mandulia +RTS -? -RTS' for help)
  -?, -h, --help         print this help text
  -V, --version          print program version

and some key bindings documented in the README:

interactive defines the following key bindings:

Escape

quit

PageUp

zoom in

PageDown

zoom out

cursors

move around

End

stop moving

Home

return to base

[ ]

change speed

#

reset speed

Source code is GPLv3+ licensed, Git repository at maximus/mandulia on Gitorious, or you can download mandulia-0.4.tar.gz.

Mandulia video (47MB, Ogg Theora)

As I briefly explained there's a connection between the Mandelbrot set and Julia sets, of which this project here takes advantage. An earlier prototype (100MB, Ogg Theora) had an alternating square/diamond lattice, which was visually a bit unpleasant. So thanks to some people on #haskell IRC, I got pointed to substitution tilings, and in particular the Ammann A3 tiling caught my eye while browsing the thingy. What struck me was the semi-regular spacing of the light blue/grey tiles, and the fact that there is a fixed point within each of those in the substitution rules: this fit my goal of an aperiodic point cloud with new points appearing in the gaps (1MB, Animated GIF) when zooming in (maintaining stability of the existing points).

The main program is written in Haskell, with the Julia Set renderer implemented in C (I found that GPU processing is actually less efficient than using the CPU for this problem, maybe I'm just doing it wrong but it seems like there are a lot of conditional loops and not enough data-parallelism). I learned some stuff about Control.Concurrent things in Haskell, like using nested MVar's to build a job scheduler and resource reallocator and whether to use safe or unsafe foreign imports with the threaded Haskell runtime (I think safe means it's slower for short calls but for long calls like rendering a fractal it works out faster as the runtime isn't blocked).

Source code is GPLv3+ licensed, Git repository at maximus/mandulia on Gitorious.

(For the sake of this post, it is better to consider the language non-profane.)

Brainfuck is an esoteric programming language based on a minimal Turing machine: it has a data tape and a code array, the instructions in the code modify the data in cells on the tape. Despite the presence of only a few simple instructions in the language it is theoretically possible to compute everything that can be computed within the Church/Turing model of computation; further an even more minimal derivative of Brainfuck, Boolfuck, restricts the data on the tape to consist only of single bits (either 0 or 1, instead of bytes from 0 to 255).

Now, consider changing the structure of the data 'tape', in particular extending it into a 2D hyperbolic plane. Hyperbolic geometry allows many parallel lines to a given line through a given point not on the line, with the consequence that the hyperbolic plane has exponentially more space than the common flat Euclidean plane.

Now, it is possible to tile the hyperbolic plane with congruent shapes, in many more ways than flat space can be tiled. For example, with one particularly shaped pentagonal tile, one can tile the hyperbolic plane in uncountably infinitely many ways. To navigate in this new arrangement of cells in this hyperbolic pentagonal tape, one adds three new instructions to the language: in addition to moving the cursor in the data tape East and West, new instructions allow moving North, South-East and South-West.

This best visualised in the Poincaré half-plane model of hyperbolic geometry, in which shapes are distorted to give a horizon at infinity to the South (compare with the Poincaré disk model in which there is a circular horizon at infinity, most popularized by M C Escher's Circle Limit I-IV prints).

This 2D hyperbolic modification of Boolfuck I call Hyperfuck, for fairly obvious reasons.

Now, the pentagonal tiling described above has a kind of tree-like memory property: if you proceed from a starting location and head North, you might enter the pentagon above from either its South-East edge or its South-West edge: this allows information to be encoded in the geometry of the initial configuration of the system. By modifying data in the cells and moving suitably, comparing the results of the movements with what you expect allows you to recover the stored bitstream. Further, this information need have no bound, for example one could encode an infinite library of every book ever written and that ever will be written (compare with Borges, the Shakespeare Typing Monkeys, and the text entry software Dasher).

However, a suitable source of information if a programmer is to be able to write useful programs more easily needs to be widely known and predictable: thus I propose the ASCII bitstream of a certain authorized version of the Christian Bible.

This liturgical modification of Hyperfuck I call King James Hyperfuck, for fairly obvious reasons.

A simple program can reconstruct the text; its source code:

+[^\;^+]

The current King James Hyperfuck interpreter is implemented in Haskell, using OpenGL to visualize the data tape, code tape, and output text. It also sends event information (the current instruction being executed) to Pure-data for sonification. There is also a (currently unfinished) SVG and ECMAScript version to run inside a web browser. The original prototype was implemented in C. Source code is available at maximus/kjhf at gitorious.org.

A minimal fractal music video for Cathedral Algorithms.

Download options:

• DVD (968MB)
• Ogg (79MB)
• MP4 (69MB)

The main code to render the video is ~185 lines of Haskell, available as part of maximus/dr1 at gitorious.org.

Bowntz is a little world of colliding circles, implemented in Haskell using OpenGL for video and SuperCollider3 for audio. Its simulation algorithm is based on directly calculating the time until the next collision rather than stepping through time at a constant rate.

Bowntz source code repository at Gitorious

git clone git://gitorious.org/bowntz/bowntz.git

Bowntz (demo 1) video

brain massage generative music video // fractal video made with Fl4m6e // organic audio made with Pure-data using GridFlow and Zexy

Downloads:

• DVD (287MB)
• Ogg (24MB)
• MPEG4 (21MB)

FAuSt has a nifty n-ary block diagram parallel composition operator, in which you give it a count and a function from numbers to block diagrams. Something with the same meaning as:

parallel n f = foldr1 Par (map f [0..(n-1)])

If only life were so simple: that won't compile, because of my decision in Heist to express the type system of block diagram algebra within Haskell's type system. So after about 4 hours of head scratching, I finally figured out how to generalize from this:

par0 f = empty
par1 f = par0 f `Par` f 0
par2 f = par1 f `Par` f 1
par3 f = par2 f `Par` f 2
par4 f = par3 f `Par` f 3
...

to the point where this compiles successfully:

import Types.Data.Num.Decimal (d2, d4)
import Heist (Bd (Seq), parallel, add)
import Dot (toDot)

main = putStrLn(toDot(

    parallel d4 (const add) `Seq`
    parallel d2 (const add) `Seq`
    add

  ))

Check out the code to see the horrible gory details, where at the time of writing (Subversion revision 1582) the relevant excerpt is lines 222-256 of Heist.hs:

svn co https://code.goto10.org/svn/maximus/2009/heist heist

At this point I was happy and delighted, but not for long. I tried to compile a program containing parallel d8 add and waited, and waited, only to find (7.5 minutes later...) that GHC reached my operating system's limit of 3GB per process and gave up. It was a similar story when explicitly limiting the heap size. I ran some tests and it seemed that the compilation time grew exponentially with the parallel count.

The "shortcut" of embedding FAuSt semantics inside Haskell to avoid having to write a parser and type checker and all the rest turned out to be a rather more "scenic route" through type level programming, but in summary: theoretically beautiful, epic fail in practice.

So I was playing around with FAuSt, a pure functional dataflow language for audio stream processing. At first I thought it was great, but then the compiler bugs bit hard: declaring two variables with the same name in the same scope made the compiler segfault [2010-01-13 update: I have not been able to reproduce this crash, so I'm probably wrong about the cause], and for some reason it thought the index variables in my parallel constructs were duplicates too. I was trying to build a 16x16 matrix multiplication to implement my favourite reverb... perhaps too ambitious? Simple things are very concise in FAuSt, though: I implemented Pure-data's vcf~ in roughly the same number of characters as the number of lines of code that the C implementation had:

vcf(q,f,x) = w ~ (_,_) with {
  a = max(f * 2 * PI / SR, 0);
  r = max(1 - a / q, 0);
  c = r * cos(a);
  s = r * sin(a);
  k = (2 - 2 / (q + 2)) * (1 - r);
  w(u,v) = (k * x + c * u - s * v) , (s * u + c * v);
};

So I thought to myself, with Haskell's awesomeness I should be able to embed the core of the FAuSt block diagram algebra semantics in a typesafe way. Much headsploding followed, figuring out how to bluff my way through type-level programming, until I managed to express the preconditions and results of each of the five block diagram composition operators.

It might help to know what a block diagram is and how they can be composed. For that I recommend Chapter 3 of A FAuSt Tutorial (PDF). I decided that GADTs were what I needed, with constructors for each of the composition operators, and one for primitives:

-- abstract block diagrams
data Bd t where
  Rec   :: (...) => Bd i -> Bd j -> Bd k
  Par   :: (...) => Bd i -> Bd j -> Bd k
  Seq   :: (...) => Bd i -> Bd j -> Bd k
  Split :: (...) => Bd i -> Bd j -> Bd k
  Merge :: (...) => Bd i -> Bd j -> Bd k
  Prim  :: (...) => String  -> k -> Bd k

I've omitted the contexts here, because they are quite long, expressing the necessary preconditions on the shapes of the block diagram inputs and the resulting output block diagram shape. I'm using "shape" to mean a pair of type-level lists, for example a standard (+) :: Float -> Float -> Float operator lifted to a block diagram could have a shape like (Float :. Float :. Nil, Float :. Nil), depending on how spicy you like your curry.

FAuSt semantics for the recursive composition operator A~B are described informally like this (where (i|o)X is the number of inputs|outputs of X):

• the first iB outputs of A are fed into the inputs of B
• the outputs of B are fed into the first oB inputs of A
• the remaining iA-oB inputs of A are the inputs of A~B
• the oA outputs of A are the outputs of A~B

From this description it's easy to see the requirement that oA>=iB and iA>=oB, moreover the types of the inputs must match the types of the outputs they are connected to. This precondition on the compatibility of the shapes, and the shape of their composition, can be expressed like this:

-- recursive composition compatibility and result shape
type family IsRecCompat x y
type instance IsRecCompat (xi, xo) (yi, yo) = (Take (Length yi) xo `Equals` yi) :&&: (Take (Length yo) xi `Equals` yo)
type family RecShape x y
type instance RecShape (xi, xo) (yi, yo) = (Drop (Length yo) xi, xo)

Here Take, Drop, Length, and Equals operate on type-level lists. The full type of the Rec constructor is now:

  -- recursive composition constructor
  Rec   :: (NaturalT (Length ai), NaturalT (Length ao),
            NaturalT (Length bi), NaturalT (Length bo),
            NaturalT (Length ci), NaturalT (Length co),
            i ~ (ai,ao), j ~ (bi,bo), k ~ (ci,co),
            IsRecCompat   i j ~ True, RecShape   i j ~ k)
        => Bd i -> Bd j -> Bd k

where the first 4 lines express that the lengths of the lists are non-negative numbers and that the shapes are pairs of lists - not sure how to simplify this boilerplate which is present for all the operators, but it seems to work ok.

With the fixities as in FAuSt...

infixl 4 `Rec`
infixl 3 `Par`
infixl 2 `Seq`
infixl 1 `Split`
infixl 1 `Merge`

...and suitable definitions of primitives, it's possible to write expressions that look like a more verbose FAuSt, and manipulate them in various ways:

-- example
main = putStrLn (toDot(
  castF `Par` castF
  `Split`
  ( addF `Par` subF `Par` mulF
     `Split`
     (addF `Par` addF `Par` subF `Par` divF `Par` negF)
  ) `Par` addF
  `Merge`
  addF `Par` castI
  ))

I won't talk about the toDot code right now, as it's not too impressive compared to the rest of the code - but if you want to look the whole lot is in my messy Subversion repository:

svn co https://code.goto10.org/svn/maximus/2009/heist heist

And why the name Heist? Maybe I'm trying to steal FAuSt's thunder! Or it could just be that it's got an H and an S in it...

The latest experiment in 4D visualisation fresh from the labs. Downloads:

• DVD (85MB)
• Ogg (6.05MB)
• MPEG4 (7.45MB)

What Reflex does first is: given a {p,q,r} specification of a 4D polytope, calculate the fundamental region (being a collection of 4 planes through the origin), from which calculate the symmetry group (being the group closure of the reflections in those planes).

What Reflex does next is: given four non-negative numbers (being the relative distances from the planes bounding the fundamental region), find the point thus represented, further find additional points at the centres of neighbouring 1-lines, 2-surfaces and 3-solids, and then find all images of that figure under the action of the symmetry group.

What Reflex finally does is a stereographic projection to 3D, assuming the original 4D figure is in the surface of a sphere, followed by a normal perspective projection to 2D - with a selection of the figures' subcomponents displayed (currently each of the six 2-surfaces can be toggled on/off - earlier versions did the same with the four 1-lines).

Reflex runs in realtime at reasonable frame rate and CPU usage provided I turn off the interpolation completely (which stops the curves being so smooth, alas) and don't try to render the mammoth {5,3,3} (which has a symmetry group with 14400 elements).

But nothing new under the sun, essentially this is all just a clone of The Symmetriad. I might write up some of the maths in an article later, meanwhile you can find the totally undocumented code in my mess of a Subversion repository:

svn co https://code.goto10.org/svn/maximus/2009/dualist reflex

(Hint: you need to generate the .4d files first, then ./reflex-jack.sh will fire up a Pd GUI to send commands to Reflex. SuperBonusPoints for anyone who gets it working, like the title says, just a preview...)