Extending PureData with Haskell

Introduction

Pd (aka PureData)

A real-time graphical programming environment for audio, video, and graphical processing. Processing objects are connected together with patch cords, much like a modular synthesizer from days of yore.

Externals (providing additional processing objects) can be written in several languages:

C (Pd's native API)
C++
Python
Ruby
Scheme
and probably others...

Haskell

A general purpose, purely functional programming language featuring static typing, higher order functions, polymorphism, type classes, and monadic effects.

HsExt

A Pd ⇔ Haskell bridge, and the topic of this talk.

Writing Externals In C

Pd's Object System

Class-based object-orientedness.
Classes are registered at runtime, with:
- name
- constructor
- destructor
- and some other things...
An attempt to create an object of an unregistered class triggers a search for files that are likely to contain code that registers such a class.

Resources

m_pd.h: The public header file for external authors to use. 642 lines of sparsely commented definitions. Very scary.
IEM External HOWTO: A document that aims to explain how to write externals in C. Straightforward.

Building Bridges

Problems

Haskell's RTS
- must be started before any Haskell code can be used.
Pd's awkward C API
- typecasting C function pointers
- global mutable state (the constructor problem)
Haskell's purity and strong typing
- no typecasting
- no global mutable state

Solutions

Single Entry Point
- Pd guarantees to call only our setup function at most once
- we can start the Haskell RTS
- we can safely initialize a Data.IORef with our state
- we can provide this guarantee to user code too
GIMME Pd API
- Pd has a rudimentary type checker
- we can tell Pd not to check our types: GIMME the raw data

Entry Point

A Pd external is a shared library. Invoking pd -path /path/to/hsext -lib hsext will start Pd and load the file /path/to/hsext/hsext.pd_linux, calling the function hsext_setup() found within it.

Haskell's RTS must be started before any Haskell code can be used. This is not a problem for standalone executables as the startup code does this, but we need a shared library and thus we need a small amount of C code to do it ourselves.

bootstrap.c

#include "HsExt_stub.h"
extern void __stginit_HsExt(void);

/* our "real" entry point, a foreign export from Haskell */
extern void hsext_init(void);

/* this function is called at most once (by Pd) */
extern void hsext_setup(void) {

    /* start Haskell RTS */
    hs_init(0, 0);
    hs_add_root(__stginit_HsExt);
    
    /* call our Haskell entry point */
    hsext_init();
}

Setup Control Flow

libhsext is some utility C code to make things less painful.

libhsext.h (excerpt)

/* function pointer types */
typedef void (t_newf  )(t_symbol *, int, t_atom *);
typedef void (t_freef )(t_libhsext *);  /* Pd object instance */
typedef void (t_inletf)(int, int, t_symbol *, int, t_atom *);

/* called by HsExt.hsext_init */
extern void libhsext_setup(
  t_newf newf, t_freef freef, t_inletf inletf
);

libhsext_setup() stores Haskell function pointers for calling later when Pd requires it.

HsExt.hs (excerpt)

-- constructor
type NewMethod = Ptr Symbol -> CInt -> Ptr Atom -> IO (Ptr Object)
foreign import ccall "wrapper" wrapNew :: NewMethod -> IO (FunPtr NewMethod)
-- destructor
type FreeMethod = Ptr Object -> IO ()
foreign import ccall "wrapper" wrapFree :: FreeMethod -> IO (FunPtr FreeMethod)
-- data flow
type InletMethod = InstanceID -> InletNumber -> Ptr Symbol -> CInt -> Ptr Atom -> IO ()
foreign import ccall "wrapper" wrapInlet :: InletMethod -> IO (FunPtr InletMethod)

-- entry point
hsext_init = do
  sref <- newIORef s0                               -- initial state
  newMethod <- wrapNew (new sref)                   -- embed state in functions
  freeMethod <- wrapFree (free sref)
  inletMethod <- wrapInlet (inlet sref)
  libhsext_setup newMethod freeMethod inletMethod   -- pass to the C code

Create Control Flow

Plugins

When an hsext object is instantiated in Pd, the first creation argument is used to form the name of a Haskell source code file to compile and load.

hsext's PureData API defines a Creator resource type, which is passed the remaining creation arguments.

The creator may return an Instance, which contains some properties of the Pd object.

PureData API (excerpt)

-- send a message from an outlet
type Outlet = CInt -> Ptr Symbol -> [Atom] -> IO ()

type Method =  Outlet        -- allows the instance to output via its outlets
            -> CInt          -- number of the inlet that received the message
            -> Ptr Symbol    -- incoming message selector
            -> [Atom]        -- incoming message arguments
            -> IO ()

data Instance = Instance {
  iInlets  :: CInt,          -- number of inlets
  iOutlets :: CInt,          -- number of outlets
  iMethod  :: Method }       -- function to call when data arrives at an inlet
  
type Creator = [Atom] -> IO (Maybe Instance)

Inlet Control Flow

When the Pd object receives a message at an inlet, Pd calls a method in libhsext.

libhsext looks into the inlet structure to find the ID of the owning Instance, and passes this information into the Haskell core of hsext.

The core calls the one method defined in the Instance record, passing in an outlet method.

If the user code needs to output from its outlets, it calls that outlet method.

The outlet function calls a libhsext function which sends the message back to Pd.

Example: Swap

Pd's internal swap object can only swap float messages.

We can write a more generic version that swaps arbitrary messages.

Swap.hs

module Swap (creator) where

import Data.IORef
import PureData

creator :: Creator
creator args = do
  bang <- gensym "bang"
  state <- newIORef (bang, [])
  return (Just Instance{
    iInlets = 2,
    iOutlets = 2,
    iMethod = inlet state})

inlet state outlet 0 s m = do
  (rs,rm) <- readIORef state
  outlet 1 s m
  outlet 0 rs rm

inlet state outlet 1 s m = do
  writeIORef state (s,m)

Issue With hsext-0.1

Creator resource
- no state shared between instances
- only one class per source file
- solution: Setup resource
loader functionality
- [hsext RealClassName] is ugly
- solution: Pd has sys_register_loader
one method for all inlets
- no pattern matching on message selectors (Ptr Symbol)
- solution: registering methods for selectors (like Pd's C API)
non-extensible
- what if we want to pass more than an outlet function?
- solution: pass a record to the Setup resource
incomplete mapping of Pd's API
- clock callbacks
- receiver names
- table access
- digital signal processing

Plans For hsext-1.0

API record created by single entry point (with any required state references embedded) and passed to the Setup resource.

module PureData where
-- the master record of Pd API functions
data PureData  =   Pd {
  -- common symbols
  bang            ::  Symbol,
  float           ::  Symbol,
  symbol          ::  Symbol,
  pointer         ::  Symbol,
  list            ::  Symbol,
  -- symbol manipulation
  gensym          ::  String -> IO Symbol,
  peeksym         ::  Symbol -> IO String,
  -- class management
  newClass        ::  Symbol -> ((Symbol, [Atom]) -> IO Object)
                             -> (Object -> IO ()) -> IO Class,
  -- object management
  newObject       ::  Class -> IO Object,
  freeObject      ::  Object -> IO (),
  -- inlet management
  newInlet        ::  Object -> IO Inlet,
  freeInlet       ::  Inlet -> IO (),
  -- add methods to inlets
  addBang         ::                         IO ()   -> Inlet -> IO Inlet,
  addFloat        ::  (Float             ->  IO ())  -> Inlet -> IO Inlet,
  addSymbol       ::  (Symbol            ->  IO ())  -> Inlet -> IO Inlet,
  addPointer      ::  (Pointer           ->  IO ())  -> Inlet -> IO Inlet,
  addList         ::  ([Atom]            ->  IO ())  -> Inlet -> IO Inlet,
  addMethod       ::  Symbol -> ([Atom]  ->  IO ())  -> Inlet -> IO Inlet,
  addAnything     ::  ((Symbol, [Atom])  ->  IO ())  -> Inlet -> IO Inlet,
  -- outlet management
  newOutlet       ::  Object -> IO Outlet,
  freeOutlet      ::  Outlet -> IO (),
  -- send messages to outlets
  outBang         ::  Outlet                      -> IO (),
  outFloat        ::  Outlet -> Float             -> IO (),
  outSymbol       ::  Outlet -> Symbol            -> IO (),
  outPointer      ::  Outlet -> Pointer           -> IO (),
  outList         ::  Outlet -> [Atom]            -> IO (),
  outAnything     ::  Outlet -> (Symbol, [Atom])  -> IO ()
} -- data PureData

Reimplementing Swap In hsext-1.0

module Swap (setup) where
import PureData
import Data.IORef
import qualified Data.Map as M              -- need to map objects from C to Haskell

setup pd = do                               -- Setup resource
  nameS <- (gensym pd) "Swap"
  classR <- newIORef Nothing                -- class state to embed in de/constructors
  classP <- (newClass pd) (nameS) (new classR) (free classR)
  writeIORef classR (Just (pd, classP, M.empty))

new classR _ = do                           -- Constructor
  Just (pd, classP, m) <- readIORef classR
  objectR <- newIORef Nothing               -- instance state reference
  stateR <- newIORef (bang pd, [])          -- instance mutable state
  objectP <- (newObject pd) classP          -- create object, inlets, methods, outlets
  inlet0 <- (newInlet pd) objectP >>= (addAnything pd) (swap0 objectR)
  inlet1 <- (newInlet pd) objectP >>= (addAnything pd) (swap1 objectR)
  outlet0 <- (newOutlet pd) objectP
  outlet1 <- (newOutlet pd) objectP
  writeIORef objectR (Just (pd, stateR, inlet0, inlet1, outlet0, outlet1))
  writeIORef classR (Just (pd, classP, M.insert objectP objectR m))
  return objectP

free classR objectP = do                    -- Destructor
  Just (pd, classP, m) <- readIORef classR  -- Pd destructor is per-class not per-object
  objectR <- M.lookup objectP m             -- so we need a Map to find stuff to free
  writeIORef classR (Just (pd, classP, M.delete objectP m))
  Just (_, _, inlet0, inlet1, outlet0, outlet1) <- readIORef objectR
  (freeOutlet pd) outlet1                   -- Clean up
  (freeOutlet pd) outlet0
  (freeInlet  pd) inlet1
  (freeInlet  pd) inlet0
  (freeObject pd) objectP

swap0 objectR msg = do                      -- Left inlet method
  Just (pd, stateR, _, _, outlet0, outlet1) <- readIORef objectR
  state <- readIORef stateR
  (outAnything pd) outlet1 msg
  (outAnything pd) outlet0 state

swap1 objectR msg = do                      -- Right inlet method
  Just (_, stateR, _, _, _, _) <- readIORef objectR
  writeIORef stateR msg

Issue With hsext-1.0

Benefits

powerful
flexible
extensible
"idiomatically Pd"

Problems

verbose
tedious

Solutions

combinators for defining classes with similar properties
- setup = floatBinOpClass (+)
- setup = stringUnOpClass reverse
- etc
some hardcore type system voodoo could eliminate the need for a Map in every class?

Haskell Data Through Pd Patch-Cords

StablePtrs and Pd pointer atoms

space leak -- Pd objects might store it indefinitely, no way to know
crashes -- Pd objects expect pointer atoms to point to Pd structures

Unacceptable.

Internal stack and special symbol

depth first message passing => stack
send an unusual Pd symbol signifying "get the value from our stack"
storing the symbol in Pd and sending later => undefinedness
(but no crash or space leak)
strong static typing => Data.Dynamic for stack values
(which loses polymorphism)

Acceptable compromise.

Dynamic Method Dispatch By Type

data Dispatcher = Dispatcher (IntMap Dynamic) (Maybe (Dynamic -> IO()))

dispatch :: Dispatcher -> Dynamic -> IO ()
dispatch (Dispatcher methods fallback) dyn = do
  key <- typeRepKey (dynTypeRep dyn)
  case (IntMap.lookup key methods) of     -- find a method for our type
    Just method -> do
      case (dynApply method dyn) of       -- apply it
        Just r   -> case (fromDynamic r) of
          Just r'  -> r'                  -- run the action
    Nothing -> case fallback of
      Just f   -> f dyn                   -- unspecific fallback method

Haskell Really In Pd

Haskell Code Written Inline In Pd Object Boxes

...would be awesome. Especially once we can send complex Haskell structures through Pd patch cords, it would get tedious to have to write verbose classes to deconstruct them.

Haskell Types vs Pd Inlet Count vs Pd Text Handling

How many inlets should (+) :: Float -> Float -> Float have?
How many inlets should (+) :: Float -> (Float -> Float) have?

Keep in mind that in Haskell these two types are identical.

Pd is not very good at text, some characters are special (or forbidden):
\ { } , ; $

Conclusion: we need to preprocess to solve these issues.

Automatic Marshalling

Say we want: flip (:) :: [Float] -> Float -> [Float]

for a candidate type of [Float] we need to check the message:
- selector is list, float (1-element list) or bang (empty list)
- all atoms in the list are floats
if these conditions are met, marshall the Pd message to a Haskell list
if Haskell types are not compatible to Pd types, use Dynamic values

Hyperspace Exploration

Screenshot from a live performance using hsext to rotate and project a 4D cube to 3D (rendered with Gem, an OpenGL external for Pd).

The End

Useful Links

PureData Community Portal: http://puredata.info
Haskell Community Portal: http://haskell.org
IEM PureData External HOWTO: http://iem.at/pd/externals-HOWTO/HOWTO-externals-en.html
HsExt Subversion Repository: https://devel.goto10.org/listing.php?
repname=maximus&path=%2Fhsext%2F&rev=0&sc=0
svn co https://devel.goto10.org/svn/maximus/hsext hsext
Live A/V Performance At Shunt: http://www.archive.org/details/ClaudiusMaximus_-_Live_At_Shunt_2007-06-22
My Website: http://claudiusmaximus.goto10.org
GOTO10: http://goto10.org

Contents

Useful Links

Extending PureData with Haskell

Introduction

Writing Externals In C

Pd's Object System

Resources

Building Bridges

Problems

Solutions

Entry Point

bootstrap.c

Setup Control Flow

libhsext.h (excerpt)

HsExt.hs (excerpt)

Create Control Flow

Plugins

PureData API (excerpt)

Inlet Control Flow

Example: Swap

Swap.hs

Issue With hsext-0.1

Plans For hsext-1.0

Reimplementing Swap In hsext-1.0

Issue With hsext-1.0

Benefits

Problems

Solutions

Haskell Data Through Pd Patch-Cords

StablePtrs and Pd pointer atoms

Internal stack and special symbol

Dynamic Method Dispatch By Type

Haskell Really In Pd

Haskell Code Written Inline In Pd Object Boxes

Haskell Types vs Pd Inlet Count vs Pd Text Handling

Automatic Marshalling

Hyperspace Exploration

The End

Useful Links