haskell

In Haskell, functions must always return the same consistent type. There is also no concept of nil or null built into the language. This is not meant to handicap you, and the expressiveness and polymorphic-ness of Haskell’s types mean it certainly does not. One way to handle such situations where functions (conceptually) may or may not return a value is through the Maybe type.

data Maybe a = Just a | Nothing

Maybe is a perfect and simple solution for this situation. It says that, for all values of type a, we can construct values that are either Just that or Nothing.

This type is also perfect for illustrating some of Haskell’s more math-heavy concepts. If we take all the potential a values as one category and all the potential Maybe a values as another, then we can use this type to describe the Functors of Category Theory. If we also think about the difference between some a and its Maybe a counterpart as some sort of state to be managed throughout an execution chain, then we can also use this type to describe a Monad. In both cases, the benefits are more concise code and a greater understanding of these abstract concepts that we can take with us to more complex type interactions.

Functor

A Functor is a way to transform a function (more formally a morphism) that acts on one category of objects into one that can act on objects in another category. In Haskell, this concept is captured by the Functor typeclass. It states that for any type t that takes one argument (like Maybe), we can make it an instance of Functor by defining the translation function fmap:

fmap :: Functor t => (a -> b) -> (t a -> t b)

This specifies precisely how a function that acts on one set of types (a -> b) can be used on types that are wrapped versions of these (t a -> t b).

So how is Maybe a Functor?

instance Functor Maybe where
    fmap _ Nothing  = Nothing
    fmap f (Just a) = Just (f a)

Seems straight forward, if the value is Just we apply the morphism to the underlying object and rewrap the result in Just. Trying to apply a morphism to a Nothing value just results in Nothing.

Monad

A Monad is a very scary term to Haskell noobies. Mainly because the first Monad we are introduced to is IO. It’s used for any computation that affects (or draws on) the outside world. We’re told that it handles potential failures and manages state between computations. Most times we accept it as magic and blindly memorize the do notation and counter-intuitive return statements.

We can take a step back, talk about a Monad in very general terms, then describe how Maybe types work as a Monad. Given that understanding, we can get a better handle on what State and IO Monads are doing (even if we still have to think of it as a bit of magic).

A Monad is a way to chain multiple computations together and manage how that chain of actions works as a whole. The Monad laws will manage the state between these actions (ensuring dependent actions are run in the correct order since they might rely on more than just their direct arguments) and also any failing cases (if some action fails, future actions are aborted and the whole expression is a failure).

Somewhat surprisingly, any type can act as a Monad by defining a few simple functions. I’m going to show and talk about them separately because I think it can go a long way to understanding Monads in general.

instance Monad Maybe where
    -- (>>=) :: m a -> (a -> m b) -> m b
    (Just x) >>= k = k x
    Nothing  >>= _ = Nothing

Here we’re just showing how to chain two dependant actions together – that’s really all it is. The first “action” is a wrapped value (m a), the second argument is a function which acts on the unwrapped value producing a new wrapped value (a -> m b). For Maybe we just have to account for the Just and Nothing cases appropriately.

    -- (>>) :: m a -> m b -> m b
    (Just _) >> k = k
    Nothing  >> _ = Nothing

Here we’re showing how to chain two independant actions together. We’re still preserving the fact that if the first action “fails” the second action is not run, but in this case the result of the first action has no bearing on the second.

    -- return :: a -> m a
    return = Just

return is simply a way to take some non-monadic value and treat it as a Monadic action. In our case wrapping a value in Just does just that.

    -- fail :: String -> m a
    fail _ = Nothing

There’s also the concept of outright failure. For us it’s simple: Nothing is the failure case. The reason for the String argument is that Haskell allows you to include a message with the failure. There’s much contention in the Haskell community around including fail in the Monad type class, but we won’t get into that here as Maybe has a pretty simple implementation of it.

It should also be noted that the do and <- notation that everyone is used to can be “de-sugared” down to an expression using only the above 4 functions. If you’re having trouble seeing how an expression is leveraging the above laws to do what it does, it can be a good exercise to de-sugar it by hand.

Example Time

So why do we care? Well, besides using Maybe as an illustration for hard-to-grasp concepts like Functors and Monads, knowing when to use those instances of Maybe can really cut down on code clutter and lead to elegant solutions when you’ve got a lot of Maybe-heavy code.

Let’s say you’ve got a user model in your webapp with an optional email field. This field is a custom type Email but you’ve got a function for translating it to Text. You’ve also got another general function for displaying Text values on the page as Html.

Because you were thinking ahead and you knew there’d be a lot of Maybe Text values in use throughout your site, you’ve coded your display function to accept maybe values and show an empty string in these cases.

userEmail :: User -> Maybe Email
userEmail = undefined

emailToText :: Email -> Text
emailToText = undefined

display :: Maybe Text -> Html
display = undefined

In the described ecosystem your going to have a lot of core Maybe values and a lot of value-manipulation functions not in Maybe. To put this in category terms, you’ve got a lot of morphisms in the non-maybe category and a lot of objects in the maybe category. You’re going to want to fmap that.

Here’s how the code looks without leveraging the fact that Maybe is a Functor:

displayUserEmail :: User -> Html
displayUserEmail u = let me = userEmail u
                     in display $ case me of
                         Just e  -> Just (emailToText e))
                         Nothing -> Nothing

Not terrible, but notice how fmap shrinks it right up:

displayUserEmail :: User -> Html
displayUserEmail = display . fmap emailToText . userEmail

Not only does it make the code clearer and cleaner, but it serves a common purpose: you’re going to have a lot of value-manipulating functions that should operate on basic values and not care about any wrapping. Just because you’ve got a lot of these values wrapped up in Maybe, that shouldn’t stop you from using these morphisms from the other category in this one. The nature of that Maybe wrapper allows fmap to easily handle the translation for you.

Sure, you could write a small function that takes functions that operate on normal values and allows them to be used on maybe values (and I think I did just that at one point) – but this concept of a Functor abstracts all that down to a simple generic fmap that can be used with a zillion different compound “wrapper” types.

-- something like this:
prettyNow :: IO String
prettyNow = do
  now <- getCurrentTime

  return $ formatPretty now

-- can be shorter:
prettyNow = fmap formatPretty getCurrentTime

prettyNow :: IO ()
prettyNow =
    getCurrentTime >>= \now ->
        return (formatPretty now)

Using Maybe as a Monad allows for even more verbose “stair-case” code to become much more readable. For this example, we’ve got a series of functions that translate values from one type to another. Any of these functions can fail if the input is not as expected and they capture this by returning maybe values:

textToXml :: Text -> Maybe Xml
textToXml = undefined

xmlToJson :: Xml -> Maybe Json
xmlToJson = undefined

jsonToResponse :: Json -> Maybe Response
jsonToResponse = undefined

As before, here’s that code written in a way that does not leverage Maybe’s monadic properties:

textToResponse :: Text -> Maybe Response
textToResponse t = let mx = textToXml t
                   in case mx of
                       Nothing -> Nothing
                       Just x  -> let mj = xmlToJson x
                                  in case mj of
                                      Nothing -> Nothing
                                      Just j  -> jsonToRepsonse j

What do you have here? A series of dependant computations where if any one of them fails we want the whole expression to fail. Strictly using what we’ve learned in this post, we can simplify this to the following:

textToResponse :: Text -> Maybe Response
textToResponse t = textToXml t >>= xmlToJson >>= jsonToResponse

And if you prefer do notation (I do), then we could write the above like so:

textToResponse :: Text -> Maybe Response
textToResponse t = do
    x <- textToXml t
    j <- xmlToJson x
    r <- jsonToResponse j

    return r

The r <- and return r is redundant but I think it shows more clearly the interaction between the as and Maybe as.

main :: IO ()
main = do
    -- this is IO
    text <- getSomeText

    let mresponse = do
            -- but this is Maybe
            x <- textToXml t
            j <- xmlToJson x
            r <- jsonToResponse j

            return r

    -- and IO again
    sendResponse mresponse

So hopefully you’ve all learned a little bit through this post. I know it was helpful for me to write it all out. We’ve seen that Maybe is a type that is complex enough to be used in a variety of different contexts but also simple enough to illustrate those contexts in an easier to grasp way. We’ve also seen that using these higher-level qualities of Maybe can lead to smaller, easier to read code.

I’ve had some fun recently, adding full-text search support to the posts on the site to try and make a simple-but-still-useful archive.

I’d like to post a bit about the feature and how it works. It’s got a few moving parts so I’m going to break it up a bit.

This post will focus on the backend, setting up sphinx, providing content to it from a yesod application, and executing a search from within a handler. The second post will go into the front-end javascript that I implemented for a pretty simple but effective search-as-you-type interface.

Sphinx

Sphinx is a full-text search tool. This assumes you’ve got some concept of “documents” hanging around with lots of content you want to search through by key word.

What sphinx does is let you define a source – a way to get at all of the content you have in a digestible format. It will then consume all that content and build an index which you can search very efficiently returning a list of Ids. You can then use those Ids to display the results to your users.

There are other aspects re: weighting and attributes, but I’m not going to go into that here.

The first thing you need to do (after installing sphinx) is to get your content into a sphinx-index.

If you’ve got the complete text you’ll be searching actually in your database, sphinx can natively pull from mysql or postgresql. In my case, the content is stored on disk in markdown files. For such a scenario, sphinx allows an “xmlpipe” source.

What this means is that you provide sphinx with a command to fetch an xml document containing the content it should index.

Now, if you’ve got a large amount of content, you’re going to want to use clever conduit/enumerator tricks to stream the xml to the indexer in constant memory. That’s what’s being done in this example. I’m doing something a little bit more naive – for two reasons:

1. I need to break out into IO to get the content. This is difficult from within a lifted conduit Monad, etc.
2. I don’t have that much shit – the thing indexes in almost no time and using almost no memory even with this naive approach.

So, here’s the simple way:

getSearchXmlR :: Handler RepXml
getSearchXmlR = do
    -- select all posts
    posts <- runDB $ selectList [] []

    -- convert each post into an xml block
    blocks <- liftIO $ forM posts $ \post -> do
        docBlock (entityKey post) (entityVal post)

    -- concat those blocks together to one xml document
    fmap RepXml $ htmlToContent $ mconcat blocks

    where
        htmlToContent :: Html -> Handler Content
        htmlToContent = hamletToContent . const

docBlock :: PostId -> Post -> IO Html
docBlock pid post = do
    let file = pandocFile $ postSlug post

    -- content is kept in markdown files on disk, if the file can't be 
    -- found, try to use the in-db description, else just give up.
    exists <- doesFileExist file
    mkd    <- case (exists, postDescr post) of
        (True, _         ) -> markdownFromFile file
        (_   , Just descr) -> return descr
        _                  -> return $ Markdown "nothing?"

    return $
        -- this is the simple document structure expected by sphinx's 
        -- "xmlpipe" source
        [xshamlet|
            <document>
                <id>#{toPathPiece pid}
                <title>#{postTitle post}
                <body>#{markdownToText mkd}
            |]

    where
        markdownToText :: Markdown -> Text
        markdownToText (Markdown s) = T.pack s

With this route in place, a sphinx source can be setup like the following:

source pbrisbin-src
{
	type		= xmlpipe
        xmlpipe_command = curl http://localhost:3001/search/xmlpipe
}

index pbrisbin-idx
{
	source		= pbrisbin-src
	path		= /var/lib/sphinx/data/pbrisbin
	docinfo		= extern
	charset_type	= utf-8
}

Notice how I actually hit localhost? Since pbrisbin.com is reverse proxied via nginx to 3 warp instances running on 3001 through 3003 there’s no need to go out to the internet, dns, and back through nginx – I can just hit the backend directly.

With that setup, we can do a test search to make sure all is well:

$ sphinx-indexer --all # setup the index, ensure no errors
$ sphinx-search mutt
Sphinx 2.1.0-id64-dev (r3051)
Copyright (c) 2001-2011, Andrew Aksyonoff
Copyright (c) 2008-2011, Sphinx Technologies Inc 
(http://sphinxsearch.com)

using config file '/etc/sphinx/sphinx.conf'...
index 'pbrisbin-idx': query 'mutt ': returned 6 matches of 6 total in 
0.000 sec

displaying matches:
1. document=55, weight=2744, gid=1, ts=Wed Dec 31 19:00:01 1969
2. document=62, weight=2728, gid=1, ts=Wed Dec 31 19:00:01 1969
3. document=73, weight=1736, gid=1, ts=Wed Dec 31 19:00:01 1969
4. document=68, weight=1720, gid=1, ts=Wed Dec 31 19:00:01 1969
5. document=56, weight=1691, gid=1, ts=Wed Dec 31 19:00:01 1969
6. document=57, weight=1655, gid=1, ts=Wed Dec 31 19:00:01 1969

words:
1. 'mutt': 6 documents, 103 hits

Sweet.

Haskell

Now we need to be able to execute these searches from haskell. This part is actually going to be split up into two sub-parts: first, the interface to sphinx which returns a list of SearchResults for a given query, and second, the handler to return JSON search results to some abstract client.

I’ve started to get used to the following “design pattern” with my yesod sites:

Keep Handlers as small as possible.

I mean no bigger than this:

getFooR :: Handler RepHtml
getFooR = do
    things      <- getYourThings

    otherThings <- doRouteSpecificStuffTo things

    defaultLayout $ do
        setTitle "..."
        $(widgetFile "...")

And that’s it. Some of my handlers break this rule, but many of them fell into it accidentally. I’ll be going through and trying to enforce it throughout my codebase soon.

For this reason, I’ve come to love per-handler helpers. Tuck all that business logic into a per-handler or per-model (which often means the same thing) helper and export a few smartly named functions to call from within that skinny handler.

Anyway, I digress – Here’s the sphinx interface implemented as Helpers.Search leveraging gweber’s great sphinx package:

sport :: Int
sport = 9312

index :: String
index = "pbrisbin-idx"

-- here's what I want returned to my Handler
data SearchResult = SearchResult
    { resultSlug    :: Text
    , resultTitle   :: Text
    , resultExcerpt :: Text
    }

-- and here's how I'll get it:
executeSearch :: Text -> Handler [SearchResult]
executeSearch text = do
    res <- liftIO $ query config index (T.unpack text)

    case res of
        Ok sres -> do
            let pids = map (Key . PersistInt64 . documentId) $ matches sres

            posts <- runDB $ selectList [PostId <-. pids] []

            forM posts $ \(Entity _ post) -> do
                excerpt <- liftIO $ do
                    context <- do
                        let file = pandocFile $ postSlug post

                        exists <- doesFileExist file
                        mkd    <- case (exists, postDescr post) of
                            (True, _         ) -> markdownFromFile file
                            (_   , Just descr) -> return descr
                            _                  -> return $ Markdown "nothing?"

                        return $ markdownToString mkd

                    buildExcerpt context (T.unpack text)

                return $ SearchResult
                            { resultSlug    = postSlug post
                            , resultTitle   = postTitle post
                            , resultExcerpt = excerpt
                            }

        _ -> return []

    where
        markdownToString :: Markdown -> String
        markdownToString (Markdown s) = s

        config :: Configuration
        config = defaultConfig
            { port   = sport
            , mode   = Any
            }

-- sphinx can also build excerpts. it doesn't do this as part of the 
-- search itself but once you have your results and some context, you 
-- can ask sphinx to do it after the fact, as I do above.
buildExcerpt :: String -- ^ context
             -> String -- ^ search string
             -> IO Text
buildExcerpt context qstring = do
    excerpt <- buildExcerpts config [concatMap escapeChar context] index qstring
    return $ case excerpt of
        Ok bss -> T.pack $ C8.unpack $ L.concat bss
        _      -> ""

    where
        config :: E.ExcerptConfiguration
        config = E.altConfig { E.port = sport }

        escapeChar :: Char -> String
        escapeChar '<' = "&lt;"
        escapeChar '>' = "&gt;"
        escapeChar '&' = "&amp;"
        escapeChar c   = [c]

OK, so now that I have a nice clean executeSearch which I don’t have to think about, I can implement a JSON route to actually be used by clients:

getSearchR :: Text -> Handler RepJson
getSearchR qstring = do
    results <- executeSearch qstring

    objects <- forM results $ \result -> do
        return $ object [ ("slug"   , resultSlug    result)
                        , ("title"  , resultTitle   result)
                        , ("excerpt", resultExcerpt result)
                        ]

    jsonToRepJson $ array objects

Gotta love that skinny handler, does its structure look familiar?

You can see the result by visiting search/j/mutt for example.

In the next post, I’ll give you the javascript that consumes this, creating the search-as-you-type interface you see on the Archives page.

This site was previously served via lighttpd using fastcgi. My haskell source was compiled using Network.Wai.Handler.FastCGI and the binary was placed at /srv/http/app.cgi to be handled by lighttpd’s mod_fastcgi.

I decided to switch it up and let Warp serve the haskell app directly, then proxy certain urls through to it via lighttpd.

This how-to will outline the steps needed to get this setup and comment a little bit on what all the moving parts do.

Why

Compiling to fastcgi was starting to feel kind of icky. Warp is all grown up now and capable of serving content mighty quickly. Often a problem with my app would result in lighttpd silently failing and leaving troublesome pid files around.

It’s nicer to let that front-facing server sit there running, none-the-wiser that I’m constantly developing and recompiling the app that it’s forwarding to. Installing and starting a compiled Warp binary will give me greater feedback in the event something goes awry.

Fortunately, I already had the backbone of url-rewriting going on to get requests to app.cgi so I just needed to update that to pull http traffic from another port on localhost rather than actually call a CGI process to get the response.

Lighttpd 1.5 built the proxy framework with the intention of superseding mod_fastcgi and providing that feature simply by telling you to proxy to a fastcgi application in the same way you would to another domain. This meant I just had to update my syntax for 1.5, then it was almost as easy as s/fastcgi/http/ing the config.

There’s also the minor benefit that I no longer need duplicated support files (like client-session key and favicon) between development and production.

The Moving Parts

Lighttpd will rewrite / redirect urls in a few stages:

1. Certain urls will be handled by lighttpd itself. I like lighttpd for static file serving. It’s got a pretty directory listing, it’s fast, and it makes it super easy to setup a /static/private which enforces simple http-auth for access – very handy.

2. Everything else will be rewritten (once) to /proxy/$1.

3. Anything coming in for /proxy/.* (presumably via rewrite) will go to another port on localhost where my Warp server will take over.

Lighttpd can also load-balance over multiple instances of Warp (nifty!).

The Setup

First let’s get lighttpd setup. I’m using mod_proxy_core which is only available in lighttpd-1.5+. If you’re on Arch, you can install aur/lighttpd-svn.

Import some modules:

server.modules = ( "mod_rewrite"
                 , "mod_proxy_core"
                 , "mod_proxy_backend_http"
                 )

Setup the “stage-one” redirects:

# notice that by rewriting to /proxy$1 and not /proxy/$1 we get the 
# desired behavior where / becomes /proxy/ and /what/ever becomes 
# /proxy/what/ever.
url.rewrite-once = ( "^/static.*" => "$0"
                   , "(.*)"       => "/proxy$1"
                   )

Finally, setup the actual proxying:

$HTTP["url"] =~ "^/proxy.*" {
  # straight, http pass-through
  proxy-core.protocol        = "http"

  # lighttpd will manage its own queue and send requests to whichever 
  # instance has the shortest queue
  proxy-core.balancer        = "sqf"

  # these are the 5 Warp instances we'll start
  proxy-core.backends        = ( "127.0.0.1:3001"
                               , "127.0.0.1:3002"
                               , "127.0.0.1:3003"
                               , "127.0.0.1:3004"
                               , "127.0.0.1:3005"
                               )

  # strip the /proxy prefix
  proxy-core.rewrite-request = ( "_uri" => ( "^/proxy(.*)" => "$1" ) )
}

Now that we’ve got that going we need to spin up some Warp instances to serve out anything lighttpd redirects from /proxy.

Luckily the scaffolded main.hs allows us to pass a port on the command line, so we’ll just start up a bunch of instances of our app all listening on a different port.

Script It Out

I like to script this process of starting and stopping the multiple Warp instances. To facilitate this, we need to create some support directories alongside your source code:

mkdir tmp/{pid,log}

With those in place, feel free to take the following functions and incorporate them into some server management script:

instances='1 2 3 4 5'

start_devsite() {
  local n

  echo 'Starting worker processes...'
  for n in $instances; do
    devsite -p=300$n > tmp/log/$n.log 2> tmp/log/${n}_errors.log &
    echo $! > tmp/pid/$n.pid
  done
}

stop_devsite() {
  local pid n

  echo 'Stopping worker processes...'
  for n in $instances; do
    if [[ -f tmp/pid/$n.pid ]]; then
      read -r pid < tmp/pid/$n.pid
      if [[ -n $pid ]]; then
        kill $pid
        rm tmp/pid/$n.pid
      fi
    fi
  done
}

Once you execute the start function, you should see 5 processes running listening on ports 3001 through 3005. Lighttpd is already setup to forward to those apps in a load-balanced way so go ahead and see if it worked!

Recently I decided I no longer like disqus. It was a great hands-off commenting system, but it had its downsides. I decided to make a change.

I have my own commenting system which I wrote as a yesod module. The initial reason I didn’t go with this when I first moved to yesod (and lost my existing homebrew php commenting system) was because it didn’t support authentication. I knew that an unauthenticated “enter comments” box had like a 100 to 1 ratio of spam to real comments.

When I put up rentersreality, I took the time to build authentication into my comments system by natively supporting YesodAuth so that I could use my module there. With that in place, the only thing keeping me back was losing my existing comments (more on that later).

This module really speaks to the flexibility of the framework. With about 8 lines of code I (the end user) was able to add tightly integrated commenting to my yesod site. Comments are stored in my database and users are authenticated using my authentication model. Furthermore, I (the developer) was able to put together a system that integrates this way even with my limited haskell-foo thanks to the openness and robustness of existing modules like yesod-auth and yesod-persistent.

I decided to make a few other changes along with this one. First, I moved the site from sqlite to postgresql. I originally went with sqlite because I was intimidated by postgres. After moving rentersreality to postgres, I realized it was no harder to get set up and offered great benefits in speed, reliability and maintenance tools. Secondly, I had to ditch my simple auth setup (one manually added user) in favor of a more typical auth setup. Now, anyone can authenticate with their open id (to be able to comment) and I just maintain an isAdmin flag manually to allow me to do the me-only stuff.

Why did you do it?

For the sake of completeness here are the Pros and Cons of the recent shift:

Pros:

• I wrote it.

Since I develop the comments module, I know that I’ll get preferential treatment when it comes to bug fixes and features. Also, it’s awesome.

• Not javascript.

Pages load faster and usage is clean, pure GET and POST html-forms.

• Markdown

Comments are parsed by Pandoc the same way my post content itself is. This means that you have the full expressive power of this awesome markdown system (any non dangerous html is possible plus syntax highlighting and other nice features).

• Integration

Both visually and architecturally, the comments are deeply integrated into my site. This is in spite of the code itself being completely modularized and reusable anywhere. Yay haskell.

Cons:

• I lose all existing comments

I have a plan for this.

• No Edit, Quote, or Reply functionality.

Edit is something I really want to implement, but it will be hard so we’ll see; maybe I’ll just do the delete and re-comment method other sites use.

I kind of sidestepped the whole javascript edit-in-place usage scenario and in stead created a full-blown management sub-site for commenting users. By following the “comments” link in my sidebar you can see all the comments you’ve left on this site and edit/delete them at-will. It’s a really clean method which provides a lot of functionality while being a) javascript-free and b) still completely modularized out of my specific site and reusable anywhere.

Quote could be done with some javascript to just populate the box with a markdown blockquote for you. Reply (and the implied threading) would require a re-engineering that I might not be willing to go through for a while.

• No notifications or rss services

I’m not sure how much of a use there is for this on a site like mine but with the new administration sub-site it would be almost trivial to add this functionality – maybe I’ll do this soon.

To existing commenters

If you’ve commented on this site before I want to restore your comments, but I need your help.

What I need you to do is go ahead and login once, choose a username and email it to me (comments at pbrisbin dot com) along with the disqus account you commented on previously.

If you commented on the old-old system, I could still restore your comments, you’ll just have to decide the best way to let me know what comments they were (the username you used or the thread/nature of the comments, etc).

With this information, I’ll be able to reinstate your comments and link them to your new identifier on the site.

I hope you’ll help me with this; but if not, I understand.

Play around

So go ahead and use this page to try out the commenting system. See what kind of markdown results in what.

If you want any comments (here or on other pages) edited or removed, I can always be reached by email. I don’t mind running a quick sql statement on your behalf.

Let me know of any bugs you find but don’t worry about css-fails, those should get fixed almost immediately (I just need the content present to find them).

subtitle: how to stay sane when developing for the web in haskell

I recently had the chance to do a coding exercise for a job interview. Due to some misunderstandings of the task on my part I had to do it twice. I’m still embarrassed about it, but I did end up getting the job so all’s well that ends well…

Anyway, the cool thing about doing it twice is that the first time, I did it in Rails and the second time in Yesod and that gave me a chance to evaluate the two frameworks in somewhat of a side-by-side.

I’m not going to get into a big discussion on the pros and cons of each one in general – though I will say they both excelled at staying out of my way and getting me a base for the exercise very quickly. What I’d rather talk about is structure.

During my brief time hacking in Rails, I quickly grew to like the “convention over configuration” philosophy. If you create a model/foo and a controller/foo and a test/foo, then foo itself just magically works. I liked it.

I hadn’t used the Yesod scaffold tool since about 0.1, but I knew I needed to get a site up quickly so I decided to use it for this exercise. I found that the new structure was very organized and well thought out. It gave me a similar convention-driven feeling, create hamlet/foo and cassius/foo then widgetFile foo would just work.

I think the framework could use a bit more of this approach to make the yesod-scaffolding tool as versatile as the ruby one. Mechanisms like widgetFile could be more plentiful and library provided (rather than scaffolded into your Settings.hs).

The yesod scaffold basically built you an “example” site which you can rewrite to your needs. In contrast, the ruby scaffold tool(s) let you say “I have some data structure like X” and it goes and creates a bunch of code to make X work. You’re obviously still going to rewrite a lot of the generated code, but it’s not a 100% guarantee like with yesod init.

Putting on my yesod-end-user cap (vs the usual yesod-contributer one), I would love to see yesod work more like rails: yesod init should give you a simple status-page site with links to all the documentation (you could still chose tiny, or database-driven and get all that setup at this point too). Then, yesod scaffold --whatever commands could be used to build up a CRUD interface with your actual data types.

Scaffold

Enough comparison, let’s talk about yesod as it is now.

The scaffold tool sets up the following basic structure:

/path/to/site
|-- config
|   `-- ...
|-- hamlet
|   `-- ...
|-- cassius
|   `-- ...
|-- julius
|   `-- ...
|-- Handlers
|   `-- ...
|-- Model.hs
|-- YourSite.hs
|-- Controller.hs
`-- yoursite.cabal

config is going to hold your Settings.hs module along with some text files where you define your routes and models. I also like to throw the main executables’ source files in there which I’ll discuss later.

hamlet, cassius, and julius will contain the templates files for html, css, and javascript respectively. One awesome new development is the aforementioned function widgetFile which I use 100% of the time regardless of what templates the page actually calls for. If you write, say, addWidget $(widgetFile "foo"), that will splice in the templates hamlet/foo.hamlet, cassius/foo.cassius, and julius/foo.julius and just ignores non-existent files.

Model.hs, YourSite.hs, and Controller.hs are pretty self-explanatory, and are either entirely site-dependant or scaffold-generated so I’m not going to discuss them.

One other cool feature of the scaffold is how it sets up the imports and exports in YourSite.hs. It handles all of the major imports (like Yesod itself, etc) and those that need to be qualified (like Settings) and then reexports them as a single clean interface. This means that all of your Handlers, Helpers, etc can just import YourSite and be done with it. Very nice, very clean.

Handlers usually contains one module per route and only defines the route handling functions. I try to keep any support functions in either per-handler or site-wide Helpers.

One last note: do yourself a favor and keep/maintain the generated cabal file. It’s a nice way to prevent breakage (when dependencies are updated) and keep dev vs prod options straight. It’s also nice to keep all the object and interface files hidden under an ignorable dist directory.

Development

For development, I use an easy simple-server approach. The haskell is as follows:

import Controller (withServer)
import System.IO (hPutStrLn, stderr)
import Network.Wai.Middleware.Debug (debug)
import Network.Wai.Handler.Warp (run)

main :: IO ()
main = do
    let port = 3000
    hPutStrLn stderr $ "Application launched, listening on port " ++ show port
    withServer $ run port . debug

I then keep a simple shell script that runs it:

#!/bin/bash -e

touch config/Settings.hs
runhaskell -Wall -iconfig config/simple-server.hs

The touch just ensures that anything set by CPP options are up to date every time I ./devel.

Deployment

By using the cabal file, deployments are pretty easy. I use lighttpd as my server-of-choice (I also let it do the static file serving), so I need to compile to fastcgi.

My cabal file builds an executable from config/mysite.hs which looks like this:

import Controller (withServer)
import Network.Wai.Handler.FastCGI (run)

main :: IO ()
main = withServer run

Then I’ve got another shell script to make deployments a single-command operation:

#!/bin/bash -e

app="${1:-/srv/http/app.cgi}"

sudo true

# this command just adds an auto-incrementing git tag so that if there's 
# some issue, I can just checkout the last tag and redeploy. this 
# completely sidesteps the need to backup the binary itself
deptag

touch config/Settings.hs
cabal install --bindir=./

# cabal will install to ./myapp as defined in the cabal file so we just 
# stop the service and replace the binary
sudo /etc/rc.d/lighttpd stop
sudo mv myapp "$app"
sudo /etc/rc.d/lighttpd start

This approach can be easily extended to a non-local deployment. In the case of rentersreality, the site lives on a slicehost. Its deployment file looks like this:

#!/bin/bash -e

ip="${1:-rentersreality.com}"

deptag # tag deployments

touch config/Settings.hs
cabal install --bindir=./

scp ./renters "$ip":~/

ssh -t "$ip" '
  sudo /etc/rc.d/lighttpd stop        &&
  sudo mv ./renters /srv/http/app.cgi &&
  sudo /etc/rc.d/lighttpd start       &&
  sleep 3
'

I found that after executing the remote command I had to sleep so that the process could detach correctly. Things would end up in a bad state if I disconnected right away.

I must say, since moving to the more structured approach and utilizing cabal install as the main deployment step, I have had far less issues with developing and deploying my apps.

To see two sites that are currently using this structure, just browse the projects on my github.