Josh Suereth

Dead simple Kubernetes home setup

2026-09-19T00:00:00+00:00

Hello Internet. It's been a hot minute since I used to write here. Life got crazy, with personal tragedy and changing jobs and demands on my free time, I just didn't have the energy to write, but also there were important things I wanted to spend my time on: Family, Friends, Health. I encourage anyone who's a chronically online or open-source advocate to take the time they need to for themselves, for their lives offline, and health.

That said, I've recently been doing a lot of tinkering, experimenting and learning, as has the entire Software industry - being thrown on its head by AI. This has led to, for myself, a dramatically increased velocity of personal projects. I have a lot I'd love to share, but first - a bit about those personal projects.

Recently, I took a foray into building out some home infrastructure. Why? Two reasons - Quota and Learning. The first—running out of GitHub Actions quota—was a big impetus for me. I'm building out a game, and my GitHub Actions tests for a private repo were exhausting my quota in the first week of the month, with ~$5-10 per week in action fees. Doing that math with my subscriptions, we're talking ~$500 / year if I continued down the path I was going.

SO - I decided it was time to look into alternatives and I've been using a lot more k8s for work and OpenTelemetry. I decided to purchase a refurbished Dell OptiPlex (~$300) and turn it into a single-node k8s cluster, upon which I can start hoisting my unbridled enthusiasm (and pet projects). This was less than the expected yearly bill of github actions!

The "Pet Infrastructure" Trap

Herein lies the danger - Does this new cluster become the hobby, or will I be able to continue working on, e.g. my board game monte-carlo simulations?

I wanted a no-fuss system that I could get up and running that would accelerate my personal projects, not cause me to tinker around.

A few things enable this for me:

I have sufficient kubernetes knowledge for managing systems, I can spot / preempt maintenance areas.
I was willing to isolate the cluster and give Agents full debugging access to it. It's not used for production, just personal - however there's still danger here (more on that in separate blog)
Talos Linux - A product I'm extremely pleased with.

Getting the kubernetes cluster up and running with Talos was mere minutes. Getting the cluster to actually talk to GitHub Actions and be a viable Action Runner (ARC) took an entire evening - a lot of debugging.

Let's look at the actual hard decisions that had to be made - and you can evaluate if I've turned this into a "pet" or not at the end of the article.

Dead Simple, Native Kubernetes

Talos Linux is a no-fuss, minimal operating system that's designed to be kubernetes natively. Accessing your box is done through talosctl (similar to kubectl but with different functions). The entire OS is setup and configured similar to k8s, with resources and YAML files.

What I really like - is this:

Flashing a box takes minutes
Pushing config brings a new box to the same state as a previous box.
Forcing all interaction to be through declarative config, means disaster recovery is easier.
Super minimal footprint / overhead compared to other solutions for k8s.

For example, on my laptop I can now run the following:

$ talosctl get mounts
NODE             NAMESPACE   TYPE          ID          VERSION   SOURCE      TARGET   FILESYSTEM TYPE
192.168.86.243   runtime     MountStatus   EPHEMERAL   1         /dev/sda4   /var     xfs

There's a variety of "node level" resources and controls you can use to inspect the machine.

You can then use all your favorite kubectl commands, e.g here's the current load on the box:

 kubectl top node
NAME            CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)
talos-s13-64i   363m         6%       2032Mi          6%

Note: When using GitHub Actions - so far - CPU tends to spike to 100%, memory to ~30%.

Within minutes I was literally moving from "setting up the box" to "fitting kubernetes deployments and services on the box".

Secrets and Painless Disaster Recovery

The catch with Talos is that the machine config (e.g. taloscfg/controlplane.yaml which describes your control plane nodes) also contains cluster PKI certs. You can't just check them in plaintext into git and say "I'm safe".

My workflow now is to have a script which pushes this file into a secret manager (e.g. 1password, google secret manager), and another script which can pull that down on new machines. Anytime I make changes to the node (I'm still running 1 box, so it's a control plane), I need to push my disaster recovery to a secret manager.

I then have all my configuration for the kubernetes cluster stored in git. I believe, on failure of this node, I could purchase a new node and have it running in ~1 hr. This really is a huge win compared to previous "home lab" setups I've done that basically were configure once, forget everything, never use after failure.

One of the patterns I use here is to make sure any manual configuration steps, if possible, are done via a job that runs after a service is stood up, e.g. for MinIO (blob storage) on the device - I have a job which will update its configuration:

apiVersion: batch/v1
kind: Job
metadata:
  name: minio-init-buckets
  namespace: caching
spec:
  template:
    spec:
      ...
      containers:
        - name: ...
          ...
          command:
          - /bin/sh
          - -c
          - |
            echo "Here is my initial setup commands..."
            echo "We first wait for minio to be reachable, then issue `mc` commands"

My new "Pet": Storage Choices

The biggest "gotcha" I had with Talos was actually choosing storage. For GitHub Actions - I wanted to re-use caching, and this is relatively easy with MinIO and gha-cache-server. However - I ran into a lot of issues because it turns out I did not understand kubernetes storage options well and persistent volume claims on Talos were odd.

The initial plan was to use Rancher's local-path-provisioner since I'm on a single node. The issue was that this was the limit of what I understand for kubernetes. I'd never had to dive past persistent volume declarations and mounts on services - to actually understand how the provisioning works.

The short answer is - it didn't.

This is where agents came in - both helpful and possibly a little overzealous. I was able to get them to debug the issue and offer me options, but at this point I realized I had some things I need to think about, else this turn into a "pet" infrastructure project:

If I want persistent volumes to be shared across nodes, I need some way to provision them so all nodes see them, some distributed file storage. Given I only have one node now, this is overkill, but it's now in the back of my mind
The fact Talos has read-only root (it's baremetal), means some expectations on mounts were wrong in the system. Gemini and Claude helped get this sorted, BUT I'm still not certain if their solution isn't bloated misinterpretation. It seems like the solution so far has been 100% in local-path-provisioner, but there's also a usage of extraMounts we had to make inside Talos machine config.

The good news is - the fix I have is repeatable on new machines. It works, I believe it will continue to, and I was able to get 90% of my private builds migrated from GitHub Actions remote runners to my custom runner, and I can now work on my personal projects without sinking money (beyond my subscription) into GitHub.

The bad news is - this may still become a pet project for me. There's a heavy temptation to go figure out how to optimize the kubernetes cluster - which was not the point of it.

Wrapping Up

Setting up your own kubernetes cluster is dead simple with Talos. I was floored at how fast I was able to spin one up, and how minimal / low overhead the cluster is for the machine. At the rate my hobbies are ramping up, I may purchase another refurbished box in a year or so to scale out capacity, as I'm already at ~50% of what I expect this box can handle at a sustained rate.

I would recommend anyone doing heavy hobby project activity spin up your own Talos box - and use declarative config for your setups.

Effective sbt

2013-06-11T00:00:00+00:00

Here's the slides from my sbt 0.13 talk today at Scaladays [Effective Sbt]:

Functional Programming Patterns for the Asynchronous Web

2013-02-13T00:00:00+00:00

NEScala happened this year, and it was amazing! Not only was I able to give a talk, we were also able to record a scalawags podcast.

For those who missed the talk, it was about how to script together a bunch of asynchronous APIs into a new asynchronous APIs. SO, generically, we have:

trait AsynchSubService {
  def getData: Future[Data]
}
trait AsynchSubService2 {
  def getData: Future[Data2]
}

What you want to do is create a new service that can call both asynchronously and non-blocking so that we don't suck up too many threads waiting for IO (or network response).

trait UberAsynchService {
  def getData: Future[(Data, Data2)]
}

SO, how do I accomplish this across a variety of goals? The example application is one that checks github statistics to see which projects are more likely to accept pull requests from non-committers.

Here's the example application. It implements the same github statistics service in two ways:

Directly using futures
Using external abstractions.

You'll notice that the code to implement the Future-only version is almost exactly the same as the "generic" version. There's a bit of difference in supporting code, but most of that is just an implementation of scalaz.

What is different is the testing strategies. The future-based code has to use "Await" for a result, which in my experience usually leads to flaky tests in some manner or other, while the abstracted service can implement single-threaded tests.

SO, the gain of all that abstraction is the ability to substitute different runtime execution models into our business logic. Regardless of whether or not you find the gain worth the cost, the underlying patterns of joining and sequencing computations are useful for anyone delving into Scala's Future library.

Here's a copy of the slides, and I beleive a video will be posted shortly.

Coding in Style

2012-10-14T00:00:00+00:00

Last week I re-gave my Scala 2.10 talks from Scalathon at both Twitter and Foursquare. While the talk still needs some more info in places, I figure it's a good helper for folks interested in 2.10.0, and now that we're in RC phase, even more relevant.

Coding in Style - Scala 2.10 features

This talk covers some basic scala style guidelines for what is emerging as 'canonical' Scala code in the standard library, as well as how styles are altering based on new Scala 2.10 features.

Some of these features are considered 'obscure' or 'non-standard', I still try to cover what they are and how best to use them. Some features, like Dynamic types, are considered dangerous and require a language import. I'd probably save those for a dynamic setting, such as using the Scala REPL for compiling/running your configuration, where runtime is fail-fast and won't affect correctness of production.

Other features are still experimental, like the Scala reflection library and macros. Here, I try to cover some example use cases, as we're trying to elicit feedback for a final design.

slides

Map-Reduce - Lessons Learned

2012-09-26T00:00:00+00:00

Yesterday I gave a talk to PittJug about Map-Reduce and a few patterns that are useful. While I plan to update the slides with additional drawings, better examples and removed typos, I thought perhaps those in attendance would appreciate being able to see them.

This talk is meant to provide very basic introduction to map-reduce and show the power of the abstraction through a few high-level patterns. Hopefully it guides you to the important decisions you have to make when using map-reduce.

Map-Reduce - Lessons Learned

slides

ScalaDays 2012

2012-04-22T00:00:00+00:00

I thought I'd do a quick recap of ScalaDays from my perspective. ScalaDays has always been a great event, and this one was no exception.

However, before going to the bits about my impressions and talks, I'll dump the links to the two talks I gave. Yes, that's two talks. I was grabbed in the hallway and told a speaker didn't show. Dumbly I said, "Why don't we find someone to go up and distract the audience before the next talk?" This, of course, meant that I was the someone to distract the audience. So, I hastily gave a version of the Effective Scala presentation (this one without all the nervous "right"s).

I also gave a talk on Binary Resilience. The first three slides tell the story of how defining binary compatibility is the hard. After that it's just a matter of highlighting the issues and showing some alternatives to avoid java.util.Date issues. (That is, an issue where you can't remove an obviously bad class because it breaks binary compatibility).

Effective Scala

video
slides

Binary Resilience

Recap

Well, in the interest of not wasting everyone's time, I decided I'd pick out three things that struck me. These aren't the three most important things, just the first three I resolved out of introspection. Maybe that means they're the three biggest things that hit me, may not. So, in no particular order:

SkillsMatter is awesome

SkillsMatter organized the event this year, and did a wonderful job. They're changing the game for tech conferences. They had a live podcast of the conference for people to watch talks and they posted many talks live almost immediately after shooting. This helped folks not at the conference keep up with releases and news and such. Not only that, the conference ran very smoothly and the action never stopped. The value of any conference is not just the talks, but the interaction with the speakers and attendees. I don't think live videos lesson the attendance of these events, since most people know the value is the people not the presentation. SkillsMatter figured this out and brings more interaction, even for those not at the conference.

If you're going to run a tech conference, talk with SkillsMatter. Not only is everyone (i met) who works for them nice/friendly/amazing, they have some rockin anime. If you're feeling left out of ScalaDays, go check out the talks

Scala usage is on the rise

There were a lot more people at the conference than in previous years. Not only that, lots of people actively using scala at work. I thought I was lucky to bag a typesafe job, but many of these people are doing really interesting work with Scala. I learned a lot just talking with the folks and seeing what was high prioirty. We all know compiler speed is important, but I would say that was the #1 thing discussed in hallway conversations. This has its own implications. With more of us using Scala, we're past the honeymoon phase and starting to dig into the real issues in the language and look for solutions.

The Scala Community is growing

There was a time where I felt I knew all the projects going on in the Scala community. This year's ScalaDays proved that I can't even delude myself into thinking that anymore. The diversity of talks, the quality of the talks and the complexity described in each talk showed how far into each topic the community is diving. We're well past "thin wrappers on Java" and "What is the cake pattern". The downside is we need more intro or entry-level talks. Given the size, I think we need some talks for people to learn how to use technologies, like lift, play, rogue, mongo, akka, scalaz, etc. The community is rich, and we need to teach people the areas they may not be able to follow.

Fin

Basically, Scaladays was pretty amazing. I spent most of my days talking with folks and my evenings catching up on talks I missed (thankfully published online). It was exhausting, but well worth it.

Effective Scala

2012-03-16T00:00:00+00:00

Well it looks like my keynote from nescala is now live! The nescala conference is was good times last year, and this year was no exception.

Note: If you want a drinking game, just take a shot everytime I say "right".

Iteratees

2012-02-29T00:00:00+00:00

There's been a lot of hype about Iteratees in the past, as well as several good articles. I've been playing around with them in my spare time, and they have some really neat properties. Before getting into those, let's cover the basics.

Iteratees are an immutable implementation of a Consumer/Producer pattern. A Consumer takes input and produces an output. A Consumer can have one of three states:

Processing
Finished
Encountered an Error

Let's look at a possible implementation.

sealed trait ConsumerState[I,A]
case class Processing[I,A](f: I => Consumer[I,A]) extends ConsumerState[I,A]
case class Done[I,A](result: A) extends ConsumerState[I,A]
case class Error[I,A](t: Throwable) extends ConsumerState[I,A]

The Processing state contains a function that takes input from a Producer, and returns the next consumer. Remember that these are immutable, so any state change must return a new object. The Done state contains the result of a given consumer, and the FatalError state holds a single java error representing what failed. The actual consumer class will look like the following:

trait Consumer[I,O] {
  def fold[A](f: ConsumerState[I,O] => Context[A]): Context[A]
}

The fold method is the only necessary method for a consumer. The fold method takes a function that operates on any consumer state and returns a value in the iteratee Context. The Context is the mechanism that is evaluating an iteratee. This is what lets asynchronous iteratees return values in a Future and operate on different threads. h Note: Consumers are called Iteratees in Haskell/Scalaz iteratee libraries.

Next is the Producer interface. A Producer should generate input values to feed into a Consumer. The producer 'drives' the Consumer to some final state, which is returned. Let's look at the interface:

trait Producer[I] {
  def into[O](input: Consumer[I,O): Consumer[I,O]
}

The into method takes a consumer, drives whatever data is in the Producer through the consumer, and returns the resulting consumer state. Note that this resulting consumer could still accept more input.

Note: Producers are called Enumerators in Haskell/Scalaz iteratee libraries.

Let's define a Producer.once method that can take a single value and feed it into an Consumer.

object Producer {
  def once[A](value: A): Producer[A] = new Producer[A] {
    def into[O](input: Consumer[I,O]): Consumer[I,O] = input fold {
      case Processing(f) => f(value)
      case other         => Consumer(other)
    }
  }
}

The new producer simply folds over the consumer's state. If it's a processing consumer, we feed the value and return the result. If the consumer is in any other state, we return that state in a consumer.

The code has one major problem. The fold method returns values inside the Context of the iteratee. That is, we can't peek into an consumer, unless we know where/how it is running and are in that context. This way, if the consumer is running inside a Future, all processing code will join with the consumer. Let's modify the code.

object Producer {
  def once[A](value: A): Producer[A] = new Producer[A] {
    def into[O](input: Consumer[I,O]): Consumer[I,O] = Consumer.flatten(input fold {
      case Processing(f) => contexted(f(value))
      case other         => contexted(Consumer(other))
    })
  }
}

This code assumes two methods. The contexted method takes any expression, and evaluates it in the iteratee context. The signature is def contexted[A](a: =>A): Context[A]. The Consumer.flatten method can take a Context[Consumer[I,O]] and flatten it out into a regular Consumer[I,O]. (Implementation is here).

Chaining Consumers

The biggest reason to use Iteratees is their composability. That is, we should be able to chain Consumers, such that one will run and then another. We should also be able to "zip" Consumers, such that two can run against the same input. To make this happen, we need to adapt our notion of consumer state a bit. First, let's ensure that 'leftover' input from one Consumer can be fed to the next when chaining.

case class Done[I,A](result: A, remainingInput: I) extends ConsumerState[I,A]

Now, the completed state has an additional value of the remaining input that was not consumed by the first iteratee. We can use this to implement our sequencing operation andThen.

trait Consumer[I,O] {
  def andThen[O2](next: O => Consumer[I, O2]): Consumer[I,O2] = Consumer flatten this.fold {
    case Done(a, remaining)  => contexted(Producer.once(remaining) into next(a))
    case Processing(k)  => contexted(Consumer(Processing(in => k(in) andThen next)))
    case Error(e) => contexted(Consumer(Error(e)))
  }
}

The algorithm for chaining is simple:

If the first consumer is done, feed the remaining input into the next consumer and return the resulting state.
If the first consumer is still processing, feed data into it, and delay creating the second consumer.
If the first consumer is in the error state, return that error state.

The fold pattern-match above reflects exactly that.

For those, who may be curious, the above is actually the implementation for flatMap on Consumer and will let us use Consumers in for expressions. To experiment, let's implement a few Consumers, starting with peek.

The peek consumer should take in an input value, return that as its done state and also put that value back onto its remaining value so that the next Consumer in the sequence will receive it again. Hence, we just peek at the value without really consuming it.

def peek[I]: Consumer[I,I] = {
  def handler(input:I) = Consumer(Done(input, input))
  Consumer(Processing(handler))
}

The peek consumer is pretty trivial. Let's implement a head consumer that will actually consume the first element.

def head[I]: Consumer[I,I] = {
  def handler(input: I) = Consumer(Done(input, ???))
  Consumer(Processing(handler))
}

This looks just like peek, except we need some way of not placing the previous input as the remaining input. To do this, we need a way of creating an "empty" input. While there are several ways to do this, the simplest is by adding an additonal level of abstraction. Let's change our definition of consumers as follows:

sealed trait StreamValue[+I]
case class Chunk(value: I) extends StreamValue[I]
case object EmptyChunk extends StreamValue[Nothing]
case object EOF extends StreamValue[Nothing]

case class Processing[I,O](f: StreamValue[I] => Consumer[I,O]) extends ConsumerState[I,O]

Now, there is a StreamValue trait that could be one of there things: A valid input, an empty input or a marker that input is at its end. The Processing state is updated to now take in a StreamValue and return the next consumer.

Let's reimplement peek and head now.

def peek[I]: Consumer[I,Option[I]] = {
  def handler(input:I) = input match {
    case Chunk(input) => Consumer(Done(Some(input), Chunk(input)))
    case EOF          => Consumer(Done(None, EmptyChunk))
    case _            => peek
  }
  Consumer(Processing(handler))
}
def head[I]: Consumer[I,Option[I]] = {
  def handler(input:I) = input match {
    case Chunk(input) => Consumer(Done(Some(input), EmptyChunk))
    case EOF          => Consumer(Done(None, EmptyChunk))
    case _            => head
  }
  Consumer(Processing(handler))
}

The peek and head consumers now need to return optional values, because they could receive an EOF input before having a chance to see the value. The only significant difference between these two is that head will return an EmptyChunk for remaining input, whereas peek will return the original input. Now that we have these consumers, we can stream them together.

def headTwice: Consumer[I, Boolean] = 
  for {
    p <- peek
    h <- head
  } yield p == h

This will run the peek Consumer and then feed remaining input to head. The headTwice observer should always return true, since peek does not actually consume the value from the stream.

The ability to sequence Consumers using for-expressions is one of their strengths. Complex protocols can be as simple as a large for-expression.

Zipping Consumers

Two Consumers can also be 'zipped' or run off the same input. This can be handy, e.g. when checksuming a file while parsing it. Let's define a ZippedConsumer:

  class ZippedConsumer[I,A,B](first: Consumer[I,A], second: Consumer[I,B]) extends Consumer[I, (A,B)] {
      // Helper method to push data through both iteratees.
      private def drive(input: StreamValue[I]): Consumer[I, (A,B)] =
        new ZippedConsumer(Producer.once(input) into first, Producer.once(input) into second)
      def fold[R](f: ConsumerState[I,(A,B)] => Context[R]): Context[R] =
        first.fold { state1 =>
          second.fold { state2 =>
            (state1, state2) match {
              case (Consumer.Done(a, i), Consumer.Done(b, _)) => f(Consumer.Done(a->b, i))
              // TODO - Both cases are errors....
              case (Consumer.Error(e,k), other)               => f(Consumer.Error(e, k))
              case (other, Consumer.Error(e,k))               => f(Consumer.Error(e, k))
              case _                                          => f(Consumer.Processing(drive))
            }
          }
        }
      
    }

The ZippedConsumer implements two methods. The first, drive is a private helper method. If both underlying consumers are in a processing state, this will feed incoming data into them and return a new ZippedConsumer with the resulting new consumers. This method is used later. The second method, fold is the core method of Consumers. This method in turn, folds over the underlying states of the two consumers and determines a new combined state. In the event that both are completed, then the ZippedConsumer enters its completed state, returning a tuple of both results. If either consumer is in an error state, the zipped consumer is in an error state. If either consumer is still processing, the zipped consumer uses the drive method to continue processing.

The ZippedConsumer can be exposed via a zip method. Now, given a consumer protocol consisting of msgHdr and bodyProcessor and a consumer sha1sum, we can construct a new Consumer which reads in a message and validates its sha1sum:

  val protocol: Consumer[ByteBuffer, Message] =
    for {
      hdr <- msgHdr
      (body, sha) <- bodyProcessor(hdr) zip sha1sum
      if sha == hdr.sha
    } yield Message(hdr,body)

Note: the above assumes we have a valid withFilter or filter method on Consumers. This method converts the Consumer state to an error if the condition does not hold.

There's a lot of flexibility with iteratees now that we can parallel or sequentially consume data. Since every consumer is immutable, the sky's the limit on how you compose them.

Stream Conversions

The Iteratee library is starting to look nice, but there's still another concern. Say, we have a Producer of byte buffers from files. However, in my program, I'd like to write consumers against lines of input. If I have a conversion from streams of byte buffers to streams of character buffers, and another from character buffers to delineated "lines", then I should be able to chain these on a producer and use consumers of lines.

For example, here's what a line-count algorithm should look like against a file:

  val bytes = read_channel_bytes(fileChannel, directBuffers=true)
  val chars = bytes convert charset.decoder()
  val lines = chars convert lineDecoder
  val lineCount = lines into counter
  lineCount fold {
    case Done(count, _) => contexted(count)
    case _              => sys.error("NO result!")
  }

In this example, read_channel_bytes constructs a consumer of byte buffers from the file. The convert method takes a stream conversion and returns a new Producer with transformed stream. So, first bytes are converted into chars by a charset.decoder(), then chars are converted into lines by the lines decoder. Finally, lines are fed into a counter Consumer, yielding a line count.

Let's first look at the count consumer, since that's the simplest:

  def counter[I]: Consumer[I, Long] = {
    def step(count: Long): StreamValue[I] => Consumer[I,Long] = {
      case e @ EOF      => Consumer(Done(count, e))
      case c @ Chunk(_) => Consumer(Procesing(step(count + 1)))
      case EmptyChunk   => Consumer(Processing(step(count)))
    }
    Consumer(Processing(step(0)))
  }

The counter consumer keeps track of the count in the step function. This function returns the next processing step.

Now, let's look at the interface for a StreamConversion. Note: In Haskell/Scalaz these are called Enumeratees.

    trait StreamConversion[I,I2] {
      def apply[O](i: Consumer[I2,O]): Consumer[I, Consumer[I2, O]]
      def convert[O](i: Consumer[I2, O]): Consumer[I, O] =
        apply(i).join
    }

A StreamConversion has two methods: apply and convert. The convert method, seen above, can take a Consumer of one type of input stream (I) and change it into a consumer of another type (I2). The apply method is the only abstract method of a stream conversion. This method constructs a Consumer that has a result of an underlying consumer. This is essentially what a conversion is. We construct one consumer, which adapts an incoming stream and feeds into the underlying consumer, eventually returning the result. The join method, found here is a convenience method in this situation. Most of the time, we don't really care how a stream is being converted, we just want to pretend there is a new consumer.

Finally, Producers can also have a convert method that takes a StreamConversion and modifes their output. Let's take a look.

trait Producer[A] { self =>
  def into[O](c: Consumer[A, O]): Consumer[A, O]
  def convert[A2](conversion: StreamConversion[A, A2]): Producer[A2] =
    new Producer[A2] {
      override def into[O](c: Consumer[A2,O]): Consumer[A2,O] =
        Consumer flatten (self into (conversion apply c)).result
      override def toString = "ConvertedProducer("+self+" by " + conversion +")"
    }
  }

This convert method simply takes any Consumer passed in and first wraps it by the stream conversion before pushing data into it. Since everything in this framework is immutable, it's ok to re-use the same stream conversions (decoders) over and over and over again.

For some example Stream Conversions, see the charset decoder, line splitter, word splitter

Communication channel between Producers/Consumers

So far, we've seen that iteratees are great for streaming data. However, what if the stream support random access? We need a way to communicate between Producers and Consumers. So far, a Producer can find out a consumer's state simply by folding on it, and a Consumer receives information from the producer via the StreamValue hierarchy. If we want the Consumer to be able to tell the Producer to move his input stream to a new location, we can accomplish this through some kind of channel in the ConsumerState.

Since a Consumer requesting some random-access seek, or other operation, be performed is probably in the midst of processing data, it makes sense to place a new message channel on the Processing state class. Let's take a look:

trait ProcessingMessage
case class Processing[I,O](next: StreamValue[I] => Consumer[I,O], 
                           optMsg: Option[ProcessingMessage] = None) extends ConsumerState[I,O]

Now, a Consumer can include an optional processing message for Producers. All of the ZippedConsumer, andThen and other combinator logic has to be changed, such that the processing messages can be propogated. However, when complete, we can create a new Consumer called "seekTo".

def seekTo(offset: Long): Consumer[ByteBuffer, Unit] =
      Consumer(Processing(in => Consumer.done((), in), Some(RandomAccessMsg.SeekTo(offset))))

This consumer will accept one input, and immediately be done, placing that input back into the stream. It also carries the RandomAccess.SeekTo message (not shown before). This message may or may not be handled by our Producer. However, if the Producer can support the SeekTo message, it will move the stream and then continue sending input. You can see an example implementation for file channels here.

This allows us to use for-expressions to send messages to the producer. e.g., we could write the following:

def findData(key: Long): Consumer[ByteBuffer, Data] =
  for {
    index <- readIndex
    hdr = index get key
    _     <- seekTo(hdr.diskLocation)
    data  <- grabBlob(hdr.size) 
  } yield data

This consumer will first consume the index at the head of a file, and then look for the location on disk to consume the remaining data. This is a very elegant way to nest seek commands with other 'normal' consumers.

Example

Hopefully, we've shown some insight in to the design of a good iteratee library here. I'd like to finish up with an example program from my current toy branch of Scalaz where I play with iteratees.

Here's a program with will perform the same calculations as the linux command line wc. That is, it will count words, characters and lines in a file. Note that if the file does not line up with the default encoding for a platform, this will barf error messages.

import scalaz.nio.std._
import scalaz.effect.IO
import scalaz.syntax.monad._
import scalaz.nio.buffers._

/** Ensures a file is closed. */
def withFile[A](file: java.io.File)(proc: java.nio.channels.FileChannel => IO[A]): IO[A] =
  for {
    stream <- IO(new java.io.FileInputStream(file))
    result <- proc(stream.getChannel) onException(IO(stream.close()))
    _      <- IO(stream.close())
  } yield result

// Mimics WC command-line
def wcLzy(file: java.io.File): IO[String] =
  withFile(file) { c =>
    val bytes: Producer[ImmutableBuffer[Byte,Read]] = 
      bytechannels.read_channel_bytes(c, directBuffers=true)
    val chars = Producer[ImmutableBuffer[Char,Read]] =
      bytes convert charsets.decoder()
    val wordCount: Consumer[ImmutableBuffer[Char, Read], Long] = 
      charchannels.words convert utils.counter
    val lineCount: Consumer[ImmutableBuffer[Char, Read], Long] = 
      charchannels.lines convert utils.counter
    val allCount: Consumer[ImmutableBuffer[Char,Read], String] = 
      lineCount zip wordCount zip utils.lengthCounter(_.remaining) map {
        case((lc, wc), cc) => "lines: %d, words %d, chars %s" format (lc,wc,cc)
      } 
    chars into allCount result
  }
def wc(file: java.io.File): String = 
  wcLzy(file).unsafePerformIO

In this example, the Consumers are being threaded through Scalaz's IO monad. This is not a necessity of the library and in fact, my goal for the asynchronous bits is to use futures rather than IO to better represent the execution model.

I hope that this has simplified your view of Iteratees. They really are an amazingly flexible, elegant system of processing inputs and producing outputs. I'm only at the beginnings of a decent Iteratee library, and already I can see the huge power in the ability to avoid re-streaming data through combining consumers.

It's my personal opinion that Iteratees demonstrate a lot of good design principles of functional programming.

Implementing class state in terms of a 'fold' operation (Note: We combine 'fold' with classical OO inheritance here).
Sequencing workflows inside a monadic context
Combinator operators to build large things from small things
Leveraging immutability to acccomplish the above.

One aspect that I didn't get to delve into much is "Leveraging category theory for minimal implemenations with rich APIs". Perhaps in a blog post covering Scalaz7.

SBT Cookbook Talk

2012-01-11T00:00:00+00:00

Well, I was in the SF/Bay Area and gave two talks on SBT while I was there. Here's the first, which had a few slide organization issues I corrected before the second (which I hope to post shortly, assuming the video makes it online soon.) This video is really high quality, so thanks to Markana, Klout, Michael Slinn and the Scala for Startups / SF Scala meetup folks!

Here's a copy of the slides (updated for the second talk, hopefully it's not confusing).

Scala Community Awards

2012-01-10T00:00:00+00:00

UPDATE - Akka has weighed in on the award!

Recently, a lot of Scala projects have moved to github! This has drastically reduced the overhead of accepting community patches, and the community has been wonderful in making Scala and its documentation better. The number of contributions and the care in commits shows that Scala has one of the best open source communities around!

I plan to show statistics for the increase in community contributions for Scala, but I'm waiting a bit until we have some statistical relevance in contributions. Right now, it's still to early to 'prove' anything, but the numbers appear to be tracking (if you ignore the blip known as Paul Phillips before he was a Typesafe employee).

However, in the meantime, I'd like to start a quarterly tradition of recognizing those who have been invaluable in their contributions to Scala and are not members of either Typesafe, Inc. or students/teachers at EPFL. I'm calling this the "Josh Suereth thinks you're a swell contributor Award". Copyright pending.

The Scala Documentation Site

The documentation site is the brainchild of Heather Miller at EPFL. It simplifies the process of creating and updating Scala documentation for all of us. The source code is available here and all merged pull requests are updated immediately thanks to github pages.

This new site gained many contributors almost immediately, and the community has been wonderful in making sure that the documentation is high quality, relevant and up to date.

While there are MANY contributors here (Thanks so much to all of you), two names stand out as distinguished contributors:

Daniel Sobral - For his continued vigilance in ensuring free documentation and translating his great Stack Overflow answers to the documentation format
Santiago Basulto - For his translation of the tour of Scala into Spanish.

The Scala IDE

The Scala IDE has undergone a lot of clean up and stabilization efforts from my coworkers at typesafe recently. However, the community has contributed some outstanding features that I make use of all the time. In particular, two names stand out:

Mirko Stocker - for his refactoring library and continued maintenance and improvements.
Matt Russell - for scalariform and its integration into the Scala IDE.

Scala

Scala is one of the most difficult projects to contribute to in the Scala community. Due to its widespread user base, and complicated interaction of compiler features, patches must be rigorously tested, reviewed and revisited (although adding tests is usually easy). Even with a high bar, Scala receives many great contributions and has a good community around it. Since this is the first swell contributor award, I'm going to pull some old commits that I feel are significant.

Todd Vierling - For his patch to drastically reduce the size of the collections library
Rex Kerr - For his murmur hash implementation and performance considerations

Also special thanks to Seth Tissue for ensuring great Java-Scala interaction and Jason Zaugg, who offers a free code review service for all of us contributors and has a fair number of impressive patches. We have lots of users contributing testing fixes and patches, too many to name all of them here. I hope to recognize you each individually over the coming quarters, so if your name was not listed, it's not because we don't appreciate you! Everyone who has a closed pull request here deserves a beer! (Just ask me sometime and I'll provide).

Akka

I was just going to cover the core Scala projects themselves, but my coworkers have nominated their picks for the Akka project as well! With no further ado, the "Swell Contributor" awards go to:

Derek Williams - For revamping Futures and Dataflow
Martin Krasser - For the akka-camel module (with its amazing documentation).

Fin

Kudos to all those selected! I hope to one to be as awesome as you all, and keep up the good work. If anyone has any nominations, feel free to send me an email for next quarter. I'll try to choose around April (ScalaDays 2011 anyone?)

Plus-colon extractor for collections.

2011-12-28T00:00:00+00:00

I often find myself writing the following for helpful head-tail decomposition:

package scala.collection

object +: {
  def unapply[T,Coll <: SeqLike[T, Coll]](
      t: Coll with SeqLike[T, Coll]): Option[(T, Coll)] =
    t.headOption map ( h => (h, t.tail))
}

And then using it for head-tail decomposition like this:

scala> val x = Vector(1,2,3)
x: scala.collection.immutable.Vector[Int] = Vector(1, 2, 3)

scala> x match {
     |   case x +: xs => println(x); xs
     | }
1
res1: scala.collection.immutable.Vector[Int] = Vector(2, 3)

rather than the 'classical' variant:

scala> x match {
     |   case Seq(x, xs @ _*) => println(x); xs
     | }
1
res2: Seq[Int] = Vector(2, 3)

I was curious if anyone else desired a parallelism to +: in pattern matching that mimics what's available for Lists (::). If you're like me, feel free to borrow the code, but also speak up! I'll find out about adding it into the standard library.

Note: We could also have a matching :+ operator for (x.init, x.last) decomposition as well. (Only really efficient for IndexedSeq, I would guess).

Scala Fresh is Alive

2011-12-07T00:00:00+00:00

It seems a lot of posts I've meant to write for a long time have been able to be used optimally in response to other blogs.

Todays post is brought to you by David Pollak's fragility post,

So let's talk about the problem of "Binary Compatibility(TM)" in Scala.

First off, David raises a big issue in Scala. One that most of us have seen. Scala's new language features necessarily break binary compatibility of the bytecode. Adding things like Specialization change Scala's standard library enough that you cannot use code compiled against older versions with the new version. This is a two-edged sword. The Scala experience should be as smooth as possible for all customers. Things like java.util.Date shouldn't survive endlessly our standard library. This requires a careful balance between breaking binary compatibility and advancing the language. So far, I'm pretty happy with the way this has been done, but would like to see things stabilize in the future. Let's take a look at what's happened:

Scala's new binary compatible releases

Scala 2.9.1 is binary compatible with Scala 2.9.0. If you compile code against Scala 2.9.0, you can use it with the standard library from Scala 2.9.1. This will hold for all 2.9.x releases. In Scala, binary compatibility will be at the bug-fix release. All 2.10.x releases will be binary compatible with previous 2.10 releases. This means that as long as your dependencies are compiling for a given Scala minor version, then you can continue to enjoy binary compatibility of libraries.

This was made possible by the freely available Migration Manager tool. A lot of us at typesafe use the publicly available version when developing to ensure binary compatibility of libraries.

Note: This was not true for a lot of David Pollak's Scala experience, and is the result of many of us petitioning for better binary compatibility. It's my opinion that this guarantee solves 50-80% of the problem.

What happened to Scala fresh?

I've begun work on what can only be called "Scala Fresh 2.0". That is, a place where community libraries will be built and deployed against the latest version of Scala.

This can serve two purposes:

Ensure that future versions of Scala do not break community libraries
Ensure that the core libraries of Scala are available for every major version of Scala.

You can find the project publicly available here. All of the work is public and will be migrate to the "scala" user on github once complete and ready for more contributions. Feel free to contribute or offer suggestions.

People may not realize but Scala Fresh failed because I failed Scala Fresh. I had very little time between a more than full time commitment at Google, writing commitments, and kids. This should not be an issue in the future, thanks to Typesafe taking binary compatibility seriously. I now have a prototype build that you can migrate your projects into.

My coworker likens this idea to linux distribution repositories. I think it's a decent way to think of it. I, and others, are working hard to ensure the ecosystem for every major Scala release is easy to use and stable. Migrating major Scala versions should be as difficult as finding deprecation warnings and removing them.

SBT cross releasing

Mark Harrah, recognized that Scala remains generally source compatible between major releases (2.8 to 2.9). A long time ago, Mark met me at a Panera in Anne Arundel, MD to discuss a mechanism of cross deploying libraries so that they were available for every release of Scala. This was well before the 2.9.x binary compatibility days and is still in heavy use within the community. It's been pretty widely adopted, but can only go so far. It's a good stop-gap solution that we, the community, can improve on.

Binary compatibility is a community effort. I know Typesafe is doing what it can with its resources, and I'm personally tackling as much as I can (probably trying to juggle too many balls). However, if you want to help, please email me!

The JVM is amazing, don't doubt its powers Finally, I want to clarify a confusion that a lot of people have about binary compatibility. That is, that traits are HUGE ISSUES FOR BINARY COMPATIBILITY ZOMG!!!. In actuality, traits are only slightly more dangerous than interfaces + implementation pairs.

In the code below, the method colored in red was added in a bug-fix release. It is binary compatible with the previous version.

trait Foo {
  def foo = "foo"
  *def bar = "NEW BAR IS NEW BAR!"*
}

That's right, adding methods, even with implementation, does NOT break linkage. Think about it. How did Java support JDBC 4.0 interfaces running JDBC 3.0 drivers? The trick is that trait-linkage errors happen at runtime when calling a method that has no linked implementation. This is the kind of magic that Java can get away with to ensure binary compatibility and it's just as useful in Scala.

Now, there is an issue where you can break binary compatibility. Again, red italics will denote new code

trait Bar {
  def foo = {
     *bar +* " foo"
  }
  *def bar = "NEW BAR IS NEW BAR!"*
}

NOW, the implementation of a previous method calls a new method. While this new method has an implementation in the trait, compiled code against the trait does not. This is akin to the following Java scenario:

interface Foo {
  public String foo();
  *public String bar();*
}
abstract public class AbstractFoo implements Foo {
  @Override public String foo() { return *bar() + "* foo" }
}

The abstract method implements one of the trait's methods, but not the new one. However, the implementation is modified to call the new method.

Scala's collections underwent a minor change in 2.10 to improve both class file size and unintentionally binary compatibility. That is, the collections now follow this pattern:

trait Traversable[A] //....
abstract class AbstractTraversable[A] extends Traversable[A] {}
class Vector[A] extends AbstractTraversable[A] with Traversale[A] //...

In this scenario, if your collection extends an Abstract collection (rather than its own parent class), you can remain binary compatble and allow new methods to be added to a trait and be called be the implementation. Vector is now "binary resilient" to changes in the Traversable trait. There are no guarantees that a trait is not used directly, so we can't take advantage of this when fixing bugs in collections in the Scala library. It's still a nice trick to use.

You do lose some flexibility doing this. Specifically, you can't have multiple class parents unless they're linear (like Java).

So, my point is that Binary Compatibility is a community issue. The JVM and Scala do what they can. I hope to see the compiler do more in the future. The Migration Manager tool from Typesafe can detect a lot of binary compatibility issues and has been crucial to ensuring 2.9.x Scala releases are binary compatible.

I'm trying to organize projects to help ensure all libraries for Scala can be released against all the major versions and remain binary compatible. The story is changing, and has done so rapidly over the past two years.

I'd love to see Lift, and others start adopting binary compatibility standards for there releases as well. This is not just a scala-the-library or scala-the-langauge issue. Libraries matter, just as they do in Java.

So while David's post highlights an issue, my response is:

Help us (the Scala community) out! We're going there, we're doing that.

Macro vs. Micro Optimisation

2011-11-29T00:00:00+00:00

So there's recently been a bit of hype about another Colebourne article: http://blog.joda.org/2011/11/real-life-scala-feedback-from-yammer.html

I'd like to respond to a few points he makes.

First

You should evaluate Scala and pay attention to its benefits and flaws before adopting it. Yes, there are flaws to Scala. Working at typesafe makes you more aware of some of them. We're actively working to reduce/minimize/get rid of these. In my opinion, the negatives of using Scala are peanuts compared to the postives of choosing Scala over Java. I think everyone should make up their own mind about this. Not everyone is going to choose Scala. I feel bad for those who don't, but I make no effort to convince you further than showing you the 40+ Open source Scala projects I have on github. It's a language with a lot to like and a bit to dislike.

Now, to the meat of what I want to say. Don't get lost in micro optimization when discussing programming.

The blog article discusses writing high-performance code in the critical path that has crazy performance needs. This is not your every day development. Scala loses a lot of benefits in this world, because features like closures have overhead on the JVM. Hopefully when Java adopts closures, this overhead can be alleviated, but it is there right now. The set of rules from the email is known to a lot of us Scala devs when writing a performance intensive section of code.

I'll reiterate a few to agree with:

Avoid closure overhead. If you can't get the JVM to optimize away and need to allocate a closure constantly in a tight loop, this can slow down a critical section. This is correlated with: Don't use a for loop. For expressions (and loops) in Scala are implemented as closures. There are some efforts underway to inline these closures as an optimization. This performance hit isn't a permanent one. As the compiler matures, you'll see a lot of optimization work happen.
Use private[this] to avoid an additional method. Scala generated methods and fields for val/var members. Using private[this] informs the compiler that it can optimize away the method. Again, while hotspot can optimize this away if you're in a very critical section, it may be a good idea to optimize. In fact, the whole promotion to fields and methods aspect of Scala classes deserves attention from anyone writing performance critical code. The rules are also there in Java, it's just that you probably have people that already know them.
Avoid invokevirtual calls. (Coda lists this as avoid Scala's collections). The true issue here is that invoke-virtual can make a difference in performance critical sections of code. Again, this is one I think we can improve with a few final declarations and maybe an annotation or two.

Here's the big missing point in all that feedback. This is for performance critical sections of code, not for general purpose applications. I think when it comes to performance bottlenecks, you need to pull out the stops and optimize the heck out of your apps. Look at the Akka framework from typesafe. Akka uses a lot of "dirty" scala code to achieve high non-blocking concurrent performance. It's one of the most amazing libraries, and the code is pretty low level. It also supports a very high level of abstraction when writing your application. It uses PartialFunctions (which are sort of a combination of pattern matching and closures) and traits, both of which have some overhead. However the resulting application is fast. Why? The inner loops are fast and optimized and the application architecture can be optimised. In a high level language, you can take advantages of designs that you would never execute in a lower-level language because the code would be unmaintainable.

You see, most of us like to be able to read and understand what our code does. Scala has some features where you can write very expressive code in a few lines. After getting over the initial hump, this code is pretty easy to maintain. If I were to write that same code in Java, it would look odd, confusing as hell, and no one would want to maintain it.

I learned this lesson at Google.

Google has a pretty stringent C++/Java style guide. One that is tuned for high performance servers. The C++ style guide is public. The style guide frowns on things like the use of 'smart' pointers that 'garbage collect' because you can't be sure that GC will happen at a critical moment in the code.

Google also has a high performance Map-Reduce library. This thing is pretty amazing, with all sorts of crazy cool features like joining data on the map part of the map-reduce. The library followed all the google coding conventions and is generally held up as a piece of awesome software, which its.

However, writing applications with the Map-Reduce library was less than idea. I could write pretty clean code with the MR library. My Mappers were pretty light weight and minimalistic. Same with my reducers. You'd then string together a series of map-reduces in this crazy guitar inspired "patch it together" configuration and the thing would be off to the races. The downside is that the throughput of this kind of processes was sub optimal.

It wasn't anything inherent in the libraries. The libraries were fast and good. The APIs were optimised for high speed performance. However, writing optimal architectures in the framework was tough. If I wanted to do any crazy performance features, like combining map functions and reduce functions in a single map reduce run, the code got ugly fast. Not only that, it was very difficult to maintain because of all the odd bookkeeping. I have 10 outputs ordered by key, this is the one writing to that file right? KRAP.

I spent 6 months writing applications of this nature and feeling like there had to be something better. I started on a venture to write something, and as usual found that someone else already had. That's when I found out about Flume and met Craig Chambers.

FlumeJava was a map reduce library that was gaining a lot of traction at Google, but I had heard a lot of complaints about its API. Around the time I was looking at it, Flume C++ was coming into existence. My team was one of the alpha users of the C++ variant.

The C++ API was a thing of beauty. Like its Java cousin, it treats data a set of Parallel Distributed Collections. You can see Daniel and my talk on the Scala equivalent here. Converting Mappers and Reducers into this API was pretty simple. You could even do it directly by just annotating your Mappers and Reducers with types.

My team saw a 30-40% reduction in code size for some of our map-reduce pipelines. We had an 80% reduction in code for unit tests (which was by far the most amazing benefit). Not only that, the Flume library optimizes the pipeline by performing all sorts of dirty map-reduce tricks for you. In some cases we dropped a map-reduce call or two. In the worst case, we had the same number we had started with.

What was wrong with the library? It violated almost every style rule for C++ at google. That's right, smart pointers, classes with inline member definitions the works. I loved every second of it. Why? Because I was getting stuff done faster than before with less code and the pipelines ran faster. It was a crazy win.

The startup time might not have been as optimal as it could be. Flume ran an on-the-fly optimization before running your pipeline. That was being improved all the time with neat tricks. Things I didn't have to write to watch my app speed up, both runtime and startup time.

The key here is that the designers of Flume weren't focused on micro optimization but macro optimization. The inner guts of the library used the very fast and efficient Map-Reduce library and the wrappers they had were as efficient as they could make them. My code did not have to follow these rules, because the core loops were fast. When it came down to it, my code used high level concepts. Something akin to a closure and for-expressions in Scala (Note: The Scala equivalent did use for-expressions and closures with no noticeable performance hit).

There are times when writing Scala code requires care and optimization in the micro level. However, don't lose the forest for the trees. Think about the entire application architecture. Scala will open up possibilities for writing code that you'd never dream of trying to maintain in C++ or Java. Take Akka as a shining example.

And when you need the performance, listen to those techniques. Viktor Klang can probably give you a ton more. I know I learn more every time I talk with him.

SBT and Plugin design

2011-09-16T00:00:00+00:00

Sbt 0.10 brings a lot of power to the table. SBT 0.10 switched from a class/inheritance based build system into a more functional approach. For those who aren't familiar, here's the quick spiel on SBT.

Basics of SBT

In SBT, a project is composed of Setting[_] values. A Setting is sort-of a name-value pair (more of a name computation pair). In the SBT command-line you can type the name of a setting and get its value (or computed value). For example, test is a task in SBT that you can type in the command line. The setting's computation is executed and the value returned. This setting may depend on other settings for its value.

SBT provides a simple way to construct a project. In the root directory, any *.sbt file is compiled to Setting[_] values. A Setting[_] is two things: A name (Key + Scope) and a Value (or computation, called Initialize in SBT). One can construct a Setting[_] via the SBT dsl:

sourceDirectory in Compile <<= baseDirectory apply { dir =>
  dir / "src" / "main" / "scala"
}

In this example, the sourceDirectory Key (name) is assigned an Initailzation (value/computation). The <<= operator is used to construct a Setting[_] by joining a Key and an Initialiation. In the above example, the Initialization is constructed to pull the current value of the baseDirectory key and modify it for the value of the sourceDirectory key.

Note: The apply method is used on baseDirectory because both baseDirectory and sourceDirectory are SettingKey[_]s. SBT distinguishes between three types of Setting[_] values: Setting, Task and InputTask with corresponding SettingKey, TaskKey and InputKey "name" types. The three are distinguished as follows:

SettingKey - Something that is computed once on project load (or reload). like a val.
TaskKey - Something that is computed each time it is called, like a def.
InputKey - Something that takes user input to perform its task.

Configurations

SBT uses a configuration matrix to define the same task against different configurations. For example, SBT defines a task for compiling Scala code called compile. This mechanism has a bunch of required settings. However, it wouldn't be DRY to repeat all these settings for compiling test code as well. So instead, SBT defines the same settings in two different configurations, on called Test and another called Compile. To compile just tests in SBT, you can prefix a task with its configuration, e.g. test:compile.

Plugin Design

So what does this have to do with plugin design? SBT plugins need to integrate Setting[_] values into a build without conflicting with SBT default settings and other plugins. To complicate matters, SBT imports all the members of plugin classes into scope of a project using a wildcard import. This means all the plugins you use could have conflicting names that step on each other. Combined with potentially conflicting key names, plugins need to be very careful with how they define things.

Having worked on several plugins recently, I'd like to outline a strategy that I think achieves a certain elegance in definition and usage, as well as the safety one wants from a plugin.

The basic pattern is as follows. Define an object with the name you want for your plugin inside the Plugin class. For example, if I want a xsbt-suereth-plugin, I would define the following:

import sbt._
import Keys._

object SbtSuerethPlugin extends Plugin {

  object suereth {
     // Your code here
  }
}

Inside of the suereth object I hide all my definitions and code. This isolates my plugin from other sbt plugins, as long as no one names their methods "suereth".

Next, let's define a new Config object that we can use to protect our keys from other plugins.

object suereth {
  val Config = config("suereth")
  // Your settings here
}

The Config also shares the name of the plugin, so in the command line tasks and settings can be run using suereth:<your-task-here>.

The next step is define whatever custom keys your plugin will use. Let's create a blog key.

object suereth {
   ...
   val blog = SettingKey[String]("blog", "location of the blag") in Config
}

The key is automatically placed into the suereth configuration using the in method. This has two benefits:

When defining the Initialization for a Setting[_], there's no need to continue writing blog in Config. Users of the plugin can directly access suereth.blog without needing to specify suereth.blog in suereth.Config. Note: You can also reference SBT keys in your configuration by writing: val sources = Keys.sources in Config.

Finally, we can provide default values/computations for tasks and settings in our plugin. By convention, calling these settings is a good idea.

object suereth {
   ...
   lazy val settings: Seq[Setting[_]] = Seq(
      blog := "http://suereth.blogspot.com"
   )
}

Notice how the keys are access directly but are actually in the appropriate config matrix. This helps defining your plugin source code, but will also help users of your plugin. Let's look at what a build.sbt file will be for this plugin.

seq(suereth.settings:_*)

suereth.blog := "http://blog.typesafe.com"

Notice how the settings for this plugin are completely namespaced by the suereth object. We've tied the concept of a "configuration" axis for keys with accessing values in an object.

I find this mechanism of defining plugins both helpful from a development perspective and a user perspective. Curious to hear what others think.

Leaving Google for Typesafe

2011-07-22T00:00:00+00:00

So, as some of you may have found out already, I'll be leaving Google effective July 22nd and moving to a new role at Typesafe. Some of you may be wondering why, so I'm offering my reasons in a blog post for the curious:

Scala has been gaining lots of traction in the market space recently. I think Scala is and has been reaching its tipping point. I'm hoping to contribute everything I can to sure it remains a viable language for everyday development.
While Google does have 20% time to work on individual projects, this correlated to about a day a week to work on all things Scala within Google. Rare was the opportunity to do anything significant or to contribute back to the community, due to time constraints. While I was able to accomplish one pretty good thing, aligning my work with my passions seemed like the best thing to do.
It's part of a new weight loss program called "pay for your own food". This was tempered by my wife's purchase of a new grill for my birthday, which has seen heavy use in the past few weeks.

So, while I see Scala's future as bright and rosy and I can't explain how excited I am to start at typesafe, I'd also like to do a little reflection on Google and some of what makes it a great company.

Google cares about employees

This is not a lie. Google, as best as it can, tries to mean what it says. At my previous companies you'd hear "I'd love to do this, but my hands are tied". At Google, you often hear "I don't know, but let's ask someone who does" which usually turns into "Yes, go ahead". It was rare that something I asked for was denied. My perception of what was acceptible is an entirely different thing. At Google, there are people who think what you think (at over 10k engineers, it's guaranteed one will probably agree with you about something), and that engineer might have also tried to accomplish the same task as you. Go ask about it and find like-minded individuals. You'll be surprised how much

Google tries to do what they say.

The other side to this coin is that the corporate entity and by that I mean the higher ups like to do things for all Google employees. Like bonuses. They really do happen, and they really do try to treat you like a human being.

The other side to this coin is the interview process. I've seen so many negative posting about the interview process. Well, I'm here to dispell a few of those. Google's interviews will challenge your technical abilities. If this annoys you and you don't take the job, then good. You're probably not the type to herd cats deal with politics/relationships involves with a large developer base. If you can't cut the interview, then it gives you a reason to go back and practice. However, more importantly, Google would rather turn away a good candidate than hire a bad one. This mindset is very impressive. It really makes a huge difference on the ability of teams to accomplish things. I knew every employee I worked with at google was reliable to get work done, not something that's always the case externally.

In the future I know my interview style will change. No more will interviewees be allowed to just talk about experiences without showing some code. It's amazing some of the depths you can learn here. It's even great when an interviewee fails to answer the question correctly because you learn their thought process. I know for me, a few candidates who struggled were the ones I wanted sitting beside me coding, more so than the ones who blazed through a question but had a more 'better than thou' aura. So, the point here is care about your employees and care about who you hire. Your company will be in far better shape if you do this.

Culture is faster than process

Google tries to instill a culture of 'doing the right thing' its employees, rather than outlining software process to a T. There are a few 'inconveniences' that exist, but other than mandatory code reviews, a lot of the process is up to the team to do what's best. The other side to this coin is the corporate culture tries to help define and change what's best. It's amazing how one executive making a statement in an all hands meeting can suddenly alter the perceived "best way to code" and get the company to move. It's also surprisingly hard to change culture once it has been really embedded into the engineers, which is the danger. A sense of 'right' in the ways of writing software can be beneficial, but can also turn into anti-patterns if not tempered. I'd love to go into more details here, just feel free to bug me.

A Generic Quicksort in Scala

2011-06-16T00:00:00+00:00

So, I decided to create a quicksort algorithm in Scala that showcases how to write 'generic' collection methods. That is, how can we write an external method that works across many types of collections and preserves the final type.

Well, here's how you do it:

import scala.collection.SeqLike
import scala.collection.generic.CanBuildFrom
import scala.math.Ordering

object QuickSort {
  def sort[T, Coll](a: Coll)(implicit ev0 : Coll <:< SeqLike[T, Coll],
                             cbf : CanBuildFrom[Coll, T, Coll],
                             n : Ordering[T]) : Coll = {
    import n._
    if (a.length < 2)
      a
    else {
      // We pick the first value for the pivot.
      val pivot = a.head
      val (lower : Coll, tmp : Coll) = a.partition(_ < pivot)
      val (upper : Coll, same : Coll) = tmp.partition(_ > pivot)
      val b = cbf()
      b.sizeHint(a.length)
      b ++= sort[T,Coll](lower)
      b ++= same
      b ++= sort[T,Coll](upper)
      b.result
    }
  }
}

I've chosen a somewhat imperative approach to the problem. The quick sort algorithm is split into two parts: The first checks for small collections and returns them, the second picks a pivot and decomposes the collection into three pieces. Theses pieces are sorted (if necessary) and pushed into a builder, cleverly named "b". This "b" is given a hint to expect the entire collection to eventually wind up in the built collection (hopefully this helps performance). Finally, after passing the three partitions to the builder, the result is returned.

The magic here is in the rather confusing type signature:

def sort[T, Coll](a: Coll)(implicit ev0 : Coll <:< SeqLike[T, Coll],
                             cbf : CanBuildFrom[Coll, T, Coll],
                             n : Ordering[T]) : Coll

Let's decompose this a bit. T is the type parameter representing elements of the collection. T is required to have an Ordering in this method (the implicit n: Ordering[T] parameter in the second parameter list). The ordering members are imported on the first line of the method. This allows the < and > operations to be 'pimped' onto the type T for convenience.

The second type parameter is Coll. This is the concrete Collection type. Notice that no type bounds are defined. It's a common habit for folks new to Scala to define generic collection parameters as follows: Col[T] <: Seq[T]. Don't. This type does not quite mean what you want. Instead of allowing any subtype of sequence, it only allows subtypes of sequence that also have type parameters (which of course, is most collections). Where you can run into issues is if your collection has (a) no type parameters or (b) more than one type parameter. For example:

object Foo extends Seq[Int] {...}
trait DatabaseResultSetWalker[T, DbType] extends Seq[T] {...}

Both of these will fail type checkking when trying to pass them into a method taking Col[T] >: Seq[T].

To get the compiler to infer the type parameter on the lower bound, we have to defer the type inferencer long enough for it to figure this out. To do that, we don't enforce the type constrait until implicit lookup using the <:< class.

The type parameter: Coll <:< SeqLike[T, Coll] Ensures that the type Coll is a valid Seq[T]. You may be asking why this signature uses SeqLike rather than Seq.

GOOD QUESTION!

SeqLike differs from Seq in that it retains the most specific type of the sequence. This one of the magic tricks behind Scala's collections always returning the most specific type known. That type is embedded in SeqLike. To ensure that we can return the most sepcific type, we can Capture Coll as a SeqLike with Coll as the specific type. This means that filters, maps, flatMaps, partitions should all try to preserve the type Coll.

The last implicit parameter is the cbf CanBuildFrom. Because we don't know how to construct instances of type Coll (because we don't know the type Coll at all), we need to implicitly receive evidence for how to construct a new Coll with sorted data.

Let's look at the result:

scala> QuickSort.sort(Vector(56,1,1,8,9,10,4,5,6,7,8))
res0: scala.collection.immutable.Vector[Int] = Vector(1, 1, 4, 5, 6, 7, 8, 8, 9, 10, 56)

scala> QuickSort.sort(collection.mutable.ArrayBuffer(56,1,1,8,9,10,4,5,6,7,8))
res1: scala.collection.mutable.ArrayBuffer[Int] = ArrayBuffer(1, 1, 4, 5, 6, 7, 8, 8, 9, 10, 56)

scala> QuickSort.sort(List(56,1,1,8,9,10,4,5,6,7,8))
res18: List[Int] = List(1, 1, 4, 5, 6, 7, 8, 8, 9, 10, 56)

scala> QuickSort.sort(Seq(56,1,1,8,9,10,4,5,6,7,8))
res: Seq[Int] = List(1, 1, 4, 5, 6, 7, 8, 8, 9, 10, 56)

You may be asking why I've chosen Seq instead of GenSeq or GenTraversable or even GenIterable. No particular reason, besides I wanted a reasonable assurance that the collection author expects the .length and indexed access methods to be called.

So, what are the lessons to be learned here?

Use *Like subclasses to preserve the specific collection type
Defer inference using <:< to give the type checker a hope of succeeding
Provide @usecase comments for scaladoc so users won't get distracted by typesafe details.

At lest, IMHO, this is the current way of creating generic collection code.

Scalatypes Podcast

2011-06-12T00:00:00+00:00

Well, Daniel, Yuvi and I started podcasting interviews and discussions on Scala. Here's the first interview with Paul Phillips. We have a lot more content to publish, but mastering audio still takes a backseat to day jobs and writing Scala in Depth. Let us know what you think!

Parallel Distributed Collections API

2011-06-03T00:00:00+00:00

Here's my ScalaDays 2011 talk that I gave with Daniel Mahler.

Annotate your type classes

2011-03-15T00:00:00+00:00

This is a real quick post saying that as of 2.8.1, everyone should be annotating their type class traits for better error messages. Obviously, those of us who are supporting 2.8.0 or earlier are left out.

Here's an example REPL session:

Welcome to Scala version 2.9.0.r24384-b20110305105029 (OpenJDK 64-Bit Server VM, Java 1.6.0_20).
Type in expressions to have them evaluated.
Type :help for more information.

scala> import annotation.implicitNotFound
import annotation.implicitNotFound

scala> @implicitNotFound(msg = "Cannot find Serializable type class for ${T}") trait Serializable[T]
defined trait Serializable

scala> def foo[X : Serializable](x : X) = x
foo: [X](x: X)(implicit evidence$1: Serializable[X])X

scala> foo(5)
:11: error: Cannot find Serializable type class for Int
       foo(5)
          ^

The annotation has very little documentation, but apparently supports ${TypeName} templates to inject the names of types the compiler is looking for. I believe you can also use this implicit on types with no type parameters.

Happy hacking everyone!

NEScala 2011 - Implicits without the import tax

2011-02-18T00:00:00+00:00

Here's the slides for todays talk for those who wish to follow along on their laptops.

On Macro-Composition vs. Micro-Composition and Social software development.

2010-10-15T00:00:00+00:00

I was talking to a co-worker today about using C++ templates to compose functionality in objects. I also mentioned how you can encode type traits using templates. My Coworker, conveniently aliased as "Bob", asked why you would want to use type traits rather than defining an interface directly and wrapper objects.

In response, I gave rather lame answers, which I intend to amend here. The basic gist is that I feel a lot of Object Oriented languages don't pay enough attention to macro composition of components in a system.

First

I'd like to state that In a lot of my code, there are a set of classes that define the format of data, or the "data layer". This layer has a certain "mind-share" around it. It is "owned" be a group of individuals and the portion of code they own. This data is passed around to other teams of people.

Note that I specifically state "other teams" and not "other components". In a lot of software shops, we think of software being comprised of sub-components of a system, but I tend to find that the sub-components of a system are also defined by the teams that work on them. If a particular team works on several sub-components, these components tend to be cohesive and reuse a lot of code. The boundaries between these teams are important to pay attention to.

It's the boundaries where we tend to do a lot of conversions from one data definition to another. Each team wants to wrap its own "needs" around the ontology of classes. So in an OO system, this usually means wrapping data from one set of classes to another, or bunch of external methods that manipulate the data from another team. Because of this natural bent in programming, it's sometimes necessary to be able to adapt other people's specific classes into your own ontology. This is where type classes come in.

I've blogged about type classes/type traits before, so I won't bore you again. The basic gist of the idea is to define an interface for your method. For example, I'm writing something that works with tree-like data structures where each parent node could have multiple children. I define a type trait as follows:

trait TreeLike[NodeType] {
  def isParent(node: NodeType): Boolean
  def children(node: NodeType): Iterator[NodeType]
}

This allows me to take any "node" type and traverse it. Now, Let's write an algorithm where we want to see if a condition holds true down one "branch" of the tree to a leaf for every Node of this path. We can use our TreeLike type trait.

final def hasValidDepthPath[T: TreeLike](node: T, validator: T => Boolean): Boolean =
  if(validator(node)) {
    val helper = implicitly[TreeLike[T]]
    !helper.isParent(node) || helper.children(node).exists(hasValidDepth(_, validator))
  } else false

The hasValidDepthPath method uses the TreeLike type trait rather than a different class. This lets us use non-treelike structures as trees. For example, we can turn a list into a tree where each branch is the same list but missing one item of the index.

scala> def dropIndex(x : List[Int], idx : Int) = x.zipWithIndex.filter(_._1 != idx).map(_._2) 
dropIndex: (x: List[Int],idx: Int)List[Int]

scala> dropIndex((1 to 10).toList, 2)
res2: List[Int] = List(0, 2, 3, 4, 5, 6, 7, 8, 9)

implicit val decomposeList[T] = new List[T] {
  def isParent(node : List[T]) = node.length != 1
  def children(node : List[T]) = (1 to node.length).map(dropIndex(node, _))
}

Now we can use a List of items as an input to our hasValidDepthPath method.

Now to my coworkers question: Why would I want to add this extra layer?

My answer is that it helps when working with "ontologies" from different teams. That is, each team has their own way of describing data. This mechanism allows our functions to mostly ubiquitously deal with these differing items, writing adapters for each ontology. This makes more sense when your algorithms deal with many different input types. In fact, I think it makes the most sense for 'peripheral' code. That is code that interacts with other teams.

Now, for the second part of the rant.

I feel a lot of languages don't support macrocomposition well. A great example of this is the existence of the Spring Framework. Spring allows you to define interfaces, objects and their dependent components. You can then build configurations where you create specific objects, bind them to names and inject them into other objects. You are essentially defining the components of your system at a micro->macro level. Spring configuration allows you to focus on building the various components as needed at an object-by-object level until you are configuring objects that comprise major subcomponents of your system. Google Guice allows you do to a similar thing in Java using annotations.

The important thing here, IMO is that there is a distinction between defining components/classes and composing them. We define a component that uses others via some interface in one section, and we combine these components together in another.

In C++ we can also accomplish this to some extent using type traits. In C++ type traits usually take the form as templated structures. Composition is then done using template arguments. The best example of this is the STL and its usage of traits for allocation, comparison, hashing, etc.

In C++, we could define a service as follows:

template<typename Logger, typename Threading>
class Service {
 public:
   typename Threading::ScopeLock ScopeLock;
   void MyMethod() {
     Logger log;
     Threading threading;
     {
       ScopeLock scoped(mutex_);
       log.info("OMG!!!");
     } 
  }
 protected:
   typename Threading::Mutex mutex_;
};

This service is really really simple, but shows how to compose with type traits. The Threading interface is any class that defines a Mutex typedef (for mutexes), a ScopeLock typedef (for locking mutexes within a scope). This class could be reused across various threading libraries, by providing something with the correct type definitions. I would expect to see, somewhere else in code, the following "Compositional" Code:

// This code replaces a Spring XML configuration.
typedef Service<PthreadThreading, DefaultLogging> MyService;
typedef SomeHigherLevelService<...., MyService> MyHigherLevelService;

The nice aspect here is that I can use typedefs to compose "micro" components of my system and aggregate those into "macro" components. It's the same technique used for both.

In Scala, I've used the following tactic: Define a trait which implies composition. This method has a lot of boilerplate, so bear with me.

I want to define a Logger service that traits can use. To do this I define the generic Logger trait and a generic HasLogger trait.

trait SimpleLogger {
  def info(msg : String) : Unit
  def warn(msg : String) : Unit
  def fatal(msg : String) : Unit
}

trait HasSimpleLogger {
  val logger : SimpleLogger
}

This SimpleLogger provides no advanced features, just three levels of logging. The HasSimpleLogger trait defined an abstract value for the Logger. It does not instantiate the logger, it merely denotes that a class will have one (i.e. needs to be composed with one). Later, we will define some service:

trait MyService extends HasSimpleLogger {
  def serviceMethod() {
    logger.info("MyService: Someone called my method")
  }
}

This is how we define each "abstract" component. We extend each composition trait (i.e. HasFoo) and then utilize the components as necessary. Again, there will be code somewhere that composes these components together. In Scala, the mechanism of composition is the trait not a class instance:

trait MyComposedService extends MyService with HasDefaultLoggerImplementation

trait HasMyComposedService extends HasMyService {
  val mySerivce = new MyComposedService {}
}

Why do we have to compose in this manner? Because we'd like to have more control when composing large systems together. This mechanism allows us to "reconfigure" a MyComposedService with additional behavior or alternative subcomponents without having to redeclare all the subcomponents.

The downside to this approach is that it is full of boilerplate. However, notice again the distinction between defining components and aggregating behavior into our final system.

I'm of the opinion that these two functions are key to creating reusable software and they need to be simple and elegant to be used. I've yet to see a large system where I felt the composability ever reached the ideals touted by J2EE of old. In fact, I feel EJBs are horrible for microcomposition. The Spring framework further proved this by provided a microcomposition layer below Session beans that became quite popular.

In any case, these are my ramblings and musings on development. I'd love to hear back from you. I'm of the opinion that after becoming a proficient software developer, the solutions I write solve three issues:

A complex new algorithm to solve a problem "never before solved" (at least by the company I work for), or some foundational library.
Munging data between social components of software, or differing ontologies
Composing components to form a complex system.

I feel a lot of languages focus on (1). The languages I tend to prefer allow me to do (2) and (3) with ease.

What's the state of Jigsaw

2010-08-17T00:00:00+00:00

I hadn't been paying close attention, but recently a few emails questioning the future of the jigsaw project have been sent to the jigsaw-dev mailing list. The last check-in posted to the mailing list was on June 8th, and was the tail-end of a flurry of check-ins from the jigsaw team.

The whole thing is rather curious, but without any specific word from Oracle, it's all speculation. I have my own guesses, but it's probably best I keep those to myself for now.

Update

It appears that after a few months of inactivity, whatever decisions at Oracle needed to be made were made. Jigsaw appears to be under active public deveopment again.

Monkey Patching, Duck Typing and Type Classes

2010-07-13T00:00:00+00:00

Somewhat recently, there have been a bunch of articles written about Scala and how you can create something that looks like Haskell's type class feature. I'm not going to duplicate these posts, but I am going to go beyond just the mechanics of creating a type class and discuss how to utilise some of Scala's advanced features with type classes to create some pretty robust code. A lot of the post borrows ideas from my upcoming book, Scala In Depth. The examples come from the Scala ARM library.

First off, let's assume you haven't read anything at all about Scala's type classes, but you do have a Ruby background. In Ruby, you really loved duck typing. It's the best. I can just use methods and pass objects around and things work. If a class doesn't have a method I need, I can either rely on some method missing magic, or just monkey patch it. To those who don't know Ruby, monkey patching is the ability to add a method to an existing class on the fly.

What if I told you that in Scala, for a small penalty, you could create methods that are flexible, like duck typed methods, and 100% compile-time type safe. Not only that, you're also able to monkey-patch types as needed. Don't believe me? Let's look at some of the fun in the Scala ARM library.

The motivation behind the ARM library is to create a flexible, robust (from exceptions) and simple-to-use automated-resource-management library that can greatly simplify things like JDBC or I/O in Scala. The standard introduction to ARM in Scala is the withResource method outlined below.

def withResource[A <: { def close() }, B](resource : => A)(f : A => B) = {
  val r = resource
  try {
    f(r)
  } finally {
    r.close()
  }
}

The with resource method takes a by-name argument that returns an opened resource, and a function that manipulates the resource. We ensure that the close method on the resource is called after the manipulation function f. We've declared the type of the resource using structural typing. This means the compiler will accept anything that defines a close method. We can then utilize withResource as follows:

withResource(new FileInputStream(new File("myFile.txt"))) { stream =>
  // Code using stream and returning something
}

So, this code is already giving us a form of duck-typing (called structural typing in Scala). What about monkey patching? This is where using the type class pattern can help. Also note, that I prefer calling the type class pattern from haskell the "type trait" pattern in Scala. This could be confusing, as C++ has a type trait pattern as well, however the fact that I use traits to define type classes is just as confusing. In any case, please think of type traits and type classes, for the rest of this blog article as the same thing: A manifestation of type classes in Scala.

Let's modify the withResource method to use a type class instead of a structural type.

trait Resource[R] {
  def close(r : R) : Unit
}

Here is our Resource type class. It defines a single method close. We can now modify our withResource method to utilise this:

def withResource[A : Resource, B](resource : => A)(f : A => B) = {
  val r = resource
  try {
    f(r)
  } finally {
    implicitly[Resource[A]].close(r)
  }
}

Notice two things here. First, we use the context-bounds constraint on the resource type parameter (A : Context). This means that we have to look up our implicit type trait using the implicitly method. We could, instead, give our method a well-defined implicit. For now, we'll use the context-bound syntax as it's quickly starting to denote type traits in Scala.

If we attempt to call this method as things stnad, we'll get an implicit lookup error:

scala> withResource(new java.io.StringReader("HAI")) { input => () }
:9: error: could not find implicit value for evidence parameter of type Resource[java.io.StringReader]
       withResource(new java.io.StringReader("HAI")) { input => () }
                                                 ^

Now let's think through what we've done. Any type can be used in the withResource method as long as an implicit is available for the given type in the implicit lookup resolution. The simplest way to create an implicit for usage is to provide one directly. This is easy to do in the REPL:

scala> implicit object stringResourceTrait extends Resource[java.io.StringReader] {
  override def close(r : java.io.StringReader) = r.close()
}
defined module stringResourceTrait

scala> withResource(new java.io.StringReader("HAI")) { input => () }

You can see that we've satisfied the type system for the particular type java.io.StringReader. Our old method was duck typed to use anything that had a close method. We can actually get this same functionality with type traits as well. Let's provide a default type trait in the Resource Companion object. To do so, let's create a resource.scala file that looks like the following:

trait Resource[R] {
  def close(r : R) : Unit
}
object Resource {
  implicit def genericResourceTrait[A <: { def close() : Unit }] = new Resource[A] {
    override def close(r : A) = r.close()
  }
}

What we've done is provide a method "genericResourceTrait" that is available on every implicit search for any Resource[_] type. We've now gotten back to our previous functionality, and we can take this a bit farther. What if we wanted to provide a specific implementation for a type? In the case of resources, Java provides a java.io.Closeable interface that most IO classes implement. Creating a type trait implementation for this will give us the benefit of not needing to use reflection when we know a specific type. Let's take our first crack at it.

trait Resource[R] {
  def close(r : R) : Unit
}
object Resource {
  implicit def genericResourceTrait[A <: { def close() : Unit }] = new Resource[A] {
    override def close(r : A) = r.close()
    override def toString = "Resource[{def close() : Unit }]"
  }
  implicit def jioResourceTrait[A <: java.io.Closeable] = new Resource[A] {
    override def close(r : A) = r.close()
    override def toString = "Resource[java.io.Closeable]"
  }
}

We now have a duck typed, reflection based implementation and an implementation for the java.io.Closeable interface. We can prove that this works via the REPL by creating a mechanism of checking what type trait is used:

scala> def showTypeTraitUsed[A : Resource](a : A) = println(implicitly[Resource])
showTypeTraitUsed: [A](a: A)(implicit evidence$1: Resource[A])Unit

scala> showTypeTraitUsed(new java.io.StringReader("HAI"))
Resource[java.io.Closeable]

scala> showTypeTraitUsed(new { def close() : Unit = () })
Resource[{def close() : Unit }]

We see that when passed a java.io.StringReader, something that subclasses java.io.Closeable, it uses the type trait defined for closeables. When we pass an anonymous object that has a close method, it uses the structural-typed type trait instead. What about monkey patching? Well, using type traits, you can achieve a form of monkey patching. Let's pretend an integer is a resource. All we have to do is create a new type trait implementation for it. Let's make that happen:

scala> implicit object intAsResource extends Resource[Int] {
     | override def close(r : Int) = println("closing: " + r)
     | }
defined module intAsResource

scala> withResource(5)(println)
5
closing: 5

We define our close method to print out a string, so we can make sure it happens. Finally, we can use an integer in our withResource method as if it were a resource, hence monkey-patching a close method onto it. In fact, we could also use this mechanism to change the implementation of the Resource type trait in a particular context. The key to this, as described in my book, is to ensure that you place your default implicitly available type trait implementation into the correct scope, such that

They can be overridden
They do not conflict

We can also use certain Scala tricks like having method return types be based on implicits available within scope. Combining these all, you can create some rather interesting libraries, such as what is available in scala arm's typeclass-fix branch. This branch contains a rather interesting flatMap that can detect if you are mapping to a Traversable type, and return a Traversable. It will also detect if you are flatMapping to another Resource and return a new ManagedResource that will open/close that nested resource. Finally, it also supports the standard flatMap implementation. Working with type classes in Scala adds a bit of overhead in amount of code, and some time to toy with the type system but can provide great flexibility in design. I'm working on a library for interactive with Nexus, using webDAV protocol. Using a "FileLike" type trait has been helpful in the design, especially since certain methods can be optimised in particular algorithms where we know both types. It's also useful because I've adapter several different WebDAV-enabled libraries to the API before finally deciding on the library I disliked the least. So in recap, let's look at why the typeclass pattern is so powerful in Scala:

Can provide implementations for many differing types, in hierarchical fashion
Can provide new type trait implementation from any scope
Allows at-usage-site overriding of implementation details

How you should think about Delimited Continuations

2010-03-10T00:00:00+00:00

Scala 2.8.x is going to have a new delimited continuations plugins. Delimited continuations are interesting animals, and I've seen some decent articles about them. I'm going to portray here how you should think about delimited continuations in a way that will help you know how to make use of them in your coding.

Let's take a look at the wikipedia definition delimited continuations:

a composable continuation, delimited continuation or partial continuation, is a "slice" of a continuation frame that has been reified into a function.

This is basically saying that a delimited continuation is marking a particular section of a program in such a way the compiler can "reify" it into a Function (or Closure if you will). In fact, the compiler can use similar semantics to how a closure is made, the difference here is in how you make arguments to functions. Let's take a look at a simple Scala delimited continuation example and try to de-mystify:

import scala.continuations.ContextControl._
object SimpleCont {
  def main(args : Array[String]) : Unit = {
    val result = reset {
       1 + shift { k => k(k(5)) } + 1
    }
    Console.println(result)
  }
}

This simple example prints 9 to the screen. So what is happening? Well, let's look at the definition: A "slice" of a continuation frame that has been reified into a function....

Well, we know reify essentially means to make it a first-class citizen (or first-class-function). So what is our "slice of continuation frame"? It's everything inside the reset block. This is where the "delimited" portion comes in. We're using reset to as an outer-bounds for the compiler to capture code and reify it into a function. In doing so, the compiler essentially takes all shift keywords and turns them into arguments for the reified function. Let's look at what our reified function is equivalent to in the above example:

reset {
       1 + shift { k => k(k(5)) } + 1                   
  }
  val reifiedFunction = { shiftVal1 : Int => 
       1 + shiftVal1 + 1
  }

Notice how the entire shift {..} expression is converted into an argument of the reified function. That's where the continuation comes in. All code inside the reset, minus the shift expressions, is turned into continuations... Something that could be called later and/or multiple times. For the real magic, let's take a look at this shift operation.

The shift command takes a function as an argument. This function takes another function as its only parameter (higher-order) and that function is the reified continuation. So... strangely enough, the shift expressions are turned into arguments on functions passed into the functions defined in the shift expression. Yes, this blew my mind at first, but becomes very interesting and handy. Not only that, it's a true inversion of control.

In this simple example, the portion of code defined in shift becomes the main execution path, and calls the reified continuation. Let's look at the translated code side-by-side

object SimpleCont {
  def main(args : Array[String]) : Unit = {
    val result = reset {
       1 + shift { k => k(k(5)) } + 1                   
    }
    Console.println(result)
  }
}

object SimpleCont {
def main(args : Array[String]) : Unit = {

// Reified function
val k = { shiftVal1 : Int =>
1 + (shiftVal1) + 1
}
val result = k(k(5))
Console.println(result)
}

As you can see, we've completely reversed the shifts and the resets. Let's try to understand an example which involves two variables. The Continuations paper includes an example that takes the cartesian product of two lists (i.e. List(A,B) and List(C,D) => List((A,C), (A,D),(B,C),(B,D)) ). First the example code straight from the delimited continuations paper (with a few modifications for simplicity):

implicit def reflectiveList[A](xs:List[A]) = new {
      def reflect[B]() : A @cps[List[B], List[B]] = shift { k:(A => List[B]) =>
        xs.flatMap(k)
      }
    }
    
    reset {
      val left = List("x","y","z")
      val right = List(4,5,6)
      List((left.reflect[(String,Int)], right.reflect[(String,Int)]))
    }
  }
// result: cartesian product

Let's decompose this one a bit. The implicit reflective stuff is used to enabled the reflective method on lists. We'll analyze reflective soon, but for now, notice that it calls shift. This means that calls to reflective are equivalent to calls to shift. Let's try to figure out what the reified continuation is by first "inlining" all the code:

reset {
  val left = List("x","y","z")
  val right = List(4,5,6)
  List(Tuple2(
      left.reflect[Any],
  <span style="color: #111199; font-weight: bold;">right.reflect[Any]</span>
))
}

reset {
  val left = List("x","y","z")
  val right = List(4,5,6)
  List(Tuple2(
        shift { k:(A => C[B]) =>
          left.flatMap(k)
        },
        shift { k:(A => C[B]) =>
          right.flatMap(k)
        }
  ))
}

First of all, Notice the new type signatures on the shift expressions (without the generics). We'll discuss these more in depth later, but if hand-reify the continuation, we need to make sure we match the signatures expected from the shift expressions. Let's take our first crack at it:

val reified : = { (shift1, shift2) =>
  List(Tuple2(
        shift1,
        shift2 
  ))
}

Well, that seems simple enough... except we're not creating the lists in the reified function! Ah... remember, we're only reifying the continuation piece. We can directly execute the portion of the code that are directly executable. Let's try to completely unwind the code and translate it:

// Original Code
reset {
      val left = List("x","y","z")
      val right = List(4,5,6)
      List((left.reflect[(String,Int)], right.reflect[(String,Int)]))
    }
val reified : (String, Int) => List[Any] = { (shift1, shift2) =>
  List(Tuple2(
        shift1,
        shift2
  ))
}

//Original reset was here...
val left = List("x","y","z")
val right = List(4,5,6)
left.flatMap( l => right.flatMap(r => reified(l,r))

Well that's interesting.... This is the effect of apply delimited continuations and should help you begin to reason about them and how they operate, however the actual truth of the matter is, your code is transformed differently. Let's take a look at what is really happening, just so we can dig deeper if needed. Technically, there are two reified continuation (for the two shifts) so let's make our hand-reified code do the same:

//Original reset was here...
val left = List("x","y","z")
val right = List(4,5,6)
val reified1 : (String => List[Any]) = { shift1 =>
  val reified2 : (Int => Any) = { shift2 =>
        Tuple2(shift1,shift2)     
  }
  right.flatMap(reified2)
}

left.flatMap(reified1)

So this is where it gets strange. We create a function "refiied1" which is passed to the first shift expression (left.flatMap). This contains the outer continuation. We then have a second call to shift. This means there is a second continuation generated inside the first continuation. The second function is responsible for generating the tuples. This is based to the second shift expression "right.flatMap". As you can see, every time you use the shift function, you're causing the compiler to generate a new continuation to defer the next set of processing. You can also see how in "pure" form, this has a very nested structure, whereas the original code was very "flat". Also, because work is automatically defered into continuation functions, you can use this style for concise and performant asynchronous work.

Let's take a mini-dive into a continuations for automated resource management. First, let's make our sample file:

HAI!

O
MY!

Now, let's right a continuation-based program to read each line and give us the size of it. I'm purposely making the code slightly confusing here to demonstrate the inversions. Well, let's start off with a mechanism of "reflecting" or "inverting" a resource for delimited continuations.

trait Resource[+R] {
      def flatMap[U](f : R => U) : U
      def reflect[B]() : R @cps[B,B] = shift { k : (R => B) => flatMap(k) }
}

def resource[R <: Closeable](acquire : => R) = new Resource[R] {
      override def flatMap[U](f : R => U) : U = {
        val x = acquire
        try {
          f(x)
        } finally {
          System.out.println("Closing Resource!")
          x.close()
        }
      }
}

We define a simple Resource trait that gives us a flatMap (some may call this "using") and a reflect (or inverse) method. This will let us "flatten" resource usage inside a reset block. The next component is the implementation of resource for any java.io.Closeable implementation. This will take a function to acquire a resource and implement the flatMap method correctly. This should look very similar to the reflect method defined on lists... One might even say this method works to flatten any monad ;)

Now, let's get really dirty and define an inversion on java.io.InputStream to deal with lines:

def reflect[A <: InputStream](input : A) = new {
  def each_line[B] : String @cps[B,List[B]] = shift { k : (String => B) =>
    val b = new BufferedReader(new InputStreamReader(input))
    var line = b.readLine
    var list = ListBuffer[B]()
    while(line != null) {
      list += (k(line))
      line = b.readLine
    }
    list.toList
  }
}

We've defined our reflect method to work on every line within the file. It builds up a list and returns it as the final result. You'll notice the shift call takes a function requiring a String and returning a "B". The fun type-system issue we have to deal with here is you need to know the result of the reified continuation in the return value for each_line. We'll get to that more in a moment, but for now make note of this.

Alright, now let's define our simple function to manipulate every line in a file:

reset {
      val input = resource(new FileInputStream("test.txt")).reflect[Any] //We cheat on the return value here!
      val line = reflect(input).each_line[Int] //We need to know the continuation returns an Int Here!!!
      System.out.println(line)
      line.size
}

Wow... so really all we're doing is grabing our input, inverting to the "each_line" call and calling line.size after printing the line to the console.

So what's the output look like?

scala> Main.resourceTest()
HAI!

O
MY!

Closing Resource!
res0: Any = List(4, 0, 1, 3, 0)

This is great! It's doing what we wanted. Notice how we've "flattened" the code substantially compared with other methods. Contrast the above with the following:

val input = new scala.io.magic.File("test.txt")
for { 
  stream <- file.managedInput
  line <- input.lines
} yield line.size

You'll notice a few similarities (in my made up library). For-expressions are another area in scala where you're reifing portions of the code into closures and passing them to flatMap blocks. However, delimited continuations open the doors to way more than flatMap. Let's go back to that typing issue though.

If you remember, I was returning Any because I was being lazy with my type annotations when reflecting the input stream. Let's use the correct annotation:

reset {
      val input = resource(new FileInputStream("test.txt")).reflect[List[Int]]
      val line = invert(input).each_line[Int] //We need to know the continuation returns an Int Here!!!
      System.out.println(line)
      line.size
}

So... what do these type annotations mean? In the case of the resource's reflect method, the type annotation is the ultimate result from the reset call (i.e. the result of the reified continuation) because of how we defined flatMap on Resource. The type annotation for each_line describes the type the resulting List of processed lines should contain... i.e. the result of the last portion of the continuation. If we were to hand-code this reset piece we'd have:

val rcont1 : (FileInputStream => List[Int]) = { 
  inputStream =>
     val rcont2 : (String => Int) = { line =>
        System.out.println(line)
        line.size
     }
     val b = new BufferedReader(new InputStreamReader(inputStream))
     var line = b.readLine
     var list = ListBuffer[B]()
     while(line != null) {
        list += (rcont2(line))
        line = b.readLine
      }
      list.toList
  }
  resource(new FileInputStream("test.txt")).flatMap(rcont1)

So... delimited continuations certainly use far less code here. The understanding of what's going on under-the-covers perhaps needs some good terminology/consistent paradigms to be useful in library-form (i.e. if I make each_line or reflective part of a standard library). Also, it's my opinion that the type-inference barrier for the @cps annotation should be looked into. Perhaps we can invent creative ways to infer the types in an inverted fashion when using delimited continuations. In the meantime, the compiler will complain until you figure out the correct types, so it could be worse ;)

I plan to post a bit more about delimited continuations after I have some useful I/O related tools built up around them. Let me know if you find this post helpful. I'd also like to clarify that my "hand-reification" is not really how scala does reification, but close enough for understanding the concepts.

Also, there's some interesting continuation-based actors code in the paper that I'm going to try to dig into here when I have the time!

New Blog Site

2009-02-20T00:00:00+00:00

Hey, just thought I'd make an initial post to announce my switch to a statically generated blog site. Hopefully I can keep the content kicking AND still have neato features like RSS feeds, etc.

The new syte is done in Jekyll and makes it way easier for me to write content, so hopefully I'll be doing so more frequently!

Leveraging Annotations in Scala

2009-02-05T00:00:00+00:00

This post began as a discussion on #scala about how people define a public "getter" method and private "setter" method using only defs and vars in scala. The usual method in scala is:

class MyClass {
   private var x_private: Int = _
   def x = x_private
}

The downside to this is that you have two names to express one concept. I'm at the point where I'm no longer caring too much about public/private parts of classes, but I thought I'd tackle the problem to exercise my new-found compiler skills. It turns out this feat is pretty easy to accomplish via a scalac plugin.

To create your first plugin you should follow the guide here. We'll be using the Maven as our build tool, since to do otherwise would be blasphemy (for me).

To start off with, here's the POM file for our plugin:


<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
 <modelVersion>4.0.0</modelVersion>
 <groupId>org.scala-lang</groupId>
 <artifactId>private-setter-scalac-plugin</artifactId>
 <packaging>jar</packaging>
 <version>1.0-SNAPSHOT</version>
 <name>Var definition extensions for the scala compiler</name>
 <url>http://suereth.blogspot.com</url>
 <repositories>
  <repository>
   <id>scala-tools.org</id>
   <name>Scala-tools Maven2 Repository</name>
   <url>http://scala-tools.org/repo-releases</url>
  </repository>
 </repositories>
 <pluginRepositories>
  <pluginRepository>
   <id>scala-tools.org</id>
   <name>Scala-tools Maven2 Repository</name>
   <url>http://scala-tools.org/repo-releases</url>
  </pluginRepository>
 </pluginRepositories>
 <dependencies>
  <dependency>
   <groupId>org.scala-lang</groupId>
   <artifactId>scala-compiler</artifactId>
   <version>2.7.3</version>
  </dependency>
 </dependencies>
 <build>
  <plugins>
   <plugin>
    <groupId>org.scala-tools</groupId>
    <artifactId>maven-scala-plugin</artifactId>
    <executions>
     <execution>
      <goals>
       <goal>add-source</goal>
       <goal>compile</goal>
       <goal>testCompile</goal>
      </goals>
     </execution>
    </executions>
    <configuration>
     <jvmArgs>
      <jvmArg>-Xms64m</jvmArg>
      <jvmArg>-Xmx1024m</jvmArg>
     </jvmArgs>
    </configuration>
   </plugin>
   <plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>shitty-maven-plugin</artifactId>
    <executions>
     <execution>
      <goals>
       <goal>clean</goal>
       <goal>install</goal>
       <goal>test</goal>
      </goals>
     </execution>
    </executions>
   </plugin>
  </plugins>
 </build>
</project>

There's a few things in this pom, mostly to ensure that the scala-tools repositories are available too us. Also, ignore the shitty (Super-Helpful-Integration-Testing-ThingY) plugin for now. The most important part is that we're compiling to a jar file, and we're depending on the scala-compiler. The version of the compiler we depend on is the only version of scala your plugin should be used with. In all reality, the scala-compiler should probably be deifned as a 'provided' dependency and the scala-library should explicitly be depended on, but for now we'll cheat on completeness.

Next we need to make sure there is a scalac-plugin.xml file in the created JAR. This is simple in maven, just place one in the src/main/resources directory. Here is what my src/main/resources/scalac-plugin.xml looks like:

<plugin>
  <name>private-setters</name>
  <classname>org.scala_lang.privateSetter.internal.privateSetterPlugin</classname>
</plugin>

You'll notice I'm naming my plugin "private-setters" and placing it in a "internal" package. This could be my eclipse plugin development rubbing off, but this helps me know what only the compiler should see.

Next we need a way for clients of our plugin to "notify" us that they want a var with a private setter method (@varname_=@), but public getter (@varname@). Here's my initial cut at client syntax:

import org.scala_lang.privateSetter._
class TestWidget {
  @privateSetter
  var myVar = 5
}

Pretty simple really, but effective. (Also I already know how to look up annotations from my scala-mojo-support project, so it only took me a few hours to work out the scalac plugin details. This post actually took the most amount of time in the whole venture).

Now we need to define the @privateSetter@ annotation we can use in our classes. It's a fairly simple file:

package org.scala_lang.privateSetter
class privateSetter extends StaticAnnotation {
}

Next we need to write the plugin itself. All plugins contain some boiler-plate code, so we'll ignore that for the time being. (see the documentation mentioned above). The truly interesting part of this plugin is the newly defined phase.

For some history, the Scala Compiler (scalac) is composed of various "phases". Each phase has a responsibility it performs. Some phases are easy to identifier e.g. icode which converts the AST to icode for each "compilation unit". In the Scala Compiler a compilation unit corresponds to a source code file and may produce multiple class files. Here's the scalac -Xlist-phases output on my machine:

$ scalac -Xshow-phases
namer
typer
superaccessors
pickler
refchecks
liftcode
uncurry
tailcalls
explicitouter
erasure
lazyvals
lambdalift
constructors
flatten
mixin
cleanup
icode
inliner
closelim
dce
jvm
sample-phase

On to our implementation! The basic structure of our phase looks like this:

    class MakeSettersPrivatePhase(prev: Phase) extends Phase(prev) {
      
      override def name = VarAccessChanger.this.name
      import global._
      override def run {
 for (unit <- global.currentRun.units; if !unit.isJava) {
           unit.body = TreeTransformer.transform(unit.body)      
 }
      }
      ...
}

I'll mention that this class is nested inside an outer "Plugin" class which is passed the 'global' object. For those of you unfamiliar with the compiler, the "Global" object is the outer layer of the "cake" pattern used by the Scala Compiler. I'm reserving judgment on Global, but I have noticed that it's very hard to unit test any "module" you write for Global (as most of them have the self:Global => syntax).

The Transformer is a very nice class for doing AST manipulation (thanks DRM for suggesting it). This class simply transforms the AST from one form to another. It's perhaps the easiest way to implement our plugin. Let's set up a Tree Transformation that does absolutely nothing useful:

      object TreeTransformer extends Transformer {

        override def transform(tree: Tree) = tree match {
         case t =>         
           super.transform(t)
        }
      }

The structure here is we override the transform method. This method takes a tree and returns a tree. We want to transform only the setter part of a var method if it contains the privateSetter annotation. Let's apply our pattern matching skills to the test with an extractor: The "AnnotationSetterShouldBePrivate" extractor.


object AnnotatedSetterShouldBePrivate {

  def unapply(node : Tree) : Option[DefDef] = {

    def hasPrivateSetterAnnotation(annotations : List[Annotation]) : Boolean = {
      for {
           annotation <- annotations
           if annotation.tpe.safeToString == classOf[privateSetter].getName
      } {
        return true
      }
      false
    }
    
    node match {
      case x @ DefDef(mods,name,_,_,_,_) if name.toString.endsWith("_$eq") =>
         if(hasPrivateSetterAnnotation(mods.annotations)){
            Some(x)
         } else {
            None
         }
      case _ => None
    }
  }
}

First, note that our extractor (unapply method) takes in a tree and returns a DefDef. DefDef is the AST class for a "def" node. All var's are parsed into dual "def" methods (setter and getter). We define a helper method that takes a list of annotations and looks for our "privateSetter" annotation. The actually implementation of the extractor matches against the tree node, checks to see if it's a DefDef and has a tailing name of "$eq". "$eq" is the mangled form of "_=" which is the convention for scala setter methods. Note that I probably could move the if/else statement into the pattern match on the DefDef, but wasn't feeling adventurous enough this evening. In the case where we find a valid annotated setter DefDef method, we return it, otherwise return None.

Now that we have our extractor, writing the tree transformation becomes fairly simple:


object TreeTransformer extends Transformer {
    
  override def transform(tree: Tree) = tree match {
      case AnnotatedSetterShouldBePrivate(d @ DefDef(mods,name,tparams,vparams,tpt,impl)) => 
          import symtab.Flags._
          val tree = copy.DefDef(d, mods | PRIVATE,name,tparams,vparams,tpt,transform(impl))
          tree.symbol.setFlag(PRIVATE)              
          tree
      case t =>
          super.transform(t)
  }
}

As you can see, we're combining our "AnnotatedSetterShouldBePrivate" extractor with the natural extractor for the DefDef case class so that we can pull out all the constructor variables (along with the DefDef itself using the d @ syntax). Initially I tried returning just a newly constructed DefDef with "mods | PRIVATE" instead of "mods" in the constructor. You'll find this blows up horribly. The main issue is that the AST nodes contain more than just their constructor values (types and symbols being the two things I found). The Transformer class provides a "copy" value/object that you can use to "copy" various parts of the tree. The copy class contains a method for every tree node that takes an original tree and overriding constructor values. For our purposes, we're applying the PRIVATE flag to the "mods" attribute of the DefDef and two the symbol flags for the DefDef. The symbol flags are what are eventually used in the icode->bytecode conversion code.

Next we should choose what phase to run this plugin after. I've chosen the typer phase, as this ensures we at least have an AST and the types are correct. When defining a class in isolation, this is working perfectly. However when defining the class and using it with other classes, I'm running into the difficulty where the methods I'm modifying are eventually being replaced with public methods of a differing names. To look into this, we should set up some integration tests. NOW we can use the SHITTY plugin!

The maven shitty plugin allows you to execute "integration" projects that depend on the currently "building" project. You simple create a directory in src/it, add a pom.xml that depends on your project (with a version of "testing"), and a goal.txt that describes which maven goals should be executed. If the integration project's maven build succeeds, the overall project's maven build continues. If an integration project's maven build fails, the entire build fails. This works great for "positive" tests (or test where you make sure that things compile with your plugin). Let's define a relatively simple positive test. First, we take the class outlined earlier as our example syntax:

package org.scala_lang.privateSetter

class privateSetter extends StaticAnnotation {

}

Next we create a pom file for this integration test:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
 <modelVersion>4.0.0</modelVersion>
 <groupId>org.scala-tools</groupId>
 <artifactId>testPrivateSetter</artifactId>
 <packaging>jar</packaging>
 <version>1.0-SNAPSHOT</version>
 <name>test-privateSetter-scalac-plugin</name>
 <dependencies>
  <dependency>
   <groupId>org.scala-lang</groupId>
   <artifactId>private-setter-scalac-plugin</artifactId>
   <version>testing</version>
  </dependency>
  <dependency>
   <groupId>junit</groupId>
   <artifactId>junit</artifactId>
   <version>4.5</version>
   <scope>test</scope>
  </dependency>
 </dependencies>
 <build>
  <plugins>
   <plugin>
    <groupId>org.scala-tools</groupId>
    <artifactId>maven-scala-plugin</artifactId>
    <executions>
     <execution>
      <goals>
       <goal>add-source</goal>
       <goal>compile</goal>
      </goals>
     </execution>
    </executions>
    <configuration>
     <compilerPlugins>
      <dependency>
       <groupId>org.scala-lang</groupId>
       <artifactId>private-setter-scalac-plugin</artifactId>
       <version>testing</version>
      </dependency>
     </compilerPlugins>
     <args>
      <arg>-verbose</arg>
     </args>
    </configuration>
   </plugin>
  </plugins>
 </build>

 <repositories>
  <repository>
   <id>scala-tools.org</id>
   <name>Scala-tools Maven2 Repository</name>
   <url>http://scala-tools.org/repo-releases</url>
  </repository>
 </repositories>
 <pluginRepositories>
  <pluginRepository>
   <id>scala-tools.org</id>
   <name>Scala-tools Maven2 Repository</name>
   <url>http://scala-tools.org/repo-releases</url>
  </pluginRepository> 
  <pluginRepository>
   <id>snapshots.scala-tools.org</id>
   <name>Scala-tools Maven2 Snapshot Repository</name>
   <url>http://scala-tools.org/repo-snapshots</url>
  </pluginRepository>
 </pluginRepositories>

</project>

You'll notice we're making us of the "testing" version of our plugin AND the "compilerPlugin" configuration option of the maven-scala-plugin. This option is new to the (not yet released except as a SNAPSHOT) 2.10 version, and allows you to depend on any number of scalac plugins during your build. We're using it now to depend on our build. Our goals.txt simply consists of "clean compile".

Another thing that the SHITTY plugin lets you do is provide a 'validate.groovy' file with your pom.xml and goals.txt. This file is run after a build to ensure things were successful. We can use this to ensure our generated classfiles have private setters. We'll tackle that problem another day (I'm currently being lame and running javap on the .class files).

I think I've typed as much as I can for one night, I'll try to cover the remaining pieces (after I code/finish them) later. Once again, there are some issues with the plugin as I'm confusing the hell out of some of the compiler phases (not to mention being confused myself as to where things happen in some cases). If you'd like to look at the source (and perhaps contribute? ), it's available on github: private-setter-scalac-plugin

Subclassing in ExtJS

2008-12-08T00:00:00+00:00

It's been a while since I posted about ExtJS, so I thought I'd take a crack at another post here. This is going to cover the Ext.extend method, and how you go about sub-classing an Ext component.

The basic just is that we are going to create a "constructor" method for the widget and bind it to a location (e.g. Ext.ux.data.MyDataStore). Then we're going to use Ext.extend to plug in the methods of the parent objects, and override some at the same time.

For our example, I'll be using the Ext.data.DataProxy class as what we're extending.

So... first, let's define our constructor in a location:

//Ensures there are at least dummy objects leading into Ext.ux.data
Ext.namespace('Ext.ux.data');

Ext.ux.data.MyProxy = function(config){

    Ext.ux.data.MyProxy.superclass.constructor.call(this);

};

The code doesn't do a whole lot, but we have our constructor. However when executing new Ext.ux.data.MyProxy() we get an error, superclass is not defined. So... where does superclass come from? It will actually be injected into our prototype via the Ext.extend method. In fact, Ext.extend will also inject the methods "override" and "extend" into our class. Here's our new class with the Ext.extend call:

//Ensures there are at least dummy objects leading into Ext.ux.data
Ext.namespace('Ext.ux.data');

Ext.ux.data.MyProxy = function(config){

    Ext.ux.data.MyProxy.superclass.constructor.call(this);

};

Ext.extend(Ext.ux.data.MyProxy, Ext.data.DataProxy);

Now we can make use of DataProxy methods from within MyProxy using this.methodName(args). This isn't very exciting yet because DataProxy is mostly an empty class (besides configuring the Observable class for us).

The convention for adding methods into a subclass is to use the override parameter from Ext.extend. This parameter is essentially an anonymous class containing methods to inject "over-top" of the parent class. In the case where the method did not exist before, it is merely inserted. Let's implement the load method (required for DataProxy subclasses) using this approach:

//Ensures there are at least dummy objects leading into Ext.ux.data
Ext.namespace('Ext.ux.data');

Ext.ux.data.MyProxy = function(config){
    //Calls our super-class (Ext.data.DataProxy) constructor for full initialization
    Ext.ux.data.MyProxy.superclass.constructor.call(this);

};

Ext.extend(Ext.ux.data.MyProxy, Ext.data.DataProxy, {

   //Load function as defined in HttpProxy, and needed for the "DataProxy interface..."
   load: function(params, reader, callback, scope, callbackArg) {
         params = params || {}; //Ensure params exists as an object
         var result;
         try {
            //Use record reading to read "stock" data
            result = reader.readRecords({ zombieKid: ['i', 'like', 'turtles'] });
         } catch(e) {
           //We had an issue pulling in stock data, fire appropriate events
           //use callback in fail mode and bail.  This function is actually coming from
           // the Observable class which is our "grandparent" class
            this.fireEvent("loadexception", this, callbackArg, null, e);

            callback.call(scope, null, callbackArg, false);

            return;
         }
         //Inform the callback of the result of parsing our "stock" data
         callback.call(scope, result, callbackArg, false);         
   }

});

As you can see... our load method is basically taking the reader (Record reader) passed in and attempting to read some stock data (calling appropriate callbacks as necessary).

Notice the use of the this.fireEvent inside our method. Because of how JavaScript handles resolution, this will refer to our fully constructed class at the time of invocation ( as opposed to definition), therefore the fireEvent method from Observable will be accessable from our object.

The next important thing in defining your own widget is adding information to take in during construction. Our superclass constructor does not take any arguments, hence the Ext.ux.data.MyProxy.superclass.constructor.call(this); in our constructor. If perhaps our superclass constructor took a config object, we would have to change it. Luckily our current widget constructor expects a config object, so we can just pass it through to our super (assuming the configs are complementary).

...

Ext.ux.data.MyProxy = function(config){
    //Calls our super-class (Ext.data.DataProxy) constructor for full initialization
    Ext.ux.data.MyProxy.superclass.constructor.call(this, config);
}

...

Since the DataProxy class doesn't take anything in its constructor, we'll remove this, but it's an important feature to make note of. Also remember you can make subclasses which could add/change configuration information sent to parents. That's the nice part about having control of the config obejct before passing it to the superclass constructor.

Now onto taking in our own information. Let's mimic the MemoryProxy and take in some data during construction that we using during our "load" method. We'll call this element 'data' in the config and our object.

//Ensures there are at least dummy objects leading into Ext.ux.data
Ext.namespace('Ext.ux.data');

Ext.ux.data.MyProxy = function(config){
    //Pull out our "data"
    this.data = config.data

    //Calls our super-class (Ext.data.DataProxy) constructor for full initialization
    Ext.ux.data.MyProxy.superclass.constructor.call(this);

};

Ext.extend(Ext.ux.data.MyProxy, Ext.data.DataProxy, {

   //Load function as defined in HttpProxy, and needed for the "DataProxy interface..."
   load: function(params, reader, callback, scope, callbackArg) {
         params = params || {}; //Ensure params exists as an object
         var result;
         try {
            //Use record reading to read "passed in" data
            result = reader.readRecords(this.data);
         } catch(e) {
           //We had an issue pulling in stock data, fire appropriate events
           //use callback in fail mode and bail.  This function is actually coming from
           // the Observable class which is our "grandparent" class
            this.fireEvent("loadexception", this, callbackArg, null, e);

            callback.call(scope, null, callbackArg, false);

            return;
         }
         //Inform the callback of the result of parsing our "stock" data
         callback.call(scope, result, callbackArg, false);         
   }

});

The above is an almost exact duplicate of MemoryProxy.js from the ExtJS source. I thought about having a more clever tutorial about a custom widget, but I'm trying to avoid assuming too much knowledge of ExtJS internals. Anyway, I hope this makes you feel more comfortable creating your own classes and extending ExtJS widgets with custom functionality. As always, let me know if you find any errors or typos.

Masochism

2008-09-24T00:00:00+00:00

Note: Due to excessive comments about my grasp of the english languages, I've done a second round of editing for this post. It may show up on the RSS feed again, but there isn't any new content.

I've recently had to jump into a bit of (very minor) C++ development at work. This started bringing back memories of all the ?fun? it was to play around in the C++ type system. Mostly you use the C++ type system in an attempt to prevent yourself from doing bad things. Sometimes though, it was a way of doing conditional compilation in a type-safe manner.

I recently decided to outline how I tend to over-engineer my C++ code. I'll give a few ground rules/goals first:

I won't use boost. In real life, I do use boost whenever I can. For the purposes of illustration I'm doing a lot by hand. If you're a C++ developer and you don't use boost, go learn it. Now. Really, I mean right now go to www.boost.org and enter the 21st century of coding. No really, GO
I'm not doing detailed explanations (although I do some explanations) of the C++ language or its compiler workings.
This could be an attempt to ramp myself to attack learning scala's (and maybe haskell's) type sytem. C++ is the closest typed language I know that could prepare me
C++/Java feels more like masochism the more new languages I learn. I'm hoping this "simple" post about using the C++ type system in a "simple" way will prove this to you. At some point I hope to post a Scala/Haskell equivalent to this post (or perhaps someone else would like to?) just so you can see the amount of work C++ really requires (and the level of understanding required to write even a simple line).

Alright! so first off, the premise: We're trying to create a class (DynamicLibrary) that will let us dynamic load a shared library and access symbols. I'm only including code for linux (although porting to windows or "cross-platforming" isn't too bad. look into the LoadLibrary function in windows and dlopen in linux).

Before we begin, I'm going to construct something (available in boost) that should be in the utility kit of every C++ developer... a base class that removes the copy-constructor and operator= from visibility (effectively making it non-copyable)

#ifndef NONCOPYABLE_H_
#define NONCOPYABLE_H_
//============================================================================
// This class prevents copying on subclasses
//============================================================================
template<class T>
class NonCopyable
{
public:
 inline NonCopyable() {}
 inline ~NonCopyable() {}
private:
 NonCopyable(NonCopyable& other) {}
 NonCopyable& operator=(NonCopyable& other) {}
};


#endif

It's a very simple class. Most of its functions are purposefully inlined. It should get compiled out during optimization phases, AND it can reduce typing for me later when I don't want a class copied around on the stack. (Note: I could have saved one line of typing by changing to a struct and getting rid of the public: line. or putting all private functions first. Sue me, but I prefer this look).

Ok, next we're going to create a very small Meta-Programming Library. This library will be used to tell us all sorts of information about the types in C++. For now, please assume all these examples are surrounded in a namespace mpl {} block.

Here's our first bout of magic:

struct true_;
struct false_;  //Not really needed, but easier to understand if we have it

WTF? Why make two completely silly classes? This is so I can represent the notion of a boolean in the type system. From now on, I want you to think of true_ as "true in the meta-programming library" and similarly for false_. Note: In actuality I could probably use bool as a template parameter and get away with it, but this is more fun

Ok, now let's make a simple method just so we can print the boolean value of a type.

template<typename T>
struct is_true
{
 enum { value = 0 };
};
template<>
struct is_true<true_>
{
 enum { value = 1 };
};

So..... what is that? That, my friends, is a C++ meta-programming function. The first template applies to every type and "returns" a value of 0 (or false). The second is a specialization for the true_ type that will return 1. This means if we ever use the is_true structure with true_ as the argument, the "member" value will be 1. Its a little convoluted (or convolved?), but it works! How do you call it? Simply like this:

is_true<some type>::value.

That will return an actually value we can assign, test in an if statement or display on the console. We're going for displaying on the console here (console = stdout).

Ok, so let's get more interesting meta-functions. The most important for what I'd like to do later is the is_ptr function. Here's a look at it:

template<typename T>
struct is_ptr
{
 typedef false_ value;
};
template<typename T>
struct is_ptr<T*>
{
 typedef true_ value;
};

So before we saw "template specialization". That is, template<> with all types completely specified in an alternative implementation of the function. This time, we see "partial template specialization" (YAY!!?). This is where we have an alternative implementation that still has some wildcards (i.e. partially specialized). When used, the compiler will select the function from most specialized to most general based on our types, so we can get away with this.

Now, what's that funny type-def doing and why isn't "value" part of an enum? Well this meta-function takes a type as argument and returns a type (not a value). You can string together meta-functions of this kind to make even stranger (and horrendous to look at) meta-functions.

As a simple example, let's make an "if" meta-function:

template<typename COND, typename T, typename F>
struct if_
{
 typedef F value;
};
template<typename T, typename F>
struct if_<true_, T, F>
{
 typedef T value;
};

The above is a structure that will select either type T or F based on the valiue of COND (much like an if statement).

For now, let's test our first two meta-functions on a few combinations of the various incarnations of ints in C++.

 cout << "mpl::is_ptr<int*>::value = " << mpl::is_true<mpl::is_ptr<int*>::value>::value << endl;
 cout << "mpl::is_ptr<int>::value = " << mpl::is_true<mpl::is_ptr<int>::value>::value << endl;
 cout << "mpl::is_ptr<const int>::value = " << mpl::is_true<mpl::is_ptr<const int>::value>::value << endl;
 cout << "mpl::is_ptr<const int*>::value = " << mpl::is_true<mpl::is_ptr<const int*>::value>::value << endl;
 cout << "mpl::is_ptr<volatile int>::value = " << mpl::is_true<mpl::is_ptr<volatile int>::value>::value << endl;
 cout << "mpl::is_ptr<const volatile int>::value = " << mpl::is_true<mpl::is_ptr<const volatile int>::value>::value << endl;
 cout << "mpl::is_ptr<volatile int*>::value = " << mpl::is_true<mpl::is_ptr<volatile int*>::value>::value << endl;
 cout << "mpl::is_ptr<const volatile int*>::value = " << mpl::is_true<mpl::is_ptr<const volatile int*>::value>::value << endl;

And the output is:

mpl::is_ptr<int*>::value = 1
mpl::is_ptr<int>::value = 0
mpl::is_ptr<const int>::value = 0
mpl::is_ptr<const int*>::value = 1
mpl::is_ptr<volatile int>::value = 0
mpl::is_ptr<const volatile int>::value = 0
mpl::is_ptr<volatile int*>::value = 1
mpl::is_ptr<const volatile int*>::value = 1

Neat-o huh? Yeah I probably could change the output to really represent things, ie mpl::is_true<mpl::is_ptr<....>::value>::value ? "true_" : "false_". However it was already way to much typing.

Ok, now onto why we were making meta-functions to begin with. I want a somewhat type-safe way of importing symbols from a dynamicly loaded library and casting them to their appropriate types. This idea came from an old colleague at APL. I hope this solution is different enough, as I don't recall his solution.

Let's start off with our basic class structure for DynamicLib:

//ASSUME LIBRARY_HANDLE_TYPE and SYMBOL_POINTER_TYPE are defined somewhere above (os-specific)
class DynamicLib : public NonCopyable<DynamicLib> {
public:
 DynamicLib(const char*const name); //implementation is OS-specific
 ~DynamicLib();                     //implementation is os-specific
 static SYMBOL_POINTER_TYPE get_symbol_internal(const char*const name, LIBRARY_HANDLE_TYPE handle); //implementation is OS-specific.  Didn't feel like figuring out to privatize this.
private:
 LIBRARY_HANDLE_TYPE handle; //handle to library (os-specific)
};

First off, notice the use of NonCopyable. This ensures the class cannot be copied (i.e. you have to use pointers to share it). This helps us ensure that the library is loaded once and destroyed once. You could still construct it twice for the same library, but if you do that... well a maniacal penguin dressed in a business suit will come steal your fish.

Next, the get_symbol_internal method is static. This is because the public (local) method has yet to be written and needs to use the MPL library (YAY!!!?). What we'd like to do is find a way to return primitives by value, and everything else by pointer. I'm going to call anything a "primitive" if it has a valid copy constructor (boy would this be wrong in a large app). Instead of pulling in Matthew Wilson's "must_be_pod" meta-function-constraint, I think I'll just assume proper usage (like a fool). Note: A maniacal penguin did show up at my door, but I was able to dissuade it with a tennis racket and tuna fish.

Ok, so onto the symbol loading... Here's what our public symbol loading function looks like:

//function for external use
 template<typename T>
 inline T get_symbol(const char*const name) {
  //Call appropriate function based on whether type is pointer or not.
  typedef get_symbol_<T, typename mpl::is_ptr<T>::value> get_symbol_functor;
  get_symbol_functor functor; //need an instance to use operator()...
  return functor(name, handle);
 }
private:
 template<typename T, typename isptr>
 struct get_symbol_;

Well, that's simple. Too bad it won't compile when someone actually tries to use it. What's going on here? Well the get symbol function is templatized on some type. Then we try to call the get_symbol_ functor struct. What is a functor struct (besides a lame excuse for a closure/real functor)? It's a structure that (sort of) looks like a function in code (as shown above). We declare (but not define) the structure. The structure is where we will specialize our implementation for pointer types vs. non-ptr types. Since we're using the is_ptr meta function above, all we need are two partial specialization for the get_symbol_ functor. The first specialization will take true_ as its second type parameter. The second specialization will take false_ as its second type parameter. The false variant should perform a copy of the value pointed at by the symbol. The true variant can just cast and return the symbol pointer. Here's the code:

 //Repeated for clarity
 template<typename T, typename isptr>
 struct get_symbol_;

 template<typename T>
 struct get_symbol_<T, typename mpl::true_> {
  T operator()(const char*const name, void* handle) {
   //This should fail to compile for types that don't have copy-constructors
   return reinterpret_cast<T>(DynamicLib::get_symbol_internal(name, handle));
  }
 };
 //Get symbol for non-ptr types
 template<typename T>
    struct get_symbol_<T, typename mpl::false_> {
     T operator()(const char*const name, void* handle) {
   //TODO - Fix const...
   return *static_cast<T*>(DynamicLib::get_symbol_internal(name, handle));
     }
    };

So now you can see how a struct-functor is created. Basically you just define an operator() overload inside the structure, and it will start acting like a function (or at least looking like one...).

You can see how on the one hand we're de-referencing the pointer, and on the other we're just casting. I'm also going to assume here (wrongly) that the get_symbol_internal will throw an exception on error, thereby ensuring that I will never dereference an invalid pointer (feel free to laugh, or send the penguin again).

Ok, and now for the main method that uses this mess of code (note I'm using getopt.h):

int main(int argc, char*argv[]) {
 int c;
 extern char* optarg;
 extern int optind, optopt;
 string library = "";
 string symbol = "";
 string type = "";
 bool usage = false;
 while ((c = getopt(argc, argv, "l:s:t:h")) != -1) {
  switch (c) {
  case 'l':
   library = optarg;
   break;
  case 's':
   symbol = optarg;
   break;
  case 't':
   type = optarg;
   break;
  case '?':
   cerr << "Unrecognized option: -" << static_cast<char> (optopt)
     << endl;
  case 'h':
   usage = true;
   break;
  }
 }

 if (usage) {
  //TODO
  return 2;
 }

 DynamicLib lib(library.c_str());

 if(type == "int") {
  cerr << "Symbol value (as int) is: " << lib.get_symbol<int> (symbol.c_str()) << endl;
 }
 if(type == "function") {
  cerr << "Symbol value (as return of function [int ()] ) is: " << lib.get_symbol<int (*)()> (symbol.c_str())() << endl;
 }
 return 0;
}

The hardest one to understand here is possibly the function pointer. Basically I'm asking for a pointer to a no-argument function that returns an integer int (*)(). I then immediately call this function for a value to place in cerr.

Ok, now we need a shared-library to test out. Here's my dummy .cpp that I compiled into a .so (I leave that as an excersie to the user).

extern "C" {
 int MY_VAL = 5;
 int MY_FUNC() { return MY_VAL; }
}

Onto our outputs:

>  Debug/cpp-types -l ../dummy-shared-lib/Debug/libdummy-shared-lib.so -s MY_VAL -t int
Symbol value (as int) is: 5
>  Debug/cpp-types -l ../dummy-shared-lib/Debug/libdummy-shared-lib.so -s MY_FUNC -t function
Symbol value (as return of function [int ()] ) is: 5

Ok, so this is a little bit limited, but we have it working! YAY!!!?

In conclusion, I really want to get back into Scala again. The type system seems a little bit less verbose, and the language itself is less clunky.

Note: I later updated this to pull string-literals out of libraries. The fun part there was realizing that a symbol pointer to a string literal is really a [char**] that needs to be dereferenced (carefully). Basically I wrote a specialized std::string specialization of get_symbol that would perform the checks/casts and return a string. The key to testing this was ensuring that the char* literal would wind up in the symbol table of the output .so. This means you should make a dummy function that returns it, otherwise it may be optimized away. Enjoy!

Ubuntu Dev. Server - Setting up hudson

2008-08-21T00:00:00+00:00

I've decided to switch from using Continuum for Continuous Integration to Hudson. This is based on a suggestion from a friend. We'll see how things turn out.

So first, I'm going to list here (because I continually have to look it up) how to install Tomcat5.5 and Apache2 on Ubuntu server and wire the mod_jk forwarding up.

First, install apache, tomcat and mod_jk

sudo apt-get install apache2 tomcat5.5 libapache2-mod-jk

Next create a workers.properties file (mine is in /etc/apache2/workers.properties). Here's what mine looks like:

worker.list=worker1
worker.default.port=8009
worker.default.host=localhost
worker.default.type=ajp13
worker.default.lbfactor=1

Next, make sure there's a sym-link in /etc/apache2/mods-enabled to /etc/apache2/mods-available/jk.load.

Now that we know mod-jk is loaded on apache startup, it's time to actually set up apache to forward appropriately. I edited my apache2.conf file (instead of available-hosts/default). Why? No real reason. In fact this should probably be moved into available-hosts/default, but for now I'll just show you the relevant apache configuration to write in the worker you specified above:

# Tomcat Configuration
JkWorkersFile /etc/apache2/workers.properties
JkMount /hudson/* worker1

Note that I'm only mounting the hudson directory. This means that I'm only forwarding to tomcat for the tomcat application. This is because I'm using apache to host my subversion repository AND trac AND hudson (and archiva in the future).

Finally, we have to turn off (or configure) java security for tomcat 5.5 in ubuntu. I'm turning it off because I'm lazy and I don't plan to place my dev box outside my internal network. The basic just to turn it off is to open your /etc/init.d/tomcat5.5 file and look for the following line: TOMCAT5_SECURITY=yes. Change the "yes" to "no" and you should be ready to go. For more details check the link here

Installing Hudson

First of, download the hudson war into tomcat's webapps directory -

sudo wget http://hudson.gotdns.com/latest/hudson.war

Next, set up a directory for hudson. I made one in tomcat's directories:

cd /var/lib/tomcat5.5
sudo mkdir hudson
sudo chown tomcat5.5 hudson

Now, you need to add the following line somewhere in the beginning of /etc/init.d/tomcat5.5 -

JAVA_OPTS="${JAVA_OPTS} -DHUDSON_HOME=/var/lib/tomcat5.5/hudson"

Now, restart tomcat and apache; Hudson should be working at http://youraddress/hudson/

Setting up Apache and Subversion on Ubuntu

2008-05-23T00:00:00+00:00

Now onto setting up SVN and apache.

First off, make sure libapache2-svn is installed. Now to the configuration in /etc/apache2/mods-available/dav_svn.conf

Mine looks like this:

<Location /svn/project>
  DAV svn
  SVNPath /path/to/my/project/svn

  AuthType Basic
  AuthName "Subversion Repository"
  AuthUserFile /etc/apache2/dav_svn.passwd
  <LimitExcept GET PROPFIND OPTIONS REPORT>
    Require valid-user
  </LimitExcept>
</Location>

Note: I'm doing something stupid here. I'm create ONE repository for my project as opposed to using SVNParentPath where you could have multiple repositories later. Yes, this is just for one project. We'll go back and fix later. (It's a virtual machine remember!)

Next, create a password:

htpasswd2 -c /etc/apache2/dav_svn.passwd josh

Now, onto our SVN repository. Let's give apache access (while we're at it, let's fix the group from last time too)

sudo chown www-data:www-data -R /projects

Ok, now to throw some random thing in the repostiory and make sure you can see it:

sudo su www-data
cd /some/place/safe
mkdir tmp
touch tmp/README
svn import /some/place/safe/tmp file:///path/to/my/project/svn

Alright... now we restart apache (/etc/init.d/apache2 restart) and it should be up and running! if you're like me shoot over to http://localhost:8888/svn/project and you should see your SVN repostiory. (you should also make sure TRAC sees it).

Well, I think we accomplished something here! Now (if your box is on the net) you can start adding code and tracking changes. Unfortunately we have a little bit more to do before I'm ready to start my project, and that little bit is the maven infrastructure.

Side note:

I was hoping Trac could manage our passwords for SVN (and create new repositories), but it looks like that's a no. Oh well, once again we'll see how annoying it gets. If it gets too annoying, I'll make an extension. I'm hoping to find an alternative to gforge that I can host internally to a company.

Setting up an Ubuntu Development Server

2008-05-22T00:00:00+00:00

Well, we're off to a rocky start. First off, the Ubuntu server kernel expects a particular processor extension that does not exist in the virtualbox emulation: see here for a fix.

Anyway, after a little shell magic, we're up and running. NOW for software. You need to apt-get the following (not exhaustive) list:

trac apache2 libapache2-mod-wsgi svn python python-pysqlite2 openssl maven ant tomcat5.5 libapache2-mod-jk

Now, to test out some of said software. First, make sure you're using some reasonable networking scheme. If you're lame like me and used NAT because you didn't want another interface just now (or you really didn't know what you were doing at the time), the Help for virtual box provides you a method of doing port forwarding from the host to the client. You need to restart the virtual-box GUI after you're done.

Creating a directory for the project

Run the following:

mkdir /path/to/my/project

Setting up Subversion

Run the following:

svnadmin create /path/to/my/project/svn

Obviously replacing the path.

There's some more magic we have to do here eventually (and integrate in with Apache). I'm going to cover this later, as I'm running out of steam.

Installing Trac

you need to set up a trac project:

mkdir /path/to/my/project/trac
trac-admin /path/to/my/project/trac initenv

Enter a project name. I took all defaults except for Subversion location (/path/to/my/project/svn).

You then need to make a file for the WSGI module, mine looks like the following:

import os

os.environ['TRAC_ENV'] = '/path/to/my/project/trac'
os.environ['PYTHON_EGG_CACHE'] = '/path/to/my/project/trac/eggs'

import trac.web.main
application = trac.web.main.dispatch_request

My file is called main.wsgi and is located in /path/to/my/project/trac.

Apache Configuration

Now it's time to edit apache configuration (I'm lazy soo....)

sudo vi /etc/apache2/sites-available/default

Ok, inside here we need to change the app to be our TRAC project...

THEN we need to give apache ownership of the TRAC files...

chown -R www-data /path/to/my/project/trac

You now have Trac running on Apache (with an SVN repository).

Trac Plugins

Of course no software distribution is complete without the mandatory downloading of plugins. Trac is no exception. Let's pick out some fun ones:

ContinuTrac - Continuum integration. We'll work with this after we install continuum on tomcat.
GraphViz - Yes... I can make intersting graph-like structures that will be pretty and amaze people from 1980! (Actually I love graphviz wiki plugins)
WebAdmin - A necessary web admin console. Has a lot of plugins into it
TagsPlugin - Folksonomy, or Release Tagging or Neither... you figure it out
ScreenshotsPlugin - Can has Bling Pix? Kthx
TracDownloaderPlugin - For Actually releasing products for download. (vs. libraries via Archiva)

So... if I ever get enough time I might place some script around all this to auto-generate the svn repostiory, trac directory and apache configuration, along with default installs of the plugins. That way you can make new projects quickly and easily AND get all the nice management features (which is the major point of maven).

So... What did we get this episode?

Source Code Repository (At least visualizing. More work later for http/https access).
Issue Tracker
Project Wiki
File Release System (with some plugin work)

Not bad eh? If we add the "magic project creation script" with a web front end, we might even have something to fight sourceforge with. But onto the real meat of this server... the nice Maven integration.