Frameworks, Libraries and Templates in Arachne

Arachne is a web framework for Clojure. Immediately, this provokes two questions. How is it different from other Clojure web frameworks that are out there? And why use a framework at all, instead of simply composing libraries?

These are good questions, and the answers to them inform both the rationale and the design of Arachne.

First, let's define a few terms, just to get on the same page. These aren't necessarily the global definitions of these words, or even the best definitions, but they're very useful for exploring the similarities and differences between different tools in the web space.

Definition: Library

Libraries are code or programs written and maintained by someone else, which you can call from your code.

This pattern should be familiar to all programmers – unless you're writing for micro-controllers in raw assembly language, it's impossible to build anything without calling into code that you didn't write yourself.

Definition: Framework

Frameworks are code or programs written and maintained by someone else, which call your code. The fancy name for this is Inversion of Control.

Every programmer also uses frameworks pervasively, whether they realize it or not. Modern operating systems are prototypical frameworks; you write a program that follows some defined interaction protocol, and the system initializes it and hands over some degree of control.

Managed language runtimes (JVM, .NET, etc) are also inherently frameworks. So are application servers, IoC containers, game engines, any application that supports plugins, and much more.

Frameworks are almost always coupled with library-style execution patterns. It's rare to have a framework that controls the execution of a developer's code without also providing hooks and utilities that developers can explicitly call themselves. Therefore, even though the definition of a framework is "calls your code", in practice it's usually a much more elaborate handing back and forth of control.

Definition: Template

There is one more concept that often comes into play: code generation, usually from some sort of code template.

Templates are code written by someone else, and have some sort of a one-time "generate" operation that inserts their code into your project. There, it resides alongside and on an equal footing with your hand-written source.

This often takes the form of the initial generation of an entire project (e.g. rails new or lein new luminus), or only certain files (e.g. rails generate)

In addition to specific, deliberate code generation there are a wide variety of other common activities that, loosely considered, constitute templates. These include copying an existing project to initialize a new one, copying snippets of code directly, or using the code generation features of an IDE.

The Holy Grail

LEGO®, the ideal programming environment LEGO®, the ideal programming environment

All of these techniques have one thing in common: they are a tool for sharing and reusing code, with the purpose of delivering faster. They allow developers to stand upon eachother's shoulders and move into new territory, rather than starting over every time.

Contrary to some popular perceptions, programming is an inherently social activity. It is inevitable. The programmer who starts with the work of another programmer will nearly always come out ahead of the lone wolf. It's no accident that collaborative open source models have become dominant in the industry. The only path to success lies in understanding and building upon what has come before.

And so, at the most fundamental level, these concepts of "libraries", "frameworks" and "templates" are not even technical constructs. They are the social conventions and patterns about how developers communicate, share, link and combine what they build.

Implications for Web Development

Web development has some specific characteristics which govern what and how web developers share and reuse.

  • There are a relatively large number of distinct concerns that need to be addressed.
  • There is a great deal of functional overlap between projects.
  • There are strong time and budget constraints.
  • Quality, polish and visual appeal have a direct correlation with success.
  • Bugs have a direct financial cost.
  • Projects have a long lifespan, and are continuously modified and expanded.
  • Projects require tight integration with traditionally non-programming tasks (e.g, UI design, copywriting, etc.)

These constraints motivate a high levels of reuse, overall. But the shape that takes – the ratio of library to framework to template-style reuse – is highly variable. Every solution uses all of them, to some degree. But in most applications, there is a dominant notion of reuse which shapes how the application is developed.

Each pattern has its own benefits and drawbacks. Partially, the choices are aesthetic. Different developers prefer different interaction styles. But there are also some real, objective tradeoffs between the different approaches.

Templates

Ancient Mesopotamian Template, c. 2600 BC Ancient Mesopotamian Template, c. 2600 BC

The good thing about templates is that they emit specific code really, really fast.

The bad thing about templates is that they emit specific code.

The degree to which templates are useful depends on how suitable the generated code is for you. After it's generated, how much of it will you have to change? Immediately? What about over the lifetime of the project? If the code is exactly what you need, they can be a huge speed boost. If they require extensive modification, or include features that you don't want, they can be more trouble than they're worth. If the time required to modify them to suit your needs exceeds the effort to simply implement what you need directly (or using other techniques) they have a negative value proposition and it's better not to use them at all.

Templates are also interesting because although you don't write the code initially, you are still fully responsible for maintaining it. Once the generation process is complete, it's your code. And your code is a liability. Every line of code adds to your project's inertia, and every line contains potential bugs.

Offsetting this, somewhat, is the fact that templates are the only means of reuse that are self-documenting, or even educational. You have to read the docs and figure out how to call a library or integrate with a framework. But after you generate a template, it's all right there. You can see exactly what it's doing, and learn from what are (hopefully) established, solid patterns. A template can actually teach you how to use a library or framework, or at least get you started.

Libraries

Libraries are probably the simplest and most general way to reuse code. You can call them using the same programming constructs you use to call your own code, but you're not responsible for writing or maintaining it. To the degree that a library does what you want, it's pure win. It moves your project forward with little cost to you.

This is why programming based on composing libraries is extremely popular, particularly in the Clojure community. There is a strong meme that composing many small libraries, not using a monolithic framework, is the best way to write software in Clojure.

While it's true that the community has gotten a lot of mileage out of purely library-based solutions, they are not without their downsides.

Their weakness is that they don't know about your code, and they don't know about other libraries (except ones they depend upon.) They are unrestricted and widely varied in how they are called, and what data formats they require. Your program bears the responsibility for calling each library in the manner in which it expects to be called.

An application and its libraries An application and its libraries

In many situations this isn't a problem. Programming against a handful of libraries is rarely cumbersome. But given the characteristics of webapps, as described above – large, diverse, and highly dependent on reuse – the mismatches start to multiply. The effect is magnified when the libraries are structural, opinionated and impose an idiom for working with your data (as many web libraries do.)

Quickly, glue code emerges; code that exists only to integrate with libraries. The more you depend upon libraries to deliver your app's functionality (as you should, in a webapp) the stronger the effect becomes until the application is mostly just a broker between different libraries.

This isn't terrible, of course. It's a much better outcome than not having libraries available at all. The cost of writing some glue code is a relatively small price to pay for not having to re-implement primary functionality.

The worst effect is how it solidifies an application. It means you can't swap out one part for another, without replacing all the relevant glue code as well. It adds inertia that must be overcome for any changes, refactorings and improvements.

And in some cases, even that isn't so bad. After all, how often does any given application need to switch out the core libraries? For something built in house, the answer is "not often."

But what's currently holding Clojure web development back is that many of the common web frameworks and toolkits are not actually frameworks (given the definition given above). They tend to be templates, consisting of libraries and the associated glue code. And for these, it is necessary to swap out libraries, unless the bundled ones precisely meet your needs.

It's true: they can let you get started quite quickly. Just like cloning an existing, working application can get you started quickly. But once you have, modifying the emitted code to suit the needs of your current task is still a lot of work. And, particularly if you disagree with some of the technical approaches taken in the core application, that might not be worth it. By only providing an easy path for a particular stack of libraries and development styles, these toolkits limit their applicability.

If one of these solutions meets your needs, wonderful! You should use it, and be grateful and happy. But if it doesn't, you may find yourself looking for something else.

Frameworks

So, how are frameworks any better? Are they any better? The answer is no. Not inherently. Like the other approaches, they have tradeoffs.

Frameworks still have a lot of glue code; indeed, you could argue, they are the glue code. The difference is that they are glue code which is written and maintained by someone else.

In a very real sense, if you use an actual framework (using this definition), you aren't building an application. The framework is the application. All you are doing, as a developer, is writing plugins or extensions for it which customize it for your use case.

As such, frameworks often have a tendency to become massive monoliths that deliver everything and the kitchen sink. They are, of necessity, extremely opinionated - the way the framework is put together is the way that the framework is put together, and if you don't like it you have no recourse except not to use it.

The benefit of frameworks is that, when done correctly, they lower the amount of work that application developers are responsible for to the absolute minimum. Developers aren't on the hook for very much at all, not even interfacing with their chosen libraries. Their sole job is to conform to the shape and requirements of the framework, and implement their own unique functionality.

A simple framework A simple framework

Of course, the cost of this is that if the framework is going to have any flexibility to use different libraries or ways of solving problems, enabling such support is exclusively the framework's responsibility. The user does not have the flexibility to make their own decisions about what to us or how to use it. Many frameworks don't do a good job of providing enough options, or integration points, and so end up as massive crystalline edifices that aren't very flexible.

Some frameworks do actually do a good job of providing integration points and useful abstractions. They treat extensibility as a feature, and are designed from the ground up

One good example of an extensible framework is Rails. Some might argue that the mechanisms of extension it uses aren't the cleanest or most evident, but it's clear that real-world developers can use Rails very effectively for different kinds of web applications. There is a selection of gems for nearly every occasion, and by and large they don't take too much effort to get to work with eachother.

An even better example of an extensibility-oriented framework, albeit one that it isn't so popular any more, is Spring. Spring had many, many problems, but one thing it got mostly right is that everything was programmed against interfaces, and you could explicitly control what concrete classes you wanted to use at any point. Not only that, but there was no top-level, global set of interfaces or extension point. Each component defined the interfaces for the points where it integrated with other components, meaning you could drastically change functionality by swapping out objects at different levels. Because of Java and Spring's other flaws, it would be a stretch to say that building extensions or modifying behavior was easy, but it was always possible, while still retaining most the benefits of having a framework.

What about Arachne?

Given this landscape, where does Arachne fit? What patterns of reuse does it leverage, in what proportion, and how does it emphasize their strengths while mitigating their weaknesses?

Given the definitions above, existing Clojure web stacks tend to be mostly libraries, held together by a template, with maybe a few framework-style elements sprinkled in.

Arachne, on the other hand, is unabashedly a framework; it consists of a prebuilt runtime that calls the code which you provide it. Of course, it does exhibit properties of all three. It is an attempt to find a point in the space of possibilities that maximizes the desirable outcomes: development speed, ease, maintainability and quality.

Templates in Arachne

The best way to describe Arachne's relationship to code generation is to say that Arachne has templates, but it is not a template. It uses templates to get started in seconds, and to provide a ready-made examples of bare-bones applications, but not to deliver any of the core functionality. Templates of Arachne projects are fairly small, and will only contain code that ought to belong to (and be maintained by) the user, not any general-purpose or glue code.

This also means that there are multiple valid shapes for Arachne projects. Arachne is not a directory layout, or a particular namespace structure, or anything that you will find in a template. Arachne will ship with multiple options: one for a Leiningen project and one for Boot, at least.

In the future, it's also likely that there will be templates for different types of Arachne projects. For example, by using different combinations of modules, Arachne can be a static site generator, a simple RESTful web service, a form-driven CRUD app or a rich ClojureScript & React single-page application. Ultimately, if we are successful, there will be a template for each of these and more.

The current plan is for templates to be extremely low overhead, with no specific code generation tool. Neither will we rely on a specific external tool (such as Leiningen project templates.) Instead, templates will be available as simple GIT repositories which you can clone, and which include a script to rename themselves to match your project.

Libraries in Arachne

Like all frameworks, Arachne does offer some libraries. It will provide a variety of namespaces and functions that you can require and invoke, as utilities and also to interact with the framework at runtime.

However, given the plethora of quality libraries that are already available for most tasks, Arachne will not focus on re-implementing basic web functionality that already exists in library form. Web development requires hundreds of distinct tasks ranging from crypto to data persistence to string manipulation, and re-inventing even a portion of those would be clearly out of scope given the resources available.

Instead, it will focus on making it straightforward to incorporate existing libraries into an Arachne projects. You can always call libraries as libraries from code that you write, without any permission or involvement from Arachne. And for libraries that benefit from deeper integration, or help provide structure to your application (such as servers, asset converters, etc), Arachne will make it easy to write lightweight shims that turn them into Arachne modules, with full awareness of and participation in the framework.

Frameworks in Arachne

As mentioned above, Arachne is a framework. But it is a framework that takes modularity and reuse seriously in an attempt to not become monolithic. In fact, the Arachne core isn't even webapp focused at all; it's a bootstrapping system for a hierarchy of modules, which are all replaceable. Everything that Arachne does is implemented in modules, which in turn can depend upon higher-level, more abstract modules. Reuse and flexibility is baked in from the very beginning.

Arachne, with embedded and shimmed libraries Arachne, with embedded and shimmed libraries

This is so true that it's perfectly accurate to think of Arachne itself as a framework for building frameworks. It doesn't have a finite set of integration points – rather, each module can itself be extended. Every time you write an extension for Arachne, you increase the surface area for future extensions. And Arachne provides clean patterns for doing so.

Even more important, integration points for individual modules don't have to be created explicitly. Modules don't need to provide hooks or anticipate the need for every type of future extension. Instead, they explicitly define a data ontology for how they work themselves. This is observable and ultimately editable by potential extenders. Used in combination with more conventional methods of open dispatch in Clojure (multimethods and protocols), this pattern is intended to provide an unparalleled level of open extension, and will hopefully foster a large ecosystem of third-party modules, all extending eachother hierarchically.

Conclusion

If Arachne is successful, it will be because of this community of extension. If it is going to meet the massively varied needs of real-world users, it isn't enough to get started quickly. It isn't enough to have the theoretical ability to add or swap features quickly.

It will need to have all those things, coupled with the availability of real, useful extensions. If we can provide easy solutions to the most common general types of problems, then we can spend that much more time working on the features unique to our applications, and take one more small step towards the elusive goal of true, optimal code reuse.