Alpha Release


The moment has finally come… Arachne, while still far from complete, is ready to put in the hands of developers for testing and feedback. I'm excited to see what people think of it, and gather some data about what's awesome and what's not.

First, however, I'm going to state a big list of caveats, so nobody gets their expectations raised too high just yet.

  • Although many of the major underlying pieces are working, the current application development experience isn't yet close to what it will be. There's a snowball effect; each new module that becomes available will diminish the friction of application development just that much more. For example, the data abstraction layer (codename Chimera) will be huge, and an enabler for a lot of other tools that can't be started until it is ready. But it isn't quite done for this alpha release.
  • Everything is subject to change! I've developed things in a way that makes sense to me, and suits my tastes and style, but Arachne ultimately isn't about me; it's about putting power in the hands of people who want a fast and efficient path towards building powerful webapps. So if my tastes differ from yours, if there's anything about Arachne that you find awkward or distasteful, please let me know.
  • Related to the above: this is an alpha version of the software. That means that things can and will change. Until we release the first "production ready" builds, I can make no guarantee, or even put any effort, towards making sure that there are no breaking changes. If you build on top of Arachne, right now, you will be responsible for keeping up to date as things change underneath you.

With that out of the way, if you're still interested, what's in the release?

Here's what's ready:

  • Documentation
  • Code
    • The arachne-core module, with the configuration, module and runtime infrastructure.
    • The arachne-http module, which defines abstract HTTP concepts such as servers and routes.
    • The arachne-pedestal module, which provides a HTTP implementation based on Pedestal.
    • The arachne-assets module, which defines the basic concepts of a functional/immutable asset pipeline.
    • The arachne-cljs module, which uses the asset pipeline to compile ClojureScript.
    • The arachne-figwheel module, which builds on top of the CLJS module to provide a dynamic Figwheel-based development experience.
  • Community

There's still a lot to do in the near term that's missing:

  • Finish the tutorials, to cover all the modules that are working.
  • Write up the "module" documentation, which will be the best source if information for the concepts the modules introduce, their configuration schema, and the DSL they expose.
  • Write up a rudimentary config browser, to demonstrate the power of of a data-driven application to introspect application config.
  • Ship the kickstarter rewards. I've got this all lined up and ready to go, but haven't pulled the trigger on moving the physical goods yet. My apologies. I had intended to do this around the same time as the alpha release, but will do it this week instead. At any rate, backers should count on getting their stuff sometime in February.

I hope to finish up everything in the list above in the next two weeks, and then move on to the more long term roadmap. That roadmap includes, in rough priority order:

  • Finish "Chimera", Arachnes data and persistence layer. This will unlock a whole set of additional modules to work on.
  • Write modules to automatically expose APIs that can read/store data in a database. This will really start to demonstrate Arachne's immediate value for web services.
  • Write a user authn/authz module, to help users quickly bootstrap "real" webapps with real requirements and functionality.
  • Write some automated deployment tools, for "one click deploy" into cloud hosting providers. p It's hard to predict how many of these I will get done before my Kickstarter funding runs dry sometime in April, and I am no longer able to work on Arachne as much. But even when that happens I'll continue to work on it on an open-source basis (or figure out some other arrangment that will let me continue to make forward progress.)

At any rate, I hope you enjoy what we've produced so far, and don't forget to post your feedback. Thanks again for your support!



Community Update 5


Approximate Reading Time: 3 minutes

Arachne design and development is progressing since our last update. Since then, due to time constraints, we sent out a brief 'Interim Update' to the Arachne mailing list, which you can sign up for on our home page, arachne-framework.org.

On Dec 2, Luke gave a talk about Arachne at the Clojure eXchange 2016, which explains some more of the high-level reasoning about Arachne and why the status quo of composing small libraries isn't good enough.

The Steering Group met on Dec 16, when Luke shared that he is ready to announce Arachne's initial alpha release date of Jan 20, 2017.

Yay! But, What does this mean? It means that while not all of Arachne's fundamental work is done, the core work is. It's time to move this project into the public square, where you fine folks can start to play with a documented release of Arachne.

It's worth saying directly: Arachne will be alpha software. Production use would be unwise!

The purpose of this release is to open up the feedback process outside the bubble of the Steering Group and into the hands of the people that matter most – all of you!

This alpha release is primarily to start getting feedback on some of the core design elements, and to enable potential module authors to get started thinking about what and how their own plans might integrate with Arachne. Please be aware that it will not provide the "complete" Arachne experience… there are a lot of convenience modules and higher-level abstractions that have yet to be written. So if the alpha release of Arachne is missing a feature you were counting on, don't despair! It's almost certainly still in the queue for delivery. At the time we release the alpha, we will also publish a roadmap of what our current plans around next steps are.

The codebase will still be under a considerable amount of flux, so please check with us before submitting a large or far-reaching pull request. Eventually, once the core design is quite stable, we hope to reach a point where the majority of changes to Arachne are driven by the community itself.

Luke will be working on documentation between now and the release, as well as refining existing code.

As to design tweaks, Luke has been busy. Chimera has been simplified. Please see ADR-017 for more.

Also, Chimera operations are now extensible and overridable. Luke used core.match to take override expressions from the config and compile them into a fast dispatch mechanism, so that it's now possible to override Chimera's default behavior at a very, very granular level, based even on the content of message payloads. This feature is intended to provide extreme flexibility when porting data to or from a legacy database. Essentially, this provides a short-circuit to the default behavior of entity persistence.

Our next update will likely be in 3 weeks. We will resume our normal update schedule, an update every 2 weeks, after the Holidays.

My best,
Jay
@webappzero

P.S. Please tweet me with any ideas you have about Arachne or our Community.



Community Update 4


Approximate Reading Time: 2 minutes

Arachne design and development is progressing since our last update. Since then, due to time constraints, we sent out a brief 'Interim Update' to the Arachne mailing list, which you can sign up for on our home page, arachne-framework.org.

The Steering Group met on Nov 04, and Luke presented the concepts behind Chimera, Arachne's proposed data abstraction model.

We met again this Monday, where Luke showed us working Chimera code, which is well underway. Many questions were asked, many potential use cases were discussed, and Luke's design satisfied the professional scrutiny of his peers.

Chimera will be ready for more serious prodding and poking by the Steering Group in approximately 3 weeks, due to the pending Holiday and Luke's travel to London to speak about Arachne at Clojure eXchange 2016 on Dec 2.

Arachne's public alpha release has a soft target of January 2017. This date assumes that no major redesign is necessary.

Regarding minor design tweaks, Luke is unifying the design of Arachne Modules and Applications which will eliminate many of their current differences and provide more consistency between them. In fact, an Arachne application will now also be an Arachne Module.

Nola Stowe @clojuregeek and I will be at Clojure/Conj in Austin with Arachne stickers for anyone who asks, so please say hello if you're attending.

Our next update will likely be in 3 weeks. We will resume our normal update schedule after the Holidays.

My best,
Jay
@webappzero

P.S. Please tweet me with any ideas you have about Arachne or our Community.



Community Update 3


Approximate Reading Time: 3 minutes

Arachne design and development is progressing since our last update.

The Steering Group met on Oct. 14 and Luke presented his work on error reporting. Luke explained that his work on error reporting was motivated to smooth over several friction points that he was experiencing while developing Arachne. This work could potentially benefit the larger Clojure ecosystem and will likely be factored out of Arachne Core. Here's an example screenshot of Arachne's error message handling:

Sample verbose error message from Arachne

Arachne's Database Abstraction Layer (DAL) is taking form, conceptually. An ADR will be published soon with more details.

A new ADR regarding project templates has recently been published.

Luke will be presenting his work on Arachne at Clojure eXchange 2016 on Dec. 2, in London.

My progress on the Static Site Generator has been slow, because I'm learning lots of new things. Russ Olsen has kindly pitched in, not only in writing code, but mentoring me along the way.

A pre-alpha template now exists for both Arachne Modules and Arachne Apps. To clarify, Arachne Modules have modest requirements on the presence and location of certain files (see ADR-004), but Arachne Apps can be any arbitrary Clojure code that calls the appropriate Arachne libraries. Even so, it's still helpful to have a starting template for Arachne Apps. The Arachne App directory structure shouldn't be taken as a proposed convention, just one possible instance that happens to work as a starting point.

These materials are not in polished form, and Arachne docs have yet to be written.

The Steering Group will be meeting this Friday, so please look forward to another update in a week.

My best,
Jay
@webappzero

P.S. Please tweet me with any ideas you have about Arachne or our Community.



Community Update 2


Approximate Reading Time: 2 minutes

This update will be brief because most of our progress since our last update has been in getting started on new work.

Luke continues his work on the Database Abstraction Layer (DAL). He has also completed the research and initial implementation of user-friendly error messages.

The error message research investigated several interesting aspects of creating friendly error messages:

  • decoupling the presentation of errors to users from the propagation of errors through the program
  • errors from the command line
  • errors from logs
  • errors from the REPL
  • stacktrace cleanup tools
  • verbose spec explanations
  • UI-based errors

ADR 13 goes over these considerations in more detail and proposes a decision for Arachne's error reporting functionality.

My work on Arachne's static site generation capabilities began on Monday, Oct 10. The immediate goal is to replace this site with an Arachne site as soon as possible.

That's all the news we have for this update. Our next update should be published in two weeks. We aim to publish an update every other Wednesday.

Jay
@webappzero

P.S. Please tweet me with any ideas you have about Arachne or our Community.



Community Update


Picture: Arachne demo of web handler Picture: Arachne demo of web handler

Approximate Reading Time: 5 minutes

Hello Arachne Enthusiasts!

Arachne is taking form. What does its future hold? After sharing some news and updating you on our progress, this post peers into Arachne's future. Our last post was in early July, so Luke has been feeling torn between writing a community update and writing Arachne itself. That's why I'm writing you instead.

My name is Jay Martin. I'm a fledgling web programmer with a sweet tooth for Clojure and Datomic. I'll be helping Luke focus on building Arachne by serving as a community liaison.

So, what does a community liaison do? Mostly listen.

Arachne's community liaisons will listen to feedback from the Community and relay that feedback to Luke in digestible form, striving to honor the feedback's spirit and substance.

Also, we want to learn as much as we can about Luke's mental model for building software systems with Arachne. We intend to share that mental model with you, clearly and concisely.

Luke is passionate about fostering a culture of learning and knowledge sharing, as is the Steering Group.

You can expect updates on our progress about once every two weeks.

Also, keep an eye out for our first technical screencasts and Wiki articles covering Arachne fundamentals. We'll post links to new resources here on this site as they become available.

Many people have stepped forward to offer Luke their help. Luke is deeply inspired by this outpouring of support. The Steering Group sees a risk to the project's success by opening up the development process too early, while Luke is simply getting ideas out of his head and into an initial project structure. The opposite risk for Arachne is that of isolating the development process from the Community, alienating the very people who will help Arachne reach its full potential – you, and every single supporter, code contributor, Kickstarter, developer, designer, author, blogger, speaker and tire kicker out there.

We aim to strike a pragmatic balance between these competing risks and we'll rely on you to let us know when we're off track. If you have ideas about how to improve Arachne, please open a GitHub issue. If you have ideas about how to make our Community Experience the very best in this quadrant of the galaxy, please send me a tweet @webappzero.

We'll open up more lines of communication, as needed to best support Arachne's community with the care and attention it deserves. We'll include Community Developers and Designers in the creative process once Arachne's fundamental pieces are more concrete. The idea being that it's easier to talk en masse about improving tangible code than intangible concepts. Presently, that's the role of the Steering Group. Luke shares his ideas and progress and then listens to our feedback. Gradually, the Community as a whole will replace the Steering Group in that role.

Speaking of Arachne's core pieces, here's a list of things that Luke has gotten done, adding to his original work, since officially starting work on Arachne on July 18th:

  • implemented continuous delivery via CircleCI
  • fully Speced the core module
  • implemented an ontology for config schemas
  • reified config & runtime entities in the config database
  • wrote utilities for error messages
  • factored apart the abstract HTTP & Pedestal modules
  • added provenance info to config & init scripts
  • designed Spec-based config & runtime validation
  • wrote 12 Architectural Decision Records (ADRs)

…and finally, saving the best for last:

  • created three working demos of Arachne's core functionality!

Yes! We've got demos. They reveal the inner-workings of the low-level Arachne system. These are largely undocumented so only the adventurous should expect to receive a learning outcome by attempting their execution. Also, keep in mind that these primitives are very low level and much friendlier DSLs (Domain Specific Languages) can and will be built on top of what you'll see there now.

My first impression of Arachne, based on seeing the working demos, is that of a system which vanquishes mystery and embraces accountability. Arachne's validated, acyclic, runtime system graph ignited atop a queryable, validated, schema compliant config value feels like a new level of control and awareness when initializing a software system. And while Arachne is marketed as a web framework, at its core its a framework of frameworks, a novel way of organizing a software application with the potential to influence the way we write server, desktop and even mobile software.

This level of control comes with a price: learning a new way of doing things that is, as yet, unproven in the real world of shipping software. Existing Clojure Libraries and software will need to be wrapped in an interface to comply with the Arachne Module protocol in order to reap the benefits described in the ADR on abstract modules.

When I saw my first Arachne demo I felt a little overwhelmed by all the new words like 'ontology', 'config schema', 'component' and how they all fit together. At that time, I'd never even used a Stuart Sierra Component, so it was a lot to take in.

For some, this price will appear too steep. Fortunately for Arachne, our Industry is awash in change. Driving us toward this new future is the mere possibility that the inherent frustrations of building and managing software systems could be significantly mitigated. Many have already opted for a future that includes Arachne by voting for its existence with their time, talent and treasure. For that we cannot thank all of you enough!

Arachne's immediate future will see Luke writing the Database Abstraction Layer (DAL). The DAL will include the common set of database primitives and commands shared by today's prevalent database systems: entities, queries, etc. I'll be working on the Static Site Generator which will allow us to dogfood Arachne by using it to host this site in the near future.

Concluding with a bit of philosophical conjecture about Arachne's future:

I recently learned about solving problems using recursion from a web page by Carnegie Mellon University:

To solve a problem recursively means that you have to first redefine the problem in terms of a smaller subproblem of the same type as the original problem. […] This step is often called a recursive leap of faith. Before using a recursive call, you must be convinced that the recursive call will do what it is supposed to do. You do not need to think how recursive calls work, just assume that it returns the correct result.

Arachne isn't pulled from the ether. It benefits from a host of innovations, built by people that answered countless challenges and shared their life's work with the world. It has been refined by many generous and thought provoking conversations within the Clojure and programming Communities. Something else will come after Arachne and It will be better than It could have been because of what Arachne is and what it is not. Ultimately, Arachne's success will depend on our collective willingness to convince ourselves that the pleasure it promises will outstrip the pain it hides and move ourselves to that self-evident and demonstrable leap of faith we call – action!

Jay
@webappzero

P.S. Tweet me with any ideas you have about Arachne or our Community. Or just to say hi!



Retrospective and Risk Assessment


Last Friday, the steering group held its first retrospective and risk assessment. The goal of this meeting was to analyze what's gone right and what's gone wrong so far throughout this lead-in period, and to try and forecast the major risks to avoid as Luke starts full-time development.

The discussion proceeded with each person identifying a potential risk, and brief discussion of each item.

Near Term Risks

  • Luke is the only person full-time on Arachne: will the steering group have time to be sufficiently prepared for meaningful input?
  • Is everyone clear on their roles in the project, and what's expected of them?
    • The only expectation from the steering group is that they spend a couple hours every few weeks thinking about the project, and identify any screw-ups or mis-steps they can see. Additional levels of involvement are welcome but not necessary.
  • Not only do we need to make sure existing roles are clear, but we need to make sure these stay clear as we ramp up community involvement, and let community members participate meaningfully without derailing forward progress or becoming chaotic.
  • We could be either too ambitious, or not ambitious enough. We need to make sure we thread a fairly narrow corridor, and try to stay in the sweet spot that produces something genuinely new and useful, while not getting mired down in attempting to solve every problem in the world.
  • We need to make sure that there is enough of a feedback loop to help Luke find that sweet spot. That feedback loop should be more rapid than just these bi-weekly sync-up meetings. What are some ways people who are interested can get involved in a more active way?
  • There is a risk that we could go off on technical tangents instead of solving the real/core problems that people have.
    • Having a real client from day 1 (Apex) is a mitigation here.
  • So far, we've been weighted a bit heavily towards up-front design rather than iterating on an MVP.
    • That's true, we will try to move more towards an iterative model once we're working full-time and the core modules are done. We definitely want to prefer an iterative, agile model of development.
  • Luke might try to take on too much instead of delegating or seeking help.
    • Steering group can try to help more, as they have time. Ideas for things that they could do to help Luke focus on core tasks: website/docs admin, shipping Kickstarter rewards. Anything else?
  • We need to be careful not to get lost aiming too high and trying too much to "change the world" - we can do as much as we have time for, but need to remember that we're actually on the hook to deliver a web framework first.
  • We need to get the foundations in place so we and others can use Arachne as a platform, from which to build the really cool/big stuff.
  • As a counterpoint, let's be careful to actually aim high enough to do something meaningful; let's not be so focused on not flying into the sun that we stay stuck in the mud.
  • What exactly are we afraid of being too ambitious about?
    • Aiming too far ahead and designing for too much generality could slow development down. If we always choose the technical approach that is "50% more work, but better for the future…" that could put our schedule at risk.
    • We might try to be more than a web framework before we even become a web framework. This increases risk in many ways, both from a schedule point of view and running the risk that we're not solving a real problem (or not solving it in the right way). We need to focus on building and iterating on a web framework first, and going from there.
  • Luke's output is about to increase 15x as he goes full-time. Are we ready for this?
    • Questions will shrink in scope as time goes on; but these early questions are vitally important.
    • Ever design decision will have an ADR for review.
    • Code reviews from steering group are desirable but optional.
    • Will soon be a public repo, can "crowdsource" code review.

Long Term Risks

  • Open source project, especially those that aspire to have community involvement, risk an explosion of configurations, which can lead to lots of complexity.
  • Backwards compatibility can be hard to maintain - how long do we want to support a bad decision made early on?
    • We can either make the choice to preserve or break backwards compatibility, but we should make these choices explicitly rather than letting them linger ambiguously.
  • If we don't hit some critical mass for adoption, Luke will have to write everything, which will limit the amount/quality/utility of what Arachne (and it's modules) can offer.
  • There will be other challenges we can't foresee. We need to make sure we deal with enough of the known problems that we have time for unknown and unforeseen issues.
  • We risk having bad or out-of-date documentation. We need to remain vigilant and set up processes such that the docs closely match the code. It can be very frustrating to the community if the documentation is inadequate or out-of-sync.
  • We need to make sure we stay organized, and will probably need a more powerful backlog and issue-management system than Trello in the long term.
  • Even if we technically succeed, if we haven't succeeded socially, we will have failed. Arachne's community needs to be more than just an arrogant group of white dudes looking down their technical noses: we need to be welcoming, beginner-friendly and diverse.


Update and Schedule


I am pleased to announce that we finally have a concrete schedule for starting real work on Arachne. By July 18 I will have wrapped up my current consulting engagements and turned my full attention to Arachne.

During this time, I will also be working closely with Apex Data Solutions. Apex is a healthcare technology company that has already invested substantially in Arachne, via a generous donation to the Kickstarter that significantly helped to ensure that the campaign would succeed.

The current plan is to consult with them for two days of every week, specifically around Arachne and how it can further their technological and business goals. During this time I can work either on Arachne's core (as it is relevant to Apex's needs), Apex-specific modules and functionality, or other software as the need arises.

The remaining three days will be dedicated exclusively to general open-source Arachne work, building it out according to the vision outlined in the Kickstarter pitch. The Kickstarter funds will only be used to pay me on these days when I'm focused purely on open-source. Task prioritization during these days will still be governed by the steering group, to maintain focus on the core general-purpose product.

By joining forces in this way, we get two significant benefits:

  • Arachne will have a real world, rigorous use case from inception, giving an opportunity to "dogfood" the product in realtime.
  • Presuming this arrangement continues, this gives at least 10 months of runway for working on Arachne, as opposed to the ~4 that the Kickstarter funds alone would have provided.

I think this is a great mutually beneficial opportunity, and I look forward to working with Apex as well as developing the core web framework.



Steering Group #2


These are the notes for the second meeting of the Arachne steering group, which occurred on Friday, May 6.

The attendees this week were:

  • Tim Ewald
  • Jay Martin
  • Russ Olsen
  • Stuart Sierra
  • Luke VanderHart

This was a smaller, shorter meeting than the last one. Because Luke's efforts the last couple weeks were spent on public-facing and Kickstarter-related materials, there was no new technical content to discuss in depth. We did, however, cover a number of other topics.

Also of note, this was our first meeting to include Jay Martin, who backed the Kickstarter campaign at the $10k level and will be joining the steering group going forward. He has a background in IT and finance, and the Arachne project resonated with him after getting back into web development with Python and Clojure. He is enthusiastic about the project and eager to be involved in development, educational materials, and ensuring that the project remains easy and accessible to beginners.

Topic: Engaging better with the Clojure community

Although public reception of the project has been largely positive, there have also been some negative reactions. We spent some time discussing the community dynamics and how we can ensure we remain friendly, inclusive and supportive of the Clojure web-development community in its entirety. Some suggestions were:

  • Stay positive
  • Focus on the novel features and vision of Arachne, not any perceived deficiencies in alternatives.
  • Answer questions and stay engaged, but put the primary focus on releasing software that will speak for itself. Show, don't tell.
  • Focus on users who already feel the need for something different. Those who are happy with the status quo will remain happy; Arachne should focus on helping those who want a more framework-heavy approach and growing the webdev community, and is not in any way a contradiction or threat to the existing ecosystem.

Topic: Engaging better with the non-Clojure web development community

So far, we haven't managed to engage at all with the non-Clojure web development community. We discussed why this is; the consensus was that there's no reason anyone should care, given that nothing really exists yet. Once Arachne is released, we can reach out to grow our user base by telling a compelling story and providing some demos of Arachne's capabilities. We can't expect to grow outside the Clojure community until we have this.

We also discussed how people evaluate frameworks based both on "hygiene factors" (doing basic things in a easy and clean way), and "selling points" (unique capabilities that bring people to the framework.)

Some ideas:

  • The Rails-esque "write a blog in 15 minutes" is now table stakes for a modern web framework; we need something comparable.
  • Create demos showcasing the "hygiene factors" - ease,
  • Create demos showcasing the "selling points." Some examples for Arachne could be:
    • Modularity: live-code swapping out major components of an app.
    • Introspectability: demo a dashboard exploring the config of an application.
  • Demo a one-button deploy of the system.
  • Tell a good story about security, baked in from the ground up.

Topic: brainstorming asset pipelines

Asset pipelining is an extremely important and somewhat difficult functionality for any web framework.

Some questions:

  • Do we build something from scratch in Clojure, or offload to an existing Ruby or Node tool (the approach taken by Phoenix, which offloads to Brunch)
  • Do all the tools need to run in the JVM or are we OK shelling out to external processes? What are the implications of that for setup complexity?
  • What are the actual "must have" tools? Can we get away with not supporting tools that only have an implementation in one language?
  • What are existing Clojure tools in this space? (Boot tasks, twixt, stefon, etc.)
  • What strategy should we use for cache busting?
  • For development workflows, should we rebuild assets using filesystem watchers, or build on-demand when a request for an asset is received (or some hybrid approach?)

It would be nice if the asset pipeline were complete and robust enough that a project using only the asset pipeline component was a complete, usable static site generator.

Ultimately we came out with lots more questions and few or no firm answers, but it was good to have the discussion.

Topic: abstract data layer

At this point in the conversation, we were getting pressed for time, so we had little opportunity to do more than bring up the topic for future thought.

In general, the idea is to have a means of specifying persist-able domain entities in a way that is agnostic to any particular database. This means defining a common schema format and persistence API that will work with JDBC, Datomic, various NoSQL databases, etc.

This has three primary use cases.

  • Modules sometimes need to persist data, but they should not be strongly coupled to any particular choice of storage, for greater modularity. Such modules need to be able to program exclusively against an abstract data storage API.
  • Modules should be able to re-purpose an entity definition for uses other than persistence. For example, a module might take an entity definition schema and expose it as a RESTful resource, or generate a form to edit it.
  • Users should not hesitate to exercise the full power of their chosen database, rather than the greatest common denominator of database features, which will be the most the abstract layer can do (by necessity). However, the most common 70% of database interactions should still be extremely easy via the abstract API.

Several points were raised:

  • To make this problem tractable, we need to be aggressive about limiting the scope of the abstract data layer. It simply can't do everything well and still be manageable.
  • We need to have a good story about schema generation, but also about connecting to existing databases with pre-existing arbitrary schemas.
  • We need to be able to handle sets of columns/attributes that are constructed at query time as well as concrete tables; there is not a necessarily a 1:1 mapping between your application's concept of an "entity" and the data store's concept (e.g, a Datomic entity or a SQL table)

At this point, we used up our allotted hour and adjourned until the next meeting.



Frameworks, Libraries and Templates in Arachne


Arachne is a web framework for Clojure. Immediately, this provokes two questions. How is it different from other Clojure web frameworks that are out there? And why use a framework at all, instead of simply composing libraries?

These are good questions, and the answers to them inform both the rationale and the design of Arachne.

First, let's define a few terms, just to get on the same page. These aren't necessarily the global definitions of these words, or even the best definitions, but they're very useful for exploring the similarities and differences between different tools in the web space.

Definition: Library

Libraries are code or programs written and maintained by someone else, which you can call from your code.

This pattern should be familiar to all programmers – unless you're writing for micro-controllers in raw assembly language, it's impossible to build anything without calling into code that you didn't write yourself.

Definition: Framework

Frameworks are code or programs written and maintained by someone else, which call your code. The fancy name for this is Inversion of Control.

Every programmer also uses frameworks pervasively, whether they realize it or not. Modern operating systems are prototypical frameworks; you write a program that follows some defined interaction protocol, and the system initializes it and hands over some degree of control.

Managed language runtimes (JVM, .NET, etc) are also inherently frameworks. So are application servers, IoC containers, game engines, any application that supports plugins, and much more.

Frameworks are almost always coupled with library-style execution patterns. It's rare to have a framework that controls the execution of a developer's code without also providing hooks and utilities that developers can explicitly call themselves. Therefore, even though the definition of a framework is "calls your code", in practice it's usually a much more elaborate handing back and forth of control.

Definition: Template

There is one more concept that often comes into play: code generation, usually from some sort of code template.

Templates are code written by someone else, and have some sort of a one-time "generate" operation that inserts their code into your project. There, it resides alongside and on an equal footing with your hand-written source.

This often takes the form of the initial generation of an entire project (e.g. rails new or lein new luminus), or only certain files (e.g. rails generate)

In addition to specific, deliberate code generation there are a wide variety of other common activities that, loosely considered, constitute templates. These include copying an existing project to initialize a new one, copying snippets of code directly, or using the code generation features of an IDE.

The Holy Grail

LEGO®, the ideal programming environment LEGO®, the ideal programming environment

All of these techniques have one thing in common: they are a tool for sharing and reusing code, with the purpose of delivering faster. They allow developers to stand upon eachother's shoulders and move into new territory, rather than starting over every time.

Contrary to some popular perceptions, programming is an inherently social activity. It is inevitable. The programmer who starts with the work of another programmer will nearly always come out ahead of the lone wolf. It's no accident that collaborative open source models have become dominant in the industry. The only path to success lies in understanding and building upon what has come before.

And so, at the most fundamental level, these concepts of "libraries", "frameworks" and "templates" are not even technical constructs. They are the social conventions and patterns about how developers communicate, share, link and combine what they build.

Implications for Web Development

Web development has some specific characteristics which govern what and how web developers share and reuse.

  • There are a relatively large number of distinct concerns that need to be addressed.
  • There is a great deal of functional overlap between projects.
  • There are strong time and budget constraints.
  • Quality, polish and visual appeal have a direct correlation with success.
  • Bugs have a direct financial cost.
  • Projects have a long lifespan, and are continuously modified and expanded.
  • Projects require tight integration with traditionally non-programming tasks (e.g, UI design, copywriting, etc.)

These constraints motivate a high levels of reuse, overall. But the shape that takes – the ratio of library to framework to template-style reuse – is highly variable. Every solution uses all of them, to some degree. But in most applications, there is a dominant notion of reuse which shapes how the application is developed.

Each pattern has its own benefits and drawbacks. Partially, the choices are aesthetic. Different developers prefer different interaction styles. But there are also some real, objective tradeoffs between the different approaches.

Templates

Ancient Mesopotamian Template, c. 2600 BC Ancient Mesopotamian Template, c. 2600 BC

The good thing about templates is that they emit specific code really, really fast.

The bad thing about templates is that they emit specific code.

The degree to which templates are useful depends on how suitable the generated code is for you. After it's generated, how much of it will you have to change? Immediately? What about over the lifetime of the project? If the code is exactly what you need, they can be a huge speed boost. If they require extensive modification, or include features that you don't want, they can be more trouble than they're worth. If the time required to modify them to suit your needs exceeds the effort to simply implement what you need directly (or using other techniques) they have a negative value proposition and it's better not to use them at all.

Templates are also interesting because although you don't write the code initially, you are still fully responsible for maintaining it. Once the generation process is complete, it's your code. And your code is a liability. Every line of code adds to your project's inertia, and every line contains potential bugs.

Offsetting this, somewhat, is the fact that templates are the only means of reuse that are self-documenting, or even educational. You have to read the docs and figure out how to call a library or integrate with a framework. But after you generate a template, it's all right there. You can see exactly what it's doing, and learn from what are (hopefully) established, solid patterns. A template can actually teach you how to use a library or framework, or at least get you started.

Libraries

Libraries are probably the simplest and most general way to reuse code. You can call them using the same programming constructs you use to call your own code, but you're not responsible for writing or maintaining it. To the degree that a library does what you want, it's pure win. It moves your project forward with little cost to you.

This is why programming based on composing libraries is extremely popular, particularly in the Clojure community. There is a strong meme that composing many small libraries, not using a monolithic framework, is the best way to write software in Clojure.

While it's true that the community has gotten a lot of mileage out of purely library-based solutions, they are not without their downsides.

Their weakness is that they don't know about your code, and they don't know about other libraries (except ones they depend upon.) They are unrestricted and widely varied in how they are called, and what data formats they require. Your program bears the responsibility for calling each library in the manner in which it expects to be called.

An application and its libraries An application and its libraries

In many situations this isn't a problem. Programming against a handful of libraries is rarely cumbersome. But given the characteristics of webapps, as described above – large, diverse, and highly dependent on reuse – the mismatches start to multiply. The effect is magnified when the libraries are structural, opinionated and impose an idiom for working with your data (as many web libraries do.)

Quickly, glue code emerges; code that exists only to integrate with libraries. The more you depend upon libraries to deliver your app's functionality (as you should, in a webapp) the stronger the effect becomes until the application is mostly just a broker between different libraries.

This isn't terrible, of course. It's a much better outcome than not having libraries available at all. The cost of writing some glue code is a relatively small price to pay for not having to re-implement primary functionality.

The worst effect is how it solidifies an application. It means you can't swap out one part for another, without replacing all the relevant glue code as well. It adds inertia that must be overcome for any changes, refactorings and improvements.

And in some cases, even that isn't so bad. After all, how often does any given application need to switch out the core libraries? For something built in house, the answer is "not often."

But what's currently holding Clojure web development back is that many of the common web frameworks and toolkits are not actually frameworks (given the definition given above). They tend to be templates, consisting of libraries and the associated glue code. And for these, it is necessary to swap out libraries, unless the bundled ones precisely meet your needs.

It's true: they can let you get started quite quickly. Just like cloning an existing, working application can get you started quickly. But once you have, modifying the emitted code to suit the needs of your current task is still a lot of work. And, particularly if you disagree with some of the technical approaches taken in the core application, that might not be worth it. By only providing an easy path for a particular stack of libraries and development styles, these toolkits limit their applicability.

If one of these solutions meets your needs, wonderful! You should use it, and be grateful and happy. But if it doesn't, you may find yourself looking for something else.

Frameworks

So, how are frameworks any better? Are they any better? The answer is no. Not inherently. Like the other approaches, they have tradeoffs.

Frameworks still have a lot of glue code; indeed, you could argue, they are the glue code. The difference is that they are glue code which is written and maintained by someone else.

In a very real sense, if you use an actual framework (using this definition), you aren't building an application. The framework is the application. All you are doing, as a developer, is writing plugins or extensions for it which customize it for your use case.

As such, frameworks often have a tendency to become massive monoliths that deliver everything and the kitchen sink. They are, of necessity, extremely opinionated - the way the framework is put together is the way that the framework is put together, and if you don't like it you have no recourse except not to use it.

The benefit of frameworks is that, when done correctly, they lower the amount of work that application developers are responsible for to the absolute minimum. Developers aren't on the hook for very much at all, not even interfacing with their chosen libraries. Their sole job is to conform to the shape and requirements of the framework, and implement their own unique functionality.

A simple framework A simple framework

Of course, the cost of this is that if the framework is going to have any flexibility to use different libraries or ways of solving problems, enabling such support is exclusively the framework's responsibility. The user does not have the flexibility to make their own decisions about what to us or how to use it. Many frameworks don't do a good job of providing enough options, or integration points, and so end up as massive crystalline edifices that aren't very flexible.

Some frameworks do actually do a good job of providing integration points and useful abstractions. They treat extensibility as a feature, and are designed from the ground up

One good example of an extensible framework is Rails. Some might argue that the mechanisms of extension it uses aren't the cleanest or most evident, but it's clear that real-world developers can use Rails very effectively for different kinds of web applications. There is a selection of gems for nearly every occasion, and by and large they don't take too much effort to get to work with eachother.

An even better example of an extensibility-oriented framework, albeit one that it isn't so popular any more, is Spring. Spring had many, many problems, but one thing it got mostly right is that everything was programmed against interfaces, and you could explicitly control what concrete classes you wanted to use at any point. Not only that, but there was no top-level, global set of interfaces or extension point. Each component defined the interfaces for the points where it integrated with other components, meaning you could drastically change functionality by swapping out objects at different levels. Because of Java and Spring's other flaws, it would be a stretch to say that building extensions or modifying behavior was easy, but it was always possible, while still retaining most the benefits of having a framework.

What about Arachne?

Given this landscape, where does Arachne fit? What patterns of reuse does it leverage, in what proportion, and how does it emphasize their strengths while mitigating their weaknesses?

Given the definitions above, existing Clojure web stacks tend to be mostly libraries, held together by a template, with maybe a few framework-style elements sprinkled in.

Arachne, on the other hand, is unabashedly a framework; it consists of a prebuilt runtime that calls the code which you provide it. Of course, it does exhibit properties of all three. It is an attempt to find a point in the space of possibilities that maximizes the desirable outcomes: development speed, ease, maintainability and quality.

Templates in Arachne

The best way to describe Arachne's relationship to code generation is to say that Arachne has templates, but it is not a template. It uses templates to get started in seconds, and to provide a ready-made examples of bare-bones applications, but not to deliver any of the core functionality. Templates of Arachne projects are fairly small, and will only contain code that ought to belong to (and be maintained by) the user, not any general-purpose or glue code.

This also means that there are multiple valid shapes for Arachne projects. Arachne is not a directory layout, or a particular namespace structure, or anything that you will find in a template. Arachne will ship with multiple options: one for a Leiningen project and one for Boot, at least.

In the future, it's also likely that there will be templates for different types of Arachne projects. For example, by using different combinations of modules, Arachne can be a static site generator, a simple RESTful web service, a form-driven CRUD app or a rich ClojureScript & React single-page application. Ultimately, if we are successful, there will be a template for each of these and more.

The current plan is for templates to be extremely low overhead, with no specific code generation tool. Neither will we rely on a specific external tool (such as Leiningen project templates.) Instead, templates will be available as simple GIT repositories which you can clone, and which include a script to rename themselves to match your project.

Libraries in Arachne

Like all frameworks, Arachne does offer some libraries. It will provide a variety of namespaces and functions that you can require and invoke, as utilities and also to interact with the framework at runtime.

However, given the plethora of quality libraries that are already available for most tasks, Arachne will not focus on re-implementing basic web functionality that already exists in library form. Web development requires hundreds of distinct tasks ranging from crypto to data persistence to string manipulation, and re-inventing even a portion of those would be clearly out of scope given the resources available.

Instead, it will focus on making it straightforward to incorporate existing libraries into an Arachne projects. You can always call libraries as libraries from code that you write, without any permission or involvement from Arachne. And for libraries that benefit from deeper integration, or help provide structure to your application (such as servers, asset converters, etc), Arachne will make it easy to write lightweight shims that turn them into Arachne modules, with full awareness of and participation in the framework.

Frameworks in Arachne

As mentioned above, Arachne is a framework. But it is a framework that takes modularity and reuse seriously in an attempt to not become monolithic. In fact, the Arachne core isn't even webapp focused at all; it's a bootstrapping system for a hierarchy of modules, which are all replaceable. Everything that Arachne does is implemented in modules, which in turn can depend upon higher-level, more abstract modules. Reuse and flexibility is baked in from the very beginning.

Arachne, with embedded and shimmed libraries Arachne, with embedded and shimmed libraries

This is so true that it's perfectly accurate to think of Arachne itself as a framework for building frameworks. It doesn't have a finite set of integration points – rather, each module can itself be extended. Every time you write an extension for Arachne, you increase the surface area for future extensions. And Arachne provides clean patterns for doing so.

Even more important, integration points for individual modules don't have to be created explicitly. Modules don't need to provide hooks or anticipate the need for every type of future extension. Instead, they explicitly define a data ontology for how they work themselves. This is observable and ultimately editable by potential extenders. Used in combination with more conventional methods of open dispatch in Clojure (multimethods and protocols), this pattern is intended to provide an unparalleled level of open extension, and will hopefully foster a large ecosystem of third-party modules, all extending eachother hierarchically.

Conclusion

If Arachne is successful, it will be because of this community of extension. If it is going to meet the massively varied needs of real-world users, it isn't enough to get started quickly. It isn't enough to have the theoretical ability to add or swap features quickly.

It will need to have all those things, coupled with the availability of real, useful extensions. If we can provide easy solutions to the most common general types of problems, then we can spend that much more time working on the features unique to our applications, and take one more small step towards the elusive goal of true, optimal code reuse.



Arachne Technical Overview


This quick, informal presentation gives a basic overview of the Arachne system at a technical level, and explains the broad strokes of the module system, the configuration system, and how the runtime launches.

Arachne Technical Overview



Steering Group Kickoff


These are the notes from the first meeting of the Arachne steering group.

The attendees were:

  • David Nolen
  • Jamie Kite
  • Luke VanderHart
  • Mike Nygard
  • Nola Stowe
  • Russ Olsen
  • Stuart Sierra

Please note: the prose here is a digest of what was said tidied up for third parties to read, not a literal transcript. Pleasantries and chit-chat that would not be meaningful for publication has been ommitted.

Update on Kickstarter

Luke: We're at approximately $9,500, which is good, although the rate of new backers has slowed somewhat over the last couple days. Several companies have expressed interest in a corporate sponsorship, and I have plans to release more materials such as blog posts and screencasts to drive continued interest.

Russ: The current plan is for the episode of the Cognicast featuring Arachne to be released within a couple weeks.

Introductions

Luke - I'm a software developer, using Clojure since 2008. Arachne isn't my idea, by any means… we've needed something like this for a long time. But I'm the one who has decided to try to make it happen, now.

Russ - I'm the VP of consulting at Cognitect, a longtime Java and Ruby programmer. I have wanted something like Arachne since the second day I started working with Clojure.

Jamie - I'm an engineering manager at Github. I'm a Rubyist at heart and have been using Rails for many years, with a brief Clojure interlude while working for Cognitect. I have deeply felt the need for a sane way to build web applications in Clojure.

Nola - I am an independent consultant, with extensive experience in Rails, and PHP before that.I worked with Java and GWT when I worked at Google, but definitely prefer Rails. Recently I've become burned out on monolithic, huge apps that are hard to update and so have gotten into Clojure and Clojure over the past few years. However, it can be difficult and take a long time to get started so I'm looking forward to Arachne.

Stuart - I work for Cognitect, and have been using Clojure approximately forever. I don't like webapps, and am glad Luke is doing Arachne so I don't have to write them any more.

Mike - I'm primarily a backend developer, and have worked in a large variety of tech stacks over the years including Java, Rails and Clojure. I'm hoping to use Arachne as a common architecture and a common layout to bring some standardization to Clojure projects.

David - I'm an engineer at Cognitect, I've been doing Clojure as a hobby and professionally by a long time. I'm a major contributor to ClojureScript. Setting up new web stuff in Clojure is a massive pain, and I'd like to see a reasonable path towards making it a bit easier.

Project Goals

For Arachne itself, the consensus was that the goals stated on the Kickstarter page reflects what we're looking for from the Project.

Steering Group Goals

Luke: I don't want to be a open source benevonent dictator. I'm looking to this group to keep me accountable, and to help decide the technical direction for the project. I have some strong opinions about how things should work, but if a plurality of this group tells me something is a good idea, or a terrible idea, I will adhere to those guidlines. Essentially, this group of people are the project stakeholders.

Jamie & Mike: What does that look like specifically?

Luke: There's no upper bound on how much you can contribute. At a minimum, members of this group should attend meetings whenever they can, and review published resources and documentation to audit for adherence to the stated goals.

Michael: It would be good to document architectural decisions explicitly.

Consensus: Let's write ADRs (Architecture Decision Records) and distribute them for review prior to steering group meetings.

Russ: I'm looking forward to providing an outside perspective, reviewing the published materials and vetting them to make sure it's actually easy to use (which is the hard part.)

Logistics

For communication on the the steering and core development groups, we will use the following mediums:

  • ADRs checked into the source repository
  • Trello
  • Slack
  • A bi-weekly remote meeting, every other Friday

Tech so Far

Editor's note: We will be releasing a screencast going over these technical concepts in the near future - these notes serve as a record of the discussion, but are not intended to fully explain the underlying tech. Please be patient and wait until that more complete explanation is available before forming strong opinions.

Luke: It isn't really far enough that I can show working end to end. A lot of the pieces are there, and there's a bunch of code already, but it hasn't all been put together yet. Here's the overall approach:

  • configuration
    • a Datomic-style database value
      • Works on Datomic Free & Datascript
    • built before the runtime starts
    • describes the application
      • everything that could possibly be data
  • configuration schema
    • ontology & meaning of application concepts
    • defined piecewise by modules included in your app
    • modules can depend on other modules, & have access to the schema of their dependencies

Mike: Do you have examples of this schema?

Luke: Yes, I'll show some examples at the end of the meeting if we have time.

Jamie: If the config is immutable, how do plugins contribute additional entities (for example, routes) to the application?

Luke: Each module has an opportunity to query the config and add additional data during its initialization.

  • startup sequence
    • build application schema from module schemas
    • loads programmer-supplied configuration
    • applies module config in dependency order
    • build runtimes
      • entity instantiation
      • dependency injection
    • start runtime
      • Lifecycle/start
  • user-supplied config
    • imperative DSL in config "scripts"
    • DSL is defined by modules
    • DSL consists of forms that (ultimately) bottoms out to the configuration data
    • Config value is a pure function of user config scripts
    • Loading pure data from EDN also possible

Luke: shows an example project directory layout

Luke: Note that the config scripts live in a different directory from the application source code to emphasize the distinction between configs and application source. Configs are DSL scripts that generate configuration data, which is a completely separate process from your running application (and could even run in a different JVM at a different time and place.)

Jamie: In what order do user config scripts get loaded?

Luke: Probably the application will invoke one, and it will explicitly load the others.

Mike: Is this the runtime view or the deployment view?

Luke: For deployment you could either run out of the project directory, largely the same as development, or put all this stuff into a JAR file.

Nola: What about database migrations? Where do they go?

Luke: I've put some thought into that, and I'd like to talk it over with you all. I'll write up an ADR and distribute it prior to our next meeting so we can talk it over.

Jamie: What about the names of these directories? Are they final? Maybe we should name the config directory "config" instead of "app."

Luke: The names are up for grabs. We can chose different names that make more sense. I named it "app" instead of "config" because these aren't static config EDN files, they're actually code (although different from system code.)

Russ: Rails programmers are used to having executable, "code-like" config, shouldn't be a problem to call it "config."

Code Dive

Editor's note: At this point, we looked over a bunch of code for the concepts described above and talked about it for another hour. It was mostly explanatory, and conversational enough that it's difficult to write detailed notes.

At a high level, the topics covered were:

  • Dependency injection, object instantiation, and Stuart Sierra's Component library.
  • The reliance of this design on Module authors doing good data modeling and providing well-thought-through DSLs.
  • What do good DSLs look like?
  • How do you interact programmatically with the DSL? Answer, you don't: Programmatic access should interact directly with the underlying config database and bypass the DSL entirely.
  • We need to refactor the HTTP modules into two separate modules: one that encodes HTTP concepts in an abstract way, and one that implements those using Pedestal.
  • We will have to spend some time, with this group, deciding on the concept of "replacability" of modules in general. There are some general architectural patterns here that we should be aware of so we can structure our modules appropriately (i.e, when are "abstract modules" necessary to define a common data model, which is implemented in a concrete way by other modules?)

Eventually, we ran over time, and one by one people dropped off as they had to leave. We will resume the conversation next time by vetting some ADRs I will hopefully have written by then.