Steering Group Kickoff

These are the notes from the first meeting of the Arachne steering group.

The attendees were:

  • David Nolen
  • Jamie Kite
  • Luke VanderHart
  • Mike Nygard
  • Nola Stowe
  • Russ Olsen
  • Stuart Sierra

Please note: the prose here is a digest of what was said tidied up for third parties to read, not a literal transcript. Pleasantries and chit-chat that would not be meaningful for publication has been ommitted.

Update on Kickstarter

Luke: We're at approximately $9,500, which is good, although the rate of new backers has slowed somewhat over the last couple days. Several companies have expressed interest in a corporate sponsorship, and I have plans to release more materials such as blog posts and screencasts to drive continued interest.

Russ: The current plan is for the episode of the Cognicast featuring Arachne to be released within a couple weeks.

Introductions

Luke - I'm a software developer, using Clojure since 2008. Arachne isn't my idea, by any means… we've needed something like this for a long time. But I'm the one who has decided to try to make it happen, now.

Russ - I'm the VP of consulting at Cognitect, a longtime Java and Ruby programmer. I have wanted something like Arachne since the second day I started working with Clojure.

Jamie - I'm an engineering manager at Github. I'm a Rubyist at heart and have been using Rails for many years, with a brief Clojure interlude while working for Cognitect. I have deeply felt the need for a sane way to build web applications in Clojure.

Nola - I am an independent consultant, with extensive experience in Rails, and PHP before that.I worked with Java and GWT when I worked at Google, but definitely prefer Rails. Recently I've become burned out on monolithic, huge apps that are hard to update and so have gotten into Clojure and Clojure over the past few years. However, it can be difficult and take a long time to get started so I'm looking forward to Arachne.

Stuart - I work for Cognitect, and have been using Clojure approximately forever. I don't like webapps, and am glad Luke is doing Arachne so I don't have to write them any more.

Mike - I'm primarily a backend developer, and have worked in a large variety of tech stacks over the years including Java, Rails and Clojure. I'm hoping to use Arachne as a common architecture and a common layout to bring some standardization to Clojure projects.

David - I'm an engineer at Cognitect, I've been doing Clojure as a hobby and professionally by a long time. I'm a major contributor to ClojureScript. Setting up new web stuff in Clojure is a massive pain, and I'd like to see a reasonable path towards making it a bit easier.

Project Goals

For Arachne itself, the consensus was that the goals stated on the Kickstarter page reflects what we're looking for from the Project.

Steering Group Goals

Luke: I don't want to be a open source benevonent dictator. I'm looking to this group to keep me accountable, and to help decide the technical direction for the project. I have some strong opinions about how things should work, but if a plurality of this group tells me something is a good idea, or a terrible idea, I will adhere to those guidlines. Essentially, this group of people are the project stakeholders.

Jamie & Mike: What does that look like specifically?

Luke: There's no upper bound on how much you can contribute. At a minimum, members of this group should attend meetings whenever they can, and review published resources and documentation to audit for adherence to the stated goals.

Michael: It would be good to document architectural decisions explicitly.

Consensus: Let's write ADRs (Architecture Decision Records) and distribute them for review prior to steering group meetings.

Russ: I'm looking forward to providing an outside perspective, reviewing the published materials and vetting them to make sure it's actually easy to use (which is the hard part.)

Logistics

For communication on the the steering and core development groups, we will use the following mediums:

  • ADRs checked into the source repository
  • Trello
  • Slack
  • A bi-weekly remote meeting, every other Friday

Tech so Far

Editor's note: We will be releasing a screencast going over these technical concepts in the near future - these notes serve as a record of the discussion, but are not intended to fully explain the underlying tech. Please be patient and wait until that more complete explanation is available before forming strong opinions.

Luke: It isn't really far enough that I can show working end to end. A lot of the pieces are there, and there's a bunch of code already, but it hasn't all been put together yet. Here's the overall approach:

  • configuration
    • a Datomic-style database value
      • Works on Datomic Free & Datascript
    • built before the runtime starts
    • describes the application
      • everything that could possibly be data
  • configuration schema
    • ontology & meaning of application concepts
    • defined piecewise by modules included in your app
    • modules can depend on other modules, & have access to the schema of their dependencies

Mike: Do you have examples of this schema?

Luke: Yes, I'll show some examples at the end of the meeting if we have time.

Jamie: If the config is immutable, how do plugins contribute additional entities (for example, routes) to the application?

Luke: Each module has an opportunity to query the config and add additional data during its initialization.

  • startup sequence
    • build application schema from module schemas
    • loads programmer-supplied configuration
    • applies module config in dependency order
    • build runtimes
      • entity instantiation
      • dependency injection
    • start runtime
      • Lifecycle/start
  • user-supplied config
    • imperative DSL in config "scripts"
    • DSL is defined by modules
    • DSL consists of forms that (ultimately) bottoms out to the configuration data
    • Config value is a pure function of user config scripts
    • Loading pure data from EDN also possible

Luke: shows an example project directory layout

Luke: Note that the config scripts live in a different directory from the application source code to emphasize the distinction between configs and application source. Configs are DSL scripts that generate configuration data, which is a completely separate process from your running application (and could even run in a different JVM at a different time and place.)

Jamie: In what order do user config scripts get loaded?

Luke: Probably the application will invoke one, and it will explicitly load the others.

Mike: Is this the runtime view or the deployment view?

Luke: For deployment you could either run out of the project directory, largely the same as development, or put all this stuff into a JAR file.

Nola: What about database migrations? Where do they go?

Luke: I've put some thought into that, and I'd like to talk it over with you all. I'll write up an ADR and distribute it prior to our next meeting so we can talk it over.

Jamie: What about the names of these directories? Are they final? Maybe we should name the config directory "config" instead of "app."

Luke: The names are up for grabs. We can chose different names that make more sense. I named it "app" instead of "config" because these aren't static config EDN files, they're actually code (although different from system code.)

Russ: Rails programmers are used to having executable, "code-like" config, shouldn't be a problem to call it "config."

Code Dive

Editor's note: At this point, we looked over a bunch of code for the concepts described above and talked about it for another hour. It was mostly explanatory, and conversational enough that it's difficult to write detailed notes.

At a high level, the topics covered were:

  • Dependency injection, object instantiation, and Stuart Sierra's Component library.
  • The reliance of this design on Module authors doing good data modeling and providing well-thought-through DSLs.
  • What do good DSLs look like?
  • How do you interact programmatically with the DSL? Answer, you don't: Programmatic access should interact directly with the underlying config database and bypass the DSL entirely.
  • We need to refactor the HTTP modules into two separate modules: one that encodes HTTP concepts in an abstract way, and one that implements those using Pedestal.
  • We will have to spend some time, with this group, deciding on the concept of "replacability" of modules in general. There are some general architectural patterns here that we should be aware of so we can structure our modules appropriately (i.e, when are "abstract modules" necessary to define a common data model, which is implemented in a concrete way by other modules?)

Eventually, we ran over time, and one by one people dropped off as they had to leave. We will resume the conversation next time by vetting some ADRs I will hopefully have written by then.