Some future facing ideas for Blitz

[ryardley]

Some future facing ideas for Blitz

These are my disorganised thoughts on a possible future Blitz that would become the most ubiquitous and useful framework on the web. I am not saying we have to follow through with all these ideas maybe they are not right for the Blitz community but just that I think this is what a modern fullstack framework should probably look like. Here I focus pretty much only on the structural blitz framework as opposed to the code generation side.

This is rambling I repeat myself these ideas here are rough and feedback is most welcome. I don't know all the details there are inconsistencies here and I haven't thought through the whole picture but I think an approach like this is worth considering even though there are probably a few really tough problems to work through. In the meantime I will try and take any suggestions and work this up to a proper RFC

TLDR

• Create an isomorphic domain layer
• Generate prisma schema from entities in the domain layer
• Inject an IO api into the Usecases within the Domain Layer for triggering mutations/queries
• Use transparent event sourcing under the hood to provide an "offline" eventually consistent mode and allow the server to send messages to the client.

Core responsability

Blitz framework's core responsability is Data flow management and our goal should be to minimise data management friction.

Thoughts on good architecture

Martin Fowler talks about architecture as being "the set of design decisions that must be made early in a project.”*. Personally I think the goal of those early decisions should allow developers to defer bigger decisions until later. So architecture is a really important consideration for Blitz as we will need to accommodate most application problems within the architecture decisions we follow. Early on we were talking about how hexagonal/clean architecture is a good starting point for how blitz should be structured. Here I am going explore how this idea of clean architecture could apply to the tech stack we use at Blitz and talk about some issues with the way we are working and hopefully provide a suggestion as to how we can improve where we currently are at with it.

Applications generally are a combination of code that represents domain logic as well as code that shuffles data to and from input and output devices. A common IO device might be a database on one end and the DOM on the other.

We could consider this with the following application structure

┌────────────────────────┐
│                        │
│                        │ Render Data <-> DOM
│        RENDERER        │ eg. React
│                        │
│                        │
├────────────────────────┤
│                        │
│                        │ DOMAIN data + validations <-> Render Data
│       PRESENTER        │ eg. React final form
│                        │
│                        │
├────────────────────────┤
│                        │
│                        │ Business/Application logic
│         DOMAIN         │ Combination of rules (functions)
│                        │ and data structures (types,objects)
│                        │
├────────────────────────┤
│                        │
│                        │ DOMAIN Objects <-> Persist Lang (SQL)
│      PERSISTENCE       │ Eg. Prisma
│                        │
│                        │
├────────────────────────┤
│                        │
│                        │ Persistance protocol <-> storage
│       DATASTORE        │ Eg. Postgres
│                        │
│                        │
└────────────────────────┘

This sort of idea is defined within clean architecture; Notice that the domain layer doesn't depend on anything - no rendering logic - no database logic.

In the React / NextJS / Prisma context it might look a little like this:

Here "Entities" might be considered a combination of "functions and data structures". So your NextJS page might use the data validations within the "User" entity whilst the code that makes the call to prisma should return data of the type "User" to the "Domain" layer. Hence the domain code retains influence between persistence and presenter.

           ┌────────────────────────┐
           │                        │
           │                        │ Render Data <-> DOM
           │        RENDERER        │ eg. React
           │                        │
           │                        │
┌──────────┼────────────────────────┤
│          │                        │
│          │                        │ DOMAIN data + validations <-> Render Data
│          │       PRESENTER        │ eg. React final form within a NextJS Page
│          │                        │
│          │                        │
│          ├────────────────────────┤
│          │                        │
│  DOMAIN  │                        │ Business/Application logic
│ TYPES /  │         DOMAIN         │ Combination of rules (functions)
│VALIDATION│                        │ and data structures (types,objects)
│          │                        │
│          ├────────────────────────┤
│          │                        │
│          │                        │ DOMAIN Objects <-> Persist Lang (SQL)
│          │      PERSISTENCE       │ Eg. Prisma
│          │                        │
│          │                        │
└──────────┼────────────────────────┤
           │                        │
           │                        │ Persistance protocol <-> storage
           │       DATASTORE        │ Eg. Postgres
           │                        │
           │                        │
           └────────────────────────┘

So the question arises from where do we define our Entities?

Can we use prisma to define our entities?

When prisma was presented originally it occurred to me there was a chance that there might be the promise of a "data type" "source of truth". We could use types from Prisma to inform all the types within our application.

Prisma potentially created a set of entity types that could be shared across the stack as prisma strongly types its return data. Some of the early examples we have been using with blitz try to make db queries in the frontend by sending down where structures to the backend via blitz queries:

import {useQuery} from "blitz"
import getProject from "app/projects/queries/getProject"
function App() {
  const [project] = useQuery(getProject, {where: {id: 1}})
}

export default async function getProject({where}: GetProjectInput, ctx: Record<any, unknown> = {}) {
  // Can do any processing, fetching from other APIs, etc
  const project = await db.project.findOne({where})
  return project
}

This is a pretty powerful idea but what I have found in practice is that by doing this we quickly run into the object relational mapping impedence problem.

The ORM impedence mismatch

When we have a complex form dealing with many connected relationships upserting the data within a form becomes difficult. Transactions help with part of this but writing a reasonably complex prisma upsert doing sub upserts for many tables is not for the faint hearted. It might become easier in the future as prisma matures but currently I have found it painful. Ideally you can provide a domain object to a persistence function and then that data is persisted by magic.

There is another problem namely getting types out of prisma in a format that is easily understood by a form library. Prisma has a few typescript tools to do this however we still need to remove all our relationship keys which will exist on our data and managing subtleties of the difference between our domain model and our database model can lead to make work. I have found that my data would often have extra data in it (foreign key ids etc) that was not in the types provided and this would cause problems during save. I think prisma is a great tool and the way you can ensure your db and typescript are in sync with it is great but it still has some refinement when it comes to being able to easily persist a pain object structure. The fact is that prisma still is a leaky abstraction when it comes to what it brings to the table for object relational mapping.

So how do we approach the ORM impedance problem. Well, traditionally there are two solutions:

1. Use objects everywhere
2. Use relational data everywhere

I wanted using prisma types as the source of entities and have explored trying #2 but found it difficult to achieve success mainly because there is no form library out there that works natively with schema based relational data that I have found.

So maybe others have has more success but in the end after exploring these ideas I am left concluding we should be careful with trying to make prisma central to our application architecture. I kind of knew this a priori but I wanted to explore the idea as the allure of global types and shared validations is a major reason why I consider TypeScript to be such a great fullstack platform in the first place.

The data schema often does not represent our domain

Then there is day to day data management artifacts. The problem with using Prisma as the source of our typescript types is agile data migration. As databases get migrated and managed you end up with data field polution. This is so we can manage down migrations in the case of a rollback. Agile data migration means that as we need to change our data model while our application grows, a good database engineer will tend to do things like rename a column instead of delete it initially so that a down migration in the case of reverting the change will work better. So the database schema might resemble our domain model it does not always represent it. This would mean that every time there is a database column change we would then need to filter out the change in our domain types and that menas more code and more type management code to shoehorn Prisma types into our domain.

Persistence is not just the database and data is not always relational

There are also a few other things that had been bugging me. Applications don't just save and retrieve state from a relational database they also could save state to a user session, client-side persistence such as localStorage , an in memory database, upload images to services like cloudinary, S3 or GraphCMS or they might save and retrieve data from cookies, a message queue, or even a blockchain. This could all be related to the application performing its key business logic and these are all types of persistence. Where the data comes from or goes as far as the app domain is concerned is generally irrelevant and those syncs are basically just an IO devices.

Secondly can we expect we can deal with relational data when we are persisting that non database data? I think the natural answer there is no. What about getting data from a GraphQL service? That data is going to be denormalized. If I serialize a complex object to a file and send it to S3, that object could be a tree and need not be relational. I might encrypt a token from an arbitrary object tree and save it as a cookie. These could be setup to be relational but it would be unnatural and it doesn't make sense to enforce them to be as it would mean that our core app has undue influence on the services that support it.

Create a new domain layer

So I think the answer is that we need to be working with universal typescript data types and objects at the heart of our application from a source of truth that we as application developers define. Every place of persistence will handle the serializing of the objects that get sent to it differently.

We can use Zod to generate universal functions that accept objects of type any and give us a strictly typed object back along with universal exported typescript types. We have a layer that does this it could be the entity layer denoted by a ./types folder. This is where we generate the universal types used by the particular application context we are in. Within each entity type file you could export a ZodSchema and a set of functions that handle business rules that depend on that entity. ./types could also be called ./entities to disambiguate it from the transient types we need in our apps but it might work well to plunk it all into ./types however what it is called probably doesn't matter to much.

We then need to isolate persistence. I propose that to do that, we should put persistence behind a function API that accepts and returns entities. I would call these functions that persist data "persistors". These are mostly going to look like our current mutations although if you have been bleeding your queries and prisma types to the frontend that you are going to want to do a bit of refactoring to isolate them.

We can use the current system of having ./mutations and ./queries to designate persistor functions although we need a way to designate the environment we are working in so prefacing them with certain keywords could make it simpler.

You can use "persistors" to take an entity and persist to anything including clientside concerns so we need a way to create and indicate to the system there are clientside persistors to persist to browser cookies or localStorage or memory etc. One idea would be to add a ./io/browser/{mutations,queries} folder and a ./io/server/{mutations,queries} folder to signal clearly in what environment the code is to be run. By adding a "persistor" function in the correct location that function is effectively an event listener and will be called when triggered from the domain API.

How can we support offline first applications?

Because of our hard dependency on Prisma offline first applications are difficult to support in a flexible all encompassing way. One simple solution is to throw prisma out and simply develop persistence using CouchDB with PouchDB on the frontend but then most of your business logic is in React and this diminishes the value of using Blitz here as you would have to come up with your own architecture and you app might theoretically not need data fetching or a server cache. Blitz should be considered a fancy message bus that ships data around the app. If we think about it like this we eventually come to the idea of an event driven solution as opposed to a function driven approach. It is possible however to use functions as proxies for events where multiple subscribers subscribe to events. When you couple this with a datastructure like automerge then we can theoretically have the ability to build offline apps. Using events it becomes easier to send messages to the client to build things like financial trading applications where you filter a stream of data in realtime and send it off to the client without having to support a websocket server necessarily.

What happens if we combine Clean Architecture with transparent Event Sourcing?

So when talking about managing persistence there is the consideration of how to design and manage offline persistence. There are a few ways you can manage this. You can try and sync data so the server and the client use exactly the same data via some kind of system using CRDTs or the OT algorithm but there is an inbuilt assumption that any particular client and server requires exactly the same type of data. You probably don't want your auth table with your user hashes anywhere on your client and there are probably heaps of data the server is tracking that the client doesn't need to have uploaded to it.

What we want to do is consider the client server relationship as simply two microservices in a distributed system. This comes with all the issues you get in a distributed system such as eventual consistency but if you think about it we are already dealing with eventual consistency by using react query to manage our data fetching.

So what is the the best way to manage state in a large distributed system? Well it would probably be using some sort of event sourcing.

An event sourced architecture is very powerful - if we build this correctly we can have a huge number of benefits - but it can also be very complex so we must hide that complexity from our users wherever possible which is why I suggest considering the event dispatch to simply be a proxied function call.

We already generate code as part of our build step we can utilize that approach along with a simple dependency injection technique to create a cross environment domain layer. This means the domain layer will work in a browser on the server or in a phone.

Currently Blitz has queries and mutations which can only be run server side and are dependent on RDMS data and a persistence framework like prisma. We should ask Blitz users to refactor their applications by adding an isomorphic 'Domain' layer on top of those serverside 'perstistor' functions.

Rules

• If you want an offline cache you need to write (or generate!) client side persistors to handle it - Why? This enables developers to accomodate any persistence pattern they want from using a React reducer to Redux to indexDB
• Any kind of offline cache requires the tradeoff of eventual consistency (we are already making that with react query in the current blitz implementation
• All domain queries/commands are universal/isomorphic and have their dependencies injected
• You cannot call persistors outside of domain code
• Persistors know their execution environment based on naming convention

What would this look like?

A domain usecase calls our persistor functions and would look something like this:

// app/messaging/domain/sendMessage.ts

// sendMessage() is run on the server or the client and can be considered a domain command.
export default function sendMessage(input, {io}:IO) {
  // validate incoming data
  const message = Message.parse(input.message)
  
  // Get full User entities from persistence
  const [fromUser, toUser] = await io.getUsers([message.fromId, message.toId])
  
  // Check business rules by calling methods on Entities
  if(fromUser.allowedToSendMessages() && toUser.canReceiveMessagedFrom(fromUser)) {
    // Call a mutation on the store and handle the result by manipulating client state
    // The following persists the `messageSent` event in the eventually consistent store
    await io.sendMessage(message) 
  }
}

Because all dependencies are injected this becomes easy to test:

import {fakeIo} from 'blitz'

// provides all store typescript types

test('sendMessage', async () => {
  const io = fakeIo({
    getUsers: jest.fn(async () =>  [cantSendUser, user2]),
	sendMessage: jest.fn(),
  }) 
 
  await sendMessage({message: {
    fromId: 123,
    toId: 234,
    message: "Hello, World!" 
  }}, {io})
  
  expect(io.sendMessage).not.toHaveBeenCalled()
})

The queries may have separate serverside and clientside routines to accommodate for the different read models they are using:

// app/users/queries/getUsers.ts
import { db } from 'blitz/db'
export default async function getUsers(ids: string[]) {
  return await db.users.findMany(...}
}

// Clientside 
// app/users/client/getUsers.ts
import { client } from 'some-clientside-state-thing'

export default async function getUsers(ids: string[]) {
  return await client.find(ids)
}

If there is no clientside query then the default is to run an rpc and fallback to the serverside read model.

// Clientside 
// app/users/client/getUsers.ts
import { rpc } from 'blitz/rpc'

export default async function getUsers(ids: string[]) {
  return await rpc.getUsers(ids) // pseudocode
}

If we are in offline mode things get interesting. That means the application has a view model stored in the client as a Redux store or some kind of localStorage data or a clientside in memory database or a mobile device filesystem or whatever. This read model is built up based on the events that it receives over a long polling connection (probably with something like browser channel) When it is detected that the users device is offline a snapshot of the view model state is created and all domain logic is run on the client (hence why domain logic is isomorphic). Then domain commands are

Storing data and conflicts

The store.sendMessage(message) call locally publishes an event to a replicated event store which then eventually publishes that event on the server

If you are using an offline cache of some sort there will eventually be conflicts but managing them is part of the domain model. This means we need to add a conflict checker that will check an incoming event against potential outgoing events and then bubble up to fail specific user actions to rectify the situation. (Think when you have been offline and you receive a sync that is irreconcilable). One example is when there was a call to store.changePurchaseOrder({shippingAddress: '123 More Street'}) made remotely but locally a purchasing manager edited an existing order to change the address and add a new yacht to the purchase order: store.changePurchaseOrder({shippingAddress: '1026 Reynard Street', newItems: [{type: "yacht", price: 13000000}]}) It would make sense for the business to accept the order change with a much higher value item added to the order despite the possibility of the address being wrong rather than say no to the second change. You might also have a business situation where you are saying no we only follow optimistic locking and whatever comes last wins. That is also a solution but it is one we need our developers to make so if we were to go live with an eventually consistent model we should probably introduce introduce a conflicts construct in the domain that manages these situations and allows developers to control those situations which would be the main change to the developer API

I thought you can't do events over serverless without an event bus

Yes you can. With Blitz, to satisfy the requirement of having an easy hosting solution we need to be able to work with a single database and vercel serverless.
Initially we could put together a distributed event bus using polling to surface events to clients while storing events as JSON in the DB. If you have the luxury of a websocket server there will be less chat. You can also limit traffic for event polling by only collecting events during the revalidate cycle of SWR for simple apps. We can allow plugins to substitute proper scalable systems Kafka / RabbitMQ etc at a later point.

But hold on that is a whole lot of extra complexity mostly handled by Blitz what am I getting for it?

• An app that feels like writing a monolith but can be easily massively scaled because it is event based. (we can eventually write the components to do that)
• Well structured offline cache solution that doesn't lock in vendors.
• Ability to take advantage of the scalability of serverless architecture.
• Application is fully auditable
• Clients act the same way any addition to the system will act - need to create an async content parser and ingestor - just write domain logic and persistors the way you do currently. The pattern is easy to learn and universally applicable.

Ideas in bullet points

Philosophy

• Blitz should be responsible for abstracting away the complexities of passing data around your application
• We want to explore and encourage developers to follow good patterns that mean the apps they build are as flexible and future proof as possible without having them add more complexity than otherwise required

Problems

• When we share prisma datatypes past the database we run into ORM impedance issues. Prisma can map data coming out of the database but does quite a poor job of handling object mapping particularly with regards to the way you write mutations in prisma. This has the effect of adding more complexity than is optimal in a Blitz app. So what are our options?
  1. Ensure we use relational data everywhere
     ❌ JS libraries generally expects object data which means that despite attempting not to map from relational to object data we will at some point have to map between the two.
  2. Ensure we use object data everywhere
     ❌ Prisma does oneway mapping from relational to object but demands a relational model for writes at least at this time so this is off the table unless we use another persistence vector.
  3. Accept this and map between objects and relational data
     ✔️ pretty much the only thing we can do
• Clientside cache invalidation is an implementation detail and should be automatic - React query style data fetching is too low level. You should not have to worry about this in the client until you crack the hood and need to do something special. We should avoid providing such a low level API to users but allow them the ability to go low level should they need.
• Blitz is great for form driven apps but how do we create messaging, p2p or blockchain backed apps that have similar persistence requirements but require radically different implementations? How do we provide this functionality without having to re-write our business code?

Ideas

• Add a universal (isomorphic) ./usercases layer - defines business rules and invariants as entities interact with each other based on user interaction
• Add a universal (isomorphic) ./entities layer - defines data types and validation - these are the universal shared types for your app.
• Compile a Prisma schema from the entities we define - remember entities are the app domain and prisma db schema will have persistence concerns (such as renamed fields) as opposed to purely domain concerns. Also by compiling to Prisma we can possibly also in the future compile to a clientside data store to work with offline apps or make adaptors to support any type of persistence.
• Add an ./io folder for repository style input output persistence. Mutations and queries can safely all live here but also writing to and from session should be here as it is a form of persistence. All database / prisma dependencies live below here and don't creep up to the domain layer this includes prisma types. This io layer contains any code that will persist data. The code determines it's execution environment through metadata so there should be adapters to UI, Browser and HTTP concerns here that are all injected via TS types into the domain. We could use special folders or decorators to determine that execution environment.
• Provide an API to supply metadata to all ./io persistors that can create an object graph which can determine event flow / SWR policy etc.
• Utilize TS type generation (ala Prisma) to accomplish io dependency injection in the domain layer.
• Remove session/ctx access from ./io/mutations and ./io/queries .
• Eventually move towards an event sourced architecture under the hood.
• All mutations become eventually consistent and fire events although we can cover this up by returning an event to the client.
• Queries receive data from projections the main one being our monolithic database but others might be a clientside cache
• Consider client cache simply as another projection (facilitated by CRDT) of the same event stream.
• Write or use a distributed event bus that runs in a serverless setting which under the hood can have the option to run in a serverful environment or using websockets.

Advantages

• Develop like monolith. Scale like serverless / microservices.
• Object relational mapping is managed cleanly and consistently
• Domain code is isolated into a highly testable and manageable system
• Apps immediately auditable
• Removes hard dependency on prisma from the framework and can now support esoteric persistence such as neo4j, mongo, blockchain, s3 etc by just having the developer write the persistence code and define their interface they want which blitz can then distribute/inject into the domain code.
• Follows hexagonal/clean architecture more closely which gives us plenty of reference material to guide new users who are not fluent in the patterns
• Can create development tools to allow time travel debugging of all state changes
• Applications can get an audit log automatically this can be sent to persistence
• Developers should be able to write a plugin to kafka or rabbit to add more scale to the system.
• Covers use cases for startups that don't want to throw out all their code as they scale
• Event sourced architectures have a side effect of providing a natural way to manage an eventually consistent offline mode assuming we can create a distributed serverless event bus (pretty sure this is possible)

Caveats and workarounds

• There is no need to move to event sourcing immediately
• Extra layer for domain. For passthrough code means we can get what appears to be boilerplate. Perhaps prisma compilation helps also we can determine a convention or defaulting to avoid this.
• Moving people to the new architecture could be a challenge. The way I see it we could start by removing ctx from mutations/queries that have been moved to the ./io folder. That way we have kind of a "get started" mode with poor architectural concerns (ala current situation) with m/qs as normal and a "scaled" mode when you move those m/q folders to ./io to be consumed by the ./domain use cases.
• Event sourcing provides a natural pattern to create an eventually consistent offline mode but to accomplish offline mode, developers would need to create client side ./io mutations and queries that persist data in a client cache.
• Event sourcing means we are dealing with an eventually consistent model but we are kind of doing that already with react-query's SWR pattern.

[flybayer]

Finally got around to reading this! Thank you for the thorough write up!

You have a lot of good points and good ideas.

I definitely feel what you are saying about Prisma not working out super very well as the source of truth for domain types. And I can totally see the benefit and appeal of adding a new domain layer as the source of truth for types and validation.

Caveat: Extra layer for domain. For passthrough code means we can get what appears to be boilerplate. Perhaps prisma compilation helps also we can determine a convention or defaulting to avoid this.

People really love how simple it is now. It'd be awesome if we can somehow keep almost exactly what we have now as the base starting point, but then have a path to easily upgrade parts of your app to work offline and with event sourcing.

Something else to consider is the actor model and how possibly xstate could fit into this.

If you haven't checked it out, Amplify DataStore has an interesting take on offline data. If I understand correctly you essentially sync a certain slice of data to the client.

[ryardley]

Something else to consider is the actor model and how possibly xstate could fit into this.

Interesting. Yeah it would be nice to be able to support various distributed concurrency models eventually. I kind of get the actor model but I need to explore it some more and get some hands on experience with it. There's https://nact.io/ I remember looking into for a gig a little while ago but we never went ahead with that model.

If you haven't checked it out, Amplify DataStore has an interesting take on offline data. If I understand correctly you essentially sync a certain slice of data to the client.

Nice I haven't seen this yet. I wonder if it is possible to find something similar that isn't vendor specific?

[flybayer]

Here's an interesting project to help with schema changes over time: https://www.inkandswitch.com/cambria.html

[eric-burel]

Hi, reposting from Slack:

At Vulcan we base all the logic on a schema system. The schema describes the field of a model. It contains a lot of information: the field types, the form input to be used, the access permissions, field resolver (we are in the context of graphql), etc.

There's a mix of server-side stuff and common stuff (less often client-only since we have SSR) within the same schema.

Since most of the data fetching logic lives in the graphql context object passed as argument to the resolver, in the form of data loaders or data source, most of the time that's ok. You don't need to actually import the object.
At worst if you have server-only code you can do a "require()" from within your resolver (though I have zero idea what impact it has on the app). But that's not a very scalable approach.

I've written an RFC about how we could better support such isomorphic/global models: VulcanJS/vulcan-npm#14

The goal is to kinda create something in-between Meteor full-stack packages (but in NPM) and Next full-stack pages

I think it can be interesting for you, given your reflections about using Prisma as a source of truth or finding other approaches. In Vulcan case our approach is based on JavaScript, with simpl-schema as the base syntax and a lot of Vulcan specific fields.
Our approach is probably very simplified compared to what you described, but it has proven at least sufficient to build small to medium sized infrastructure.