retrospective.txt

{
    title       : Retrospective: 2016
    description : What changed in Myrddin-land over 2016.
}

Retrospective 2017
------------------

The year sucked. But work got done.

Retrospective 2016
------------------

2016 was a good year for Myrddin, with 574 commits to the compiler and
standard libraries, hundreds more commits to other Myrddin related
repositories and projects, several contributors and pull requests, ports to
new operating systems and a number of interesting projects using Myrddin.

The summary, for those who don't care to read the whole page:

- 574 new commits on the compiler, and hundreds more in libraries and other
  code.
- Users and hackers appeared.
- The standard libraries their most glaring holes.
- OpenBSD support got added.
- Plan 9 support got fully fleshed out.
- Thread support landed.
- Cryptography libraries grew more features.
- Lots of bugs got fixed.

This year, the compilers and libraries seem to have matured enough that
it's possible, even likely, to write programs without hitting any serious
issues, which is no small feat for mostly one person reinventing most of the
world. There are still known holes, deficiencies, and bugs, but they feel
like they're small enough to tackle, especially with the fresh blood
contributing and testing.

Andrew Chambers has written a few projects in Myrddin, the most impressive of
which is his C compiler, `qc`. On top of being small and cleanly written, it's
almost useful. It's, capable of natively using system headers on my FreeBSD
box to compile nontrivial amounts of C code, using the QBE backend to generate
optimized code. It lives at [https://github.com/andrewchambers/qc](
https://github.com/andrewchambers/qc)

Quentin Carbonneaux has been valuable discussing ideas and laguage direction.
In addition, he's helped helped discover and debug a number of issues in the
type system. Additionally, it's likely that when the Myrddin compiler gets
self hosted, it will be on his backend, [qbe](http://c9x.me/compile/)

### New Myrddin Code

I realized I have nearly 200,000 lines of Myrddin sitting on my hard drive.
Lots of it is duplicated, autogenerated, like my experiments with XCB
bindings, or a result of failed experiments. Still, there is a great deal
of code that exists.

When it comes to tools and code written in Myrddin, there has been a good
deal of work done looking inwards, writing Myrddin to support writing more
Myrddin. There are a number of tools to do things like dump memory statistics,
which are only useful for Myrddin.

However, other generally useful tools have been written, including
[contbuild](https://github.com/oridb/contbuild). Contbuild is a continuous
builder which is running on the Myrddin website. Other neat tools include a
regex debugger, useful for learning why regexes either match or fail to match
a pattern. And as the language has matured, I've found myself writing small
utilities in it, instead of pulling out either C or Python.

And while it is not yet ready to be released, there is work on a parser
generator that fills ne needs for Lex and Yacc. This parser generator will be
designed to generate output for more common languages, hopefully becoming a
competent replacement for Lex and Yacc for Myrddin, C, C++, and whatever other
languages people decide to add support for.

[Libthread](https://github.com/oridb/mc/tree/master/lib/thread) has landed
this year. The library itself is very basic, providing only thread spawning
and mutexes as things stand, but the basics are in place, and the way forward
to provide a solid implementation on multiple platforms is clear. Of course,
the higher level interfaces for threading are still somewhat up in the air,
pending discussion.

	use thread
	const threadcount = {
		var tid, threads

		threads = [][:]
		for var i = 0; i < Nherd; i++
			tid = thread.spawn({
				std.put("nthreads: {}\n", threads.len)
			})
			std.slpush(&threads, tid)
		;;
	}

Right now, it works well on FreeBSD and Linux, and acceptably on Plan 9.
Doing a good job on OSX and OpenBSD is a work in progress.


[Libcrypto](https://github.com/oridb/mc/tree/master/lib/crypto) has gathered
more functionality, including a secure RNG based off of OpenBSD arc4random,
and a constant time implementation of AES. Hopefully over the next year, it
will grow to become a viable base for a complete TLS implementation in
Myrddin.

	use crypto
	const encrypt = {key, iv, data, result
		var ctx
		crypto.chacha20keysetup(&ctx, key)
		crypto.chacha20keysetup(&ctx, iv)
		crypto.chacha20encrypt(&ctx, data, result)
	}

[Libescfmt](https://github.com/oridb/mc/tree/master/lib/escfmt) exists, and
can handle escaping text for a number of different outputs. It's rather nifty,
because it actually has no API for formatting, but instead uses libstd hooks
for formatting.

	const escape = {
		std.put("shell-escaped text: {}", escfmt.sh("foo & bar"))
		std.put("url-escaped text: {}", escfmt.url("foo & bar"))
	}

[Libmparse](https://github.com/oridb/mparse) was written, and can parse
Myrddin source. There are a number of incomplete parts, most notably the
handling of usefiles and of type inference, but it's a good base for dealing
with Myrddin source code.

[Libjson](https://git.eigenstate.org/ori/libjson.git) has not been shipped with
Myrddin yet, but it exists on the internet, works, and has over 300 tests to
verify that it catches all sorts of edge cases.

	const json = {
		var text, json

		text = std.try(std.slurp("test.json"))
		json = std.try(json.parse(text))
		std.put("{}\n", json)
	}

[Libinifile](https://github.com/oridb/mc/tree/master/lib/inifile) has been
added, to allow for simple configuration. Ini files are a rather restricitve
format, and parsing them is very simple, but because they are so restrictive,
they are easy to understand and use.

[Libhttp](http://git.eigenstate.org/ori/libhttp.git) now works, at least on
the client side. It could stand to be cleaned up and tested, and a server side
implementation should be added. Once those are done, it will also land in the
standard set of libraries shipped with Myrddin.

[Libavl](https://github.com/mpu/avl-myr.git) was written by Quentin,
implementing AVL trees. The code there was contributed earlier this year, and
caught a number of bugs in the type checking code, because of the specific
nature of the recursion in the type checker.

[Libstd](https://github.com/oridb/mc/tree/master/lib/std) has covered most of
the glaring holes that were present at the start of the year, and presents
nearly all the functions I find myself wanting as I write programs.

- Basic floating point support is done, both
  [input](https://github.com/oridb/mc/blob/master/lib/std/fltparse.myr) and
  [output](https://github.com/oridb/mc/blob/master/lib/std/fltfmt.myr). It
  should be accurate, rounding input correctly, and producing the shortest
  correct output when formatting.

- Formatting values now [descends into
  them](https://github.com/oridb/mc/blob/master/lib/std/fmt.myr#L259),
  producing a sane output for nearly all types. This makes printf debugging
  far more pleasant. This is also some of the ugliest code in the Myrddin
  standard library.

- Traits and iterators have landed. This allows things like `for path in
  std.bydirentry(dir)` to work. [Custom iterators are pretty easy to
  implement](https://github.com/oridb/mc/blob/master/lib/std/iterutil.myr).

- Libstd is now thread safe for all calls with unshared arguments. All
  functions in libstd either appropriately lock their global state, or have no
  shared state to protect.

- [Type introspection](https://github.com/oridb/mc/blob/master/lib/std/introspect.myr)
  has landed. The API is shit, and needs to be reworked.

- [Path manipulation](https://github.com/oridb/mc/blob/master/lib/std/pathjoin.myr)
  is now shipped with libstd.

- And much much more.

### Language Changes

The biggest change to the language is that the changes to the language weren't
big. Myrddin has actually stabilized quite a bit. There are pending changes
to resource management. Additionally, there are a few restrictions around
trait scoping that I would like to lift. However, I am largely happy with how things have
turned out, and don't find myself wanting to make many changes.

The most impactful change this year to how code is written comes through
pointer chasing pattern matching. You can now use the address operator in
pattern matches to match what is pointed to by a pointer:

	match intptr
	| &123:	std.put("the value pointed at by intptr is 123\n")
	| _:	std.put("the value pointed at is not 123\n")
	;;

In this case, the change is pretty pointless. However, if you want to match
a complex data structure, for example, when tiling a syntax tree, this kind
of chasing into data structures is extremely valuable.

Traits have also been made more powerful, allowing implementations over
generics. When a trait is implemented over a generic, the most specific
implementation is selected. This allows something that allows the most
intereesting attributes of sfinae -- specialization of implementations
for specific types -- without the clusterfuck of metaprogramming that
traditionally comes with it.

### Looking Forward

The next year is looking bright. The major goals coming up include:

- A Real(tm) release
- Documentation
- Self hosting
- Complete toolchain
- Trivial cross-compilation
- More programs
- Solid, batteries-included set of libraries
- Start looking outwards

Myrddin is nearly due for a release. Multiple users have been hacking
on and with it, and bugs are rare enough that the language is usable.
The language isn't stable, but it's settled down. And there are enough
systems that many interested developers can use it.

Of course, right now there is very little documentation. The API documentation
on this site is fine, but the language spec is not complete and not up to
date.  There is no tutorial, and there are few resources for learning the
language and tools. For a release, that is going to have to change.

Once that's out of the way, I'm going to split my time between improving
the libraries and weaning the implementation off of the GNU toolchain and
the C language.

There's already a good deal of work towards self-hosting, with a parser
library well underway. All that's left is to build a compiler on top of it.
This will probably be either using QBE, or implementing a new SSA backend.

A new toolchain has also been started in collaboration with k0ga, author of
the Suckless C compiler. The goal is that we will have compatible toolchains
that are portable across platforms, and fit our needs better than the rather
clunky and non-portable native toolchains. With multiple people working on it,
this should hopefully go quickly.

When it comes to libraries, there are a number of things that still need to be
worked on. First is improving the libraries used for web technologies. HTTP
needs to be finished. TLS also needs to be implemented. Improving threading
and async I/O come next.

There are also other libraries that end up getting used all over the
place, like floating point math. Right now, I have no implementation of
things like `sqrt()` or `sin()`, and that needs to get fixed.

I also plan to finish off a number of other tools that I have had on my
back burner for ages. Getting a releasable version of the `Hairless` parser
generator would be fantastic, for example.

Finally, Myrddin should be used for a number of tools that stand on their own,
instead of just tools used to support more Myrddin development.

Overall, it's been a fun year of work, and I'm looking forward to using my
own tools heavily going forward!