Alternatively, just write software in a Lisp variant (with good macros, clojure is quite meh here) or Ocaml or Haskell.
These languages put you at the level of writing a language for each problem domain immediately and with a comprehensive and useful toolkit. These languages also do this while directly competing with all but the most carefully tuned of C++/C environments. Steel Bank Common Lisp and the Glasgow Haskell Compiler are good references here.
These languages are often considered strange because of this, and feel somewhat alien. But once you realize you're building a language to model your problems, suddenly tons of stuff makes more sense. Previously alien concepts like Macros and Monads are outside the typical language's experience precisely because the language authors created these contexts and put you inside them.
This article is sort of fundamentally wrong that it's difficult to write small, purpose built languages. It's not, and even outside the functional and homoiconic meta-syntactic world we have seen code generators deployed regularly. Successful products and libraries are build using these techniques all the time, and with a modern toolchain it delivers excellent results. It's just that other more restricted and guided approaches are often introduced earlier in people's learning curve and sets the expectations for them subsequently.
> Alternatively, just write software in a Lisp variant (with good macros, clojure is quite meh here) or Ocaml or Haskell. These languages put you at the level of writing a language for each problem domain immediately and with a comprehensive and useful toolkit.
Great comment. And if a domain-specific-language (DSL) of really custom syntax is needed, an alternative is to use Racket (a Lisp-family language), which allows for creating DSLs of all kind. Racket is already used for designing your own programming language and trying it, so I'm surprised that the author did not mention suggest Racket as a test-bed for the new language.
> This article is sort of fundamentally wrong that it's difficult to write small, purpose built languages. It's not, and even outside the functional and homoiconic meta-syntactic world we have seen code generators deployed regularly.
Correct. For example, most of the common systems used today do use a 'code generator' internally: Every software that uses an ORM library (i.e. Hibernate), is already using a sort of domain-specific-language (i.e. HQL) that, via a code generator, translates to another language (i.e. SQL). So most major platforms (Java, .NET, etc) already have one or many popular libraries that do implement some sort of (limited) DSL and code generator.
I think the aim of the article was to create a new compiler from the ground-up.
I think the correct choice for syntax is something that can't be boiled down to three pieces of advice or one article. It is a topic that would require very deep discussion.
Does Racket support non-Lisp-like syntax for DSLs written in it? I've been hearing really great things about how it has the best macro system on earth, but Haskell has spoiled me to the extent that I'm a bit lazy (no pun intended at all) to look at things outside of that broad area (so Idris, PureScript, etc.) nowadays, but Racket seems to be interesting enough that (to continue the pun) I can afford a bit of IO for it.
Most of my interest stems from this excellent project in Racket that's shaping up to be quite something:
>Does Racket support non-Lisp-like syntax for DSLs written in it?
Yes! You can override the reader so that it parses anything, and there are modules for writing grammers[1][2]. Still, it is usually much easier to just use the built in s-expression parser, and if you're writing a lot of lisp, you probably don't mind s-expressions anyways :)
Hackett has interested me too, I hope that progress can continue to be made, it would be a nice alternative to Typed Racket (which is great, but very complex).
I'm working with the author of the "Type Systems as Macros" paper right now, we're working on implementing a linear language (along the lines of Rust) using Racket's macro facilities and the turnstile[3] package. Being able to embed arbitrary type systems in Racket really shows the insane power of the macro system.
I'll second the use of Lisp as the implementation language. I've also used Prolog. For low level code, e.g. a virtual machine, you'll also need a low level language such as C or assembly language.
If you're designing a language as a learning exercise, or a domain-specific one, my advice is to go right ahead, using a Lisp variant. If you don't know one, now is the time to learn one.
If you're developing a general-purpose language, ask yourself how your language will be better than, or at least different to, every general purpose language in current use. Be ambitious. As languages belong to families (e.g. Algol variants, Lisp variants, functional languages, visual dataflow languages, etc.), you should aim for your language to be the best in its family.
Your language should also be one you'd use yourself.
I'm developing Full Metal Jacket (http://web.onetel.com/~hibou/fmj/FMJ.html). It's very different from almost everything else, so I'm not expecting popularity overnight.
The Dr Dobb's article is way too conventional for my requirements. My language doesn't require any parsing. Syntax, type, and memory errors can't happen. The ideal target machine would just run the code directly without any need for compilation, though I will at some point need to cross-compile it onto the von Neumann architecture, and for that dataflow analysis is unnecessary, making optimal object code simpler to generate.
This is the direction I'd like to see computing go.
>This article is sort of fundamentally wrong that it's difficult to write small, purpose built languages. It's not,
This "correction" is mischaracterizing Walter Bright's article. He's writing about industrial "professional languages" and that context is made clear multiple ways:
1) he's referencing his D Language and presumably languages like it, 2) desire for the compiler to display quality error messages, 3) a good runtime library. The comparables to D Language would be Java/C#/Golang/Rust. All those languages took years by a team of people to get to v1.0.
> we have seen code generators deployed regularly.
But no professional language's (like D's) canonical compiler I know of is the direct output of yacc or ANTLR.
It seems like you have made reasonable and correct statements about programming languages -- for someone else's article -- but not specifically Mr. Bright's. The context really matters here.
> He's writing about industrial "professional languages" and that context is made clear multiple ways
Embedded languages can have all the features you've named. Although I do agree that usually they piggyback on an existing bytes->bytecode system. But you mention YACC and ANTLR as the mechanisms for doing this whereas someone using Lisp or Haskell need not use an external parser, which lowers the cost.
I can implement a C interpreter in Haskell or Lisp. We routinely make language extensions to support all sorts of things, and we do it with excellent error messages and supportive runtime libraries.
Maybe if the author had titled the essay, "So you want to build a runtime and optimizer then attach a lexer and parser to the top" I'd have rephrased. But he didn't, and I took the opportunity to make my point.
> It seems like you have made reasonable and correct statements about programming languages -- for someone else's article -- but not specifically Mr. Bright's. The context really matters here.
Maybe if people want to make nuanced points they should stop using clickbaity titles and leaving major tenants unsaid save for "contextual" queues applied ad hoc by others in 3rd party forums?
Actually, this is one of the most important, albeit usually implicit, features of a good programming language: how easy it makes to create a DSL (without leaving it). This is because ideally you would first find or create a language to express, in the most adequate way, the problem you are trying to solve, and only then you solve it. That is why pseudo-code is useful. The OO support in programming languages made creating a DSL somewhat easier, but it is true - nothing can compare with Lisp in this regard. (So, I guess, this makes Lisp the best programming language of all time.)
> how easy it makes to create a DSL (without leaving it).
Pretty much impossible. If you don't leave your original language, you don't
use domain specific language, you just have general purpose language with
domain specific API.
But what is an API? Is it just a set of functions? Even then, it is already a language, with its own vocabulary. Or, is it a set of classes, interfaces, etc.? Then, even more so. And so on. Limitations and particulars of the syntax of a host language do not matter as much as you may think. Also, if you look close enough, you will realize that most of the languages already are, in fact, combinations of various built-in sublanguages, each having its own "subsyntax". What matters is the ease with which the host allows to create a new sublanguage that suits a particular purpose.
You're missing the point. Some programming languages (Lisp family in particular) allow you to build DSLs directly, with the tools the language itself provides.
And then, inside this "DSL" in Lisp, can you have a completely different
language, with significant spaces and unmatched parentheses? Or is it just
Lisp data structures that are then interpreted and you're pretending to
use language other than Lisp because of its flexible data structures notation?
At the age of 40, I strongly believe that I'd have been much more productive, had I invested at least as much energy into creating languages as I did into learning how to tame C++. Doing so in a structured way, more elegant than writing more parsers and compilers in C, sounds like a good idea.
I always liked C++, but having had the luck of experimenting many alternative concepts during the university (FP, LP, GC system programming languages) I never bought into the idea of using the C/lex/yacc trio.
Always preferred more productive ways to prototype programming languages.
Many of lex/yacc flaws (flaky syntax, global mutable state…) have nothing to do with their fundamentals. Also, LALR is bad, you want a generalised parser that can work on any context free grammar (Bison has such a mode).
We need no-nonsense lexer and parser generators, that are easy to use, easy to deploy, not too hard to implement, and generate fast parsers. My own studies¹ suggest it can be done. Unfortunately, I'm busy earning a salary right now.
Back when I was in the university, we weren't allowed to pick Lisp, Prolog or Caml Light for our compiler design projects, because they would make the whole project too easy. :)
Reminds me of my compiler class, and how I cheated by picking as a project symbolic differentiation and convincing the PhD to let me use Lisp. Needless to say, a single EBNF->lambda macro from a library, and the project became trivial :).
Monads aren’t as special as macros though, all you need is functions and higher-kinded types. The rest is carefully crafting a lawful monadic API, which is something you could probably do in most languages (but none of the standard ones have kinds other than *, if even that). Monads are a fairly simple (to implement and use) but strange (to learn to appreciate) abstraction, nothing more.
You can leave away higher-kinded types if you leave away types altogether, but I don’t think anyone has had a practical benefit from a monadic effects system or transformer stack in e.g. Javascript. I’m not even sure the plain »monad« concept even exists as an abstraction instead of in docstrings.
> Monads aren’t as special as macros though, all you need is functions and higher-kinded types. The rest is carefully crafting a lawful monadic API, which is something you could probably do in most languages (but none of the standard ones have kinds other than *, if even that). Monads are a fairly simple (to implement and use) but strange (to learn to appreciate) abstraction, nothing more.
I don't really see what you're saying here. I didn't mean to equivocate the structures except in terms of their "alien-ness".
But Monads are "special", in that literally everyone uses them but almost never sees them from the "outside" unless they're using a language that offers techniques to model them. Writing Javascript? You're in a monad with helper tools of a specific type. Writing C? Same deal, different shapes.
Lisp Macros do something different (and I alluded to this with the "meta-syntactic" mention), but similarly change the thought process from "how do I write this logic" to "how do I describe this logic".
> . I’m not even sure the plain »monad« concept even exists as an abstraction instead of in docstrings.
Since they are annihilated in the Haskell runtime as well that shouldn't be terribly surprising. It's a modeling technique to build programs algebraically. It's like asking if Generics exist at runtime. They do inasmuch as they are relevant to the interpretation of the program.
The key insight is that all languages have monads, but very few have monad. It’s up to the programmer to see the pattern in e.g. Javascript. But other than having higher kinds there’s not much that keeps most languages from having them the same way e.g. Haskell has.
The »plain monad« part was directed at Javascript, not Haskell. Haskell, like JS, can work perfectly fine without monads, but it still works perfectly fine with them ;-)
Even without knowing about monads, you can simply implement »bindIO« and »pureIO«. That’s what most other languages do – they have bindMaybe, bindList, bindAsync, …
I'm approaching an intermediate level in Haskell, but I don't really see how writing a program in Haskell is any more like writing a language than writing a program in, say, python. Could you elaborate or point to some DSLs written on top of Haskell? I'm very curious.
Here is a fantastic article to get you started down this path. Note that the author, in his introduction, submits that the approach to FizzBuzz in this paper is a "somewhat tongue-in-cheek solution." Nevertheless, by applying this kind of thinking to what is, in some ways, a deceptively simple-looking problem, the paper serves as a great starting point for using DSLs as an approach to problem-solving.
There is another article about FizzBuzz I wrote roughly 5 years ago (forgive the pronouns, I need to rebuild the site and I'm procrastinating on the css).
This uses a slightly different set of abstractions with the same problem.
But perhaps more specifically, when you select a monad stack (group of effects) to compose to solve a problem you're building the features of the language and its effects. And if you use the "final tagless" or "Free" approaches you're doing that even more directly.
You can think a monad defines a DSL. Let's take for example the Rand[0] monad:
type Rand g = RandT g Identity
newtype RandT g m a = ...
-- with
runRandT :: (Monad m, RandomGen g) => RandT g m a -> g -> m (a, g)
newtype MyType = Rand StdGen
-- now you can define
intGreaterThan :: Integer -> MyType Integer
intGreaterThan = ...
randomName :: Integer -> MyType String
randomName length = ...
-- an finally convine the two above
nRandomNames :: Integer -> Integer -> MyType [String]
nRandomNames length n
| n <=0 = return []
| otherwise = do
tailNames <- nRandomNames (n-1)
headName <- randomName length
return (headName:tailNames)
These functions of type `...->MyType a` can be viewed as a DSL where the Random generator state is abstracted away.
#include <stdio.h>
int
main(int argc, char **argv)
{
double i, s;
s = 0;
for (i = 1; i < 100000000; i++)
s += 1/i;
printf("Almost infinity is %g\n", s);
}
Lennarts-Computer% gcc -O3 inf.c -o inf
Lennarts-Computer% time ./inf
Almost infinity is 18.9979
1.585u 0.009s 0:01.62 97.5% 0+0k 0+0io 0pf+0w
And now the Haskell code:
import BASIC
main = runBASIC' $ do
10 LET I =: 1
20 LET S =: 0
30 LET S =: S + 1/I
40 LET I =: I + 1
50 IF I <> 100000000 THEN 30
60 PRINT "Almost infinity is"
70 PRINT S
80 END
And running it:
Lennarts-Computer% ghc --make Main.hs
[4 of 4] Compiling Main ( Main.hs, Main.o )
Linking Main ...
Lennarts-Computer% ./Main
Almost infinity is
18.9979
CPU time: 1.57s
As you can see it's about the same time. In fact the assembly code for the loops look pretty much the same. Here's the Haskell one:
Variable capture is a problem with lisp macros that lisp and clojure programmers have tools and strategies to manage.
A number or languages use hygienic macros to avoid the variable capture problem: Scheme, Racket, Dylan, Elixir, Rust, Julia & Perl 6 (which has hygienic AND unhygienic macros)
Racket has more advanced macro facilities specifically aimed at making new languages languages and DSLs. Syntax-parse is very interesting in how it addresses the problem of error reporting with macros.
> Scheme, Racket, Dylan, Elixir, Rust, Julia & Perl 6 (which has hygienic AND unhygienic macros)
Just want to add that Racket, and most Schemes also support both hygienic and unhygienic macros, through defmacro. [0][1]
Though, their use is heavily discouraged. (And it may be worth pointing out that Guile's defmacro is actually implemented using syntax-case, which is hygienic most of the time, but flexible enough to let you do madcap things. [2]).
Racket has hygenic template macros, which are quite nice. Common Lisp offers a whole bunch of machinery around macros and more types of macros (most famously symbol macros).
Clojure picks the worst of all worlds by making reduced-power backtick macros, not stack tracing the macro output on failures, and then having a culture that discourages the use of macros.
Do you know of any template based macro systems which are more powerful than syntax-rules, but without going to the other extreme of full turing completeness? As I understand it, syntax-rules doesn't even let you concatenate strings to create new identifiers - everything in the output must come verbatim from the input or the template.
These languages put you at the level of writing a language for each problem domain immediately and with a comprehensive and useful toolkit. These languages also do this while directly competing with all but the most carefully tuned of C++/C environments. Steel Bank Common Lisp and the Glasgow Haskell Compiler are good references here.
These languages are often considered strange because of this, and feel somewhat alien. But once you realize you're building a language to model your problems, suddenly tons of stuff makes more sense. Previously alien concepts like Macros and Monads are outside the typical language's experience precisely because the language authors created these contexts and put you inside them.
This article is sort of fundamentally wrong that it's difficult to write small, purpose built languages. It's not, and even outside the functional and homoiconic meta-syntactic world we have seen code generators deployed regularly. Successful products and libraries are build using these techniques all the time, and with a modern toolchain it delivers excellent results. It's just that other more restricted and guided approaches are often introduced earlier in people's learning curve and sets the expectations for them subsequently.