crush

Rust LINK

Crush is an attempt to make a command line shell that is also a powerful modern programming language.

Crush

Crush is an attempt to make a traditional command line shell that is also a modern programming language. It has the features one would expect from a modern programming language like a type system, closures and lexical scoping, but with a syntax geared toward both batch and interactive shell usage.

What features of a traditional shell does Crush retain?

The basic structure of the Crush language resembles a regular shell like bash.

How to invoke commands, pass arguments and set up pipelines are unchanged, as is the central concept of a current working directory. This means that trivial invocations, like ls or find .. | count look the same, but under the hood they are quite different, and nearly everything beyond that is different.

What does Crush do so differently, then?

Scratching the surface

Let's start with two trivial commands; listing files in the current directory, and checking how many files are in the current directory:

crush> ls
user         size modified                  type      file
liljencrantz 2279 2020-03-07 13:00:33 +0100 file      ideas
liljencrantz 4096 2019-11-22 21:56:30 +0100 directory target
...

crush> ls | count
14

This all looks familiar. But appearances are deceiving. The ls command being called is a Crush builtin, and the output is not sent over a unix pipe but over a Rush channel. It is not understood by the command as a series of bytes, but as a table of rows, and Crush provides you with SQL-like commands to sort, filter, aggregate and group rows of data.

crush> ls | sort ^size
user         size  modified                  type      file
liljencrantz    31 2019-10-03 13:43:12 +0200 file      .gitignore
liljencrantz    75 2020-03-07 17:09:15 +0100 file      build.rs
liljencrantz   491 2020-03-07 23:50:08 +0100 file      Cargo.toml
liljencrantz   711 2019-10-03 14:19:46 +0200 file      crush.iml
...

crush> ls | where {type == "directory"}
user         size modified                  type      file
liljencrantz 4096 2019-11-22 21:56:30 +0100 directory target
liljencrantz 4096 2020-02-22 11:50:12 +0100 directory tests
liljencrantz 4096 2020-03-16 14:11:39 +0100 directory .idea
liljencrantz 4096 2020-02-15 00:12:18 +0100 directory example_data
liljencrantz 4096 2020-03-14 17:34:39 +0100 directory src
liljencrantz 4096 2020-03-14 19:44:54 +0100 directory .git

Because Crush output is a stream of rows with columns, actions like sorting by an arbitrary column or filtering data based on arbitrary logical expressions operating on these columns is easy, and because the components used to do this are generic and reusable, you can trivially do the same to data from any source, such as json files, http requests, etc.

Reading and writing files

In traditional shells, I/O is done as binary streams. Because Crush streams are typed, I/O happens differently. Crush has command pairs used for serializing and deserializing various file formats. Use e.g. json:from and json:to to deserialize and serialize json data, respectively. These commands all work like you'd expect:

# Dump the output of the ls command to the file listing.json in json format
crush> ls | json:to ./listing.json

# Read the file Cargo.toml as a toml file, and extract the dependencies-field
crush> (toml:from Cargo.toml):dependencies

# Fetch a web page and write it to a file
(http "https://isitchristmas.com/"):body | bin:to ./isitchristmas.html

If you don't supply an input file to any of the deserializer commands, the command will read from the input, which must be a binary or binary stream, e.g. (http "https://jsonplaceholder.typicode.com/posts/1"):body | json:from.

If you don't supply an output file to one of the serializer commands, the command will serialize the output to a binary stream as the pipeline output:

crush> list:of 1 2 3 | json:to
[1,2,3]

One of the Crush serializers, Pup, is a native file format for Crush. The Pup-format is protobuf-based, and it's schema is available here. The advantage of Pup is that all crush types, including classes and closures, can be losslessly serialized into this format. But because Pup is Crush-specific, it's useless for data sharing to other languages.

Operators for comparison, logical operations and arithmetical operations

Crush allows you to perform mathematical calculations on integer and floating point numbers directly in the shell, mostly using the same mathematical operators used in almost any other programming language.

crush> 5+6
11
crush> 1+2*3
7

The only exception is that the / operator is used for constructing files and paths (more on that later), so division is done using the // operator.

crush> 4.2//3
1.4000000000000001

Comparisons between values are done using >, <, <=, >=, == and !=, just like in most languages. All comparisons between values of different types are false.

crush> 4 > 5
false

The and and or operators are used to combine logical expressions:

crush> false or true
true
crush> if some_file:exists and (some_file:stat):is_file {echo "yay"}

Crush also has operators related to patterns and matching. =~ and !~ are used to check if a pattern matches an input:

# The % character is the wildcard operator in globs
crush> %.txt =~ foo.txt
true
# This is how you construct and match a regular expression
crush> re"ab+c" =~ "abbbbbc"
true

Regexps also support replacement using the ~ (replace once) and ~~ (replace all) operators, which are trinary operators:

crush> re"a+" ~ "baalaa" "a"
balaa
crush> re"a+" ~~ "baalaa" "a"
bala

Type system

As already mentioned, many Crush commands operate on streams of tabular data. The individual cells in this table stream can be any of a variety of types, including strings, integers, floating point numbers, lists, binary data or another table stream.

crush> ps | head 5
pid ppid status   user cpu  name
  1    0 Sleeping root 4.73 /sbin/init
  2    0 Sleeping root    0 [kthreadd]
  3    2 Idle     root    0 [rcu_gp]
  4    2 Idle     root    0 [rcu_par_gp]
  6    2 Idle     root    0 [kworker/0:0H-kblockd]

Some commands of course output a single value, such as pwd, which outputs the current working directory as a single element of the file type.

Variables of any type

Variables must be declared (using the := operator) before use.

crush> some_number := 4      # The := operator declares a new variable
crush> some_number * 5
20

Once declared, a variable can be reassigned to using the = operator.

crush> some_number = 6
crush> some_number * 5
30

Like in any sane programming language, variables can be of any type supported by the type system. There is no implicit type conversion. Do note that some mathematical operators are defined between types, so multiplying an integer with a floating point number results in a floating point number, for example.

crush> some_text := "5"
crush> some_text * some_number
Error: Can not process arguments of specified type

Named and unnamed arguments

Crush supports named and unnamed arguments. It is often possible to use one, the other or a combination of both. The following three invocations are equivalent.

http uri="http://example.com" method="get"
http "http://example.com" "get"
http "http://example.com" method="get"

It is quite common to want to pass boolean arguments to commands, which is why Crush has a special shorthand syntax for it. Passing in --foo is equivalent to passing in foo=true.

Subshells

Sometimes you want to use the output of one command as an argument to another command, just like a subshell in e.g. bash. This is different from using the output as the input, and is done using ():

crush> echo (pwd)

Closures

In Crush, braces ({}) are used to create a closure. Assigning a closure to a variable is how you create a function.

crush> print_greeting := {echo "Hello"}
crush> print_greeting
Hello

Any named arguments passed when calling a closure and added to the local scope of the invocation:

crush> print_a := {echo a}
crush> print_a a="Greetings"
Greetings

For added type safety, you can declare what parameters a closure expects at the start of a closure.

The following closure requires the caller to supply the argument a, and allows the caller to specify the argument b, which must by of type integer. If the caller does not specify it, it falls back to a default value of 7.

crush> print_things := {|a b: integer = 7|}

Additionally, the @ operator can be used to create a list of all unnamed arguments, and the @@ operator can be used to create a list of all named arguments not mentioned elsewhere in the parameter list.

crush> print_everything := {|@unnamed @@named| echo "Named" named "Unnamed" unnamed}

The @ and @@ operators are also used during command invocation to perform the mirrored operation. The following code creates an lss function that calls the ls command and passes on any arguments to it, and pipes the output through the select command to only show one column from the output.

lss := {|@args @@kwargs| ls @args @@kwargs | select %file}

Types

Crush comes with a variety of types:

• lists of any type,
• dicts of any pair of types type, (Some types can not be used as keys!)
• strings,
• regular expressions,
• globs,
• files,
• booleans,
• integer numbers,
• floating point numbers,
• structs, which contain any number of named fields of any type,
• tables, which are essentially lists where each element is the same type of struct,
• table streams, which are like tables but can only be traversed once,
• binary data,
• binary streams, which are like binary data but can only be traversed once,
• types, and
• commands, which are either closures or built in commands.

Crush allows you to create your own types using the class and data commands.

Exploring the shell

When playing around with Crush, the help and dircommands are useful. The former displays a help messages, the latter lists the content of a value.

crush> help
sort column:field

    Sort input based on column

    Example:

    ps | sort ^cpu
crush> dir list
[type, truncate, remove, clone, of, __call_type__, __setitem__, pop, push, empty, len, peek, new, clear]

The content of your current working directory lives in your namespace

All the files in the current working directory are part of the local namespace. This means that e.g. . is a file object that points to the current working directory. The / operator is used in Crush to join two file directory element together.

This means that for the most part, using files in Crush is extremely simple and convenient.

crush> cd .. # This does what you'd think
crush> cd /  # As does this

The right hand side of the / operator is a label, not a value, so ./foo refers to a file named foo in the current working directory, and is unrelated to the contents of any variable named foo.

Namespaces, members and methods

Members are accessed using the : operator. Most other languages tend to use ., but that is a very common character in file names, so Crush needed to find something else.

Most types have several useful methods. Files have exists and stat, which do what you'd expect.

crush> .:exists
true
crush> .:stat
{is_directory: true, is_file: false, is_symlink: false, inode: 50856186, nlink: 8, mode: 16877, len: 4096}
crush> (.:stat):is_file
false

Semi-lazy stream evaluation:

If you assign the output of the find command to a variable like so:

crush> all_the_files := (find /)

What will really be stored in the all_the_files variable is simply a stream. A small number of lines of output will be eagerly evaluated, before the thread executing the find command will start blocking. If the stream is consumed, for example by writing

crush> all_the_files

then all hell will break loose on your screen as tens of thousands of lines are printed to your screen.

Another option would be to pipe the output via the head command

crush> all_the_files | head 1

Which will consume one line of output from the stream. This command can be re-executed until the stream is empty.

More SQL-like data stream operations

Crush features many commands to operate om arbitrary streams of data using a SQL-like syntax. These commands use field-specifiers like ^foo to specify columns in the data stream that they operate on:

ps | where {user == "root"} | group ^status | aggr proc_per_status={count}

(Note that the aggr command is currently broken.)

Unlike in SQL, these commands all operate on input streams, meaning they can be combined in any order, and the input source can be file/http resources in a variety of formats or output of commands like ps, find.

Globs

The * operator is used for multiplication, so Crush uses % as the wildcard operator instead. ? is still used for single character wildcards.

crush> ls %.txt
user         size  modified                  type file
liljencrantz 21303 2020-03-30 13:40:37 +0200 file /home/liljencrantz/src/crush/README.md
crush> ls ????????
user         size modified                  type file
liljencrantz   75 2020-03-07 17:09:15 +0100 file /home/liljencrantz/src/crush/build.rs

The operator %% is used for performing globbing recursively into subdirectories. Another way of looking ath the same syntax is to say that % and ? match any character except /, whereas %% also matches /.

# Count the number of lines of rust code in the crush source code
crush> lines src/%%.rs|count

Wildcards are not automatically expanded, they are passed in to commands as glob objects, and the command chooses what to match the glob against. If you want to perform glob expansion in a command that doesn't do so itself, use the :files method of the glob object to do so:

crush> echo (%%.rs):files

Regular expressions

Regular expressions are constructed like re"REGEXP GOES HERE". They support matching and replacement:

crush> re"ab+c" =~ "abbbbbc"
true
crush> re"a+" ~ "baalaa" "a"
balaa
crush> re"a+" ~~ "baalaa" "a"
bala

Lists and dicts

Crush has built-in lists:

crush> l := (list:of 1 2 3)
crush> l
[1, 2, 3]
crush> l:peek
3
crush> l:pop
3
crush> l:len
2
crush> l[1]
2
crush> l[1] = 7
crush> l
[1, 7]
crush> help l
type list integer

    A mutable list of items, usually of the same type

    * __call_type__  Return a list type for the specified element type
    * __getitem__    Return a file or subdirectory in the specified base directory
    * __setitem__    Assign a new value to the element at the specified index
    * clear          Remove all elments from the list
    * clone          Create a duplicate of the list
    * empty          True if there are no elements in the list
    * len            The number of elements in the list
    * new            Create a new list with the specified element type
    * of             Create a new list containing the supplied elements
    * peek           Return the last element from the list
    * pop            Remove the last element from the list
    * push           Push an element to the end of the list
    * remove         Remove the element at the specified index
    * truncate       Remove all elements past the specified index

and dictionaries:

crush> d := (dict string integer):new
crush> d["foo"] = 42
crush> d["foo"]
42
crush> help d
type dict string integer

    A mutable mapping from one set of values to another

    * __call_type__  Returns a dict type with the specifiec key and value types
    * __getitem__    Return the value the specified key is mapped to
    * __setitem__    Create a new mapping or replace an existing one
    * clear          Remove all mappings from this dict
    * clone          Create a new dict with the same st of mappings as this one
    * empty          True if there are no mappings in the dict
    * len            The number of mappings in the dict
    * new            Construct a new dict
    * remove         Remove a mapping from the dict

Time

Crush has two data types for dealing with time: time and duration.

crush> start := time:now
crush> something_that_takes_a_lot_of_time
crush> end := time:now
crush> echo ("We spent {} on the thing":format end - start)
4:06

The mathematical operators that make sense are defined for time and duration. Subtracting one time from another results in a duration. Adding two duration results in a duration. Multiplying or dividing a duration by a integer results in a duration.

Materialized data

The output of many commands is a table stream, i.e. a streaming data structure consisting of rows with identical structure. Some commands, like cat instead output a binary stream.

These streams can not be rewound and can only be consumed once. This is sometimes vital, as it means that one can work on data sets larger than your computers memory, and even infinite data sets.

But sometimes, streaming data sets are inconvenient, especially if one wants to use the same dataset twice.

crush> files := ls
crush> files
user         size  modified                  type      file
liljencrantz  1307 2020-03-26 01:08:45 +0100 file      ideas
liljencrantz  4096 2019-11-22 21:56:30 +0100 directory target
liljencrantz  4096 2020-03-27 09:18:25 +0100 directory tests
liljencrantz 95328 2020-03-24 17:20:00 +0100 file      Cargo.lock
liljencrantz  4096 2020-02-15 00:12:18 +0100 directory example_data
liljencrantz    31 2019-10-03 13:43:12 +0200 file      .gitignore
liljencrantz 13355 2020-03-29 03:05:16 +0200 file      README.md
liljencrantz  4096 2020-03-27 11:35:25 +0100 directory src
liljencrantz   479 2020-03-24 17:20:00 +0100 file      Cargo.toml
liljencrantz  4096 2020-03-29 01:29:52 +0100 directory .git
liljencrantz  8382 2020-03-29 00:54:13 +0100 file      todo
liljencrantz    75 2020-03-07 17:09:15 +0100 file      build.rs
liljencrantz   711 2019-10-03 14:19:46 +0200 file      crush.iml
crush> files

Notice how there is no output the second time files is displayed, because the table_stream has already been consumed.

Enter the materialize command, which takes any value and recursively converts all transient components into an equivalent but fully in-memory form.

crush> materialized_files := (ls|materialize)
crush> materialized_files
user         size  modified                  type      file
liljencrantz  1307 2020-03-26 01:08:45 +0100 file      ideas
liljencrantz  4096 2019-11-22 21:56:30 +0100 directory target
liljencrantz  4096 2020-03-27 09:18:25 +0100 directory tests
liljencrantz 95328 2020-03-24 17:20:00 +0100 file      Cargo.lock
liljencrantz  4096 2020-02-15 00:12:18 +0100 directory example_data
liljencrantz    31 2019-10-03 13:43:12 +0200 file      .gitignore
liljencrantz 14420 2020-03-29 03:06:02 +0200 file      README.md
liljencrantz  4096 2020-03-27 11:35:25 +0100 directory src
liljencrantz   479 2020-03-24 17:20:00 +0100 file      Cargo.toml
liljencrantz  4096 2020-03-29 01:29:52 +0100 directory .git
liljencrantz  8382 2020-03-29 00:54:13 +0100 file      todo
liljencrantz    75 2020-03-07 17:09:15 +0100 file      build.rs
liljencrantz   711 2019-10-03 14:19:46 +0200 file      crush.iml
crush> materialized_files
user         size  modified                  type      file
liljencrantz  1307 2020-03-26 01:08:45 +0100 file      ideas
liljencrantz  4096 2019-11-22 21:56:30 +0100 directory target
liljencrantz  4096 2020-03-27 09:18:25 +0100 directory tests
liljencrantz 95328 2020-03-24 17:20:00 +0100 file      Cargo.lock
liljencrantz  4096 2020-02-15 00:12:18 +0100 directory example_data
liljencrantz    31 2019-10-03 13:43:12 +0200 file      .gitignore
liljencrantz 14420 2020-03-29 03:06:02 +0200 file      README.md
liljencrantz  4096 2020-03-27 11:35:25 +0100 directory src
liljencrantz   479 2020-03-24 17:20:00 +0100 file      Cargo.toml
liljencrantz  4096 2020-03-29 01:29:52 +0100 directory .git
liljencrantz  8382 2020-03-29 00:54:13 +0100 file      todo
liljencrantz    75 2020-03-07 17:09:15 +0100 file      build.rs
liljencrantz   711 2019-10-03 14:19:46 +0200 file      crush.iml

When the table_stream is materialized into a table, it can be displayed multiple times.

Flow control

Of course Crush has an if command, as well as for, while and loop loops, that can be controlled using break and continue.

crush> help if
if condition:bool if-clause:command [else-clause:command]

    Conditionally execute a command once.

    If the condition is true, the if-clause is executed. Otherwise, the else-clause
    (if specified) is executed.

    Example:

    if (./some_file:stat):is_file {echo "It's a file!"} {echo "It's not a file!"}

for [name=]iterable:(table_stream|table|dict|list) body:command

    Execute body once for every element in iterable.

    Example:

    for (seq) {
        echo ("Lap {}":format value)
    }

Calling external commands

Obviously, one needs to sometimes call out to external commands. Currently, the functionality for doing so in Crush is somewhat primitive. If an internal command of a given name does not exist, Crush looks for external commands, and if one is found, it is used. But Crush does not hand over the tty or emulate a tty, so interactive terminal programs do not work, and commands that prettify their output with escape sequences may fail.

This part of Crush should be considered a proof of concept, but still, most non-interactive commands work as expected:

crush> whoami
liljencrantz

Crush features several shortcuts to make working with external commands easier.

• Firstly, subcommands like git status are mapped into method calls like git:status. That way you do not have to quote the subcommand name, e.g. git "status".
• Secondly, named arguments are transparently translated into options. Single character argument names are turned into options with a single hyphen, and multi-character argument names are turned into GNU style long options with two hyphens, e.g. git:commit m="hello" is converted into git commit -m "hello" and git:commit message="hello" is converted into git commit --message "hello".
• Thirdly, named arguments with a value of boolean true are simply turned into options without a value, so for example git:commit --a --append (or git:commit a=true append=true for that matter) is converted into git commit -a --append.

Further work is required when it comes to job control, terminal emulation and various other integration points.

Executing remote commands

To run a closure on a remote host, use the remote:exec command:

remote:exec {uptime} "example.com"

The closure will be serialized, transferred to the remote host using ssh, deserialized, and executed on the remote host (the crush shell must be in the default path on the remote host). Once the command has been executed and the output of the closure is serialized, transferred, deserialized on the local machine and used as the output of the remote:exec command.

To run a closure on multiple remote hosts, use remote:pexec instead.

Creating custom types

You can create custom types in Crush, by using the class command:

Point := (class)

Point:__init__ = {
    |x:float y:float|
    this:x = x
    this:y = y
}

Point:len = {
    ||
    math:sqrt this:x*this:x + this:y*this:y
}

Point:__add__ = {
    |other|
    Point:new x=(this:x + other:x) y=(this:y + other:y)        
}

p := (Point:new x=1.0 y=2.0)
p:len

Crush supports single inheritance (by passing in the parent to the class command). The class command will create a new struct, that contains a method named new. When called, new will create a new instance of the class. If the __init__ method is defined, new will call it, and pass on any parameters to it.

Add methods by adding them to the class, add member variables by adding them to the instance (this) in __init__.

Similarity to PowerShell

Crush shares the majority of its design goals with PowerShell. I consider PowerShell one of the coolest and most interesting innovations to ever come out of Microsoft. That said, I've found using PowerShell in practice to often feel clunky and annoying, especially for interactive use. I also feel that tying a shell to COM objects is a poor fit.

I wanted to do something similar but with a more streamlined syntax, and with what I felt was a more suitable type system.

Similarity to Nushell

On the surface, Crush looks identical to nushell, but less polished. Crush lacks syntax highlighting, tab completion and has a worse screen rendering. But that is because the focus of Crush right now is to create a well defined, powerful and convenient language that supports things like arithmetic operations, closures, loops and flow control while remaining useful for interactive use.

Future work

There are plenty of langage ideas waiting to be tried out. Pattern matching and error handling are among the most obvious. Also, the error handling in Crush itself is currently very rudimentary.

About the codebase

I am teaching myself rust by writing Crush. I still have plenty to learn. :-)

Building and installing Crush

Crush should work on any modern Unix system. Install rust,

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

clone this repository,

git clone https://github.com/liljencrantz/crush.git

and run

cd crush; cargo build

and you should have a working binary to try out.

Have fun!