Weave

A language for data pipelines

Weave is a scripting language that makes data transformation trivial to express. Read files. Transform data. Write results. In a few lines.

Get Started

View on GitHub

First-Class File Formats

CSV, JSON, YAML, XML, TOML, and INI are built into the language. The most common cases are preconfigured so you can just read and go.

config = read("app.toml", :toml)
db_host = config[:database][:host]
users = read("users.csv", :csv)
users *> ^(u) { puts(u[:name]) }

Pipeline Operators

Chain operations with pipe (|>), map (*>), and reduce (&>). Data flows left to right, like a sentence.

read("sales.csv", :csv)
    *> ^(row) { row[:amount] }
    &> ^(v, acc: 0) { acc + v }
    |> print

Shell Integration

Glueing commands together should be easy. Run shell commands with backticks and get back structured output.

result = `kubectl get pods -o json`
if result[:success] {
    pods = result[:output] |> json_parser
    pods[:items] *> ^(p) {
        puts(p[:metadata][:name])
    }
}

Beautiful Weave

Weave's syntax is designed to feel natural for data work.

Structured data, no setup

Just read and go. Common datafile formats are built into the language.

Weave

config = read("config.json", :json)
puts(config[:database][:host])

import json

with open('config.json') as f:
    config = json.load(f)
print(config['database']['host'])

jq -r '.database.host' config.json

data, _ := os.ReadFile("config.json")
var config map[string]any
json.Unmarshal(data, &config)
db := config["database"].(map[string]any)
fmt.Println(db["host"])

Pipelines read like sentences

Data flows in natural reading order: read, transform, reduce, print. Each step feeds the next.

Weave

read("sales.csv", :csv)
    *> ^(row) { row[:amount] }
    &> ^(v, acc: 0) { acc + v }
    |> print

import csv
with open('sales.csv') as f:
    total = sum(float(row['amount'])
        for row in csv.DictReader(f))
print(total)

awk -F',' '
  NR>1 {sum += $3}
  END {print sum}
' sales.csv

f, _ := os.Open("sales.csv")
r := csv.NewReader(f)
r.Read() // skip header
total := 0.0
for rec, _ := r.Read(); rec != nil; rec, _ = r.Read() {
    v, _ := strconv.ParseFloat(rec[2], 64)
    total += v
}
fmt.Println(total)

Functions that build functions

Closures and lambdas make it easy to create specialized, reusable tools on the fly.

Weave

fn make_multiplier(factor) {
    ^(x) { x * factor }
}

triple = make_multiplier(3)
[1, 2, 3] *> triple    # [3, 6, 9]

def make_multiplier(factor):
    def multiply(x):
        return x * factor
    return multiply

triple = make_multiplier(3)
[triple(x) for x in [1, 2, 3]]  # [3, 6, 9]

make_multiplier() {
    local factor=$1
    eval "${2}() { echo \$((\$1 * $factor)); }"
}
make_multiplier 3 triple
for i in 1 2 3; do triple "$i"; done

func makeMultiplier(factor int) func(int) int {
    return func(x int) int {
        return x * factor
    }
}

triple := makeMultiplier(3)
for _, v := range []int{1, 2, 3} {
    fmt.Println(triple(v))  // 3, 6, 9
}