Syntax Highlighting

-t compiles a grammar into a TextMate grammar — the JSON format VS Code and many other editors use for syntax highlighting — from whichever rules you’ve tagged with (as scope.name), as described in The Px Language. Untagged rules are pure grammar, invisible to -t, transparently inlined wherever a tagged rule calls them. When a tagged rule calls another tagged rule, the callee keeps its own colour for its own span — that’s the only way to get more than one colour out of a single rule; Px deliberately has no way to tag part of a rule’s body.

px -t px.px > px.tmLanguage.json

How a rule becomes a pattern

A tagged rule that doesn’t call another tagged rule (inside a repetition, at least) becomes a flat match pattern — its whole compiled regex, one colour. A rule that does needs a begin/end region instead, so the callee can be included with its own colour in between.

The interesting case is a rule like Px’s own literal — content only, no delimiters of its own (see pitfalls.md again: any unanchored content pattern will happily swallow text it shouldn’t). Used bare, literal’s pattern has no way to know where a real Px string literal starts. The generator solves this the same way you would by hand: it scans the rest of the grammar for how literal is actually called — finds '\'' literal '\'' in atom — and anchors it as begin: "'", end: "'" instead of a bare match. class gets the same treatment from '[' class ']'. This is exactly the lookahead lesson from the pitfalls page, just applied by the generator instead of by you.

The actual output

Here’s what px -t px.px produces right now — eight tagged rules, none of them warned-about or skipped:

{
  "scopeName": "source.px",
  "patterns": [
    { "include": "#space" }, { "include": "#scope_annotation" },
    { "include": "#scope_name" }, { "include": "#label" },
    { "include": "#name" }, { "include": "#literal" },
    { "include": "#property" }, { "include": "#class" }
  ],
  "repository": {
    "literal": { "begin": "'", "end": "'", "name": "string.quoted.single.px" },
    "class":   { "begin": "\\[", "end": "\\]", "name": "constant.other.character-class.px" },
    "name":    { "match": "[A-Za-z_][\\w_]*", "name": "entity.name.function.px" }
  }
}

(trimmed for space — the full repository has all eight rules.)

Live demo

Below is px.px highlighting itself, and grammars/json.px highlighting with the exact same grammar (the tags live entirely in px.px; every .px file benefits for free). The renderer is a small hand-written tokenizer implementing just the two pattern shapes above — not a full Oniguruma engine, but enough to show precisely what a real editor would do with this output.

• spacecomment.line.double-slash.px
• scope_annotationkeyword.other.annotation.px — whole (as ...) span, one colour
• scope_nameentity.other.attribute-name.px
• labelkeyword.other.annotation.px — shares scope_annotation's colour
• nameentity.name.function.px — every identifier
• literalstring.quoted.single.px
• propertyconstant.character.escape.px
• classconstant.other.character-class.px

Using the output

The generated .tmLanguage.json is a standard TextMate grammar. VS Code loads it through the smallest possible kind of extension — no code, just a package.json declaring the language and pointing at the grammar file. Many other editors (Sublime Text, JetBrains via a plugin, the CodeMirror/Monaco vscode-textmate library) consume the same format directly.

Building the extension folder

mkdir -p px-syntax/syntaxes
px -t px.px > px-syntax/syntaxes/px.tmLanguage.json

px-syntax/package.json:

{
  "name": "px-syntax",
  "displayName": "Px",
  "version": "0.0.1",
  "engines": { "vscode": "^1.60.0" },
  "contributes": {
    "languages": [
      {
        "id": "px",
        "extensions": [".px"],
        "aliases": ["Px"]
      }
    ],
    "grammars": [
      {
        "language": "px",
        "scopeName": "source.px",
        "path": "./syntaxes/px.tmLanguage.json"
      }
    ]
  }
}

scopeName here has to match the "scopeName" field inside the generated JSON exactly — -t derives it from the grammar’s base filename (px.px → source.px), so if you rename the .px file, regenerate before re-testing.

Installing it for local testing

The fastest loop while you’re iterating — no packaging step, no marketplace — is to drop the folder straight into VS Code’s extensions directory and reload:

ln -s "$(pwd)/px-syntax" ~/.vscode/extensions/px-syntax-0.0.1

(%USERPROFILE%\.vscode\extensions on Windows; VS Code Insiders and other variants use their own sibling folder.) Then in VS Code: Cmd/Ctrl+Shift+P → Developer: Reload Window, open any .px file, and the colours above should be exactly what you see. Every time you regenerate px.tmLanguage.json, reload the window again to pick up the change — no reinstall needed, since it’s a symlink.

Packaging it properly

Once it’s worth sharing, package it into a real .vsix with Microsoft’s own extension CLI:

npm install -g @vscode/vsce
cd px-syntax
vsce package                              # writes px-syntax-0.0.1.vsix
code --install-extension px-syntax-0.0.1.vsix

That .vsix is also what you’d upload to the VS Code Marketplace, or hand to someone else to install the same way.