Hey everyone, my name is Abin Simon
and this talk is about "Tree-sitter:
Beyond Syntax Highlighting."
For those who are not aware of what Tree-sitter is,
let me give you a quick intro.
Tree-sitter, at its core, is a parser generator tool
and an incremental parsing library.
What it essentially means is that it gives you
an always up-to-date
AST [abstract syntax tree] of your code.
In the current Emacs frame, what you see to the right
is the AST tree produced by Tree-sitter
of the code that is on the left.
For example, if you go to this "if" statement,
you can see it goes here.
It is also really good at handling errors.
For example, if I were to delete this [if statement],
it still parses out a tree as much as it can,
but with an error node.
Now let's see how we can query the tree
to get the information that we need.
Let's first try to get all the identifiers in the buffer.
It highlights all the identifiers in the buffer,
but let's say we want to get something
a little more precise.
Let's say we wanted to get this "i" here.
This, in our case, would be this identifier
inside this assignment expression
inside this "for" statement.
We can write it out like this.
I hope this gives you a basic idea
of how Tree-sitter works and how you can query
to get the information that you need.
First of all, let's see how Tree-sitter can help us
with syntax highlighting.
This is the default syntax highlighting by Emacs for SQL.
Now let's see how Tree-sitter helps.
This is the syntax highlighting in Emacs
which Tree-sitter enabled.
You'll see that we're able to target
a lot more things and highlight them.
That said, you don't always have to
highlight everything.
I personally prefer a much simpler theme.
Now let's see how Tree-sitter helps you simplify
adding custom syntax highlighting to your code.
This is a Python file which has
a class and a few member functions.
Anyone who has used Python will know that
the "self" keyword, while it is passed in as an argument,
it has more meaning than that.
Let's see if you can use Tree-sitter
to highlight just the "self" keyword.
If you look at the Tree-sitter tree,
you can see that this is the first identifier
in the list of parameters for a function definition.
This is how you would query for the first identifier
inside parameters inside a function definition.
Now, if you see here, it also matches "cls",
but let's restrict it to match just "self".
Now we have a Tree-sitter query that identifies
the first argument to the function definition
and is also called "self".
We can use this to apply custom highlighting onto this.
This is pretty much all the code
that you'll need to do this.
The first block here is essentially to say to
Tree-sitter to highlight anything with python.self
with the face of custom-set.
Now the second block here essentially is
how we match for that.
Now if you go back into a Python buffer
and re-enable python-mode, we'll see that "self"
is highlighted differently.
How about creating text objects?
Tree-sitter can help there too.
For those who don't know, text objects
is an idea that comes from Vim,
and you can do things like select word,
delete word, things like that.
There are other text objects like line and paragraph.
For each text object, you can have operations
that are defined on them.
For example, delete, copy, select, comment,
all of these are operations that you can do.
Let's try and use Tree-sitter to add more text objects.
This is a plugin that I wrote
which lets you add more text objects into Emacs.
It helps you code aware text objects
like functions, conditionals, loops, and such.
Let's see an example scenario of how
something like this could come in handy.
For example, I can select inside this condition
or inside this function and do things like that.
Let's say I want to take this conditional,
move to the next function, and create it here.
What I would do is something like
delete the conditional, move to the next function,
create a conditional there, and paste.
Let's try another example.
Let's say I want to take this and move it to the end.
If I had to do it without text objects,
I'd probably have to go back to the previous comma,
delete till next comma, find the closing bracket,
and paste before.
That works, but let's see
how Tree-sitter can simplify it.
With Tree-sitter, I can say delete the argument,
go to the end of the next argument, and then paste.
Tree-sitter essentially helps Emacs
understand the code better semantically.
Here is yet another use case.
I work at a remote company,
and I often find myself being in a call
with my teammates, explaining the code to them.
And one thing that really comes in handy
is the narrowing accessibility of Emacs.
Specifically, the fancy-narrow package.
I use it to narrow just the function,
or I could narrow to the conditional.
Next to the end, the list would be code folding.
This is a package which uses Tree-sitter
to improve the code folding functionalities of Emacs.
Code folding has always been this thing
that I've had a love-hate relationship with.
It usually works most of the time,
but then fails if the indentation is wrong
or we do something weird with the arguments.
But now with Tree-sitter in the mix,
it's a lot more precise.
I can fold comments, I can fold functions,
I can fold conditionals. You get the idea.
I work with Kubernetes, which means I end up
having to write and read a lot of YAML files.
And navigating big YAML files is a mess.
The two main problems are figuring out where I am,
and two, navigating to where I want to be.
Let's see how Tree-sitter can help us with both of this.
This is an example YAML file.
To be precise, this is the values file
of the Redis helm chart.
I'm somewhere in the file on tag under image,
but I don't know what this tag is for.
But with the help of Tree-sitter,
I've been able to add this information
into my header line.
If you see in the header line,
you'll see that I'm under sentinel.image.
Now let's see how this helps with navigation.
Let's say I want to enable persistence on master node.
So with the help of Tree-sitter,
I was able to enumerate every field
that is available in this YAML file,
and I can pass that information onto imenu,
which I can then use to go to exactly where I want to.
Also, since we're not dealing with
any language specific constructs,
this is very easy to extend to
other similar languages
or config files in this case.
So for example, this is a JSON file,
and I can navigate to location or project.
And just like in YAML, it shows me where I'm at.
I'm in projects.name,
or I'm inside projects.highlights.
Or how about Nix?
This is my home.nix file.
Again, I can search for services,
and this lists me all the services that I've enabled.
How about just services.description?
So this is all the services
that I've enabled and have descriptions.
Now that we have seen this for config files,
let's see how similar things apply for code.
Just like in config files,
I can see which function I'm under,
and if I go to the next function, it changes.
Okay, here is something really awesome.
This is probably one of my favorites,
and one of the things that actually made me understand
how powerful Tree-sitter is, and got me into it.
I work with a lot of Go code,
and anyone who has worked with Go will tell you
how repetitive it is handling errors.
For those who don't write Go,
let me give you a rough idea of what I'm talking about.
If you want to bubble up the error,
the way you would do it is just to return the error
to the function that called it.
Over here, you can either return nil or an empty value,
and at the end, you return error.
Let's try and use Tree-sitter to do this.
Using the help of Tree-sitter, let's make Emacs
go back, figure out what the return arguments are,
figure out what their default values are,
and automatically fill in the return statement.
It would look something like this.
In my case, it filled in the complete form,
it figured out what the return arguments are,
what their types are,
and what their default values are,
and filled out the entire return.
And since this is a template,
I can go to the next function, do the same thing,
next function, do the same thing,
next function, do the same thing.
Here is a really fascinating use case of Tree-sitter,
structural editing.
You might be aware of plugins like paredit,
which seems to "know" your code.
This sort of takes it onto another level.
It is in its early stages, but what this lets you do
is completely treat your code as an AST,
and edit as if it's a tree instead of characters.
I am not going to go much in depth into it,
but if you're interested, there is a talk
from last year's EmacsConf around it.
I'm just going to end this with one last tiny thing
that I found in the tree-sitter-extras package.
It's this tiny macro called tree-sitter-save-excursion.
It works pretty much like save-excursion, but better.
It uses the Tree-sitter syntax tree
instead of just the code
to figure out where to restore the position.
My main use case for this was with code formatters.
Since the code moves around a lot
when it gets formatted,
save-excursion was completely useless,
but this came in handy.
I'll just leave you off with
what the future of Tree-sitter looks like for Emacs.
So far, every Tree-sitter related feature
that I've talked about is powered by this library.
But there is talk about Tree-sitter coming into the core.
It will most probably be landing in Emacs 29,
and if you want to check out the work on Tree-sitter
in core Emacs, you can check out
the features/tree-sitter branch.
You'll probably see more and more features and packages
relying upon Tree-sitter, and even major modes
being powered by Tree-sitter.
And that's a wrap from me. Thank you.