

For my undergraduate project I decided to port [VITS](https://arxiv.org/abs/2106.06103), an end-to-end text-to-speech model, to C++ so it could run on mobile devices without a GPU. The result is [vits.cpp](https://github.com/maxilevi/vits.cpp), built on top of [ggml](https://github.com/ggerganov/ggml), the same tensor library used by llama.cpp.

The goal was simple to state: take a modern neural TTS model and make it run fast enough to be useful on a phone.

# Why VITS?

VITS stands for Variational Inference with adversarial learning for end-to-end Text-to-Speech. Compared to older two-stage pipelines that required a separate acoustic model and vocoder, VITS is trained end-to-end and produces noticeably more natural-sounding output.

It was also a practical choice. The pretrained models are openly available, the architecture is well documented, and the original PyTorch implementation is clean enough to be a useful reference when building something from scratch.

## Model architecture

VITS is composed of three main submodels that work together in sequence.

**Prior encoder.** This part of the model takes phonemes as input and encodes them into latent vectors. Working in phoneme space rather than raw text makes the representation more regular and language-agnostic.

**Stochastic duration predictor.** Given the encoded phonemes, this model estimates how long each phoneme should last in the output audio. This is what gives the synthesized speech its natural rhythm and prosody rather than sounding mechanically even.

**Decoder (HiFi-GAN).** The decoder takes the outputs of the prior encoder and the duration predictor and produces raw audio. Under the hood this is the HiFi-GAN V1 generator, a model designed specifically for high-fidelity waveform generation.

The stochastic part matters. VITS samples from the learned distribution rather than always picking the most likely output, which is part of why the resulting speech sounds varied and natural rather than robotic.

# Building on ggml

Rather than writing a tensor library from scratch, I built on top of [ggml](https://github.com/ggerganov/ggml). ggml handles memory management, quantization, and low-level math operations, which meant I could focus on mapping the VITS architecture to operations the library already understood.

The approach was similar to how llama.cpp works: define the model as a graph of ggml operations, load pretrained weights into the graph, and then run inference by executing that graph. The model weights were exported from the original PyTorch implementation using a small Python script.

This was mostly straightforward. The trickier parts were a few operations that ggml did not have built-in, which had to be implemented manually, and making sure the numerics matched the reference implementation closely enough to produce recognizable speech.

# Convolutions: the bottleneck

Once a basic working version was running, profiling made the problem obvious: the decoder was consuming almost all of the inference time, and inside the decoder, convolutions were the culprit.

HiFi-GAN works by repeatedly upsampling the data. At each step it applies a `Conv1dTranspose` to increase resolution, followed by a stack of `Conv1d` operations for feature mixing. By the time the model reaches the final audio output, it is handling tensors with hundreds of millions of floating point values — even for short phrases.

This pattern is expensive on the CPU for reasons that go beyond raw compute.

## CPU sucks for convolutions

The access patterns for `Conv1d` and `Conv1dTranspose` are awkward on CPU. A naive implementation ends up making many small, scattered reads from memory as it slides the kernel across the input. At the sizes VITS produces during decoding, this creates a lot of cache pressure and the CPU spends a disproportionate amount of time waiting for memory rather than doing arithmetic.

GPUs handle this well because they have enormous memory bandwidth and their hardware is designed around exactly this kind of operation. A CPU inference engine does not have that luxury.

## Solution: im2col

The standard solution to this is `im2col`, which rearranges the input tensor into a form where convolution becomes a single matrix multiplication. Instead of sliding a kernel over scattered memory locations, the input patches are first collected into a dense matrix, and then the convolution is computed as one large matrix multiply.

The advantage is that matrix multiplication is something CPUs are much better at. Good implementations use SIMD instructions and are designed to keep data in cache across a large computation, which is exactly the access pattern that benefits from the hardware.

A `Conv1dTranspose` can be expressed as a `Conv1d` with modified padding and stride, so the same technique applies to both operations. Once both are reformulated as matrix multiplications, they benefit from the same optimized kernel.

## Dot product benchmarks

As part of optimizing the matrix multiplications that underpin everything else, I ran benchmarks comparing different dot product implementations:

| Implementation     | Mean time (ms) | Std dev (ms) | Mean error | Std dev error |
|--------------------|----------------|--------------|------------|---------------|
| Naive float32      | 13.13          | 0.23         | 1029.44    | 2.10          |
| SIMD float32       | 29.35          | 0.25         | 11665.7    | 7.30          |
| GGML float32       | 13.05          | 0.31         | 1029.44    | 2.10          |
| GGML float16       | 6.56           | 0.13         | 248975     | 22.47         |
| Naive double       | 26.45          | 0.66         | 0          | 0             |

A few things stand out. The SIMD float32 implementation was actually slower than the naive float32 version. SIMD should in theory be faster, but for this particular access pattern the vectorization overhead and the lack of cache-friendly layout meant it did not help. This matches the general observation that memory bandwidth, not raw compute, was the real bottleneck.

GGML float32 matched the naive float32 speed but with much better integration into the rest of the graph. GGML float16 was about twice as fast at the cost of a large increase in numerical error. For TTS, the tolerance for numerical error is meaningful: the output is audio, and some loss in precision does not necessarily produce audible degradation. Whether float16 is acceptable depends on the specific model and use case.

The naive double implementation was the slowest by a large margin, which confirmed that staying in float32 or float16 was the right call.

## Final thoughts

Getting VITS running on CPU at a usable speed was a good exercise in understanding where time goes in neural inference. The model architecture itself is not the bottleneck. The bottleneck is always somewhere in the data movement, and the solution is usually to find a way to reformulate the computation so it fits the hardware better.

The im2col approach for convolutions is a well-known technique for a reason: it converts an awkward memory access pattern into something the CPU can handle efficiently. The benchmark results for dot products reinforce the same idea from a different angle — raw compute is rarely the limiting factor, and the naive implementation is often competitive precisely because it avoids the overhead of abstractions that were not designed for the specific problem.

The full code and pretrained model weights are available at [github.com/maxilevi/vits.cpp](https://github.com/maxilevi/vits.cpp).


How I ported VITS, an end-to-end neural TTS model, to C++ using ggml for mobile devices, and what it took to make it fast enough to be usable.

Writing a CPU only text to speech engine


[Project Hedra](https://store.steampowered.com/app/1009960/Project_Hedra/) is a game I wrote from scratch using OpenGL which I develop in my spare time.

What I think is the most interesting aspect about it are the worlds. They are completely [procedurally generated](https://en.wikipedia.org/wiki/Procedural_generation) at runtime and the only data saved for them is the original seed.

![](/blog/hedra_lake.jpg)

# World structure

Since the world is theoretically infinite it's impossible to load it all at once. Instead, a `32x32` grid of "chunks" (very similar to chunks in Minecraft) is loaded around the player. New chunks are created when the player moves and the old ones discarded.

Chunks are pieces of the world that hold an offset and information about `32x32x128` voxels. These are stored in a hash table and can be quickly queried by their offset.

## Storing voxel information in memory

A voxel is the smallest unit of world information inside the engine, it's needed to build the meshes for the world and calculate the placements of structures.

Each voxel needs to hold information about its density and its type. The density of a voxel represents "how filled" that specific spot is. A density `< 0` represents an empty or air voxel while a density of `1` is a filled voxel. The type represents the material and properties of that voxel.

A naive approach would be making the voxel struct contain a float for its density and a byte for this type.  This would result in `5 bytes` (8 when we add padding) per voxel. Although this looks good when having many voxels like in our case, it expands to `1GB` [1] of memory just for storing voxel information.

A better approach would be to use a [Half precision floating point](https://en.wikipedia.org/wiki/Half-precision_floating-point_format) which occupies `2 bytes` and reduces our voxel size to `4 bytes` when we add padding. Even though we just cut the memory needed in half I think we can do better.

The approach I choose was to shorten the density information to the range `[-2, 2]` and store all this in an unsigned short with the following format:
```
BlockType    Sign      Significant
0000         0         0000000000
```
Where
* `BlockType` stores the voxel type.
* `Sign` stores the sign of the density.
* `Significant` stores up to 3 significant numbers in the density in integer format. This gives us enough precision to create a smooth-looking world.

This results in just 2 bytes per voxel, needing only `256MB` of memory for storing the voxel information.

# Generating the structure

A typical approach to generating these density values I talked about previously is using 3D [Perlin Noise](https://en.wikipedia.org/wiki/Perlin_noise). Perlin noise is a very commonly used noise for generating procedural worlds. With enough layering, you can make realistic-looking surfaces just from this simple function.

Hedra takes a similar approach but instead uses [Simplex Noise](https://en.wikipedia.org/wiki/Simplex_noise) because of the lower computational overhead.  This noise is generated using [SIMD](https://github.com/Auburn/FastNoiseSIMD) to improve its performance during runtime.

A bunch of layers of noise are generated both in 2D and 3D some of these are generated in a lower resolution and then upscaled using trilinear interpolation. Finally, these are all mixed to create a single density value.

![](/blog/hedra_terrain.jpg)

# Paths, rivers, and environment

After the base terrain is generated the environment, path and rivers need to be placed. In the case of rivers and path
layers of noise with very big amplitude are generated. These are then clamped to a range as to form "rings". Many of these
noise rings with different seeds are layered together forming a network of paths and rivers.

The environment consists of stuff like trees, rocks, grass, reeds, etc. These are placed either randomly
using the chunk's offset as a seed for the RNG or in the case of grass and trees using layers of perlin noise in order
to form patches of grass and trees.

# Meshing the terrain

![](/blog/hedra_terrain_wireframe.jpg)

Finally, the last step is to combine all the data generated and create the mesh. Because of the smooth looks of the game, this step is usually very different from other voxel engines.

First, the voxel information is polygonized into a mesh using [Marching Cubes](https://en.wikipedia.org/wiki/Marching_cubes). This process is run for all the voxels in each chunk, grabbing the neighbor information and marching a cube through it to determine the correct surface. At the same step neighbor information and some noise is used to determine the color of the generated surface.

A similar thing is done for the water mesh. Then this information is combined with the hand-made meshes we placed earlier. This mesh is then optimized and simplified depending on the chunk LOD (using the awesome [meshoptimizer](https://github.com/zeux/meshoptimizer)), stitching together borders with other chunks to avoid holes in the terrain.

As the last step, the chunk mesh which is divided into 3 different buffers (water, static, and instance) is uploaded into the corresponding GPU. The separation is needed because they use different shaders and rendering techniques.

![](/blog/hedra_terrain_full.jpg)

# Next
Some ideas for other blog entries:
* Rendering engine in Project Hedra
* Placing human-generated content in an algorithmically-generated world


[1] `32x32` chunk grid of `32x32x128` voxels each = `32x32x128x32x32x8 = 1073741824 bytes`.


A simple overview of how worlds are generated and meshed in my open world RPG "Project Hedra"

Quickly generating procedural worlds at runtime


For my personal page I wanted something clean and minimalistic that could be hosted as a static site and with the simplest publishing process possible.

I also wanted to avoid having a database, an admin panel or any sort of CMS. The content is small, the layout is simple and most of the time I just want to write a markdown file, push it and be done with it.

# Technologies

## Next.js

The site is built with [Next.js](https://nextjs.org/). For something this small it might sound like overkill at first, but it gives me a very nice development experience and static generation out of the box.

The interesting part for the blog is that every post page is generated at build time. The blog index calls `getPosts()` to read all the files inside the `posts/` directory and the individual post page uses `getStaticPaths()` plus `getStaticProps()` to generate one route per markdown file.

That means there is no backend serving blog content dynamically. Once the site is built, it is just static files.

## Tailwind

For styling I used [Tailwind CSS](https://tailwindcss.com/). I like it for small personal projects because it makes it very easy to iterate on spacing and typography without having to spend time naming CSS classes.

Most of the layout is just a handful of utility classes and the blog post body uses the typography plugin so markdown content gets decent defaults for headings, paragraphs and links.

The end result is pretty close to what I wanted: very little custom CSS and a UI that stays simple.

## Loading blog posts

Posts live as plain `.md` files inside the `posts/` folder. Each file starts with a tiny frontmatter block:

```
---
title: Post title
description: Short description
hidden: false
date: 2021-06-25
---
```

I wrote a very small parser for this in `common/posts.js`. It reads the file, splits the header from the body and returns an object containing the metadata and the markdown contents.

This is intentionally simple. I do not need tags, categories, authors, nested metadata or anything fancy, so keeping it small felt better than bringing in extra dependencies.

The markdown body is rendered on the page using `react-markdown`, which means writing a new post is basically just creating a new file and adding some content.

## Publishing

The publishing flow is also intentionally minimal. Tailwind is compiled into a single stylesheet, Next.js builds the site, and then the project can be exported as static HTML.

So adding a post looks roughly like this:

1. Create a markdown file in `posts/`
2. Add the title, description, date and visibility in the header
3. Write the post body in markdown
4. Build and export the site
5. Deploy the generated static files

This setup is probably not the most powerful one, but I think it is a good fit for a personal website. It is easy to understand, easy to modify and does not require me to maintain any infrastructure beyond static hosting.

## Final thoughts

I really like small websites that do one thing well. In this case that means a homepage, a projects page and a blog with markdown posts.

There are still a couple of things I would like to improve over time, like richer markdown support and better metadata handling, but for now this does exactly what I wanted: it stays out of the way and lets me write.

A quick overview of the stack and structure behind this personal site and blog.

How I built this blog


# Porting C++ code to CUDA

Recently I spent some time moving parts of my [raytracer](https://github.com/maxilevi/raytracer) from regular C++ code to CUDA so the GPU could handle the expensive ray intersection work.

At a high level the goal sounds simple: take the code that shoots rays and checks for collisions and run it on the GPU instead of the CPU.

In practice, the interesting part is that CUDA is not just "faster C++". Some patterns that feel very natural on the CPU become awkward or expensive once the code has to run inside a kernel.

This post is a quick overview of what had to change and why.

## The original CPU structure

The raytracer already had a pretty standard structure. A `Scene` contains geometry, a `Camera` casts rays, and those rays are tested against the scene to compute the color of each pixel.

The expensive part is collision detection. For any non-trivial scene, testing every ray against every triangle would be far too slow, so the engine uses a [BVH](https://en.wikipedia.org/wiki/Bounding_volume_hierarchy). That lets us discard large parts of the scene early by first testing the ray against bounding boxes.

On the CPU this kind of code is very comfortable to write recursively. A simplified version looks like this:

```cpp
bool Hit(Node* node, const Ray& ray) {
if (!node->box.Hit(ray)) return false;

if (node->is_leaf) {
return HitTriangles(node, ray);
}

return Hit(node->left, ray) || Hit(node->right, ray);
}
```

That is easy to read and easy to extend. Unfortunately it is not a great fit for GPU execution.

## CUDA is a different environment

When I first approached the port I made the mistake of thinking mostly in terms of syntax. CUDA lets you write code that looks a lot like C++, so it is tempting to assume you can move whole classes over with just a few annotations.

That is not really how it goes.

The important difference is that CUDA code runs in a separate execution environment with its own memory and its own constraints. Data has to be copied to the GPU, kernels need simple memory layouts, and some features that are convenient on the CPU are much less convenient on the GPU.

For this project, the two main friction points were:

1. Recursive traversal
2. Object-oriented data structures with pointers

Those two things show up everywhere in a CPU raytracer.

## Why recursion becomes a problem

Recursive tree traversal is the most natural way to walk a BVH, but it is not something I wanted to rely on in CUDA kernels.

NVIDIA does support recursion in device code in some cases, but it comes with practical limitations and is generally something you want to avoid in hot code like ray traversal. It is harder to reason about stack usage and it pushes the implementation away from the simple data-parallel model that GPUs like best. The CUDA programming guide also calls out recursion-related limits and stack considerations [here](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#nesting-and-synchronization-depth).

So instead of trying to preserve the recursive CPU traversal exactly, I rewrote it as an iterative depth-first search using an explicit stack.

This is the key idea behind the GPU BVH implementation in [`gpu_bvh.cu`](https://raw.githubusercontent.com/maxilevi/raytracer/master/src/volumes/gpu_bvh.cu).

## Flattening the BVH

Another important change was the representation of the tree itself.

On the CPU, a BVH node can naturally point to its left and right child using regular pointers. That is flexible, but it is not an ideal layout to copy to the GPU. Pointer-heavy structures are awkward to serialize, difficult to move between host and device, and usually not great for memory locality.

The solution I used was:

1. Build the BVH on the CPU
2. Traverse that CPU BVH once
3. Generate a flat array of GPU nodes and a flat array of GPU triangles
4. Copy those arrays to GPU memory

That is exactly what the `TraverseBvh` and `FromBvh` functions are doing in the repository.

Instead of storing child pointers, each `GPUBvhNode` stores integer indices for its children. Leaf nodes store indices to triangles, while internal nodes store indices to other nodes. This makes the structure much easier to upload and traverse on the GPU.

In other words, the CPU version owns a tree of objects, while the GPU version owns a few compact arrays.

## Replacing recursion with an explicit stack

Once the BVH is flattened, traversal becomes an iterative loop.

The nice thing about depth-first traversal is that the recursive call stack can be replaced by a very small fixed-size array. In the GPU code I used:

```cpp
const int MAX_STACK_SIZE = 64;
```

The comment in the source explains the reasoning well: for a binary tree, the height grows roughly with `log2(N)`, so a stack of 64 entries is enough for an absurdly large number of nodes.

The traversal then looks roughly like this:

```cpp
GPUBvhNode stack[MAX_STACK_SIZE];
size_t count = 0;
stack[count++] = root;

while (count > 0) {
GPUBvhNode node = stack[--count];

if (!node.Hit(ray, t_min, closest_so_far))
continue;

if (!node.is_leaf) {
stack[count++] = right_child;
stack[count++] = left_child;
} else {
// test leaf triangles
}
}
```

This is still a depth-first search, just written in a way that works better inside a CUDA kernel.

The traversal algorithm stayed the same. The main change was representing the BVH as flat arrays and replacing recursive calls with an explicit stack.

## Materials and object ownership

The same kind of flattening happened for materials.

On the CPU it is convenient for triangles to reference material objects directly, but on the GPU you want a much simpler representation. In the port, materials are created on the CPU, converted into `GPUMaterial` values, stored in a flat array and then referenced by index when triangles are uploaded.

There is a small but important detail in the code: materials are deduplicated before being copied. The `AddTriangle` helper keeps a map from CPU material id to GPU material index so the same material is only uploaded once.

That is a practical detail that shows up often in CUDA code. Once the data is moved into flat arrays, you need to keep track of indices, deduplicate shared data and copy everything to device memory explicitly.

## What actually stayed the same

One thing I like about this port is that the high-level renderer design did not need to be thrown away.

The scene is still a scene. Rays are still rays. A BVH is still a BVH. The intersection math is the same, the shading model is the same, and the overall rendering pipeline is still recognizable.

The CPU version can afford richer object graphs and recursive logic because that makes the code easier to express. The GPU version benefits from compact arrays, explicit indices and iterative traversal because that makes execution simpler and memory access more predictable.

What changed when I moved parts of my raytracer from regular C++ code to CUDA kernels.

Porting CPU C++ code to CUDA


This week's most of my efforts where focused in adding more content into the game. More specifically I worked on adding a new dungeon into the overworld. Here is a picture of the inside layout of a future dungeon:

![](/assets/posts/post3/dungeon_layout.jpg)

So adding the model and the collisions into the game was pretty easy and fast because of groundwork that was done a few months before. (This "groundwork" was porting the engine to use [bulletphysics](https://en.wikipedia.org/wiki/Bullet_(software))). Here is a picture of the dungeon ingame:

![](/assets/posts/post3/ingame0.png)

Instead, the development bottleneck was making the AI work smoothly. Currently the AI for the mobs uses [AStar](https://en.wikipedia.org/wiki/A*_search_algorithm) which is an algorithm for finding a shortest path from a grid of points. This works fine in the overworld terrain however when mobs are inside structures there are too many grid points to sample so the AI becames too slow to do anything.

My first idea for solving this issue was to use [navigation meshes](https://en.wikipedia.org/wiki/Navigation_mesh) this would allow me to clearly define "walkable zones"
that the AI can traverse. Then I would build a [graph](https://en.wikipedia.org/wiki/Graph_theory) out of those "walkable zones" and use it as waypoints. [0]
Here is an ingame picture:

![](/assets/posts/post3/navmesh.png)

Now finding shortests paths isn't something very hard to do since it's well explored topic. For solving this particular problem I used [Dijkstra's algorithm ](https://en.wikipedia.org/wiki/Dijkstra%27s_algorithm)

Here is a simple explanation from [wikipedia](https://en.wikipedia.org/wiki/Dijkstra%27s_algorithm):

_Suppose you would like to find the shortest path between two intersections on a city map: a starting point and a destination. Dijkstra's algorithm initially marks the distance (from the starting point) to every other intersection on the map with infinity. This is done not to imply that there is an infinite distance, but to note that those intersections have not been visited yet. Some variants of this method leave the intersections' distances unlabeled. Now select the current intersection at each iteration. For the first iteration, the current intersection will be the starting point, and the distance to it (the intersection's label) will be zero. For subsequent iterations (after the first), the current intersection will be a closest unvisited intersection to the starting point (this will be easy to find)._

![](https://upload.wikimedia.org/wikipedia/commons/thumb/5/57/Dijkstra_Animation.gif/220px-Dijkstra_Animation.gif)

[Source](https://en.wikipedia.org/wiki/Dijkstra%27s_algorithm)

I managed to implement it and now the AI is working correctly inside the buildings.

With nothing more to show I will end this week's post :)


Notes:

[0] This is basically applying AStar but on graph thats not built in realtime.

A look at Project Hedra's first dungeon and how navigation meshes plus Dijkstra's algorithm improved indoor AI.

Navigation meshes for project hedra


If you are seeing this it's because the NextJs backend is loading all the markdown files.

More coming soon.

Blog