sh*shqa

sh*shqa

How I Stopped Worrying and Migrated This Website to a Decentralized Net

Aug 18, 2024

I want to create a blog… But before that, I need to over-engineer my Mojo Dojo Casa Blogging Platform from scratch!

image.png

Background

At the time of writing this post, I am located in Russia. And, if you’ve heard recent news… More and more tech turns into a pumpkin (but this is another long story). The idea for this post came up when I was tinkering with the Radicle — an open-source, peer-to-peer code collaboration stack built on Git — looking for a better alternative to GitHub, which is also in a hurry to turn into a pumpkin.

Radicle extends the Git interface, allowing you to push your repo to the decentralized network of peers. You do rad init inside your repository, and it receives a unique repository ID (RID), which further serves as the public identifier (unlike <username>/<repo-name> in centralized alternatives).

Then you push to the magical rad remote

git push rad <your-branch>

And voila, your repository starts to spread across the network!

image.png

Seeing fancy frames and colored symbols in the terminal I became determined to migrate all the fruits of my labor from the GitHub to the Radicle.

Migrating this website

This blog (with zero posts) was always powered by Hugo and hosted on the GitHub Pages. Radicle doesn’t currently offer any CI/CD options (but is developing some) and doesn’t offer any «Pages» service. I could use Cloudflare Pages or something similar, but

  1. Who knows, Cloudflare can turn into a pumpkin as well ¯\_( ͡° ͜ʖ ͡°)_/¯
  2. Without CI/CD that would require installing a special CLI and integrating it into my local workflow
  3. I just wanted to dig deeper and discover some decentralized solutions

Disclaimer: my final setup isn’t fully decentralized (and uses Cloudflare, heh), but close enough.

My final goals were:

  • The website should be hosted under my domain shishqa.xyz
  • I should be able to release content without ssh-ing somewhere, using complex workflows, or opening dashboards on third-party websites. Preferably, just using Git and Hugo
  • No money :)
  • The website should be generally available from any location with OK-ish load times

The solution came surprisingly fast.

Hosting website directly on the Radicle network

In the cycle of moving repositories to the network I accidentally pushed a Hugo output directory with rendered HTML files. I’ve noticed this while browsing the repo on one of the nodes:

image.png

You know what happened next. I clicked raw view and to my surprise the browser showed a rendered HTML page instead of raw source code (like GitHub does). Some links were broken, some styles and fonts didn’t render, I’ll cover some tweaks I’ve made later. But the first thought was Wow, is this a decentralized ungoverned Pages??? or is it?

image.png

And I can confirm that at least by referencing a related discussion in the Radicle community. But there are some difficulties:

Node serving capabilities

The node itself doesn’t do much. In my previous Pages setups, I had links like shishqa.xyz/whoami. But if I pass the whoami path to the node it will respond with an error, because there is no whoami file in the repo, but only a whoami.html. This problem was solved quite easily by enabling the following options in the Hugo configuration:

  1. relativeURLs = true allowed me to serve the content under the prefix /raw/<rid>/<commit_hash>/public/.
  2. uglyURLs = true solved the problem with .html suffix.

URL

The URL I was redirected to has the following structure:

https:// <seed_url> /raw/ <your_RID> / <commit_hash> / <path_inside_repo>.

There are some problems with it:

  1. When I update content in the main branch I want it to be shown to readers. But, if you know git, you know that a new commit means a new commit hash. So, the URL changes each time I update something, and users are not automatically redirected from the previous commit hash to the next.

Btw, this is a killer feature itself, because I can see my website on any commit without the need to deploy any environments. For example, I work on this post in a branch and can push a commit to the branch and open it in the browser to preview it on the radicle node.

If the <commit_hash> could be replaced with for example <branch_tag> (e.g. main), the URL I want you to open won’t change with push to the main branch. The same discussion gives hope that it will be implemented someday in the Radicle nodes. But there comes the next problem.

Edit: After publishing, cloudhead@ suggested a better solution in this comment. One can simply use :rid/head/:path to get the latest contents!

  1. OK, <your_RID>, <path_inside_repo>, and <branch_tag> won’t change in time. But what to do with seeds? They are hosted by different organizations and enthusiasts (I love this part about the Internet so much), but how to implement the discovery? It is obvious that each seed in the network is not so reliable and doesn’t have any SLA. And I don’t want to list 10 mirrors everywhere, I want to use my domain name.

I’ve googled a lot and had a lengthy chat with my GPT, and figured out that a) generally, DNS providers don’t allow you to serve something like ash.radicle.garden / raw / rad:z4NUTDwCFyoExPEHanFqvSYKu2Wsk / main / raw / public / XXX under my.domain / XXX (this is called the iframe redirect), Nginx can do that, but b) there are no free Nginx-as-a-Service solutions. I decided to stick with Cloudflare’s free offerings for now.

Configuring a Cloudflare Worker

Cloudflare Workers is the platform, which allows running serverless code for each request. Free-tier is not so generous, but is enough for now (I have less than 10 readers at the moment).

I’ve set up a website and configured Cloudflare nameservers so that I own the DNS record in another registar, but configured DNS in Cloudflare. This also gave me free analytics and some level of protection (but of course I control myself and don’t get used to it, this is a platform that can turn into a pumpkin!)

Then I set up a worker, that triggers each time the user requests a URL shishqa.xyz/XXX. The worker has the following parts:

  1. Seed selection. For now, I decided to stick with a simple solution: select a seed manually. By default, I offer ash.radicle.garden, but it can be changed by going to shishqa.xyz/at/<seed>/XXX. In the worker, I parse it the following way:

    const url = new URL(request.url);

const urlRegex = /https:\/\/shishqa\.xyz(?:\/at\/([^\/]+))?\/?(.*)/;
const match = url.toString().match(urlRegex);

if (!match) {
    return new Response(`Error: bad url`, {
        status: 400,
    });
}

const seed = match[1] || 'ash.radicle.garden';
var suffix = match[2] || '';

  2. After selecting a seed, I need to get the latest available commit. For this, I use a Radicle API call:

    Edit: this step can be omitted now, simply use head instead of commit hash in url.

    const repoInfo = await fetch(`https://${seed}/api/v1/projects/${rid}?v=4.0.0`);
if (!repoInfo.ok) {
    return FallbackMirrors(seed, suffix);
}
const headRevision = (await repoInfo.json())['head'];

    Here headRevision is the latest revision of the default branch. The FallbackMirrors function returns a small HTML page with the list of other mirrors you can try (go to https://shishqa.xyz/at/poop/log/000-moving-away-from-github.html)

  3. After these steps, I can fetch the original page from the seed node:

    const originUrl = `https://${seed}/raw/${rid}/${headRevision}/public/${suffix}`;

const originPage = await fetch(originUrl);
if (!originPage.ok) return FallbackMirrors(seed, suffix);

return new Response(originPage.body, originPage);

  4. Frankly speaking, there is one more step before querying. With the setup above you wouldn’t be able to query just shishqa.xyz, because of the reason I mentioned: there is no such file. So I need to modify the suffix a bit:

    const extensionPattern = /.*\.[^\/]*$/;
if (suffix.length == 0) {
  suffix += 'index.html';
} else if (suffix.endsWith('/')) {
  suffix = suffix.slice(0, -1) + '.html';
} else if (!extensionPattern.test(suffix)) {
  suffix += '.html';
}

    Now url shishqa.xyz will point to public/index.html, shishqa.xyz/abc will point to public/abc.html and shishqa.xyz/abc/ will point to public/abc.html as well.

That’s all! You can find the full worker code here. And the repo with the website is here.

Final thoughts

This setup isn’t perfect. I still use Cloudflare Workers with a small quota, but the overall setup is completely free, except I need to pay for my domain. It is always possible to fall back from the domain URL to a Radicle node URL. When referencing files by a branch tag will be implemented, this will be generally usable on Radicle without Cloudflare.

Another caveat is that there is still a small number of seeds in the Radicle network and no seeds in Russia (I’ll probably try to set up one). This means that the network may become unavailable someday. But this will be another story.

What a day to be alive! Thanks for reading ☀️!