fortuned, the second
- 2017-06-06T11:09:07+02:00
So, it's been a while since I wrote about that fortune cookie server. I had sort of gotten bored of it, you know how that happens. But over the long weekend, I found some time to revisit that - this time as a proper API. Since I like to make fun of this whole "frob as a service" thing. And because I can.
So I came up with this API for fortune cookies. It's all on GitHub, doesn't have an SLA, because that would be silly, but it did just get its first release. Long weekend and all.
Like any good microservice thingy, you can query it with curl:
curl https://api.ef.gy/fortune
No more disqus!
- 2016-05-15T22:17:16+02:00
Disqus is actually surprisingly annoying to inline, and I wasn't really happy with how it pretty much forces you to also load Google Analytics, so I've killed it on the site.
The DNT setting in your browser still has an effect, though. It controls whether or not iframes get to load in the page or whether they turn into links for you to follow. For YouTube, etc.
Seeing as how most of the discussion on contents of this site aren't really happening here, and the disqus plugin is broken half of the time anyway, this shouldn't really affect anyone. If you do want to add comments to an article, just do so on Facebook or Google+ and tag the author or something :).
Right. Now carry on, then!
Getting down to business with a Raspberry Pi PIco UPS
- 2016-01-19T21:10:33+00:00
You may remember the picture of my shiny little To-enabled Pi on the last blog post about monitoring the very same device through Tor. It had just received a shiny new PIco UPS backpack with a LiPO battery from pimodules.com. It turns out, however, that I first had to hack that a bit to make it usable for my purposes.
For one thing, it was a wee bit too big to fit in the current case, so I had to make room by getting a new one. Which obviously made the Pi rather happy:
For another, the device actually requires you to run a Python script as root so that the firmware can figure out if your Pi is booted or not, and to get it to shut down if a power failure is imminent. Now, I don't know if you follow my Twitter, but on there I'd previously quite proudly declared that I'd gotten rid of Python on the sucker, and I wasn't about to put it back on permanently just to get the UPS to work.
Secure Prometheus Monitoring via SSH over Hidden Service
- 2016-01-10T12:32:58+00:00
So you've set up monitoring via Prometheus, e.g. by following a guide like this one. The obvious next step is to make sure all the servers you want to monitor this way are actually reachable by Prometheus - and you definitely want to make sure your monitoring data isn't tampered with in transit.
This is reasonably straightforward if you control or trust the network, but if your monitoring targets are on remote networks and will traverse unsafe networks, such as the internet... well... not so much, then. There, you basically need to solve three problems:
- Make it possible for Prometheus to scrape its target, no matter where it is.
- Authenticate the target, and Prometheus to the target.
- Prevent malicious users from messing with the data in flight.
In particular the reachability is a bit of doozie - the inspiration for all this was basically that I wanted to monitor my portable, battery-powered Raspberry Pi Tor gateway from one of my servers. The typical way to do that is to play with my firewall at home and use NAT to allow inbound connections to the Pi, but since the thing is portable that wouldn't always work - for instance when the Pi is resting comfortably in my backpack at work.
A Primer on div-Abuse
- 2016-01-02T11:45:10+00:00
div is one of the universal staples of web design. It really shouldn't be. The same goes for span. It's not like these tags are never the right tool for the job, but most of the time they're used they really shouldn't have been.
The problem is that using these tags when they wouldn't be necessary turns your HTML into a meaningless mess, which at the same time will make CSS to style it a lot harder to read and write than it would otherwise have to be - which in turn leads to people thinking that CSS is hard, when really it's not. So I've compiled this handy list to highlight the more common abuse cases out there. It also requires extra bandwidth, and I've yet to see a single site doing any of the following that then actually set up proper compression to fix it - or the extra whitespace and HTML comments people are so fond of, at that.
Please note that I made none of these examples up. I literally keep seeing them every time I use an HTML inspector on just about any random site.
for marking paragraphs
Using a Battery-powered Raspberry Pi as a Tor Gateway
- 2015-12-26T11:47:35+01:00
Time to have some fun with a shiny new Raspberry Pi! Obviously one of the things I had to do, was to make it work as an AP for my iPad Pro to connect to, so I could SSH in and do some actual work.
It turns out that it's pretty straightforward to go from there to a neat little Tor gateway. For, you know, whenever you'd need that. The finished product looks something like this:
To create one of these handy little buggers, you'll need the following:
Feedly Can't Handle Full XHTML Content in atom:content
- 2015-12-26T09:06:26+01:00
I got a heads up about this a few days ago. As it turns out, the most excellent feed reader Feedly can't handle atom:content nodes with full XHTML content, i.e. of the form:
<atom:content type="application/xhtml+xml">
<xhtml:html ...>
<!-- ... -->
</xhtml:html>
</atom:content>
The content gets rendered all sad and unformatted, with just the literal text content of the nodes rendered. Like so:
How the iPad Pro Replaced My Laptop
- 2015-12-20T13:17:12+00:00
My old personal laptop was due for a refresh around the time the iPad Pro was announced. While I still really like it, the previous four years of abuse are starting to take their toll on the poor thing, with the display doing flickery things no display ever should and the battery starting to last less than three hours. I guess four years is a good run for a laptop these days.
I went browsing for a suitable replacement when I noticed that I really only ever do three things with it: Use a web browser, SSH into things to work and watch videos. Apparently I'm not a power user, because all of these things are quite doable on an iPad, other bloggers' assertion that it's "not quite ready for power users" notwithstanding.
Turns out it's just a big iPad, and that's all it ever needed to be
1.2k USD and several weeks of waiting for the thing to ship later, I'm actually finding myself liking this thing a lot more than I thought I would. Yeah, it does sound expensive when I'm putting it like that, but to be honest that's the same price range as the Chromebook Pixel 2, or any other decent laptop at that.
[PHP] Just Say "No" To PHP's mysql(i) Interface
- 2015-10-07T00:03:13+01:00
I've recently "inherited" some PHP scripts for an otherwise pretty cool project. Now, I used to get paid for my PHP, back in the day. Doing medical trials. Back when PHP was considered the duck's nuts and you couldn't tell people otherwise. Even though some people really should've.
We spent a lot of time making sure our code was not exploitable. Naturally I assumed that everyone else was also taking those precautions. Because, you know, you always assume everyone else is doing the same stuff you're doing.
Anyway, this code made me realise that there's still people out there using PHP's mysql(i) interface. In rather... injectable ways. Basically creating SQL queries doing string concatenation. Which is bad, mmmkaaay? So I figure someone should reiterate this again: Do NOT, _EVER_ do that!
Just say NO. There's never a legit reason for this. Use PDO with prepared statements instead. The prepared statement part being really important here.
On The Spot: a game about tokenism
- 2015-09-07T09:14:33+01:00
I wanted to show you a game I made about a year ago at a hackathon. I was one out of two people there who weren't men, and that gave me the idea to make something that illustrates the concept of tokenism - that is, what happens if you have a group where a very small percentage belongs to an oppressed group. You've probably heard expressions like "the token black person", "the token woman" or things like that. That's tokenism. I ended up making a prototype of a game in Processing, and I kept the game abstract on purpose, because whether you're talking about sexism, racism, ableism or any other form of oppression, the tokenism mechanisms are pretty much the same. You're never really able to blend in. You stand out, whether you want to or not. Because there are so very few of your group, everyone else sees you as a representative of all $kind-of-people - and if you mess up, that'll reflect badly on your whole group, not only on you. You're constantly on the spot.
So, I made this game. This week I was at another hackathon and ended up fixing a few bugs in the prototype, throwing processing.js at it and putting it up on the web because what is even the point of having a game like this if people can't play it? It's here - feel free to play it, although it's still got a good few bugs in it. Anyway, as I was working on this and talking to people, I got at least one person to actually look up tokenism, so it's serving its purpose already. My work here is done ;)
The Case of the Incomplete HTTP Implementation
- 2015-06-08T21:00:00Z
People write the darndest things. Early last year I wrote a little piece on why I felt that FastCGI was rather pointless. As expected, responses were quite mixed - some people agreed, some people got their panties in a bunch. So far, so good. 'tis the Internet, after all.
So, what's interesting about this? Well, the stance of some of the comments, of course. Let's recap: in TFA, I was trying to point out that FastCGI was rather pointless as HTTP can fill the exact same position, only more legibly so. To put my money where my mouth was, I linked to a really short and concise HTTP implementation I threw together a while ago, and that I was using for some projects as backends behind nginx frontends. I honestly didn't expect any kind of reaction out of that, but hey, I'm notoriously wrong at judging which content will fly and which won't.
The Comments
Apart from nods of agreements there's strange ones cropping up all over the place every other month. Not entirely unexpectedly, some of the more elaborate ones were on Hacker News and The Something Awful Forums. The latter is a first, I believe. Yay! Maybe :)
Executable C++ Scripts
- 2015-03-01T19:30:00Z
Taking a Note from TCC's Book
For quite some I've been meaning to write/convert more scripts in C++. But doing so is rather annoying, since you have to compile the scripts. Well, OK maybe it wouldn't be all THAT annoying, but it is somewhat. Plus, if you want to give the script to others, you have to either compile it for them or make them use your makefile and possibly explain quite a bit along the way. It's a hassle.
Apparently I'm not the only person who would like this feature, seeing as TCC (the Tiny C Compiler) has this very feature by providing a -run command line flag. As a minor extension, the compiler even supports a shebang with this flag. But alas, it's not clang. And it definitely doesn't support C++11. Or C++. And clang++ will barf on a shebang line (I've tried).
So, there I was, about to file a feature request for such a flag in clang++, when it dawned on me: I can just wrap this up in a tiny shell script. I guess sometimes the easy solutions are those that you can do with the tools you already have. Behold, the following tiny "hello world" programme:
Perspective Projections: Beyond 3D
- 2014-12-23T13:50:00Z
I've been pushing this one off for years now. Not because I find it boring or hard to express, but rather because I spent most of my time actually fiddling with Topologic and its WebGL frontend, which uses the formulae you're about to see.
So what is this article about? Well, you may recall how your computer can do 3D graphics, even though your display is only 2D. The way that works is that your graphics card can do certain types of matrix and vector maths that turn points in 3D space into points in 2D space, and some additional bits that draw triangles. These operations, specifically, are called projections - and in the case of video games, they tend to be perspective projections in particular.
Going further, these projections are not actually limited to 3-space. All they do is remove a dimension from your source vectors. And what OpenGL in particular does is easily extended to projecting from n-d to (n-1)-d. This allows us to, kind of, "see" space as though it was four-dimensional. Or five-dimensional, etc. We simply need to chain these projections and find a way of coming up with projection matrices in arbitrary dimensions.
This article is exactly about that: coming up with the matrices needed for the projections, and how to apply them to vectors. It'll get kind of math-y, so... you've been warned. In case you need to refresh your memory, have a look at my previous article on Homogeneous Coordinates, and the one on Normal Vectors in Higher Dimensional Spaces. I'm also assuming you still remember your standard vector and matrix maths from linear algebra 101.
How to Track SSH Offenders with Google Analytics
- 2014-08-15T13:55:00Z
I've got a bit of a strange one today. A while ago I noticed that the rate of erroneous SSH connections had increased quite a bit since I got new servers. It's not that I'm worried about anyone actually getting in, thanks to some fairly steep hardening of those servers. However, I still wanted to figure out where all these... let's call them offenders are coming from and how frequent these attempted break-ins were. Fortunately we kinda have a thing for that, called Google Analytics.
Usually this is used to log web site hits and do some fancy reporting, but thanks to the Measurement Protocol you can actually track just about anything else you might want to track. And hits on your SSH server aren't all that different from hits on your website - might be good to correlate those two metrics as it were. To follow along at home, all you need is a Unix-y (e.g. Linux) server running OpenSSH and a Google Analytics account with Universal Analytics enabled - to be able to push data using the Measurement Protocol. Oh, and you'll need bash, sed and curl on your server. curl is kind of nonstandard on some systems, so check your package manager if you don't have it.
The whole trickery is condensed in this little shell script, which you should save with an apt name, possibly track-ssh-offenders:
#!/bin/bash
# track-ssh-offenders
# -------------------
# Upload SSH authentication errors to Google Analytics.
#
# See https://ef.gy/google-analytics-ssh for further details.
# Change this to the tracking ID you wish to use.
# WARNING: Make a second property with a new tracking ID for testing! You
# CANNOT remove data from Analytics, so make sure it works correctly
# BEFORE you put this in with your main website data.
TID=UA-NNNNNN-1
host=$(hostname)
dl=ssh:$host
tail -F /var/log/auth.log -n +0|
while read line; do echo "$(expr $(date "+%s") - $(date -d "$(echo $line|cut -f 1,2,3 -d ' ')" "+%s"))000 $line"; done|
sed -run \
-e "s/([0-9]+) .+ sshd.([0-9]+).: Invalid user (.+) from (.+)/v=1\&tid=$TID\&cid=\2\&t=event\&ec=Service\&ea=SSH\&el=InvalidUser\&uip=\4\&qt=\1\&p=$dl/p"\
-e "s/([0-9]+) .+ sshd.([0-9]+).: Received disconnect from (.+): .* Bye Bye .preauth./v=1\&tid=$TID\&cid=\2\&t=event\&ec=Service\&ea=SSH\&el=PreAuthDisconnect\&uip=\3\&qt=\1\&p=$dl/p"\
-e "s/([0-9]+) .+ sshd.([0-9]+).: Connection closed by (.+) .preauth./v=1\&tid=$TID\&cid=\2\&t=event\&ec=Service\&ea=SSH\&el=PreAuthClose\&uip=\3\&qt=\1\&p=$dl/p"\
-e "s/([0-9]+) .+ sshd.([0-9]+).: Received disconnect from (.+): .* Normal Shutdown.* .preauth./v=1\&tid=$TID\&cid=\2\&t=event\&ec=Service\&ea=SSH\&el=PreAuthNormalShutdown\&uip=\3\&qt=\1\&p=$dl/p"\
-e "s/([0-9]+) .+ sshd.([0-9]+).: .* \[(.+)\] .* POSSIBLE BREAK-IN ATTEMPT.+/v=1\&tid=$TID\&cid=\2\&t=event\&ec=Service\&ea=SSH\&el=PossibleBreakInAttempt\&uip=\3\&qt=\1\&p=$dl/p"\
| xargs -L 1 curl http://www.google-analytics.com/collect -o /dev/null -s -d
How to Harden SSH with Identities and Certificates
- 2014-08-07T16:13:00Z
Introduction
Whether you just need to feel in power or you actually use shells for day-to-day tasks, the Secure Shell [SSH] is probably the most important administrative access tool to your servers. It's also one of the least secured mission-critical services on most UNIX servers. Why? Because for some reason people are still using mere passwords to protect their root accounts. That's not quite as bad as using telnet, but not by too much. You might as well be using plain FTP to transfer data to your server... oh, wait, that's another article.
Using passwords for your remote servers exposes you to a whole class of unnecessary security risks, which are easily avoided by either switching to SSH identities or SSH certificates. This article will cover both, since they're conceptually very similar. We'll be working very closely with OpenSSH's configuration files, hopefully explaining some of the more intimidating options you might encounter.
Why Passwords are Bad for You
Finite-State Machines with recursive SQL
- 2013-09-23T16:28:00Z
Really, everything is a database problem.
One of the reasons I like PostgreSQL is that it supports recursive queries through common table expressions. This gives you the tools to properly use recursive data structures, like graphs, in SQL and perform queries on them. Neat, huh? You could, for example, represent a network of some kind in your database - modeling it as a graph is an obvious choice, and if you want to traverse it, solve a reachability problem on it or something, recursive SQL may be your tool of choice.
Now, regular expression matching is really a graph traversion problem - that is, if your regular expression actually is regular. Most of the modern regex libraries in various programming languages provide features like backreferences that exceed the capability of regular languages, but we're working with the definition of regular expressions in formal language theory, which is, given a finite alphabet:
- The empty set is a regular expression.
- The empty string and all characters of the alphabet are regular expressions.
- if a is a regular expression, so is its Kleene stara*
- if a and b are regular expressions, their concatenation ab and their alternation (a|b) are regular expressions
One more thing about finite-state machines: they come in two flavours, deterministic and nondeterministic. With deterministic FSM, there must be exactly one starting state (though there may be multiple accepting states), and every state must have exactly one transition for every character. Nondeterministic FSM can have multiple starting states, multiple (or no) transitions for a character from the same state, and "epsilon transitions" - that is, you can go from one state to another without reading a character. (If you can have multiple transitions per character, the latter is just syntactic sugar, but it comes in handy at times) With nondeterministic FSM, while reading a string you might have to guess which state to start in, or to which state to transition, at some point, but as long as there is a set of guesses that leads you to an accepting state, the string is an element of the language.
sr4.sql: a SQLite based Shadowrun character generator
- 2013-07-10T13:44:00Z
So I'm still working on that thing...
With the PDF release of 5th edition just around the corner, you'd think it's obsolete, but I'm looking forward to checking out Fifth Edition and translating it to SQL, too... well, the latter, not so much. It's pretty clear the rules are designed to be interpreted by humans, not computers, so the SQL is getting more complex and bloated than I'd like. I heard they streamlined the rules quite a bit in Fifth Edition though, so this bodes well for sr5.sql.
Anyway I've been working on the Fourth Edition version, as time permitted, and I've got a couple of new features for you:
- BP cost for skill specialisations
- BP costs for spells and complex forms
- BP costs for registered sprites at character generation
- Skill groups
- Essence cost computation
- Reduction of maximum Magic and Resonance attribute by Essence loss
- native languages for characters
So how do I use this?
SQL: Game of Life
- 2013-03-14T22:19:00Z
EVERYTHING is a Database Problem
Right, so it's π-Day (the lame one, anyway), we just had πzza and are watching Harry πtter and the Philosopher's Stone. Everything gets better with large quantities of π so the obvious thing to do is, of course, to follow up on yesterday's XSLT Game of Life implementation.
As mentioned yesterday, the XSLT version is extremely slow. It could probably be sped up a bit. Maybe by including dead cells explicitly and working with standard XPath functions instead of looking up attributes. Maybe some other way.
But the far geekier thing to do is, of course, to write another version in a real language: SQL. Cue mad laughter.
(SLC) SSD Write Endurance Considered... Sufficient
- 2013-02-18T21:42:00Z
The Numbers
Ever wonder how long your shiny new SLC SSD will last? It's funny how bad people are at estimating just how long "100,000 writes" are going to take when spread over a device that spans several thousand of those blocks over several gigabytes of memory. Not to mention even newer drives that can take even more of a beating before going down. So yeah, let's crunch some numbers for that: