tag:blogger.com,1999:blog-60694392993052247302024-03-08T09:04:31.157+01:00zoom.nuThings that zoom - the ehrmygerd! of coding.Anonymoushttp://www.blogger.com/profile/01044897049984478240noreply@blogger.comBlogger65125tag:blogger.com,1999:blog-6069439299305224730.post-14816833561799925902017-09-13T19:35:00.000+02:002017-11-05T19:13:56.549+01:00pid 1 rageOk so most of the linux world have gone bonkers and converted to systemd (except slackware - the original rude boy). <br />
I tried to like systemd (well, not really) but it is impossible. It is a gigantic tire fire of badness, reminds me of the clusterf*** that is pulse-audio (hint, hint, know what I mean).<br />
<br />
So I thought to myself; if I must have a gigantic monolithic opaque pid 1 process (flying in the face of all that is unix) then by jove it must be something else. Enter RancherOS as one of the dists that do not use systemd. It has a glorious solution, it runs docker as pid 1 and most of the OS services are actually docker containers. It is so fantastically insane that it is really brilliant.<br />
<br />
So I'm in the process of converting "the cluster", i.e. my 3 raspberry pi machines to rancher OS. It is not straight forward but this is the process I use. So caveat # 1. My workstation is a windows machine for the reason that I must play starcraft 2 and that does not run on linux. So I got a cheap USB connected SD card reader to write my SD cards. I use Rufus as the utility of choice to write images to the SD card. It has worked really well.<br />
<br />
However, the RancherOS on the pi does not auto-expand the root partition. There is also the little snag of having to write the initial cloud-config.yml file with the SSH-keys so I can get at the pi after it has booted. In dire need of a linux machine that can hack the SD-card I tried virtualization. Luckily VirtualBox can expose USB devices to a guest. So I booted up my virtual linux, mounted the SD card reader through USB and by magic it appears. So I can run gparted on the sd card from the virtual linux machine and resize the rootfs. After that it is just a matter of creating a /var/lib/rancher/conf/cloud-config.yml with all the SSH keys on the SD card. Plug it into the pi and boot.<br />
<br />
And it freaking works. After a while the pi snagged an IP from the dhcp-server and I could SSH into it. Now to do this a couple of more times and then all my servers are on RancherOS and systemd is just a bad memory on the bare metal servers. It will still haunt many of the docker images but maybe I can force supervisord to be pid 1 :-)Anonymoushttp://www.blogger.com/profile/02908752593907979820noreply@blogger.com0tag:blogger.com,1999:blog-6069439299305224730.post-51814072461560556742013-11-12T18:47:00.002+01:002013-11-12T18:47:51.611+01:00gradleI've just quickly tried out gradle as a build system. I have three observations so far.<br />
<br />
1) Netbeans have a pretty good integration<br />
<br />
2) Repository configuration is <i>in</i> the build script. That is in my book a pretty big f*ckup, maven got rid of that with maven 2. I know I can write my own build script but a new build system where there is no convention for configuration of things that are specific to the build machine and not the source code... The irony of using Ivy and obscuring everything that is good with Ivy.<br />
<br />
3) The run task... It is currently not possibly (said the forum 1 year ago and that's all I could find on this subject) to pass command line arguments to the application. You offer a "run" task but it is unusable.<br />
<br />
If I have to write those things myself I might as well stick to bare Ant + Ivy.Anonymoushttp://www.blogger.com/profile/02908752593907979820noreply@blogger.com1tag:blogger.com,1999:blog-6069439299305224730.post-10923859725940803682013-10-30T21:05:00.001+01:002013-10-30T21:05:57.640+01:00Designing OO APIsI'm trying to find a good pattern to designing an API/Framework that let's clients subclass to inherit convenient behavior.<br />
These are my requirements:<br />
<br />
The framework owns the collection of "Things". Like this:<br />
<span style="font-family: Courier New, Courier, monospace;">class Framework {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> void addAThing(Thing t)...</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<br />
A "Thing" is probably a composition of other stuffs. So it must have a Stuff getStuff() method. The framework uses a Things Stuff sometimes.<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">interface Thing {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Stuff getStuff() ;</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<br />
So one implementation of Thing is the simple version that gets stuff from the outside.<br />
<span style="font-family: Courier New, Courier, monospace;">class Thing1 implements Thing {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> private final Stuff myStuff ;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> public Thing(final Stuff myStuff) { this.myStuff = checkNotNull(myStuff) ; }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> public Stuff getStuff() { return this.myStuff }</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<br />
The problem I have is that there can be many different Things, and the actual creation of Stuff is something that I want the subclasses to be able to implement. So I'll help them along with this:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">abstract class ParentThing implements Thing {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> private final Stuff myStuff ;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> public ParentThing() { </span><br />
<span style="font-family: Courier New, Courier, monospace;"> this.myStuff = checkNotNull(createStuff()); </span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> public Stuff getStuff() { return this.myStuff }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> protected abstract Stuff createStuff() ;</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<div>
<br /></div>
<div>
The problem with that is that I'm relying on subclasses to be very nice about the implementation of createStuff() since the method is called in the constructor. It shouldn't for example register "this" as a callback to some other thread in createStuff since "this"-instance might not have left the constructor when the callback occurs.</div>
<div>
<br /></div>
<div>
So maybe this then:</div>
<div>
<br /></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">interface Thing {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> void initialize() ;</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> Stuff getStuff() ;</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">}</span></div>
<div>
<br /></div>
<div>
and then I'll fix it so that framework calls initialize before it uses the Thing, like so</div>
<div>
</div>
<div>
<span style="font-family: Courier New, Courier, monospace;">class Framework {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> void addAThing(Thing t){ t.initialize(); ... }</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<span style="font-family: Times, Times New Roman, serif;"><br /></span>
<span style="font-family: Times, Times New Roman, serif;">But then I can't use final in the ParentThing class anymore, leaving me with another state of Things, "created but not initialized yet". </span><br />
<span style="font-family: Times, Times New Roman, serif;"><br /></span>
<span style="font-family: Times, Times New Roman, serif;">So maybe I actually should have </span><br />
<span style="font-family: Times, Times New Roman, serif;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">interface ThingBuilder {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Thing createAThing() ;</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<span style="font-family: Times, Times New Roman, serif;"><br /></span>
<span style="font-family: Times, Times New Roman, serif;">and then</span><br />
<span style="font-family: Times, Times New Roman, serif;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">class Framework {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> void addAThing(ThingBuilder t)...</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Times, Times New Roman, serif;">Might not be so bad with closures, but without closures/lambda it gets a bit messy on the client side.</span><br />
<span style="font-family: Times, Times New Roman, serif;"><br /></span>
<span style="font-family: Times, Times New Roman, serif;">Ideas are welcome. How to design an OO API? </span></div>
Anonymoushttp://www.blogger.com/profile/02908752593907979820noreply@blogger.com5tag:blogger.com,1999:blog-6069439299305224730.post-80746460878347137122013-04-04T09:04:00.000+02:002013-04-04T09:04:48.103+02:00Debian Squeeze, chrome and firefox woesOk so I am guilty of running stable software, I use Debian Squeeze. It is rather conservative in updating versions of things, doing a lot of testing, and for that reason everything "just works", even if I can't run the latest games and so on. <br />
<br />
Google chrome has another strategy, they update their version of the browser all the time. Recently I got updated to Version 26.0.1410.43. <br />
Now chrome says that I am running on an unsupported version of my operating system (Debian 6). It has a link to a page that states what versions are supported. That page says that Debian 6 is supported.<br />
Yeah<br />
Testing<br />
<br />
So given the recent public relations stunts that Google has pulled (like ditching google reader - anyone remember orkut, still running....) I wasn't to keen on cutting them any slack, I'll just go back to firefox despite the abbysmal WebGL performance. <br />
<br />
Firefox isn't included in Debian because of ... principles. Firefox will not license their logo in a Debian friendly manner because it's like trademark and their property and stuffs. Debian will not include software that have restricted licenses and trademarks and stuffs.<br />
Principles, I can actually dig that. I'll just download it myself.<br />
<br />
Happy like a little child finding a fresh puddle of mud on a spring day I headed over to mozilla.org to get me some firefox goodness.<br />
Downloaded the version 19, unpack, click binary, wait for things to break (they usually do). But it ran. That's pretty nice of them.<br />
Only it looked really crappy but I didn't have time to look into what that was about.<br />
Until now, I'm home with a cold and between sneezing and other unpleasantness I tried to get Java-applets running. Only to find out that the reason firefox looks like, pardon the language, shite, and that the java plugin doesn't even get recognized is because it is a 32-bit build.<br />
This is the year of our grace 2013.<br />
I went 64-bit before my niece was born and she is in school now. I mean, come on!<br />
The stupid, stupid, stupid, download page automatically detects I'm on linux and starts a download without asking and it picks the wrong arch. Ok, accidents happen, so where is the download link then? There freaking isn't any!<br />
The solution was to google my way through countless questions until finding <a href="ftp://ftp.mozilla.org/pub/firefox/releases/" target="_blank">ftp://ftp.mozilla.org/pub/firefox/releases/</a><br />
Klicking my way to the 64 bit version 19 beta build. Unpack that, then let firefox auto-update to version 20 and THEN the plugins work.<br />
It looks good now also because it doesn't try to load GTK from the <i>wrong </i>32-bit libraries that it obviously can't find since this is, as said, a 64-bit clean machine.<br />
Yeah<br />
TestingAnonymoushttp://www.blogger.com/profile/02908752593907979820noreply@blogger.com8tag:blogger.com,1999:blog-6069439299305224730.post-86770227357184968852013-01-30T22:38:00.000+01:002013-05-14T12:14:34.648+02:00Minimum viable web framework<p>In some projects I've been working on, there's a small part of common code and thinking that one could call a minimum viable web framework (well, technically, it's a library and not a framework, but I digress). There's absolutely nothing new in it, and if you've ever done even one Servlet app, you know it already. I see that as an advantage.
<p>My minimum viable web framework consists of 8 lines of code and one principle. The code (quoted below) is a static method (on some utility class) that takes a path to which to forward a request, the request and response objects, and then a vararg list of alternating names and values to put as request attributes. The principle is: if something produces any HTML, then that's all that thing does (inverted: if you don't produce HTML, <i>you don't produce HTML</i>). Anything that produces HTML (in my cases, JSPs) only reads values prepared for it from the request attributes (and if you have something that produces some output that isn't HTML, then it's by definition a special case and is exempt from this rule). It's OK to have lists and other simple data structures as request attributes, and it's OK to use simple looping constructs (e.g. a JSTL core <code>forEach</code>) in the rendering, but you can't do things like calling methods from there.
<pre style="background:#fff;color:#000"><span style="color:#00f;font-weight:700">public</span> <span style="color:#00f;font-weight:700">static</span> <span style="color:#00f;font-weight:700">void</span> forward(<span style="color:#00f;font-weight:700">String</span> path, <span style="color:#00f;font-weight:700">HttpServletRequest</span> request, <span style="color:#00f;font-weight:700">HttpServletResponse</span> response, <span style="color:#00f;font-weight:700">Object</span><span style="color:#00f;font-weight:700">.</span><span style="color:#00f;font-weight:700">.</span>. attributes)
throws <span style="color:#00f;font-weight:700">ServletException</span>, <span style="color:#00f;font-weight:700">IOException</span>
{
<span style="color:#00f;font-weight:700">for</span>(<span style="color:#00f;font-weight:700">int</span> i<span style="color:#00f;font-weight:700">=</span><span style="color:#0000cd">0</span>; i<span style="color:#00f;font-weight:700"><</span>attributes<span style="color:#00f;font-weight:700">.</span>length; i<span style="color:#00f;font-weight:700">+=</span><span style="color:#0000cd">2</span>) {
request<span style="color:#00f;font-weight:700">.</span>setAttribute((<span style="color:#00f;font-weight:700">String</span>)attributes[i], attributes[i<span style="color:#00f;font-weight:700">+</span><span style="color:#0000cd">1</span>]);
}
request<span style="color:#00f;font-weight:700">.</span>getRequestDispatcher(path)<span style="color:#00f;font-weight:700">.</span>forward(request, response);
}
</pre>
<p>Together, the code and the principle give a simple way of keeping application code and page formatting separate, and I've been quite happy with it in my projects. There's a lot of things missing that you might expect a framework to provide (e.g. input validation or data access). I see those as nice to have and that the only <i>essential</i> part of a web framework is facilitating the separation of logic and rendering.
<p>So, what do you think - is this web framework both minimal and viable? If it's not minimal - how can we minimise it? If it's not viable, then what's missing? And last but not least: does it zoom?Anonymoushttp://www.blogger.com/profile/01044897049984478240noreply@blogger.com5tag:blogger.com,1999:blog-6069439299305224730.post-50872848269116182013-01-22T21:39:00.001+01:002013-01-22T21:39:57.412+01:00OpenTSDBAt my daytime gig I have come into contact with two applications that really zooms.<br />
<br />
One is <a href="http://www.splunk.com/" target="_blank">Splunk</a> and I'm sure the developers of that application drink awesome-sauce for breakfast. It is so awesome I won't even write more about it here because I just don't know what words to use. Suffice it to say that it is a log analysis tool that actually beats 'find, xargs, awk, grep' and all of those.<br />
<br />
The other application I've came into contact with is <a href="http://opentsdb.net/index.html" target="_blank">OpenTSDB</a>. <a href="http://opentsdb.net/index.html" target="_blank">OpenTSDB</a> is a time series database, that means you put metrics into it that you want to plot over time. <a href="http://opentsdb.net/index.html" target="_blank">OpenTSDB</a> uses <a href="http://hbase.apache.org/" target="_blank">hbase</a> as a database. That is the Apache hadoop database.<br />
OpenTSDB has a web front end for plotting the data points that uses <a href="http://www.gnuplot.info/" target="_blank">gnuplot</a>. It is rather simplistic as a web front but it is mostly bug free and despite a few little quirks it "just works" exactly like you want it to. It does one thing (send a query to OpenTSDB and plot a PNG-image) and it does it good.<br />
<br />
We use OpenTSDB to monitor our servers and applications. We put numbers into it like, how much heap has the server, what is the number of busy threads, how many messages was put on the message queue, what was the response time for each web service call.<br />
It is extremely helpful in monitoring and post-mortem analysis of application behaviour. I mean like, <i>really</i> useful. I can correlate exactly the number of packets the load balancer sends to a certain host at 5s intervals with the number of busy threads, the heap size, the cpu load and a load of application specific metrics extracted from the JVMs using JMX.<br />
The JMX-collector, developed in house, is written by some pretty clever guys to be fast but you can also write one, it isn't that hard. Remember to do everything async. and use caching and you're good to go.<br />
<br />
I just can not stress enough how incredibly useful OpenTSDB is for not only monitoring what happens now but what happened that sunday when a couple of servers want haywire and didn't respond. Given the precision of the correlation it is very easy to find the relevant log entries from the time stamps.<br />
<br />
To put in to perspective how good opentsdb & hbase is. We dump I would say more than 1 metric/second (usually we poll a specific metric each 5 sec or so - depending on what it is).<br />
From each server, and we have > 35 servers.<br />
To a single opentsdb + hbase server.<br />
<br />
And it just freaking works<i>.</i> I can actually see exactly what the heap size was for host X on christmas eve.<br />
<br />
Just tonight I installed OpenTSDB on my local machine/server, I don't <i>need</i> it at home but just to pay tribute to it.Anonymoushttp://www.blogger.com/profile/02908752593907979820noreply@blogger.com0tag:blogger.com,1999:blog-6069439299305224730.post-81553145673334723742012-11-21T22:47:00.000+01:002012-11-21T22:47:23.630+01:00Tracking down memory corruption by mprotecting your ADTsIn C, it's customary to design your code around Abstract Data Types, that is, modules that consist of a header file that declares the external interface of the module (consisting of an opaque struct and a set of functions operating on that struct), and an implementation file (which has the full declaration of the structure, the definitions of the functions in the header and any helper functions). The header would be something like this:
<code><pre>
#ifndef INCLUDE_GUARD_STACK_H
#define INCLUDE_GUARD_STACK_H
#include <stddef.h>
#include <stdbool.h>
struct stack;
struct stack* stack_create(size_t size);
bool stack_push(struct stack* stack, int i);
bool stack_pop(struct stack* stack, int* i);
void stack_destroy(struct stack* stack);
#endif
</pre></code><p>The implementation, then, is:
<code><pre>
#include "stack.h"
#include <stdlib.h>
struct stack
{
int* elements;
size_t used;
size_t allocated;
};
struct stack* stack_create(size_t size)
{
struct stack* stack = malloc(sizeof(struct stack));
stack->elements = malloc(size * sizeof(int));
stack->used = 0;
stack->allocated = size;
return stack;
}
bool stack_push(struct stack* stack, int i)
{
if (stack->used == stack->allocated) {
goto error_full;
}
stack->elements[stack->used++] = i;
return true;
error_full:
return false;
}
</pre></code><p>I'll leave the implementation of the rest of the functions to your imagination. Since only the forward declaration of the struct is in the header, no code outside the implementation can access the members of the struct.<p>Now, assume we have a memory corruption fault somewhere in the rest of the program which when triggered corrupts the <code>elements</code> pointer but doesn't have any other effects. Our program them seems to be working fine until some later time when it suddenly crashes due to an invalid memory access in <code>stack_push</code>. We'd really like to get the program to abort at the point of the original corruption of the <code>elements</code> pointer, but how can we do that?
<p>One way of solving that is to use the fact that since the structure is opaque, there is no way that any code outside our implementation file has any legitimate use of touching any of the memory to which <code>struct stack*</code> points. Since no other code has any business accessing that memory, then maybe we can have the OS help us preventing it from doing that? Enter <a href="http://pubs.opengroup.org/onlinepubs/7908799/xsh/mprotect.html">mprotect</a>.
<p>The <code>mprotect</code> function lets us control what types of access should be permitted to a region of memory. If we access the memory in any other way, the OS is free (and in some cases even required) to abort our program at the spot. If we keep the memory inaccessible at all times except for when we use it inside our implementation functions, then chances are we can catch the memory corruption as it happens. The <code>mprotect</code> man page does say that the memory it protects has to be page aligned, though. How do we do that? Via <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/posix_memalign.html">posix_memalign</a> and <a href="http://pubs.opengroup.org/onlinepubs/007908775/xsh/getpagesize.html">getpagesize</a>, like so:
<code><pre>
#include "stack.h"
#include <stdlib.h>
#include <unistd.h>
#include <sys/mman.h>
struct stack* stack_create(size_t size)
{
struct stack* stack;
posix_memalign((void**)&stack, getpagesize(), sizeof(struct stack));
posix_memalign((void**)&stack->elements, getpagesize(), size * sizeof(int));
stack->used = 0;
stack->allocated = size;
protect(stack);
return stack;
}
</pre></code><p>Now we just have to implement and use the <code>protect</code> function mentioned above and its inverse, <code>unprotect</code>:
<code><pre>
static void protect(struct stack* stack)
{
mprotect(stack->elements, stack->allocated * sizeof(int), PROT_NONE);
mprotect(stack, sizeof(*stack), PROT_NONE);
}
static void unprotect(struct stack* stack)
{
/* Unprotect in the reverse order, or we crash and burn trying to read stack->elements */
mprotect(stack, sizeof(*stack), PROT_READ|PROT_WRITE);
mprotect(stack->elements, stack->allocated * sizeof(int), PROT_READ|PROT_WRITE);
}
bool stack_push(struct stack* stack, int i)
{
unprotect(stack);
if (stack->used == stack->allocated) {
goto error_full;
}
stack->elements[stack->used++] = i;
protect(stack);
return true;
error_full:
protect(stack);
return false;
}
</pre></code>
<p>Since this ends up modifying bits in the MMU, it may not be suitable to have enabled on performance critical code, so an <code>#ifdef NDEBUG</code> switch that selects an empty implementation of <code>protect</code> and <code>unprotect</code> for non-debug builds could be advisable.<p>So does protecting your ADTs via <code>mprotect</code> zoom? Well, it does come in handy at times, and there's not much disadvantage to using it, so my verdict is: Zooms!Anonymoushttp://www.blogger.com/profile/01044897049984478240noreply@blogger.com0tag:blogger.com,1999:blog-6069439299305224730.post-63586436250048699162012-10-28T13:07:00.000+01:002012-10-28T14:18:25.603+01:00Compiling with LLVM<a href="http://llvm.org">LLVM</a> is, among many other things, a reusable compiler backend implemented as a library. In this post, I'll show how the <a href="http://www.llvmpy.org/">llvmpy</a> Python bindings can be used to write an optimising cross-compiler in a few lines of Python.
As a starting point, let's re-use the old parser and interpreter from the <a href="http://blog.zoom.nu/2010/05/parser-combinators-in-python-part-3.html">Python Parsing Combinators</a> series. Since the grammar only has operators and numbers, we'll need to extend it a bit - there's not much point in spending time compiling something that isn't going to take any input. Let's add named values like so:
<pre style='color:#000000;background:#ffffff;'>identifier <span style='color:#808030; '>=</span> RegEx<span style='color:#808030; '>(</span><span style='color:#0000e6; '>"[a-zA-Z_][a-zA-Z0-9_]*"</span><span style='color:#808030; '>)</span><span style='color:#808030; '>.</span><span style='color:#e34adc; '>apply</span><span style='color:#808030; '>(</span>Identifier<span style='color:#808030; '>)</span>
simple <span style='color:#808030; '>=</span> number <span style='color:#808030; '>|</span> paren <span style='color:#808030; '>|</span> identifier
</pre>
Next, we'll need an implementation for this new node in our Abstract Syntax Tree:
<pre style='color:#000000;background:#ffffff;'><span style='color:#800000; font-weight:bold; '>class</span> Identifier<span style='color:#808030; '>(</span>object<span style='color:#808030; '>)</span><span style='color:#808030; '>:</span>
<span style='color:#800000; font-weight:bold; '>def</span> <span style='color:#e34adc; '>__init__</span><span style='color:#808030; '>(</span>self<span style='color:#808030; '>,</span> s<span style='color:#808030; '>)</span><span style='color:#808030; '>:</span>
self<span style='color:#808030; '>.</span>name<span style='color:#808030; '>=</span>s
<span style='color:#800000; font-weight:bold; '>def</span> <span style='color:#e34adc; '>eval</span><span style='color:#808030; '>(</span>self<span style='color:#808030; '>,</span> env<span style='color:#808030; '>)</span><span style='color:#808030; '>:</span>
<span style='color:#800000; font-weight:bold; '>return</span> env<span style='color:#808030; '>[</span>self<span style='color:#808030; '>.</span>name<span style='color:#808030; '>]</span>
<span style='color:#800000; font-weight:bold; '>def</span> <span style='color:#e34adc; '>__repr__</span><span style='color:#808030; '>(</span>self<span style='color:#808030; '>)</span><span style='color:#808030; '>:</span>
<span style='color:#800000; font-weight:bold; '>return</span> <span style='color:#e34adc; '>repr</span><span style='color:#808030; '>(</span>self<span style='color:#808030; '>.</span>name<span style='color:#808030; '>)</span>
</pre>
As you can see if you compare this <code>eval</code> to the old ones, we now take a dictionary as argument to eval, and that it maps names to numbers. The old <code>eval</code> implementations should be extended to pass along this environment, but <code>Identifier</code> is the only one that has a use for it.
<p>Interpreting is now done thus:
<pre style='color:#000000;background:#ffffff;'>expr<span style='color:#808030; '>.</span>parse<span style='color:#808030; '>(</span><span style='color:#0000e6; '>"a+1"</span><span style='color:#808030; '>,</span> <span style='color:#008c00; '>0</span><span style='color:#808030; '>)</span><span style='color:#808030; '>.</span>next<span style='color:#808030; '>(</span><span style='color:#808030; '>)</span><span style='color:#808030; '>[</span><span style='color:#008c00; '>0</span><span style='color:#808030; '>]</span><span style='color:#808030; '>.</span><span style='color:#e34adc; '>eval</span><span style='color:#808030; '>(</span><span style='color:#800080; '>{</span><span style='color:#0000e6; '>"a"</span><span style='color:#808030; '>:</span><span style='color:#008c00; '>4</span><span style='color:#800080; '>}</span><span style='color:#808030; '>)</span>
</pre>
Now, the topic of this post is compilation, not interpretation. Let's start by importing the llvmpy bindings:
<pre style='color:#000000;background:#ffffff;'><span style='color:#800000; font-weight:bold; '>from</span> llvm <span style='color:#800000; font-weight:bold; '>import</span> <span style='color:#808030; '>*</span>
<span style='color:#800000; font-weight:bold; '>from</span> llvm<span style='color:#808030; '>.</span>core <span style='color:#800000; font-weight:bold; '>import</span> <span style='color:#808030; '>*</span>
<span style='color:#800000; font-weight:bold; '>from</span> llvm<span style='color:#808030; '>.</span>ee <span style='color:#800000; font-weight:bold; '>import</span> <span style='color:#808030; '>*</span>
</pre>
In LLVM, you have modules (roughly object files) that contain functions, which take arguments and contain a basic block. Basic blocks contain instructions, which can in turn contain things like basic blocks or call functions. Creating a module is simple:
<pre style='color:#000000;background:#ffffff;'>m <span style='color:#808030; '>=</span> Module<span style='color:#808030; '>.</span>new<span style='color:#808030; '>(</span><span style='color:#0000e6; '>"m"</span><span style='color:#808030; '>)</span>
</pre>
The function we want to add to the module should correspond to the expression we're compiling. It should return an int, and its arguments should be the identifiers that are used in the expression. We can find the identifiers by adding a method to the AST nodes that return the union of all identifiers in the sub-tree (it's trivial, so I'll spare you the implementation). Let's sort them ASCIIbetically and use that as our arguments. We're going to need a mapping from identifier name to function arguments, so let's build that up as well.
<pre style='color:#000000;background:#ffffff;'>identifiers <span style='color:#808030; '>=</span> sorted<span style='color:#808030; '>(</span><span style='color:#e34adc; '>list</span><span style='color:#808030; '>(</span>ast<span style='color:#808030; '>.</span>identifiers<span style='color:#808030; '>(</span><span style='color:#808030; '>)</span><span style='color:#808030; '>)</span><span style='color:#808030; '>)</span>
ty_int <span style='color:#808030; '>=</span> Type<span style='color:#808030; '>.</span><span style='color:#e34adc; '>int</span><span style='color:#808030; '>(</span><span style='color:#808030; '>)</span>
ty_func <span style='color:#808030; '>=</span> Type<span style='color:#808030; '>.</span>function<span style='color:#808030; '>(</span>ty_int<span style='color:#808030; '>,</span> <span style='color:#808030; '>[</span>ty_int<span style='color:#808030; '>]</span> <span style='color:#808030; '>*</span> <span style='color:#e34adc; '>len</span><span style='color:#808030; '>(</span>identifiers<span style='color:#808030; '>)</span><span style='color:#808030; '>)</span>
f <span style='color:#808030; '>=</span> m<span style='color:#808030; '>.</span>add_function<span style='color:#808030; '>(</span>ty_func<span style='color:#808030; '>,</span> <span style='color:#0000e6; '>"f"</span><span style='color:#808030; '>)</span>
args <span style='color:#808030; '>=</span> <span style='color:#800080; '>{</span><span style='color:#800080; '>}</span>
<span style='color:#800000; font-weight:bold; '>for</span> i <span style='color:#800000; font-weight:bold; '>in</span> <span style='color:#e34adc; '>range</span><span style='color:#808030; '>(</span><span style='color:#e34adc; '>len</span><span style='color:#808030; '>(</span>identifiers<span style='color:#808030; '>)</span><span style='color:#808030; '>)</span><span style='color:#808030; '>:</span>
f<span style='color:#808030; '>.</span>args<span style='color:#808030; '>[</span>i<span style='color:#808030; '>]</span><span style='color:#808030; '>.</span>name <span style='color:#808030; '>=</span> identifiers<span style='color:#808030; '>[</span>i<span style='color:#808030; '>]</span>
args<span style='color:#808030; '>[</span>identifiers<span style='color:#808030; '>[</span>i<span style='color:#808030; '>]</span><span style='color:#808030; '>]</span> <span style='color:#808030; '>=</span> f<span style='color:#808030; '>.</span>args<span style='color:#808030; '>[</span>i<span style='color:#808030; '>]</span>
</pre>
Now, function bodies consist of a basic block (i.e. a series of statements). You can add statements to a basic block using a builder.
<pre style='color:#000000;background:#ffffff;'>bb <span style='color:#808030; '>=</span> f<span style='color:#808030; '>.</span>append_basic_block<span style='color:#808030; '>(</span><span style='color:#0000e6; '>"entry"</span><span style='color:#808030; '>)</span>
builder <span style='color:#808030; '>=</span> Builder<span style='color:#808030; '>.</span>new<span style='color:#808030; '>(</span>bb<span style='color:#808030; '>)</span>
</pre>
Now we're ready for the interesting parts. For each AST node type, we add a method for compiling to LLVM opcodes. Since LLVM uses Static Single Assignment, each opcode results in the assignment of a variable, and this variable is then never modified. Every method on the builder that appends an instruction to the basic block returns the resulting value, which makes it natural to do the same for our <code>compile</code> method: we return the value that will hold the results of the computation. As arguments, let's take the builder that we use for appending the instructions, the type to use for our numbers, and the argument mapping we built up. Identifiers are simple: we just return the argument that corresponds to the identifier:
<pre style='color:#000000;background:#ffffff;'><span style='color:#800000; font-weight:bold; '>class</span> Identifier<span style='color:#808030; '>(</span>object<span style='color:#808030; '>)</span><span style='color:#808030; '>:</span>
<span style='color:#696969; '>#...</span>
<span style='color:#800000; font-weight:bold; '>def</span> <span style='color:#e34adc; '>compile</span><span style='color:#808030; '>(</span>self<span style='color:#808030; '>,</span> builder<span style='color:#808030; '>,</span> ty_int<span style='color:#808030; '>,</span> args<span style='color:#808030; '>)</span><span style='color:#808030; '>:</span>
<span style='color:#800000; font-weight:bold; '>return</span> args<span style='color:#808030; '>[</span>self<span style='color:#808030; '>.</span>name<span style='color:#808030; '>]</span>
</pre>
Numbers are also easy - we just return a constant value.
<pre style='color:#000000;background:#ffffff;'><span style='color:#800000; font-weight:bold; '>class</span> Number<span style='color:#808030; '>(</span>object<span style='color:#808030; '>)</span><span style='color:#808030; '>:</span>
<span style='color:#696969; '>#...</span>
<span style='color:#800000; font-weight:bold; '>def</span> <span style='color:#e34adc; '>compile</span><span style='color:#808030; '>(</span>self<span style='color:#808030; '>,</span> builder<span style='color:#808030; '>,</span> ty_int<span style='color:#808030; '>,</span> args<span style='color:#808030; '>)</span><span style='color:#808030; '>:</span>
<span style='color:#800000; font-weight:bold; '>return</span> Constant<span style='color:#808030; '>.</span><span style='color:#e34adc; '>int</span><span style='color:#808030; '>(</span>ty_int<span style='color:#808030; '>,</span> self<span style='color:#808030; '>.</span>n<span style='color:#808030; '>)</span>
</pre>
Now, how do we do our two remaining AST node types, addition and multiplication? We can do this by first emitting the code for the left hand side of the operation, remembering the value that will hold the results for that. Then we do the same for the right-hand side, and finally, we emit a multiplication instruction that takes the resulting values for the left hand side and the right hand side as its operands.
<pre style='color:#000000;background:#ffffff;'><span style='color:#800000; font-weight:bold; '>class</span> Mul<span style='color:#808030; '>(</span>BinOp<span style='color:#808030; '>)</span><span style='color:#808030; '>:</span>
<span style='color:#696969; '>#...</span>
<span style='color:#800000; font-weight:bold; '>def</span> <span style='color:#e34adc; '>compile</span><span style='color:#808030; '>(</span>self<span style='color:#808030; '>,</span> builder<span style='color:#808030; '>,</span> ty_int<span style='color:#808030; '>,</span> args<span style='color:#808030; '>)</span><span style='color:#808030; '>:</span>
a <span style='color:#808030; '>=</span> self<span style='color:#808030; '>.</span>a<span style='color:#808030; '>.</span><span style='color:#e34adc; '>compile</span><span style='color:#808030; '>(</span>builder<span style='color:#808030; '>,</span> ty_int<span style='color:#808030; '>,</span> args<span style='color:#808030; '>)</span>
b <span style='color:#808030; '>=</span> self<span style='color:#808030; '>.</span>b<span style='color:#808030; '>.</span><span style='color:#e34adc; '>compile</span><span style='color:#808030; '>(</span>builder<span style='color:#808030; '>,</span> ty_int<span style='color:#808030; '>,</span> args<span style='color:#808030; '>)</span>
<span style='color:#800000; font-weight:bold; '>return</span> builder<span style='color:#808030; '>.</span>mul<span style='color:#808030; '>(</span>a<span style='color:#808030; '>,</span> b<span style='color:#808030; '>)</span>
</pre>
Addition is of course the same, except for using <code>builder.add</code>.
We can now let the basic block of our function return the value that will hold the result of evaluating our entire AST by doing
<pre style='color:#000000;background:#ffffff;'>builder<span style='color:#808030; '>.</span>ret<span style='color:#808030; '>(</span>ast<span style='color:#808030; '>.</span><span style='color:#e34adc; '>compile</span><span style='color:#808030; '>(</span>builder<span style='color:#808030; '>,</span> ty_int<span style='color:#808030; '>,</span> args<span style='color:#808030; '>)</span><span style='color:#808030; '>)</span>
</pre>
We can now generate machine code from this, and run it directly:
<pre style='color:#000000;background:#ffffff;'>ee <span style='color:#808030; '>=</span> ExecutionEngine<span style='color:#808030; '>.</span>new<span style='color:#808030; '>(</span>m<span style='color:#808030; '>)</span>
env<span style='color:#808030; '>=</span><span style='color:#800080; '>{</span>
<span style='color:#0000e6; '>"a"</span><span style='color:#808030; '>:</span> GenericValue<span style='color:#808030; '>.</span><span style='color:#e34adc; '>int</span><span style='color:#808030; '>(</span>ty_int<span style='color:#808030; '>,</span> <span style='color:#008c00; '>100</span><span style='color:#808030; '>)</span><span style='color:#808030; '>,</span>
<span style='color:#0000e6; '>"b"</span><span style='color:#808030; '>:</span> GenericValue<span style='color:#808030; '>.</span><span style='color:#e34adc; '>int</span><span style='color:#808030; '>(</span>ty_int<span style='color:#808030; '>,</span> <span style='color:#008c00; '>42</span><span style='color:#808030; '>)</span>
<span style='color:#800080; '>}</span>
args<span style='color:#808030; '>=</span><span style='color:#808030; '>[</span><span style='color:#808030; '>]</span>
<span style='color:#800000; font-weight:bold; '>for</span> param <span style='color:#800000; font-weight:bold; '>in</span> identifiers<span style='color:#808030; '>:</span>
args<span style='color:#808030; '>.</span>append<span style='color:#808030; '>(</span>env<span style='color:#808030; '>[</span>param<span style='color:#808030; '>]</span><span style='color:#808030; '>)</span>
retval <span style='color:#808030; '>=</span> ee<span style='color:#808030; '>.</span>run_function<span style='color:#808030; '>(</span>m<span style='color:#808030; '>.</span>get_function_named<span style='color:#808030; '>(</span><span style='color:#0000e6; '>"f"</span><span style='color:#808030; '>)</span><span style='color:#808030; '>,</span> args<span style='color:#808030; '>)</span>
</pre>
This will build up a function in memory, which we can pass some arguments and call. This is really quite amazing. You call some functions to describe how to do something, and then you get a function back that you can execute directly. This can be used for all sorts of interesting things, like partial specialisation of a function at runtime, to take one example. Let's say you have to do a fully unindexed join on two strings in a database. For each value in the first table, you will need to do a string comparison with every single value in the other table. Since none of the strings we're comparing are known at compile time, there's little the compiler can do except hand us its very best version of a general strcmp. With LLVM, we can do better: for each string in one of the tables, generate a specialised string comparison function that compares a string in the other table to that particular one. Such functions can be optimised quite a bit.
<p>
The fun doesn't stop here either. In addition to generating functions in memory, we can serialise them to bitcode files using the <code>to_bitcode</code> method of LLVM modules. That generates a .bc file for which we can generate assembler code for some particular architecture, and then use the platform assembler and linker to generate an executable. To do that, however, we first need a main function. Since the function we generated for our expression above takes ordinary integers, we'll have to generate a main function that will take inputs form somewhere (e.g. the command line), convert that to integers, pass them to the function, and then make the result known to the user. To make things easy, let's just call the libc function atoi on argv[1], argv[2],... and use the result of the function as the exit code of our program.
<pre style='color:#000000;background:#ffffff;'>ty_str <span style='color:#808030; '>=</span> Type<span style='color:#808030; '>.</span>pointer<span style='color:#808030; '>(</span>Type<span style='color:#808030; '>.</span><span style='color:#e34adc; '>int</span><span style='color:#808030; '>(</span><span style='color:#008c00; '>8</span><span style='color:#808030; '>)</span><span style='color:#808030; '>)</span>
ty_main <span style='color:#808030; '>=</span> Type<span style='color:#808030; '>.</span>function<span style='color:#808030; '>(</span>ty_int<span style='color:#808030; '>,</span> <span style='color:#808030; '>[</span>ty_int<span style='color:#808030; '>,</span> Type<span style='color:#808030; '>.</span>pointer<span style='color:#808030; '>(</span>ty_str<span style='color:#808030; '>)</span><span style='color:#808030; '>]</span><span style='color:#808030; '>)</span>
main <span style='color:#808030; '>=</span> m<span style='color:#808030; '>.</span>add_function<span style='color:#808030; '>(</span>ty_main<span style='color:#808030; '>,</span> <span style='color:#0000e6; '>"main"</span><span style='color:#808030; '>)</span>
main_bb <span style='color:#808030; '>=</span> main<span style='color:#808030; '>.</span>append_basic_block<span style='color:#808030; '>(</span><span style='color:#0000e6; '>"entry"</span><span style='color:#808030; '>)</span>
bmain <span style='color:#808030; '>=</span> Builder<span style='color:#808030; '>.</span>new<span style='color:#808030; '>(</span>main_bb<span style='color:#808030; '>)</span>
argv<span style='color:#808030; '>=</span>main<span style='color:#808030; '>.</span>args<span style='color:#808030; '>[</span><span style='color:#008c00; '>1</span><span style='color:#808030; '>]</span>
int_args<span style='color:#808030; '>=</span><span style='color:#808030; '>[</span><span style='color:#808030; '>]</span>
<span style='color:#800000; font-weight:bold; '>for</span> i <span style='color:#800000; font-weight:bold; '>in</span> <span style='color:#e34adc; '>range</span><span style='color:#808030; '>(</span><span style='color:#008c00; '>1</span><span style='color:#808030; '>,</span> <span style='color:#e34adc; '>len</span><span style='color:#808030; '>(</span>identifiers<span style='color:#808030; '>)</span><span style='color:#808030; '>+</span><span style='color:#008c00; '>1</span><span style='color:#808030; '>)</span><span style='color:#808030; '>:</span>
<span style='color:#696969; '># atoi(argv[i])</span>
s <span style='color:#808030; '>=</span> bmain<span style='color:#808030; '>.</span>load<span style='color:#808030; '>(</span>bmain<span style='color:#808030; '>.</span>gep<span style='color:#808030; '>(</span>argv<span style='color:#808030; '>,</span> <span style='color:#808030; '>[</span>Constant<span style='color:#808030; '>.</span><span style='color:#e34adc; '>int</span><span style='color:#808030; '>(</span>ty_int<span style='color:#808030; '>,</span> i<span style='color:#808030; '>)</span><span style='color:#808030; '>]</span><span style='color:#808030; '>)</span><span style='color:#808030; '>)</span>
int_args<span style='color:#808030; '>.</span>append<span style='color:#808030; '>(</span>bmain<span style='color:#808030; '>.</span>call<span style='color:#808030; '>(</span>atoi<span style='color:#808030; '>,</span> <span style='color:#808030; '>[</span>s<span style='color:#808030; '>]</span><span style='color:#808030; '>)</span><span style='color:#808030; '>)</span>
bmain<span style='color:#808030; '>.</span>ret<span style='color:#808030; '>(</span>bmain<span style='color:#808030; '>.</span>call<span style='color:#808030; '>(</span>f<span style='color:#808030; '>,</span> int_args<span style='color:#808030; '>)</span><span style='color:#808030; '>)</span>
</pre>
Here, <code>ty_str</code> is a character pointer, and <code>main</code> is a function from int and pointer to character pointer to int. In its basic block, we add one call to <code>atoi</code> for each identifier that our expression uses. The argument to <code>atoi</code> would in C be written as <code>argv[i]</code>, but here, we do it by first getting an element pointer ("<code>gep</code>" is short-hand for Get Element Pointer) to the string we want, and then dereference that. Think of the "<code>load</code>" as the <code>*</code> and the <code>gep</code> as the <code>+</code> in <code>*(argv+i)</code>
<p>We then add a call to our function, and return its result as our exit code. But what does <code>atoi</code> refer to here? We must first tell LLVM that we want to call something from the outside of our program. It will give us back a reference to the function, so that we can build up calls to that something.
<pre style='color:#000000;background:#ffffff;'>ty_atoi <span style='color:#808030; '>=</span> Type<span style='color:#808030; '>.</span>function<span style='color:#808030; '>(</span>ty_int<span style='color:#808030; '>,</span> <span style='color:#808030; '>[</span>ty_str<span style='color:#808030; '>]</span><span style='color:#808030; '>)</span>
atoi <span style='color:#808030; '>=</span> m<span style='color:#808030; '>.</span>add_function<span style='color:#808030; '>(</span>ty_atoi<span style='color:#808030; '>,</span> <span style='color:#0000e6; '>"atoi"</span><span style='color:#808030; '>)</span>
atoi<span style='color:#808030; '>.</span>linkage <span style='color:#808030; '>=</span> LINKAGE_EXTERNAL
</pre>
All that is left before we can generate our very own native executables is file handling and calling the right tools. On Mac OS X, it goes a little something like this:
<pre style='color:#000000;background:#ffffff;'><span style='color:#800000; font-weight:bold; '>import</span> os
<span style='color:#800000; font-weight:bold; '>if</span> filename<span style='color:#808030; '>.</span>endswith<span style='color:#808030; '>(</span><span style='color:#0000e6; '>".zoom"</span><span style='color:#808030; '>)</span><span style='color:#808030; '>:</span>
basename <span style='color:#808030; '>=</span> filename<span style='color:#808030; '>[</span><span style='color:#808030; '>:</span><span style='color:#808030; '>-</span><span style='color:#008c00; '>5</span><span style='color:#808030; '>]</span>
<span style='color:#800000; font-weight:bold; '>else</span><span style='color:#808030; '>:</span>
basename <span style='color:#808030; '>=</span> <span style='color:#0000e6; '>"a"</span>
bitcode <span style='color:#808030; '>=</span> basename <span style='color:#808030; '>+</span> <span style='color:#0000e6; '>".bc"</span>
asm <span style='color:#808030; '>=</span> basename <span style='color:#808030; '>+</span> <span style='color:#0000e6; '>".s"</span>
obj <span style='color:#808030; '>=</span> basename <span style='color:#808030; '>+</span> <span style='color:#0000e6; '>".o"</span>
executable <span style='color:#808030; '>=</span> basename
f<span style='color:#808030; '>=</span><span style='color:#e34adc; '>open</span><span style='color:#808030; '>(</span>bitcode<span style='color:#808030; '>,</span> <span style='color:#0000e6; '>"wb"</span><span style='color:#808030; '>)</span>
m<span style='color:#808030; '>.</span>to_bitcode<span style='color:#808030; '>(</span>f<span style='color:#808030; '>)</span>
f<span style='color:#808030; '>.</span>close<span style='color:#808030; '>(</span><span style='color:#808030; '>)</span>
<span style='color:#800000; font-weight:bold; '>if</span> target<span style='color:#808030; '>=</span><span style='color:#808030; '>=</span><span style='color:#0000e6; '>"x86"</span><span style='color:#808030; '>:</span>
os<span style='color:#808030; '>.</span>system<span style='color:#808030; '>(</span><span style='color:#0000e6; '>"llc -filetype=obj %s -o %s"</span> <span style='color:#808030; '>%</span> <span style='color:#808030; '>(</span>bitcode<span style='color:#808030; '>,</span> obj<span style='color:#808030; '>)</span><span style='color:#808030; '>)</span>
os<span style='color:#808030; '>.</span>system<span style='color:#808030; '>(</span><span style='color:#0000e6; '>"ld -arch x86_64 %s -lc /Developer/SDKs/MacOSX10.6.sdk/usr/lib/crt1.o -o %s"</span> <span style='color:#808030; '>%</span> <span style='color:#808030; '>(</span>obj<span style='color:#808030; '>,</span> executable<span style='color:#808030; '>)</span><span style='color:#808030; '>)</span>
<span style='color:#800000; font-weight:bold; '>else</span><span style='color:#808030; '>:</span>
os<span style='color:#808030; '>.</span>system<span style='color:#808030; '>(</span><span style='color:#0000e6; '>"llc -mtriple=arm-apple-darwin -filetype=obj %s -o %s"</span> <span style='color:#808030; '>%</span> <span style='color:#808030; '>(</span>bitcode<span style='color:#808030; '>,</span> obj<span style='color:#808030; '>)</span><span style='color:#808030; '>)</span>
os<span style='color:#808030; '>.</span>system<span style='color:#808030; '>(</span><span style='color:#0000e6; '>"ld -arch_multiple -arch arm %s -o %s -L/Developer/Platforms/iPhoneOS.platform/DeviceSupport/4.2/Symbols/usr/lib /Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS4.3.sdk/usr/lib/crt1.o -lSystem "</span> <span style='color:#808030; '>%</span> <span style='color:#808030; '>(</span>obj<span style='color:#808030; '>,</span> executable<span style='color:#808030; '>)</span><span style='color:#808030; '>)</span>
</pre>
In order to build for other targets, you can specify <code>-mtriple=xxx</code> on the llc command line. You will of course have to run a matching version ld in order to generate your executable.
<p>
So, we can build native, and even cross-compiled, executables, but what about optimisation? Turns out we get that too. The LLVM bitcode we generate for the expression <code>1+1+1+1+a+a+a+a</code> is
<pre style='color:#000000;background:#ffffff;'>define i3<span style='color:#008c00; '>2</span> @f<span style='color:#808030; '>(</span>i3<span style='color:#008c00; '>2</span> <span style='color:#808030; '>%</span>a<span style='color:#808030; '>)</span> <span style='color:#808030; '>{</span>
<span style='color:#e34adc; '>entry:</span>
<span style='color:#004a43; '>  </span><span style='color:#004a43; '>%0</span><span style='color:#004a43; '> = add i32 %a, %a</span>
<span style='color:#004a43; '>  </span><span style='color:#004a43; '>%1</span><span style='color:#004a43; '> = add i32 %a, </span><span style='color:#004a43; '>%0</span>
<span style='color:#004a43; '>  </span><span style='color:#004a43; '>%2</span><span style='color:#004a43; '> = add i32 %a, </span><span style='color:#004a43; '>%1</span>
<span style='color:#004a43; '>  </span><span style='color:#004a43; '>%3</span><span style='color:#004a43; '> = add i32 1, </span><span style='color:#004a43; '>%2</span>
<span style='color:#004a43; '>  </span><span style='color:#004a43; '>%4</span><span style='color:#004a43; '> = add i32 1, </span><span style='color:#004a43; '>%3</span>
<span style='color:#004a43; '>  </span><span style='color:#004a43; '>%5</span><span style='color:#004a43; '> = add i32 1, </span><span style='color:#004a43; '>%4</span>
<span style='color:#004a43; '>  </span><span style='color:#004a43; '>%6</span><span style='color:#004a43; '> = add i32 1, </span><span style='color:#004a43; '>%5</span>
<span style='color:#800000; font-weight:bold; '>ret</span> i3<span style='color:#008c00; '>2</span> <span style='color:#808030; '>%</span><span style='color:#008c00; '>6</span>
<span style='color:#808030; '>}</span>
</pre>
whereas the resulting x86 assembler code is
<pre style='color:#000000;background:#ffffff;'><span style='color:#e34adc; '>_f:</span> ## @f
<span style='color:#e34adc; '>Ltmp0:</span>
## BB#<span style='color:#008c00; '>0</span><span style='color:#808030; '>:</span> ## <span style='color:#808030; '>%</span>entry
leal <span style='color:#808030; '>(</span><span style='color:#808030; '>%</span>rdi<span style='color:#808030; '>,</span><span style='color:#808030; '>%</span>rdi<span style='color:#808030; '>)</span><span style='color:#808030; '>,</span> <span style='color:#808030; '>%</span><span style='color:#000080; '>eax</span>
addl <span style='color:#808030; '>%</span><span style='color:#000080; '>edi</span><span style='color:#808030; '>,</span> <span style='color:#808030; '>%</span><span style='color:#000080; '>eax</span>
leal <span style='color:#008c00; '>4</span><span style='color:#808030; '>(</span><span style='color:#808030; '>%</span>rdi<span style='color:#808030; '>,</span><span style='color:#808030; '>%</span>rax<span style='color:#808030; '>)</span><span style='color:#808030; '>,</span> <span style='color:#808030; '>%</span><span style='color:#000080; '>eax</span>
<span style='color:#800000; font-weight:bold; '>ret</span>
</pre>
Clearly, llvm has figured out what's wrong and folded the constant additions and lowered the multiplication to a left shift.
<p>
So, what's the conclusion from this? That building optimizing cross-compilers is a weekend project for a small DSL, and you can do it all in Python - and if that doesn't zoom, nothing does.Anonymoushttp://www.blogger.com/profile/01044897049984478240noreply@blogger.com0tag:blogger.com,1999:blog-6069439299305224730.post-75588649371518781592012-09-22T22:31:00.000+02:002015-03-24T22:34:18.308+01:00Goto CheckerOne of the cool new features of the upcoming <a href="http://clang.llvm.org">Clang</a> release is the <a href="http://clang.llvm.org/docs/Tooling.html">tooling infrastructure</a>. This Clang tooling infrastructure provides an easy way of writing style checkers, source-to-source rewriters, analyzers and all sorts of other tools that need to understand C and related languages.
<p>
Especially nice is the AST matching support that landed recently. With the AST matchers, you can describe patterns of code that you're interested in in a declarative DSL, and get a callback called for all instances of that pattern in the code you're parsing. It's a bit like XPath for C.
<p>
As a demonstration of this, I've written a little code checker that checks that code follows a particular error handling style, and warns of any transgressions. The error handling style I chose is the one I prefer in C, which I call the reverse label style. The reverse label style has been made popular though its use in the Linux kernel, and looks like the below example:
<code><pre>
int read_file_normal(const char* filename)
{
FILE* f = fopen(filename, "rt");
char buf[100];
if (f == NULL) {
fprintf(stderr, "Failed to open %s", filename);
goto error_open;
}
if (fread(buf, sizeof(buf), 1, f) == 0) {
fprintf(stderr, "Failed to read data from %s\n", filename);
goto error_read;
}
if (fclose(f) == EOF) {
fprintf(stderr, "Failed to close %s\n", filename);
}
return 0;
error_read:
fclose(f);
error_open:
return -1;
}
</pre></code>
For each error case, we have a <code>goto</code> to a unique label. Starting at that label is all cleanup that should be done in the current case. The labels are therefore listed in reverse order to the gotos. A bit more discussion can be found at <a href="http://blog.staila.com/?p=114">the staila blog</a>. As rightly pointed out there, it results in readable code that unfortunately easily can rot during maintenance.
<p>
Automatic code checking to the rescue. We want to find all function definitions, and in each function definition we want to see all uses of <code>goto</code> and all labels, and verify that we first have one section of <code>goto</code>s but no labels, which is then followed by a section of labels, listed in reverse order of their corresponding <code>goto</code>s in the first section, without any jumps in it. We thus set up a <code>MatchFinder</code> and add our three patterns and bind their match results to names we can use in our callback:
<code><pre>
MatchFinder mf;
GotoChecker handler;
mf.addMatcher(functionDecl(isDefinition()).bind("func"), &handler);
mf.addMatcher(gotoStmt().bind("goto"), &handler);
mf.addMatcher(labelStmt().bind("label"), &handler);
</pre></code>
When we run our tool on some code, our <code>GotoChecker::run</code> method will be called with its <code>MatchResult</code> set to one of these three things. So, we check which one it is, and if it's the start of a new function, we note that we're now in the normal section of the function (as opposed to the error handling section), and we clear our stack of labels:
<code><pre>
const clang::FunctionDecl* func =
Result.Nodes.getNodeAs<clang::FunctionDecl>("func");
const clang::GotoStmt* g =
Result.Nodes.getNodeAs<clang::GotoStmt>("goto");
const clang::LabelStmt* label =
Result.Nodes.getNodeAs<clang::LabelStmt>("label");
if (func) {
in_error_section = false;
gotos.erase(gotos.begin(), gotos.end());
}
</pre></code>
If we get a <code>goto</code>, we first check that we're not in the error handling section, and then append its label to our stack of labels.
<code><pre>
if (g) {
clang::LabelDecl* label = g->getLabel();
if (in_error_section) {
clang::DiagnosticsEngine& d =
Result.Context->getDiagnostics();
unsigned int id = d.getCustomDiagID(
clang::DiagnosticsEngine::Warning,
"Found goto to label %0 inside "
"error hanling section"
);
d.Report(g->getLocStart(), id) << label;
}
gotos.push_back(label);
}
</pre></code>
Here we can also see an example of how to produce diagnostics from a Clang tool. The <code>DiagnosticBuilder</code> returned by <code>DiagnosticsEngine::Report</code> accepts a number of different Clang types, and will format them appropriately. In this case, we're giving it a <code>NamedDecl</code>, so it knows to put it in quotes. The source location we give to <code>Report</code> also means that Clang knows to quote from the source and highlight exactly what we're objecting to.
<p>
Now, the final part of the puzzle is to handle labels. It would be really nice if we could verify that there's no way for the code to fall though from the normal section to the error handling section, but that would involve scanning backwards though the statements and checking that the last is either a return, a call to a function known not to return, or a conditional where all branches end with a return or a function knon not to return or a conditional where... Let's skip that for now, and just get the name of the label.
<code><pre>
if (label) {
if (!in_error_section) {
// TODO: Check somehow that all paths have returned
}
clang::LabelDecl* found = label->getDecl();
std::string name = found->getNameAsString();
</pre></code>
Since we're enforcing that <code>goto</code> is only used for error handling, let's check that the name of the label reflects this:
<code><pre>
if (
name.substr(0, 6) != "error_"
&& name.substr(0, 8) != "cleanup_"
&& name.substr(0, 4) != "err_"
&& name != "exit"
) {
clang::DiagnosticsEngine& d =
Result.Context->getDiagnostics();
unsigned int id = d.getCustomDiagID(
clang::DiagnosticsEngine::Warning,
"Illegal label name: %0"
);
d.Report(label->getLocStart(), id) << found;
}
</pre></code>
This feature could be debated. Maybe it should be optional, and maybe the permitted prefixes should be configurable, but that's for version 2.0.
<p>
Now all that is left is to check that the label we found matches the top of the label stack we built up from the <code>goto</code>s in the normal section.
<code><pre>
if (gotos.size() == 0) {
return;
}
clang::LabelDecl* expected = gotos.back();
in_error_section = true;
if (found != expected) {
clang::DiagnosticsEngine& d =
Result.Context->getDiagnostics();
unsigned int id = d.getCustomDiagID(
clang::DiagnosticsEngine::Warning,
"Error handling sequence mismatch. "
"Expected %0, found %1"
);
d.Report(label->getLocStart(), id) << expected << found;
}
gotos.pop_back();
</pre></code>
Apart from a handfull of boilderplate lines (~4 lines plus some includes and namespace uses), that's it. What I like is that all the parsing and scanning and other things that would normally get in our way just disappears and we can go straight to implementing our checking logic.
<p>
The full source can be had at the <a href="https://gitlab.com/magnusr/goto-checker">project page</a>. Note that it requires LLVM and Clang from trunk, but there are <a href="http://clang.llvm.org/get_started.html#build">good instructions</a> for getting that.
<p>
In summary, the Clang tooling infrastructure definitly zooms. I expect to be using this quite a bit from now on, and if you're working with C or its derivates, I think you should too.Anonymoushttp://www.blogger.com/profile/01044897049984478240noreply@blogger.com2tag:blogger.com,1999:blog-6069439299305224730.post-46598236262522313012012-09-09T10:59:00.001+02:002012-09-09T10:59:46.554+02:00The Netbeans build systemThis is more of a "venting frustration" than informative. I have been using netbeans for a while now, at least the re-branded version that is jMonkeEngine SDK. However, this is about the netbeans part of it and nothing to do with the jME SDK.<br />
I have been thinking for years that I should port Svansprogram to a Rich Client Platform instead of hacking my own. Hacking my own RCP was fun at first but now it is mostly tedious.<br />
So to learn module development on Netbeans I have started out with some small hacks to get a feel for it. Ran the Hello World things and so on.<br />
Eventually came the time to make my very own module, first I let Netbeans create the project, then I put it into my version control and then on to set up my continuous build system.<br />
The build system in netbeans is Ant. Which is all very fine by me, Ant+Ivy is my preference if I can choose whatever I like. It is always good if the IDE understands the real build system, i.e. the headless builds run on the build server. You know, the real build, that produces the installable binaries.<br />
Eclipse has a totally separate build system so you need to keep the IDE and the real build in synch using plugins and other voodoo. So if Netbeans have some build system that I can use in both the IDE and on the CI machine I'm a happy camper. So with a smile on my face I set out to run the netbeans project on the CI machine.<br />
That is when the horror starts.<br />
The way netbeans set up the builds is to have some archane and weirdly mutated Ant scripts, without documentation, importing things from secret directories. Then have some plugins in the IDE that knows the secret locations to insert values here and there. I have never in my life as a professional coder seen a more opaque and strange build system.<br />
For example, it uses an Ant plugin called CopyLibs to, you know, copy JAR-files it needs for the build. It makes me want to grab the engineer by the throat and foaming at the mouth scream at them that ANT already can copy files or why don't you just use Ivy.<br />
So in order for my CI machine to even be able to get all the required dependencies I have to pollute my Ant installation making it harder to have reproducible builds since now my entire build system is dependent on a specific version of Netbeans.<br />
It does not stop there, to make a plugin for Netbeans I must of course build and link and all that against a specific version of Netbeans. This is acheived by including build scripts from a "Harness" directory that is specific to a version of Netbeans. This is all understandable, but the location of this harness-directory is set up in a "private" property file, through one or two ant properties with closely guarded complex names. The private property file is actually hidden in $HOME.<br />
Yeah you heard me, to make the CI machine build it will have to have access to my $HOME to be able to find the harness directory that is from my _installation_ of netbeans, not from some central repository of dependecies.<br />
There are some other ant-voodoo that Netbeans hides in a FAQ to instruct people how to download the harness from the web instead of pointing the CI machine to an actual installation of netbeans. I say it again, an installation, that requires you to run an installer, and click through stuff.<br />
There is more to this story but now I need to go cry in a dark corner. It has taken me several weeks of hobby-coding time to even get this far, and then to realize that netbeans generates images when building so you have to remember to set the JVM as headless or the build fails. And oh yeah, when I did it generated blank images so my menu entries when the module is installed are empty lines.Anonymoushttp://www.blogger.com/profile/02908752593907979820noreply@blogger.com0tag:blogger.com,1999:blog-6069439299305224730.post-28628323144919560502012-06-27T21:48:00.003+02:002012-06-27T21:48:54.994+02:00Glassfish and TeamCityOn my day job we use TeamCity from JetBrains as the Continous Integration solution. It works really well, looks good and has a plug-in for eclipse even. So I thought I'd install it at home too.<br />
<br />
Now, I have glassfish as an app server. I like JEE application servers, I think they are neat, I want my glassfish to deploy all my JEE applications. So i found the WAR-download of TeamCity. I read the installation instructions which required me to set some environment flag to change the default location for data directory (I keep my variable files in /var thank you very much, not in some wierd directory in $HOME).<br />
<br />
I deploy the application and I am presented with a blank page.<br />
<br />
Looking in the log file it says that I have no database definition in my data directory - yeah, like, how the hell could I since I have no clue what you are talking about.<br />
<br />
So back to the instructions here <a href="http://confluence.jetbrains.net/display/TCD7/TeamCity+Data+Directory" target="_blank">http://confluence.jetbrains.net/display/TCD7/TeamCity+Data+Directory</a>. I suspect that I must have _all_ the configuration in place before I deploy the WAR file, at least <a href="http://youtrack.jetbrains.com/issue/TW-20362" target="_blank">http://youtrack.jetbrains.com/issue/TW-20362</a> seems to say that. So in order for me to deploy the WAR-file I must first download and run the Netty-integrated installer to create the directories?!?!<br />
<br />
Hello <a href="https://wiki.jenkins-ci.org/display/JENKINS/Glassfish" target="_blank">Jenkins</a>, goodbye TeamCityAnonymoushttp://www.blogger.com/profile/02908752593907979820noreply@blogger.com1tag:blogger.com,1999:blog-6069439299305224730.post-2530317812705540232012-05-18T20:47:00.000+02:002012-05-18T20:47:16.716+02:00Svansprogram v3.0 out nowI just released v3.0 of Svansprogram. Some bug fixes, <i>lots</i> of refactoring and sweet new graphics (icons and banners) by <a href="http://tavlorna.se/" target="_blank">Mattias Persson</a>. Get Svansprogram from <a href="http://code.google.com/p/svansprogram/">http://code.google.com/p/svansprogram/</a><br />
<br />
Oh, sorry, no packaging for Mac yet but I'll do it "real soon now" :-)Anonymoushttp://www.blogger.com/profile/02908752593907979820noreply@blogger.com0tag:blogger.com,1999:blog-6069439299305224730.post-15411791269609845752012-04-03T14:40:00.004+02:002012-04-03T15:01:11.789+02:00Installer woesI'm preparing to release v3.0 of my Multi-tail application <a href="http://code.google.com/p/svansprogram/" style="font-style: normal; ">Svansprogram</a>. One popular feature request is native launchers. Yeah, OK, so this one guy asked for it but since there are about 3 people in the world using svansprogram that is a substantial part of the user base.<div style="font-style: normal; ">Being a Linux guy I have to jump through a lot of hoops to even begin to create a windows EXE-file or an OS X launcher. I can't even test them properly! Anyway, <a href="http://launch4j.sourceforge.net/">Launch4j</a> seems nice and might even have some type of integration with maven (and I'm switching back to ant + ivy when I find the strength). There is a GUI and some XML files that you can use to work up an exe-file. The GIMP did let me create an ICO-file also. As soon as I can test this somewhere I might put in the hour or so to try this out.</div><div><div>A pal has promised to build an OS X launcher by using some sort of maven magic and an another launcher-creator-app-thingy.<div style="font-style: normal; "><br /></div><div style="font-style: normal; ">So I tackled the Linux side. I decided to distribute an installer with the TAR-ball,</div><div><i>disclaimer - Blogger ate the formatting and yes I KNOW there are symlinks and stuff this does not handle, write a bug report or something :-)</i></div><br /><div>And this people is why I LOVE linux: a few lines of BASH and here is your installer. </div><br /><blockquote style="font-style: normal; "><br />#! /bin/bash<br /><br />SVANSPROGRAM_HOME="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"<br />LOG_FILE=$SVANSPROGRAM_HOME/install.log<br /><br />log() {<br /> echo "$1" >> $SVANSPROGRAM_HOME/install.log<br />}<br /><br />log "`date`"<br />log "Svansprogram directory: ${SVANSPROGRAM_HOME}"<br /><br />#######################################<br /># Create executable file #<br />#######################################<br />EXECUTABLE="$SVANSPROGRAM_HOME"/svansprogram.sh<br />log "Creating shell script: $EXECUTABLE"<br />cat <<MESSAGEDELIMITER > $EXECUTABLE<br />java -jar $SVANSPROGRAM_HOME/svansprogram.jar<br />MESSAGEDELIMITER<br /><br />log "Setting execute permission"<br />chmod +x "$EXECUTABLE"<br /><br />#######################################<br /># Create desktop file #<br />#######################################<br />INSTALL_DIR=~/.local/share/applications<br />log "Install directory: ${INSTALL_DIR}"<br /><br /><br />if [ ! -d "$INSTALL_DIR" ]; then<br /> log "Install directory does not exist"<br /> exit 1<br />fi<br /><br />log "Creating desktop file"<br />cat <<MESSAGEDELIMITER > "$SVANSPROGRAM_HOME"/svansprogram.desktop<br />#!/usr/bin/env xdg-open<br /><br />[Desktop Entry]<br />Version=3.0<br />Type=Application<br />Terminal=false<br />Exec=$EXECUTABLE<br />Name=Svansprogram<br />Comment=A Multi tail application<br />Icon=$SVANSPROGRAM_HOME/icon_128x128.png<br />Categories=Development<br />MESSAGEDELIMITER<br /><br />log "Moving desktop file"<br />mv "$SVANSPROGRAM_HOME"/svansprogram.desktop "$INSTALL_DIR"<br /></blockquote></div></div>Anonymoushttp://www.blogger.com/profile/02908752593907979820noreply@blogger.com5tag:blogger.com,1999:blog-6069439299305224730.post-60714521882036765282012-02-08T23:08:00.002+01:002012-02-09T07:49:01.388+01:00Some laws of enterprise architectureThough the years, I've acquired a few observations on enterprise development, which I will pompously refer to as my Laws of Enterprise Architecture. They are as follows:</p>
<h2>1: In the end, it's SQL</h2>
Sure, the system you're looking at is not just a simple SQL front-end. While it does terminate some functionality in an SQL database, other things are passed on to web services and other support systems. But what do <b>those</b> do? More likely than not, they terminate functionality in an SQL database or pass the request on to other systems that do the same. Eventually, everything ends up as SQL queries. There aren't <a href="http://en.wikipedia.org/wiki/Turtles_all_the_way_down">web services all the way down</a>.
<p>As a corollary, you can determine the efficiency of the architecture by comparing the queries that are actually run for a set of functions with that you would have if you just looked at those functions and translated that into queries to an imaginary database built for the purpose. If you only need a query or two that fetch a grand total of a handful of rows, but the queries that end up hitting the actual databases are a lot more complex (and especially return a lot more data), then you're losing track of the objective somewhere along the way from the user to the database.
<h2>2: If the answer to your enterprise integration needs is a bus, then you have bigger problems</h2>
<p>If you need a bus to integrate your systems, then you either
<ol type="a"><li>pretend that you don't know what systems you have, or</li><li>actually have no control over what's deployed.</ol> If it's a), then you should open your eyes. Pretending that you don't know things that you, in reality, fully control is a sure way to ruin any design. Doing it for the entire architecture is madness. If you're working on the architecture for an enterprise system and you're in case b), then you should work a bit more on the processes. From where do these unknown systems come?
<p>That said, there are cases where a bus makes sense. If you're building a desktop operating system that will be used by millions of users running any number of strange things on their systems, then a bus does make sense. If your enterprise architecture is comparable to millions of installations of a desktop system, however, you have your plate full.
<h2>3: Premature generalization is the root of all evil</h2>
<p>Premature optimization is the root of all evil, and it applies even when you're not optimizing for runtime performance. If you prematurely optimize for generality, then you will invariably make designs that are more complex than is called for.
<p>For instance, if the cardinality of a relation between two entities is one-to-one, then don't make it one-to-many just because you think it might be useful later on. Yes, it will be painful one day if you end up having to change it, but it will be painful every single day if you needlessly generalize it now.
<p>Another example is that if you're designing a system whose purpose it is to generate HTML that will be sent out over HTTP, then it's OK to make components of the system aware of that now and then. I'm not saying you should generate HTML from the stored procedures in your database, but only that there's no need to make every (or any, actually) component capable of doing everything. If you can simplify the design by assuming, say, that the purpose of the system is what you know it is, then do so.
<h2>Conclusion</h2>
<p>So, there you go, my Laws of Enterprise Architecture. What are your laws?</p>Anonymoushttp://www.blogger.com/profile/01044897049984478240noreply@blogger.com1tag:blogger.com,1999:blog-6069439299305224730.post-66018614487391141802011-12-05T20:45:00.001+01:002011-12-05T21:54:10.306+01:00System-level component testingOne of my favourite parts of development is writing automated blackbox functional tests in the component scope. That is: writing automated tests that use a component as it is ment to be used in a system. The definition of "component" that I tend to use in this context is "as much of the code that my team is working on as makes sense and as little else as possible".
<p>
If your code just uses other libraries etc., then it's usually very simple to just mock those parts, especially if you're using a dynamic language. I won't waste your time by talking about that. Instead, I'll talk about when it gets interesting - when you're writing system-level tools in a static language.
<p>
Since the tool support (and by "tool support" I mean <a href="http://valgrind.org/">Valgrind</a>) is best on Linux, I first try to make the code build as a self-contained executable in Linux, no matter what the target OS is. For small embedded OSes, you can probably just reimplement the OS functions and you're set. For Windows, suit your self. That leaves POSIX-like OSes, in which case the code should be fairly easily buildable in Linux. Here's where the fun starts.
<p>
So you have some code that calls <a href="http://pubs.opengroup.org/onlinepubs/000095399/functions/open.html">open(2)</a> on devices that don't exist on your workstation, connects sockets using address families unknown to civilization, or does all kinds of strange things that requires root privileges and can't be easily chrooted. How can you possibly write a harness for that? <code>LD_PRELOAD</code>, that's how.
<p>
<code>LD_PRELOAD</code> is a little-used feature of the <a href="http://kernel.org/doc/man-pages/online/pages/man8/ld.so.8.html">GNU dynamic linker</a> (and others, e.g. <a href="http://developer.apple.com/library/mac/#documentation/DeveloperTools/Reference/MachOReference/Reference/reference.html#//apple_ref/c/func/dlsym">the one in Mac OS X</a>) that lets you specify a dynamic library that is injected into an executable before other libraries are loaded. That, in combination with the rule that whoever first defines a symbol wins, means that you can reimplement any function you like in a library that is part of your harness and have those functions be used instead of the versions defined in, say, libc. If the component you're testing opens /dev/thingamajig, then just add a function that looks like open(2), but instead of actually opening the device node just tells your test scripts about it. One useful way of doing that is to have the test running as a separate process and have a Unix Domain Socket (or pair of pipes if you prefer) where you can send messages about what the component tried to do and receive instructions about what to do with the call.
<p>
Since reimplementing everything can be both tedious and slow, you may want to forward uninteresting calls to the normal versions of the functions you've overridden. This can be done using <code><a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/dlsym.html">dlsym</a>(RTLD_NEXT, "some_function")</code>. That will make the dynamic linker look up the next library that has a symbol called "some_function" and give you a pointer to that. Assigning that to a function pointer variable gives you a way to call, say, the plain old libc open(2) from your magic open(2) for any file that the test scripts deem uninteresting. Something like this:
<pre><code>
static int (*real_open)(const char *path, int oflag, ... );
__attribute__((constructor))
void init_hackery(void)
{
real_open = dlsym(RTLD_NEXT, "open");
}
int open(const char *path, int oflag, ... )
{
int mode = 0;
va_list va;
va_start(va, oflag);
if(oflag & O_CREAT)
{
mode = va_arg(va, int);
}
va_end(va);
inform_scripts_about_open(path, oflag, mode);
switch(ask_scripts_what_to_do())
{
case HandleInScripts:
{
return hanlde_open_in_scripts(path, oflag, mode);
} break;
case PassOnToLibc:
{
if(oflag & O_CREAT) return real_open(path, oflag, mode);
else return real_open(path, oflag);
}
}
}
</code></pre>
With this in place, the test scripts will be informed about every single call to open, and when they decide to, they can take over and do something else (but you probably do want it to end up opening some kind of file descriptor in the process you're testing, as you may need to follow the usual rules for file descriptor numbering and reuse between both the files you fake and the ones you pass on to libc). Override the <code>read</code>, <code>write</code>, <code>ioctl</code> and <code>close</code> in the same manner (mapping the file descriptor to some mock object in your scripts), and the component can get its devices and whatnot and you get your tests. Have fun!Anonymoushttp://www.blogger.com/profile/01044897049984478240noreply@blogger.com0tag:blogger.com,1999:blog-6069439299305224730.post-85867308926306224042011-05-13T23:23:00.002+02:002015-03-24T22:27:16.402+01:00Ada Server PagesIt seems to me that sometimes, shoe-horning web applications into an Object Oriented design just for the sake of it just doesn't make sense. A class that doesn't have both state and operations is not a class. A framework built on classes that aren't really classes isn't really using the language as it should.<br /><p>When building a small site, don't you sometimes secretly think that just plain JSP and some static helper methods wasn't all that bad? You're only half wrong. As long as you have a way out to a real language when you grow out of the shell you started in, it's not wrong to use simple tools for a simple end. It's just that for a purely procedural program, maybe Java isn't the right language.<br /><p>So what's the solution? I'll tell you: <a href="https://gitlab.com/magnusr/ada-server-pages">Ada Server Pages</a>. Some might call it The Thing That Should Not Be. I call it my first Ada program in almost 15 years. It compiles AdaSP pages into ada source, builds that into shared libraries, loads them into the server and calls them to serve the pages. And it works! At least on my particular OS X box.<br /><p><a href="https://gitlab.com/magnusr/ada-server-pages">Ada Server Pages</a>. It's here. Tell your friends.Anonymoushttp://www.blogger.com/profile/01044897049984478240noreply@blogger.com1tag:blogger.com,1999:blog-6069439299305224730.post-62359984816660750722011-03-24T08:36:00.002+01:002011-03-24T09:08:41.737+01:00Defence-in-depth for data-base backed websites: ConnectionsAs we've seen, connections can on some database/os combinations be <a href="http://blog.zoom.nu/2011/02/1000-connections-in-postgresql-on-os-x.html">scarce resources</a>. If we're keeping one connection for each session, then won't that limit the site to something like 1000 sessions? Well, there is a way around that.<br /><p>In a typical site, there will be a large number of users logged in, who all have sessions in the application (especially with long session timeouts), but not that many are actually active at the same time. If we could disconnect the connections for uses who are logged in but idle, and then reconnect when the users become active, then we'd get by using only as many connections as we expect to have simultaneously active users. That number is likely to be much lower, and a site that has 1000 users actively clicking on things in a, say, 5-minute period should probably run Oracle on Solaris anyway.<br /><p>Basically we get a most-recently-used cache the size of the number of connections that our database and OS can provide for us, and the way we use it looks pretty much the same as a connection pool: get a connection, use it, and release it. The difference is that instead of blocking in the get until there is a connection for us, we may be reconnecting to the database (possibly after waiting, in case the active set is full and we're thrashing).<br /><p>In a traditional setup, the web app of course knows how to log in to the database, since it's always using the same username etc. In the setup I'm proposing, only the users themselves know how to log on to the database. The web app could technically store the passwords, but that's madness from a security perspective. Cleartext passwords are to be discarded as soon as possible. Thus, we have to try some other way to log on to the database.<br /><p>One solution that's pretty much made for this scenario is Kerberos. When a user logs in to the site, authenticate to the Authentication Service, get a Ticket-Granting Ticket, and store that one. Whenever you need to connect to the database on the behalf of the user, use the user's TGT to get a ticket for the database. Should work in theory, but in practice it can be a nightmare both to set up and to get working in Java. It's possible that this would be smoother in Windows, where I imagine you could put the users in Active Directory and be done with the first part, but whether it will work still depends on the Kerberos APIs and how your database drivers use them.<br /><p>So, if we're passing on Kerberos, we can go for PAM or just roll our own solution. With PAM we could build a module that will let us use one-time passwords, so that we authenticate with the password via PAM on login, get a cookie, and then use that cookie when reconnecting. On logout, the cookie gets invalidated.<br /><p>For my prototype, I've skipped even that and gone for pre-salted passwords. What I do is that before I send passwords to the database (including on enrollment), I hash the passwords together with a random per-user salt. That salted password is what the database sees when authenticating the users, and never the cleartext one. The salted passwords are then stored in the sessions of the users, and used when reconnecting. Thus, cleartext passwords are never stored, so an attack that would show the contents of session variables of other users would not immediately give a way passwords that the users potentially could be using for other sites.<br /><p>Now that the prototype implementation can have very large numbers of simultaneous users, the real load testing and performance comparison of using a view filter versus working directly on the tables can commence.Anonymoushttp://www.blogger.com/profile/01044897049984478240noreply@blogger.com2tag:blogger.com,1999:blog-6069439299305224730.post-14285267858155497992011-02-19T13:53:00.016+01:002011-03-02T19:13:59.807+01:001000 connections in PostgreSQL on OS XPostgreSQL is not really geared towards more than a few dozen simultaneous connections on desktop operating systems. In this post, I'll show you how to push PostgreSQL 9 on Max OS X 10.6 to handle 1000 connections on a single system running both the webserver and the database.<br /><p>The first step to getting to 1000 connections is to make PostgreSQL actually try to do it. By default, it only allows 100 simultaneous connections. Change <code>/Library/PostgreSQL/9.0/data/postgresql.conf</code> from <code>max_connections = 100</code> to <code>max_connections = 1000</code>.<p>PostgreSQL creates one process per connection, and the web server will need one TCP connection to each one. Thus, to have 1000 connections, we need to allow at least 1000 processes per user (for the PostgreSQL processes) and at least 1000 open file descriptors per process (for the app server). Each PostgreSQL process seems to use 32 fds, and with some margin in case it grows, we should permitt 35000 fds in the system, plus some for other uses (in case you want to do other things on your machine, like logging in to it). The kernel seems to be picky about power-of-two increments, so I'll round the values up to things that it will accept.<p>Change <code>/etc/sysctl.conf</code> from<code><pre>kern.maxprocperuid=512<br />kern.maxproc=2048</pre></code><p>to<code><pre>kern.maxprocperuid=2048<br />kern.maxproc=4096<br />kern.maxfiles=40960 <br />kern.maxfilesperproc=2048</pre></code><p>This will make sure the kernel reserves enough space etc. In addition, we need to allow processes to create enough sub-processes (as reported by <code>ulimit -u</code>). This limit is set by OS X's equivivalent of <code>init</code>, called <code>launchd</code>. Change <code>/etc/launchd.conf</code> (or create if you don't have it already) to say <code>limit maxproc 2000 2000</code>.<p>The changes to <code>/etc/sysctl.conf</code> and <code>/etc/launchd.conf</code> both take effect on bootup, so reboot the system and have fun with your 1000 connections!Anonymoushttp://www.blogger.com/profile/01044897049984478240noreply@blogger.com7tag:blogger.com,1999:blog-6069439299305224730.post-10097924664561962302011-02-18T06:52:00.003+01:002011-02-18T12:29:45.161+01:00Thee-tiered testingThe three-tiered program structure has been used to great benefit in many types of programs. I'm using it for my test script. It has been of great help, and I'll show you how I've done it.<p><br />The top tier in my tests is the test cases. These read like a high-level description of the steps of the test case, like this:<br /><code><pre><br />def test_create_thread():<br /> session=login("p1", "p1")<br /> other=get_forum_id(session, "Other")<br /> before=get_num_threads(session, other)<br /> create_thread(session, other, "New Thread", "Newly added thread post")<br /> after=get_num_threads(session, other)<br /> logout(session)<br /> return before+1==after<br /></pre></code><p><br />Log in, pick a forum and check how many threads there are. Post a thread and verify that the thread count has increased by one. Easy stuff. All the details about how a thread is created in the application are abstracted away, and what's left is just the code related to the test case. So what do the <code>get_num_threads</code> etc. functions in the middle layer (application adaptation might be a name for it) look like?<br /><code><pre><br />def get_num_threads(session, forum_id):<br /> return int(fetch_data(session, "forum?id=" + forum_id, "count(//html:tr[@class='thread'])"))<br /></pre></code><p><br />These functions tell the bottom layer which URL to fetch, and what parts of the results they are interested in. As you can see, there's an XPath query there. In some, like <code>get_thread_id</code>, where the text of a DOM node is too much, a regex can also be used to pick out parts of the text:<br /><code><pre><br />def get_thread_id(session, forum_id, title):<br /> return fetch_data(session, "forum?id=" + forum_id, "//html:th[@class='subject']/html:a[text()='%s']/@href" % title, "id=(\\d+)")[0].group(1)<br /></pre></code><p><br />Since I'm testing a web application, the functions in the adaptation layer are of course implemented by fetching and parsing web pages. For other applications, I expect this layer to be implemented by composing and decomposing structured messages sent on some link, direct functions calls etc., but the role of the layer is the same: provide functions that correspond to functionality in the application, so that the upper layer can talk about things like reading a post instead of details about where the post is read from etc.<br /><p><br />The bottom layer then is the workhorse functions like <code>fetch_data</code> and <code>post</code>. Here, I'll show <code>fetch_data</code>, which has proven itself to be very useful:<br /><code><pre><br />def fetch_data(session, url, xpath_query, regex=None, params={}):<br /> conn=httplib.HTTPConnection("localhost", 8080)<br /> encoded_params = urllib.urlencode(params)<br /> headers = {"cookie":session}<br /> conn.request("GET", "/myapp/" + url, encoded_params, headers)<br /> response = conn.getresponse()<br /> if response.status != 200:<br /> print response.read()<br /> raise Exception("Failed to fetch data. status=%d, reason=%s" % (response.status, response.reason))<br /> html=response.read()<br /> doc=html5lib.parse(html, treebuilder="dom")<br /> context = xpath.XPathContext()<br /> context.namespaces['html'] = 'http://www.w3.org/1999/xhtml'<br /> results=context.find(xpath_query, doc)<br /> conn.close()<br /> if regex:<br /> r=re.compile(regex)<br /> results=map(lambda node:r.search(node.value), results)<br /> return results<br /></pre></code><p><br />Build the request using the URL from the caller. Send it, verify that it was OK, and parse the response. Pick out the part that the caller is interested in, and return it. In the old days, parsing HTML was practically impossible. Only a handfull of companies had the resources necessary to write an HTML parser that could parse HTML as it <strong>is</strong>, not as it <strong>should</strong>. Even though I'm attempting to have my application only send out valid HTML, it may fail (and we should assume it does here: this is the test suite after all!), so having a parser that can handle anything would of course be nice. <p>Enter the new HTML spec, where Hixie has done an astounding job in specifying how tagsoup can be parsed in a way that is compatible with the major browsers. Since we now have a spec for parsing HTML, little parsing libs based on this keep popping up everywhere. I'm using the Python html5lib, which can produce DOM trees, which in turn support XPath queries.<br /><p>That's it for the how of three-tiered tests. Now the why: In addition to having easily readable test cases for the functionality test, it also helped with the load test. Having the middle layer in place meant that the load test I'm writing has simply been a joy to write. Had the functionality testing been done using copy-pasted HTTP/HTML-related code (as it was before I started restructuring it), I'd have to start over. Now, I had almost every function I needed, with names that make sense. Just look at it!<br /><code><pre><br />sessions=[]<br />for i in range(num_users):<br /> username="load_user_%s_%d" % (instance, i)<br /> password=username<br /> signup(username, password, "%s@example.org" % username)<br /> session=login(username, password)<br /> sessions.append(session)<br /><br />for i in range(num_actions):<br /> session=random.choice(sessions)<br /> forum_ids=get_fora(session)<br /> forum_id=random.choice(forum_ids)<br /><br /> if random.randint(1,10)<10:<br /> post_to_existing_thread(session, forum_id)<br /> else:<br /> create_new_thread(session, forum_id)<br /><br />for session in sessions:<br /> logout(session)<br /></pre></code><p>Here, <code>post_to_existing_thread</code> and <code>create_new_thread</code> are functions similar to the test cases in the functionality test. All in all, I had to add two new functions to the adaptation layer. The rest was reused, and the load test is (at least to me) plainly readable.<br /><p>So: the three-layered approach to writing tests definitely zooms. Not only should you use it for your next project: you should apply it to the tests in your current one as soon as possible!Anonymoushttp://www.blogger.com/profile/01044897049984478240noreply@blogger.com0tag:blogger.com,1999:blog-6069439299305224730.post-31373885337820143982011-01-01T19:25:00.002+01:002011-01-01T19:28:46.390+01:00Android texting the wrong person<div>This hasn't happened to me (yet) but Engadget is writing about it.</div><div><br /></div><a href="http://www.engadget.com/2010/12/31/android-still-has-horrible-text-messaging-bugs-thatll-get-you-f/">http://www.engadget.com/2010/12/31/android-still-has-horrible-text-messaging-bugs-thatll-get-you-f/</a><div><br /></div><div>and it is reported</div><div><br /></div><div><a href="https://code.google.com/p/android/issues/detail?id=9392">https://code.google.com/p/android/issues/detail?id=9392</a></div><div><br /></div><div>It is a pretty serious issue, hope it doesn't happen to me.</div>Anonymoushttp://www.blogger.com/profile/02908752593907979820noreply@blogger.com0tag:blogger.com,1999:blog-6069439299305224730.post-35753991396935688442010-11-22T10:16:00.000+01:002010-11-23T20:30:17.416+01:00Defence-in-depth for data-base backed websites: Roles<p>Role-based access control has been hugely influential in how we do authorization. So much that it's almost difficult to find an application that makes authorization decisions for users but does not have a concept of roles. Thus, a complete solution for doing application-level authorization decisions in the database must support application-level roles.<br /><p>There are two major takes on how roles should be used. On one hand, we have what I'll call big roles, like User, Administrator, Support, and so on. Users usually have a single role. On the other hand, we have small roles, like the ones in Solaris' RBAC: <code>Printer Management</code> or <code>File System Security</code>, and so on, each letting users run a handful of privileged commands. Users usually have a whole bunch of roles, and get the permissions of the union of the permissions of the roles.<br /><p>In Postgresql, there is support for role-based access control. Users have roles, and roles have permissions. We've used it when creating users in the <code></code> procedure, which among other things does something like <code>CREATE USER <var>db_username</var> WITH PASSWORD <var>password</var> IN ROLE kroll_role_user</code>. Permissions are then given to the role <code>kroll_role_user</code>, and not to individual users. We also use the search path functionality to ensure that when users ask for a named table, they get a similarly named view in the schema we set up for them.<br /><p>These two things, database roles and search paths, can be used to implement application-level big roles. By giving different classes of users different database roles, and creating view schemas that match what that role is supposed to be permitted, we can do things like allowing a moderator to set the <code>sticky</code> field of a row in the <code>threads</code> table, while still restricting the view used by regular users to reading threads and inserting new threads with <code>sticky</code> set to false.<br /><p>If we modify the application to play along, we can give users access to several roles, and by prefixing table names with the schema for a role, the application can specify what authority it is asserting for each operation. That way, the user can operate under the rules of one role by default, but escalating when doing specific operations. A bit like <code>sudo</code>.<br /><p>So what about small roles? I haven't studied it closely yet, but I believe that they could for the most part be implemented reasonably cleanly using <code>WHERE</code> clauses on the views. Any authorization rule that depends on information that is stored in the database should be possible to formulate as a <code>WHERE</code> clause (e.g. if we have some authorization matrix that the application already uses, then we can select the row for the current user, the column for the current operation, and check that the operation is permitted).<br /><p>What we get from the database can be seen as data-level permissions, and schema-level permissions. By data-level, I mean that it operates on a row-by-row basis, where we can say that even if a user can sometimes be permitted to update a row in a table, a particular row may not be permitted (e.g. users can edit rows in the <code>posts</code> table if and only if they are specified in the <code>author</code> column). The schema-level permissions are things like allowing users to read from <code>users</code>, but not write. I believe that schema-level permissions match big roles in the application, and data-level matches small roles, but saying anything definitely on this will require studies of real-world role usage in web applications.<br /><p>There may be situations where you want to combine multiple schema-level roles (i.e. when they do not match the big role idea directly), and you cannot have users select one at login, nor have the application escalate. One example could be if you have one role for letting customer service representatives see details about the specific issues that they handle and one role for giving access to summaries of entire categories of issues for statistics generation, and then have a page for displaying summary information about a specific issue. In that case you probably can't decide which schema to use beforehand. In these cases, combination roles can be created, which give the permissions of both roles. In the general case, this is not feasible since the number of roles combining two roles would be the square of the number of roles, and there could in principle be a need for any combination of roles, leading to O(n<sup>n</sup>) combination roles. In practice, it's more likely that it's one or two roles (remember, this is schema-level roles we're talking about here) that need to be combined with ordinary roles. In that case, combination roles should be possible to use.<br /><p>Combination roles can easily be implemented by having the search path for users with combination roles specify first the combination role, and then one of the ordinary ones. That way, if the ordinary role has sufficient rules for a view, then the combination role schema can just omit the view. For views like the one in the example above, a simple union between the views of the ordinary roles and a rule for inserting/updating etc via the corresponding view in the schema of one of the ordinary roles should suffice. It's only when both roles can modify the view and it's not clear from the context of the application which authority the user is asserting when doing the update that the combination role gets complicated. I don't see this as a real-world problem, but more experimentation will tell.<br /><p>This concludes the implementation part of the series. Next up will be a post on unsolved problems, and performance test results.Anonymoushttp://www.blogger.com/profile/01044897049984478240noreply@blogger.com0tag:blogger.com,1999:blog-6069439299305224730.post-28459518443334011342010-10-31T16:24:00.003+01:002010-10-31T16:27:29.622+01:00Release of zoom-desktopI've just made a new release of zoom-desktop and the applications there. Notably the tailing-application 'Svansprogram' have been updated.<br />Check it out, comments are more than welcome. Please write bug reports or feature requests on the issue tracker.<br /><a href="http://code.google.com/p/zoom-desktop/">code.google.com/p/zoom-desktop</a>jmaasinghttp://www.blogger.com/profile/09542994977944948243noreply@blogger.com0tag:blogger.com,1999:blog-6069439299305224730.post-73648796531116108022010-09-23T01:23:00.000+02:002010-09-23T01:24:21.500+02:00JavaOne 2010How to make JavaOne suck?<br />Let Oracle host it.jmaasinghttp://www.blogger.com/profile/09542994977944948243noreply@blogger.com2tag:blogger.com,1999:blog-6069439299305224730.post-44564275117032690562010-08-19T20:57:00.005+02:002010-11-25T12:06:43.031+01:00Defence-in-depth for data-base backed websites: Writing<p>Writing though views seems to be quite different in different database servers. In Postgresql, it's done using something they call rules. Rules rewrite the queries early on, before they are sent off to the planner. Views are implemented as rules, so we actually already used them in the previous post, even though it didn't show.<br /><p>If we have a table for forum threads like this:<code><pre>CREATE TABLE impl.threads<br />(<br /> id serial not null,<br /> forum integer not null,<br /> locked integer not null default 0,<br /> sticky integer not null default 0,<br /> CONSTRAINT pk_threads PRIMARY KEY (id),<br /> CONSTRAINT fk_threads_forum FOREIGN KEY ("forum")<br /> REFERENCES impl.fora (id) ON DELETE CASCADE<br />);</pre></code><p>and we want users to be able to start new threads, but not to be able to specify the <code>locked</code> or <code>sticky</code> values, we can create a rule on the view like this:<br /><code><pre>CREATE OR REPLACE RULE posts_insert AS<br />ON INSERT TO kroll_user.threads<br />DO INSTEAD<br />INSERT INTO impl.threads (forum, locked, sticky) VALUES (NEW.forum, 0, 0);</pre></code><p>For the private messages view, we can set the sender to the current user, regardless of what the user tried:<code><pre>create or replace rule messages_insert as<br />on insert to kroll_user.messages do instead<br />insert into impl.messages ("to", "from", posted, subject, text)<br />values (<br /> NEW."to", kroll_user.get_current_app_user(), NEW.posted,<br /> NEW.subject, NEW.text<br />);</pre></code><p>Conditionals get a bit trickier. Posts should only be permitted to threads that are not locked. An insert to a locked thread should be ignored. Due to a limitation in Postgresql, it's impossible for it to figure out that if it shouldn't do anything then it should do nothing, so we have to explicitly tell it:<pre><code>CREATE VIEW kroll_user.posts as select * from impl.posts;<br /><br />grant select, insert, update on kroll_user.posts to kroll_role_user;<br /><br />create or replace rule posts_insert as<br />on insert to kroll_user.posts<br />WHERE (select locked from impl.threads where threads.id=NEW.thread)<>1<br />DO ALSO<br />insert into impl.posts (thread, author, subject, body, created)<br />values (<br /> NEW.thread, kroll_user.get_current_app_user(), NEW.subject,<br /> NEW.body, now()<br />);<br /><br />create or replace rule posts_insert_default as<br />on insert to kroll_user.posts<br />DO INSTEAD NOTHING;</code></pre><p>Postgresql runs all rules that have matching <code>WHERE</code> clauses (and if there is no <code>WHERE</code>, then the rule always matches), so for a post to a locked thread, it (like <a href="http://www.youtube.com/watch?v=juFZh92MUOY">the goggles</a>) does <code>NOTHING</code>, while for a post to an unlocked thread, it both inserts the row <b>and</b> does <code>NOTHING</code>.<br /><p>Updates are handled almost identically:<code><pre>create or replace rule posts_update as<br />ON UPDATE TO kroll_user.posts<br />WHERE (<br /> select locked from impl.threads<br /> where threads.id=(select thread from impl.posts where id = OLD.id)<br />)<>1<br />DO ALSO<br />UPDATE impl.posts<br />SET subject=NEW.subject, body=NEW.body, edited=now()<br />WHERE id=NEW.id and author=kroll_user.get_current_app_user();</pre></code><p>Writing to sequences can be done in a manner similar to how we did reading: by virtualizing the accessor functions:<pre><code>create function kroll_user.nextval_threads_id_seq() returns bigint as<br />'select pg_catalog.nextval(''impl.threads_id_seq'');'<br />language sql security definer;<br /> <br />/* ... */<br /> <br />create function kroll_user.nextval(unknown) returns bigint as<br />'select case<br /> when CAST($1 as text)=''messages_id_seq''<br /> then kroll_user.nextval_messages_id_seq()<br /> when CAST($1 as text)=''posts_id_seq''<br /> then kroll_user.nextval_posts_id_seq()<br /> when CAST($1 as text)=''threads_id_seq''<br /> then kroll_user.nextval_threads_id_seq() <br /> else pg_catalog.nextval(CAST($1 as text))<br /> end;'<br />language sql;</code></pre><p>As stated in the beginning, this is how things work in Postgresql, and other database servers will require things to be done in different ways. For Oracle, MS-SQL, and DB2, the views are updatable automatically, without any extra rules. In some cases (like with the <code>locked</code> and <code>sticky</code> columns), you may have to use triggers to make certain parts non-writeable.<br /><p>Next post will move back into vendor-neutral theory again. I'll discuss role-based access control and how the database can get involved in enforcing application roles.Anonymoushttp://www.blogger.com/profile/01044897049984478240noreply@blogger.com0tag:blogger.com,1999:blog-6069439299305224730.post-34522477961248619792010-08-19T09:02:00.004+02:002010-09-04T10:09:28.316+02:00Defence-in-depth for data-base backed websites: Reading<p>Now that we can create users and log them in, let's try to let them read some data. The actual tables are locked away in the <code>impl</code> schema to which regular users have no access. Selected parts of the tables can be made available though views. As we saw in the introduction, a table containing private messages between users can be exposed as a view that selects only the messages to or from the current user like so:<pre><code><br />CREATE VIEW kroll_user.messages AS<br />SELECT id, "from", "to", posted, subject, text<br />FROM impl.messages<br />WHERE "to"=(SELECT id FROM kroll_user.current_app_user)<br /> OR "from"=(SELECT id FROM kroll_user.current_app_user);</code></pre><p>So what's this <code>SELECT id FROM kroll_user.current_app_user</code>? <code>current_app_user</code> simply a view containing the appuser/dbuser mapping for the current user:<pre><code><br />CREATE VIEW kroll_user.current_app_user AS<br />(SELECT appuser as id FROM mapping.users WHERE dbuser=current_user);</code></pre><p><code>current_user</code> is a special SQL-standardized function that is called without the usual parentheses and returns the login name of the current data base user as a string. Or, that's what Postgresql returns. There seems to be some disagreement on whether it should be the login name or the current security context, with Postgresql, DB2, and Firebird in the first camp and MS-SQL, MySQL and Oracle in the second. As long as we don't call it from a privileged stored procedure, that should not matter, however.<br /><p>If we would like to filter out columns instead of rows, then this can be done by selecting null instead of the data for the column. Row-based and column-based filtering can also be combined. For the <code>users</code> table, I want to hide the email address of all users, except the logged-in user's own address. <code>UNION</code> to the rescue:<code><pre><br />CREATE VIEW kroll_user.users AS (<br /> select id, "name", null as email, moderator from impl.users<br /> where id <> kroll_user.get_current_app_user()<br />) UNION (<br /> select id, "name", email, moderator from impl.users<br /> where id = kroll_user.get_current_app_user()<br />);<br />grant select on kroll_user.users to kroll_role_user;<br /></pre></code><p>Setting up read-only views of tables is pretty much straight-forward: decide what cells users should be allowed to see in each table, come up with <code>SELECT</code> statements that pick those, and make them into views.<br /><p>Sequences are a bit trickier. Postgresql defines a sequnce to be "special single-row table" that can be used only via the functions <code>nextval</code>, <code>currval</code>, and <code>setval</code>. Unfortunately, Postgresql only allows <code>currval</code> to be run on actual sequences, and not on views of sequences. Since the sequences are data and thus stored in the <code>impl</code> schema to which users don't have access, we have to make a little workaround. Instead of virtualizing the sequence, we can virtualize the accessor functions:<code><pre><br />create function kroll_user.currval_threads_id_seq() returns bigint as<br />'select pg_catalog.currval(''impl.threads_id_seq'');'<br />language sql security definer;<br /><br />create function kroll_user.currval(unknown) returns bigint as<br />'select case<br /> when CAST($1 as text)=''threads_id_seq''<br /> then kroll_user.currval_threads_id_seq()<br /> else pg_catalog.currval(CAST($1 as text))<br />end;'<br />language sql;<br /></pre></code><p>This creates a new function called <code>currval</code> in the user schema (which is the first schema in the users' search path) that checks if the requested sequence is the one we want to virtualize (<code>threads_id_seq</code>). If it is, then we use a privileged access function for that specific sequence (which is a database object that we can explicitly grant access to). If it isn't, then we delegate to the built-in <code>currval</code> (which in Postgesql is stored in the <code>pg_catalog</code> schema). The delegation isn't strictly necessary if users don't have access to non-virtualized sequences, but it ensures we got correct error reporting at least.<br /><p>That's about all there is to reading and filtering data, which is the majority of what most web apps do, and most of the code is plain old standard SQL. Next up will be writing data, which will lead us in to heavy vendor-specific terrain.Anonymoushttp://www.blogger.com/profile/01044897049984478240noreply@blogger.com4