Smári McCarthy

Building social, political and technical infrastucture

Bulk and Memory

I’m working a lot with huge datasets these days, and it’s becoming par for the course to end up with large files in various formats. I found myself showing up to work this morning to find a 13GB JSON file waiting for me to load it into ElasticSearch. The JSON file was an array containing some 550000 large objects. Unfortunately the file isn’t structured in such a way as to allow just throwing it at ElasticSearch’s bulk loader, so I need to parse it out and do stuff with the objects. The list as such doesn’t matter.

If you know how JSON parsers work, you’ll understand why this is a problem. If you don’t, suffice to say that there are two general approaches to parsing JSON. The common one is to iterate over the entire document, loading each entity, when fully parsed, into memory, and returning a data structure. This of course would result in the program eating up about 14GB of memory (or more, because pointers and refcounts), which is kind of okay in this case since I have more RAM than that, but is less and less acceptable as the data size grows.

The other approach is what some refer to as a “SAX-like” parser, although it’s probably better to call it an event parser. Such a parser iterates over the document, but instead of loading everything into memory resident data structures, it just announces events like “object start” or “string end”, sometimes including a value. The value is typically only filled if the event has a natural value field.

Since my weapon of choice for this project is Python, the standard module is (not surprisingly) json. It does the former type of parsing, so that’s no good. ijson is a module that does event parsing, but the events are so granular that I’d have to keep track of the entire document as a stack in order to be know what’s going on, and essentially reconstruct each part of the document in memory based on the events. That’s way too much work, and seems to me like the wrong approach for most of the processing I’m going to be doing.

I couldn’t find any modules that don’t use either of these approaches. What I need is a middle-ground parser, that allows me to specify something like: “give me all objects that are nested 1 deep”, or “give me anything that’s on the third nesting level”.

Building this kind of parser requires managing a certain amount of state. Specifically, it needs to keep track of three things:

  1. What nesting level are we at.
  2. The type of entity we’re dealing with at each nesting level.
  3. The sequence of bytes from the beginning of the place where we entered a nesting level we care about until the the end of it. (This can then be run through the standard json parser.)

For now, I’ll just do the simplest thing: skip the first line (“[”), skip the last line (“]”), and drop the trailing comma from each line. But since I expect to run into gargantuan JSON blobs every now and then, I might take a shot at writing a midway parser.

In the meantime, by all means let me know if this already exists.

It’s Time to Stop Being Hypocrites

Leaders

In a wonderful show of support for the right to freedom of expression on Sunday, world leaders descended on Paris to participate to march in solidarity with the people of France.

This event would have been even better if the leaders in question were not so selective in their support for freedom of expression. As Daniel Wickham pointed out in a long sequence of Tweets, many of those attending the march have in recent years overseen the imprisonment, torture or murder of journalists.

This of course makes for a pretty weak gesture. When Benjamin Netanyahu, Sergey Lavrov, and Ahmet Davutoğlu march in support of free speech, they are disrespecting everybody who has died for their right to do so. But rather than dwelling of the hypocricy, it’s worthwhile to consider how to move beyond it.

The first order of business would be to release political prisoners. These would include journalists such as Aziz Kayed, Ahmad al-Khatib and Mustafa al-Khawaja, Hatice Duman, Mustafa Gök and Cüneyt Hacıoğlu; Sergei Reznik and Aleksandr Alesin; the staff at the Syrian Center for Media and Freedom of Expression; Abduljalil Alsingace, Ahmed Humaidan and Hussein Hubail. A broader campaign supported by the attendees of the march could lead to the release of Khadija Ismailova, Eskinder Nega, Jean Laokolé, Mohammed al Maqaleh, Muhammad Anwar Muna, Chelsea Manning, and various other political prisoners.

Once this has been done, the universal adoption of new standards and practices around freedom of expression would be a good move. This could include such radical concepts as eliminating imprisonment as a punishment for libel, decriminalizing blasphemy, and repealing any laws which restrict media activity. This could mean a sane libel law in the UK, the liberalization of religious speech in Ireland, and the elimination of the media censorship committee in Hungary, among other great things.

The next appropriate step would be to put in place appropriate safeguards to protect the global communications infrastructure from attempts at censorship and selective availability. This could be done by enshrining the principle of network neutrality in law, repealing existing Internet censorship laws, and reforming copyright to prevent DMCA-style notice and takedown procedures from being used as a general purpose censorship device.

The are of course many other things which need to be done, but this would be a good start. How about we set some deadlines? We could agree for instance to release the prisoners by the end of January, complete the legislative reforms by the end of 2015 (it is understandable that these things take time), and then perhaps these righteous world leaders could meet again in Paris in January 2016 for a summit on where to go from there.

Alternatively, the hypocricy can continue.

Learning to Live With Perpetual Information Warfare

Since the Snowden revelations started to inform the public about the ways in which western governments have been spying on everybody, a number of international diplomatic relations have soured, and many relationships between governments and their electorates have soured.

The actions of the governments of these countries have rendered them entirely untrustworthy. Their only avenue to regaining trust is to dismantle military-surveillance artefacts that are not physical, cannot be visually accounted for, that exist in a post-scarcity economy, with no meaningful limit to how many surveillance systems can be in place and no way of counting them. It is impossible to prove that this has been done. We must therefore hereafter assume that it is going to continue forever.

I publish this now in the context of a mass murder perpetrated against a group of journalists, in the name of religion. This is a terrible deed, but it is no more terrible than some of the reactions: it is almost as if certain fascist political actors are rubbing their hands together in glee over the atrocities, for events like these can be used to lend credence to downright disgusting political agendas. Marine Le Pen of course is overjoyed by the inevitable spike the Charlie Hebdo shooting will cause to her party’s following — as Bob Altemeyer pointed out in his book The Authoritarians, people seem spring-loaded to become more authoritarian in times of crisis. But when a French political party leader, even a fascist one, calls for the reuptake of the death penalty, then it is high time for everybody to become very very afraid.

This is a transcript of a talk I gave in Warsaw in September 2014, where I discussed some of the problems that make blanket surveillance easy, some of the possible approaches to eliminating broad state surveillance capacity, and put that into the larger geopolitical context of ongoing international information warfare. This was a continuation on a series of previous lectures, consisting of “Where States Go To Die” (SHARE 2013), “Engineering Our Way Out of Fascism” (FSCONS 2013), “Humanity Scale Information Security” (NullCon 2014) and “The Political Implications of Technology” (Digital Activism Now, 2014)

6. Surveillance is Easy

When the cold war ended, suddenly a generation of people, whose primary role had been to defend against an indeterminate adversary in a war that never happened, were put into the worst possible situation for them. Peace. Relative peace, interspersed with small conflicts, but the entire logic of the nuclear bureaucracy was upended, and all the skill and talent that had been built up since the end of the second world war was suddenly rendered unnecessary.

All those idle hands. And yet, like any other artefact of military superiority from a bygone age, it was repurposed. Unlike, say, the fort at Komárom or the military base at Christiania, or the Roman roads, these people were not repurposed for the public’s benefit. They were put into various roles including policy advisory and research and development.

It is in those roles that hundreds or thousands of smart people with a Cold War mindset got into the peacetime business of preparing for the next big problem. Papers were written, drafts circulated, plans shaped. But people who are in the business of preparing for the worst aren’t very good at assuming good faith. So they came up with bad law proposals, and kept them in their rainy day boxes, just in case.

Meanwhile, a culture of fear was being cultivated. Cities were turned into panopticons. Buildings were fitted with cameras, and the cameras were fitted with face recognition software, and the face recognition software was fitted with databases containing everybody.

The overarching argument was at one point crime. Then it was drugs. Then it became terrorism. Terrorism.

When we call somebody a terrorist, we are pretending that their actions have no motives. That their only aim is terror. That there is no chance of any legitimate political argument or concern behind the atrocities. Ignoring the politics of the terrorist, and instead lumping them into vague demographics based on nationality or religion, serves two goals: First, to eliminate any chance of non-violent solutions to their political demands, and secondly, to expand the group of potential terrorists beyond a negligible group of extremists with a particular set of political demands to a large amorphous group of indeterminate membership, thereby justifying the encroachment of the civil liberties of everybody.

Then, of course, it isn’t just cameras. The state security services are staffed with smart, dilligent people, who have been working hard on protecting their nation state from all of the indeterminate enemies. Because they’re smart, they know that you cannot fight your enemy without knowing your enemy. Unfortunately, they’re not smart enough to recognize that an enemy whose membership is intentionally, through willful ignorance, made to be indeterminate, cannot ever be known.

Thus the assumption that we must all be terrorists, and we must all, therefore, be known. Everything we do must be catalogued and understood. So our phones get tapped, and our Internet monitored. Our e-mails get read by machines and filtered through stupid, inaccurate computational linguistics models, slapshod statistical methods. Our passenger name records get analyzed for patterns. All of the data produced through the course of our increasingly interconnected lives are shoved through a pipeline of quantifications.

The state wants to find the outliers, and line them up against the wall. Fear isn’t cultivated because it’s fun. It’s cultivated as a means of manufacturing compliance, regardless of how insane the rules are.

In case you missed it, we live in a world of ubiquitous surveillance now. Information warfare is being perpetrated against us.

Surveillance is easy because ignoring the politics of minorities is easy. Surveillance is easy because accepting the bent logic of the state is easy. Surveillance is easy because the post cold-war nuclear bureaucracy got bored.

5. You are making Surveillance Easy

So one might say “down with the state,” with no plan for replacement, as if nihilism had any chance of improving our situation. It does not. Not only because there are an unknown number of devices spying on our activities, and not only because there is no way to find out where they are, and guarantee that we’ve turned them all off, but also because we willingly and actively submit ourselves to the surveillance.

You are carrying a device in your pocket that constantly keeps track of where you are, and reports it back to its overlords — the phone company. The phone company also keeps track of who you call and when, and for how long, and who you message, and which websites you visit, and in which order. The phone company dilligently complies with the demands of the state. If you are in Poland, they reported you to the authorities over a million times last year. The Stasi were never that efficient.

But it gets worse. You may use Facebook, or Twitter. You might use GMail or Yahoo for your e-mail. You might use Dropbox for your files, or iCloud maybe. These systems not only spy on you, but they aggregate your information and sell it to the highest bidder. And the second-highest. And the third. Actually, one of the most common business models of the cloud is to sell your data to everybody who wants to buy it. How do you think Facebook makes money? Do you think they’re allowing you to post pictures of your lunch or observations about the weather out of the goodness of their hearts? There is, to date, little evidence that the people running Social Surveillance Networks have hearts.

Cloud providers, as they are called, do of course have privacy policies, where they make vague promises not to harm you. But the definition of harm is narrow, and the scope of potential harm is broad. When you choose to put your data in the cloud, you are choosing to risk that it might rain. They can promise it’ll never rain, but the rain still comes, as many celebrities became profoundly aware of last week.

But it gets worse: even if you don’t use GMail or Yahoo for your e-mail, there’s a high probability that your friends do. When your friend uses a centralized e-mail service, they are exposing your activities to these companies, who may then report it to the state. When your friend uses GMail, your friend is reporting you to the authorities. Automatically. The Stasi were never that efficient.

When we choose to use Social Surveillance Networks, we are choosing to allow people of dubous moral fibre, with awkward relationships with governments, to keep track of us. And yet we can’t stop using Social Surveillance Networks any more than we can stop breathing: it is how we communicate now.

The only thing we can do is to be very clear about what is permissible, and what not.

You are making surveillance easy by not being clear about what is permissible. You are making surveillance easy by accepting the bent logic of the Social Surveillance Networks. You are making surveillance easy by using the cloud. It will rain.

4. We made Surveillance Easy

So one might say “technology should protect us,” ignoring entirely the political implications of technology. Technology is neither good nor bad, nor is it neutral, as Melvin Kranzberg has pointed out.

There are two ways to enforce any rule: enforcement by policy, and enforcement by design.

When you enforce a rule through policy, then the rule is kept as long as the policy is not changed, and nobody violates the policy, and nobody forgets to enforce it. It works well while everybody is playing nice.

Getting everybody to play nice is a bit like getting everybody to eat their vegetables. Most people will do it, because they know it’s good for them, but some people will refuse, because you know, they just don’t like the taste of broccoli.

Enforcement by design is a different type of thing entirely. It is where the rule is built into the system, in such a way that the universe prevents the rule from being violated. Gravity is a rule that is enforced by design. Imagine what would happen if there were a gravity committee that met every Tuesday. There would be chaos. Thankfully, the universe is not governed by committees, and it is very good at making sure certain rules never get violated.

But the design still has to happen. To prevent surveillance, there are three methods:

  1. Decentralization. It is harder to watch everybody when nobody is in the same place. When everybody goes to one place, we call it a single point of failure. If that point fails, everything fails. And if that point surveils, everything is surveilled. Facebook is a single point of failure for over a billion people now. Twitter is a single point of failure for about 600 million people. Skype is a single point of failure for another 600 million people. GMail is a single point of failure for at least half a billion people. Decentralized networks, by comparison, are pretty much impossible to surveil, and thankfully the Internet was designed from scratch to be decentralized. Unfortunately, a lot of the businesses on the Internet think that the only way they can make money is by building single points of failure. They made the technical decision to violate one of the most important design decisions of the Internet for their own gain, and we are all paying the price.

  2. Encryption. Some mathematics are very easy to do but practically impossible to undo. This is important it allows us to send messages in secret. This is useful for banking, it is useful for commerce, but it’s also useful for political activism, or police activities, or keeping healthcare records safe. Encryption is, however, used very sparingly. Next time you visit a website, check if it says “https” at the top. If it only says “http”, without the “s”, then your communications are not encrypted. Unfortunately, HTTPS is hard to use, and it has many flaws, so most websites don’t use it. In fact, about 700 of the largest 1000 websites in the world don’t enforce HTTPS encryption. E-mail is even worse: in order to encrypt that, people are required to learn mystical magical incantations called PGP, and even those who have learned this horrible type of magic get it wrong every now and then. This is because PGP was never designed for normal people. It was designed by elitist technologists for use by elitist technologists, and for that we are all paying the price.

  3. Hardening of computational endpoints. This is a bit more complicated, but generally what it means is that we need to write better software. Unfortunately, the common approach to software development is to make something that doesn’t work and then keep poking it until it does. If buildings were made the way software is, they would look ugly, stand at odd angles, and suddenly collapse. This isn’t just because software developers are bad at developing software, it’s also because software is hard. But long story short, most software is riddled with severe bugs that make surveillance easy.

The technical community created this mess, by making poor decisions and by valuing speed and profits more than stability or security. The greybeards who built the Internet created this situation because they had faith in the system, in the nuclear bureaucracy of the cold war era. When the guys with the shiny shoes came and told them not to build in encryption, they said okay, because they thought the government was their friend.

The technical community has a lot to answer for.

We made surveillance easy by pretending that a few big centralized services weren’t a problem. We made surveillance easy by making PHP and MySQL easier to use than HTTPS and PGP. We made surveillance easy by believing in the benevolence of the governments. We made surveillance easy by writing bad code. We made surveillance easy by not caring enough about people.

3. We cannot stop Surveillance

So one might ask, “how come the public has unintentionally conspired with governments and the technical community to eliminate privacy?” The answer is democracy.

I have so far not mentioned Edward Snowden, and have been working on the supposition that he needed no introduction. But let’s imagine a world where most countries operated on the principle that its laws were created by a group of people who were selected in a fair election by the adults in each country. These people also make executive decisions, such as managing roads and waging wars. They also decide who gets to be judge. Now imagine what would happen if these people decided to do something absolutely horrible, and never tell the public. How would we ever know? As long as the guise of democracy was maintained, we have no proof that they aren’t working for our benefit.

The only reason we know that the governments of this world have been waging war on us is because Edward Snowden told us. Oh, we had our suspicions, but we had no proof. And he gave us proof of activities being conducted against us that were way beyond anything we could have imagined. But, note, he only told us of the activities of the US and UK governments, and a bit about their Five Eyes partners. There still has been no Chinese Snowden, or Russian Snowden, or Indonesian, or Nigerian, or even Polish Snowden. There is an entire world of bad stuff being done behind our backs.

Stopping surveillance is impossible, because surveillance can happen without us knowing. Projects like PRISM and TEMPORA and Boundless Informant could, in theory, be defunded, but the technology already exists and can’t be un-invented. Even if the NSA were abolished, like the Black Chamber was in the 1920’s, the technological artefacts won’t be dismantled because there is no way to prove that they have been dismantled. You can’t dismantle a piece of software, you can just stop running it. But there’s no way to prove that other people aren’t still running it.

Moreover: abolishing the NSA would do nothing to reduce the capacity of the FSB, or GCHQ, or the BND. The US may be attacking us more than everybody else, but that doesn’t mean the others aren’t attacking us.

Since we cannot stop surveillance, we must learn to live with it. We need to learn to live in a world of perpetual information warfare, where states attack each other and all of them attack us. But that does not mean that we need to accept surveillance, or make it easy, or even allow the surveillors to get away with it. Not at all.

We cannot stop surveillance, but the good news is we don’t have to.

2. We can make Surveillance Expensive

The best thing we can do in this situation is make surveillance prohibitively expensive to maintain. In order to do that, we need to be very serious about our demands. We must demand decentralization, strong encryption, and hardened endpoints. But we must also demand political accountability.

Making surveillance economically expensive will reduce the activities of the surveillance agencies. Making surveillance politically expensive will reduce the activities of the governments and the corporations.

My current estimation of how much it costs to monitor everybody is about 25 cents per person per day. It’s a rough estimate, gotten by taking a rough estimation of the budget of the largest surveillance alliance, the Five Eyes, and dividing that number by the number of people who use the Internet. It’s changed a bit over the last year: I estimated it as being around 13 cents per person per day back when Snowden first revealed this activity to us. Since then, more and more people have been adopting strong encryption, even though it’s hard, people have made greater demands of security, and things have gotten a little bit better overall.

1. Surveillance does not happen in a (Political) Vacuum

Surveillance serves political ends. The objective is control, and we are the controlled. The logic of government is the logic of normalization. Only that which can be seen can be normalized. We must always be watched. If we are not watched, government cannot work.

This has been true throughout history. Surnames were created to give authorities a better understanding of who was who, so that people could be catalogued and taxed. We have passports and ID cards, so our flow can be controlled. Biometrics are becoming more and more popular. As technology has developed, the capabilities of humans have expanded, but so have the needs of the state to have perfect visibility.

That visibility extends not only to citizens of the state in question, but to all citizens of all states. In particular, those citizens who wield political power. Historically, those people are the kings and the presidents, but also the parliamentarians, and the state officials, and so on down. But now, for all the faults of the Social Surveillance Networks, they are facilitating greater communication, which is lending more political power to the public.

Surveillance is a weapon. We are, as a species, engaged in information warfare. Bellum Omnium Contra Omnes, Hobbes said, the war of all against all, could only be avoided if there were strong centralized governments. Because, he said, humans are not angels, and we cannot be trusted. As it happens, governments are not angels either. And those with much power can be trusted even less than those with none.

0. This is a Cold War that can Never End

I’ve been calling this information warfare, but the question remains whether this can be called a war at all. I posit it can: the Internet Engineering Task Force has defined pervasive surveillance as an attack. When person attacks a person, we call it a crime. When a state attacks another state, we call it war. When a state attacks its own people, we call it a civil war — no matter how uncivil it is. Incidentally, when the people retaliate against states, it is called terrorism.

But when nobody dies from the war, and instead of broken houses and broken lives we simply live in constant fear, we call it a cold war. This is a cold war, but it’s not like the last one. In the past, the nuclear bureaucracies of the world were engaged in a standoff against each other. Now, the old nuclear burueaucracies are engaged in a standoff against us. And we’re unarmed.

One of the most interesting documents generated during the cold war era was a document generally referred to as the Long Telegram. Written by George Kennan, it is the first document to suggest the US strategy of containment, whereby the USSR would be prevented from spreading its political influence or ideology, and would be allowed to rot from the inside until the point of collapse. It is effectively the cold equivalent of a war of attrition.

This is my Long Telegram. I am calling for a war of information attrition against those in this world who would seek to wield their power against the general population, in whatever form. This needs to happen on all levels, but the opening step involves rendering ourselves illegible to the surveillance state. It’s really easy: just be as confusing to the state as possible. Break the logic of the state. If it can’t understand you, it cannot fight you.

Where States Go to Die: Military Artifacts, International Espionage and the End of Liberal Democracy

This was originally published at the Center for a Stateless Society on October 12th, 2013. It is a transcript of a talk I gave at the SHARE Boat Camp in Croatia in August 2013, on board the Galeb

Military Artifacts

All over the world, landscapes both urban and rural are littered with military artifacts from bygone times. These artifacts have completed their lifecycle as objects of power, force and control, and have either been repurposed or forgotten.

Repurposed artifacts gain new meaning in the world, as they take on new roles. The former military base of Christiania in Copenhagen became a self-organizing free town. In Keflavík, a former US Navy base was converted into a university. In Florence, a former juvenile prison was turned into a safe haven for human rights defenders. In many places, former strongholds with relatively little public value have become tourist attractions, such as the tunnels inside the rock of Gibraltar, the castle in Ljubljana and the fortress of Komárom.

Some of these military artifacts don’t need to be explicitly repurposed to retain public value. In Europe, roads built by and for Roman armies up to two thousand years ago still form many of the transportation backbones of the continent. Without roads, there could be no trade.

But as time has gone on, military artifacts have become less amenable to public repurposing. While we might find some potentially beneficial use for the odd warship, the NORAD facility in Cheyenne Mountain isn’t going to become a theme park anytime soon, and despite Arnold Schwarzenegger’s suggestions, thermonuclear devices cannot be turned into snow cone makers. And while it is also conceivable that some guy might come along one day and convert an ICBM into a spaceship for faster-than-light travel, I’m not going to hold my breath.

Nuclear Democracy

Nuclear weapons are interesting artifacts. It is a matter of public record that almost ten thousand Nuclear weapons have been constructed. How many were constructed outside of the public record is anyone’s guess. Where they are is also an open question. A Nuclear device belonging to the US military was found in the sea off the coast of Greenland a couple of years ago, and nobody could publicly explain how it got there. And that’s the US – a country that at appears to have at least a vaguely competent military and relatively stable political atmosphere. Consider the artifacts left behind from the USSR. Not all visually accounted for, I’d venture to guess.

It has been said that the Nuclear bomb is a fundamentally undemocratic device: It has widespread impact, it is unspecific as to which humans it harms, it is expensive to source materials for and complicated to build. While an Ulam-style trigger mechanism is really just a question of getting enough dynamite in the right place, plutonium isn’t something you can pick up at the next convenience store.

Contrast these to rifles: Easy to build, easy to use, limited range and action, fairly focused on a particular target, unless you’re using an AK-47, in which case the only serviceable objective is chaos. As such, they are a much more democratic form of military artifact. Although they cannot directly be repurposed beyond a certain degree, there may be legitimate use for them outside of warfare.

What unites all of the artifacts I’ve mentioned so far is that they are physical. They can be visually accounted for. They exist in a scarcity-based economy. There is an upper limit to how many nukes can be built here on Earth, there is a way of counting them.

And as determined by the START treaty, there is a way to dismantle them. Nuclear disarmament was a hotly contested and highly useful goal near the end of the Cold War, although the topic has somewhat fallen out of fashion today. It’s as if people have come to terms with the idea of certain people having the ability to wipe out all of humanity at the blink of an eye. After Obama first took office, he went and had a conversation with Putin about disarmament, but there hasn’t been much media followup since then. Are there fewer nukes now than there were five years ago? I doubt it.

But an ICBM is a relatively hard thing to hide. This we know in part because if Scotland gets independence from the UK, the net number of Nuclear powers in the world remains constant, although the identity of one of them changes: the UK’s Nuclear stockpile is for the most part poorly hidden in the highlands. So if we did at some point get serious about disarmament, we’d know where to go, modulo some degree of military ingenuity and political madness.

Utopian Indulgence

With nukes, there is an exit strategy. In recent weeks, we have been granted some rather disturbing insights into the world of surveillance. We have heard of Prism, Boundless Informant, Tempora, and other things, the goal of which is not to spy on enemies of the state, but to spy on everybody on the assumption that we are all enemies of the state.

Let us indulge in a utopian form of escapism for a moment and posit the possibility that US President Barack Obama were to appear live on all the networks tonight, terrestrial and satellite, and declare that these catch-all surveillance programs would be abandoned forthwith, that all of the collected information – several hundred billion database entries – and all of the surveillance equipment would be destroyed.

If the US government had any credibility left, there would be instant jubilation. Peace would break out and victory would be declared, of some kind. But this is not the case. The US government was already running on the fumes of its credibility by the time Chelsea Manning exposed a shocking number of war crimes perpetrated in full knowledge of the upper echelons of the US government, and in terms of credibility it sputtered to an unceremonious halt when it was exposed that they had for at least seven years been conducting massive pervasive intrusions into the privacy of hundreds of millions of people around the world, violations against the trade secrecy afforded to companies globally, and quite literal invasions into the sovereignty of possibly every country on the planet.

This is not to say that all parts of the US government are rotten – not at all. On the contrary, many people within the US government or working for it are decent people with good intentions: The existence of people such as Edward Snowden, Chelsea Manning, Thomas Drake and Bill Binney is proof of this. The problem is not with the people, as such, it is with the structures and the behavior those structures breed.

If we return to our indulgence, the onus on the US government in this situation is to prove that they have dismantled their surveillance systems. But how could this be accomplished? There is no easy answer.

Dismantling Realpolitik

One of the fundamental challenges is that the US has ratcheted up their security apparatus to a point where any loosening would be construed by some as backing down. There are countries which might conceivably wish to take advantage of any weaknesses. There aren’t a lot of avenues for reduction.

One might argue that there is a possibility for the governments – and let’s remember that it isn’t just the US government, there’s the UK, Germany, France and many others – to back out of this surveillance quietly without alerting their enemies. But that would be moot – the public would not know, and thus public opinion would not be mended, and therefore little real benefit would come of it.

The understanding here is that any action taken by any of these governments now that does not lead to a better informed public on the one hand, and better protected rights to privacy on the other, are not going to be sufficient. So what are governments to do? There aren’t a lot of options.

The Death of the Republic

We have reached an impasse. On the one hand, the actions of the governments of these countries have rendered them entirely untrustworthy. On the other hand, their only avenue to regaining trust is to dismantle military artifacts that are not physical, cannot be visually accounted for, that exist in a post-scarcity economy, with no meaningful limit to how many surveillance systems can be in place and no way of counting them.

This is a catch-22. But we have seen this kind of stalemate arise before, numerous times in numerous empires, and they always had the same result. Some issue of contention comes up, ratcheting to the point where there is no feasible outcome. Politics be damned, military action is sometimes taken. Sometimes, it’s not country-on-country action. It’s the public using all of those repurposed artifacts to their own ends.

I am deeply worried by this possibility. While the little anarchist in me would be happy to see these governments replaced, I very much prefer soft landings. The republic as we know it needs an exit strategy. This means a few different things.

A Motion for Rebirth

First, we need some new way of creating structural transparency on the protocol level. This is to say that the institutions which service us must be capable of exposing their activities directly to the public through a complete analytical mechanism. In practice this would mean that people are granted the capacity to be as well informed as they see fit.

Second, we need some new way of aggregating political will. This essentially means better collective decision making mechanisms, systems of direct democracy that allow everybody to express their social choices in a way that does not disempower them. Most direct democracy systems fulfill the requirement of allowing everybody to participate, but few fulfill the requirement of giving everybody a say. This needs to change, and until it does, there is no reasonable expectation that people will wish to participate.

The third thing is slightly more cumbersome, and more related to this discussion of military artifacts. The world’s political economy has been constructed over many centuries, imbued the logic of empire. If you take any artifact from the economy, physical or electronic, military or civilian, the chances of its creation having involved the exploitation of humans somewhere are near certainty.

We need to figure out – and here I have no boilerplate solution – new organizational structures that don’t require exploitation. I know, I know. Slightly slipping back into Utopia here.

New Logic, New Artifacts

The hard problems are kind of obvious. We’re all here because we know that they need solving. Some look to the people standing on this deck for guidance and leadership in these issues. The reality is, nobody has the answers.

What we do know is that the logic of our current societies does not lead to equality, democracy and civility. It leads to Prism, Tempora and Boundless Informant. It leads to GCHQ, NSA, and BND. It leads to Tito, Obama and Lukaschenko.

We need a new logic. This logic will only come about by the elimination of the existing states, the states that have rendered themselves untrustworthy by their actions against us. But as assuredly as the current system has generated the military artifacts of our time, the new logic will produce new artifacts, both military and civilian, and it is up to us to repurpose them to the benefit of everybody.

Passing Over Eisenhower

This was originally published at the Center for a Stateless Society on the 18th of July 2013 — it feels like years ago, so much has happened in the interim. A Portuguese translation is available. I decided to repost it now because it came to mind recently while doing a bit of a retrospective, and realized I hadn’t cross-posted it.

The Internet industries of America may just have inadvertently had their hats handed to them by the military industrial complex. Now it’s up to Europe to provide an alternative to the surveillance state.

Almost all of the major Internet industry giants are based in the United States. The reasons for this are historical and economical. The tradition of strong entrepreneurship practiced in the US since their inception, mixed with their purchasing power and history of acquiring any sufficiently profitable venture or fascinating technology from abroad, has put the US into a prime position to be the global leader in provision of Internet services.

That may just have ended. While US dominance over the roughly $11 trillion/year global Internet services market is still unchallenged, the damage that the revelations made about NSA’s vast global surveillance scheme may stymie their growth and perhaps even turn them into a localized recession in coming months and years.

The reason for this is Europe. While some Europeans are becoming increasingly comfortable with the notion of living in a surveillance state, most people on the European mainland still grow up hearing stories of totalitarian dictatorships, wars, genocides, and the Holocaust, and have a natural inclination to detest the notion of secret police. As more is learned of the US’s secret spying games – aided in part, it seems, by their English counterparts – outrage boils thickly in countries like France and Germany, where despite highly open and inclusive societies in some senses, the notions of privacy as practiced in the United States have often been thought of as quaint. While modern discourse on privacy is dominated by the philosophical foundations of the 4th Amendment, a slightly different, somewhat more subtle understanding of privacy reigns in European discourse, with an annoyingly elusive definition.

Over coming months and years, the US government’s betrayal of the people of the world will spur a new industry in Europe, not aimed necessarily at pure technological innovation, but rather simply creating secure, privacy-respecting alternatives to the software services provided by the US based companies that can no longer be trusted. We will see Czech and Hungarian startups bringing out new search engines and Croatian and Polish companies developing secure e-mail services. We’ll undoubtedly see surveillance-resistant chat software coming out of Austria and global map databases being developed in Estonia. Or something like that.

This is not to say that Europe is ready to take on such a massive task. There is a lot of soul-searching that needs to happen, both culturally and politically in Europe: while privacy is a shared value in most of the continent’s corners, due to the lingering fear of a return to totalitarianism – fueled in no small part by the ascension of the likes of Hungarian prime minister Viktor Orbán to power – there is still a phantom of apprehension in the interactions between the tribes that make up Europe that seems to foreshadow balkanization. On top of this we have a schizophrenic political class that speaks of free trade one minute and restrictions the next, amongst whom are those who get raging hard-ons at the merest mention of censoring pornography or anything else they find offensive or overly stimulating.

That said, this may well turn out to be Europe’s decade in tech, and all because the United States failed to heed an important and timeless warning: “We must guard against the acquisition of unwarranted influence, whether sought or unsought, by the military industrial complex.” Eisenhower’s parting words to a nation being enveloped in a cold war were colder still, as a man who had seen a beast grow out of hand during his years in office was urgently pointing at the writing on the wall. But the years passed and the beast grew – premonitions turning to loathsome misery with each passing President who failed to stop the surveillance state.

And now, the military-industrial complex may have destroyed the US’s Internet-industrial complex.

Just as the last two thirds of humanity are preparing to transition into cyberspace, the NSA’s actions have revealed it to be far more of a Wild West than any government feels comfortable admitting. The rule of law breaks down really fast when there’s no clear monopoly on the legitimate use of violence. There are few acts as violent as stealing everybody’s secrets. Almost two hundred countries are screaming for legitimacy, but the one that stayed the most silent – except when berating, say, Iran, for not respecting “Internet freedom” – was the one whose legitimacy had already been eradicated by their violations of the values upon which their country was founded.

Passing over Eisenhower may have been the death-knell for American democracy, but it’s exposure may sound the beginning of a new era of human rights. Those coming online for the first time a few years or decades from now may be faced with a world altogether different from the one we now live in, perhaps partly in that they will have a choice between the monitored networks of Oceania or the liberal cryptarchies of Eurasia. The market will undoubtedly have its say in what happens after that.

For now though, there is a plan emerging. The hackers and the human rights activists, the net-freedom-blah people and the technophiles have been awakening from the post-Arab spring burnout and remembering the things that need to be done to prevent the next Mubarek. Better, simpler, more usable cryptography. Peer-to-peer, verifiable, anonymous monetary systems and democratic decision making systems. Secure communications and full transparency within governance.

During the transition to this new European future, a lot of data is going to have to be stored – refugee data seeking asylum from the terrors of the Anglo-American surveillance state. While the governments of Sweden and the UK may be somewhat too eager to share the data flowing through their resident data centers with their American pals, there are a few countries, notably Iceland, who are willing to provide a strong legal environment, cheap renewable energy, and good connectivity to the rest of the world. Data centers are not the future, but they are the present, and for now there’s an amazing business opportunity out there for countries who are willing to stand up and defend data sovereignty, the notion that individuals have the right to privacy and control over the data they generate.

To those who wish to practice data sovereignty before it becomes cool, I’d say: Come to Iceland. Bring data.

Some Thoughts Gnupg

Some thoughts on working with GnuPG

A lot of people have complained about OpenPGP for a number of valid cryptographical reasons1,2. It doesn’t change the fact that it is widely used, and wildly useful. It urgently needs to be replaced with something more sensible, but for now we’re stuck with it. In practice, this also means that we are stuck with GnuPG, the most common and by far the best implementation of OpenPGP.

GnuPG is the one and only reference implementation of RFC 4880, and despite thousands of companies making use of OpenPGP in their infrastructre there is for all intents and purposes a solitary dude in Germany trying to keep it all together. Werner Koch is an absolute hero for managing to do that, and deserves our respect and support. Financially supporting the GnuPG project is also something people should be doing.

The following is however neccessary and hopefully constructive criticism of GnuPG.

One of the things I’m largely to blame for in Mailpile is the GnuPG interface. It’s a chunk of Python code that executes the GnuPG binary, tosses information at it, and figures out what to do with the output. There are lots of libraries for doing this, but after a great deal of exploration I found that all of the Python libraries that did this were insufficient for our needs, and the only thing crazier than manually forking out GnuPG in our situation would be to use the PGPME library.

PGPME is almost as confusing and annoying as calling GnuPG directly, but it also requires us to ship architecture-specific libraries to everybody, something we’re actively avoiding. Having to ship GnuPG binaries to Windows and MacOS users is bad enough, but dependency hell is a place we want to stay out of. If we were writing Mailpile in, say, C or C++, then PGPME would definitely be the library of choice, but we’re not, so it isn’t. On top of that, the available Python bindings for PGPME are very flaky (last updated in 2008!), and not developed or maintained by the GnuPG team.

As a result, we’ve got a roughly 1200 line chunk of code in Mailpile that has the fun and useful task of chatting with GnuPG, and the stupifyingly annoying task of working around all of GnuPG’s inconsistencies.

The problems with GnuPG seem to fall roughly into two broad categories: inconsistent output structure, inconsistent interfaces. These are both ripe with surprising behaviour and confusing failure modes. In addition to these categories, it appears that the larger meta problem is that no single statement about its problems is going to remain a stable statement, as these problems disappear and reappear at odd intervals as new versions are being built. The number of moving parts essentially leads to a lot of confusion about whether a particular bug exists in a particular version or not, and whether it is affected by wind speed. To wit, I have over the course of Mailpile development added, removed, and readded a workaround for a bug, although I think I’m safe to say that it does not exist post GnuPG 2.1. The comment of that workaround in the code illustrates the issue perfectly:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
def list_secret_keys(self):
       #
       # Note: The "." parameter that is passed is to work around a bug
       #       in GnuPG < 2.1, where --list-secret-keys does not list
       #       details about key capabilities or expiry for
       #       --list-secret-keys unless a selector is provided. A dot
       #       is reasonably likely to appear in all PGP keys, as it is
       #       a common component of e-mail addresses (and @ does not
       #       work as a selector for some reason...)
       #
       #       The downside of this workaround is that keys with no e-mail
       #       address or an address like alice@localhost won't be found.
       #       Therefore, this paramter should be removed when GnuPG >= 2.1
       #       becomes commonplace.
       #
       #       (This is a better workaround than doing an additional
       #       --list-keys and trying to aggregate it though...)
       #
       #       BRE: Put --fingerprint at the front and added selectors
       #            for the worlds MOST POPULAR LETTERS!  Yaaay!
       #
       retvals = self.run(["--fingerprint",
                           "--list-secret-keys", ".",
                           "--list-secret-keys", "a",
                           "--list-secret-keys", "e",
                           "--list-secret-keys", "i",
                           "--list-secret-keys", "p",
                           "--list-secret-keys", "t",
                           "--list-secret-keys", "k"])
       return self.parse_keylist(retvals[1]["stdout"])

This bug exists in the first category:

Inconsistent output structure

GnuPG generally accepts command line parameters, uses these to perform actions, and returns output. The output generally takes two forms:

  1. Line by line descriptive output, such as when listing keys
  2. Bulk output, such as when encrypting, decrypting, or signing

The line-by-line output has two modes, the normal mode where the data is tabulated with spaces into mostly nice, if somewhat confusing columns, and the --with-colons mode, where the spaces are replaced with colons, for easy parsing. This is quite clever and good. The problem arises when one intends to start parsing this data.

First, a word on discoverability. If you ever intend to do anything with GnuPG, you first need to read and internalize a document aptly titled DETAILS, which contains a lot of the details about what’s going on with GnuPG output. I have dutifully read, memorized chunks of, and bookmarked this file for posterity. It is immensely helpful. For example, it gives an example of GnuPG’s output:

1
2
3
4
5
6
7
8
9
10
$ gpg --with-colons --list-keys \
      --with-fingerprint --with-fingerprint wk@gnupg.org
pub:f:1024:17:6C7EE1B8621CC013:899817715:1055898235::m:::scESC:
fpr:::::::::ECAF7590EB3443B5C7CF3ACB6C7EE1B8621CC013:
uid:f::::::::Werner Koch <wk@g10code.com>:
uid:f::::::::Werner Koch <wk@gnupg.org>:
sub:f:1536:16:06AD222CADF6A6E1:919537416:1036177416:::::e:
fpr:::::::::CF8BCC4B18DE08FCD8A1615906AD222CADF6A6E1:
sub:r:1536:20:5CE086B5B5A18FF4:899817788:1025961788:::::esc:
fpr:::::::::AB059359A3B81F410FCFF97F5CE086B5B5A18FF4:

In order to decipher what this all means, you need to refer to rest of the document. This shows the --with-colons format, which is the way we want to be working with it.

Now here comes issue the first: this is essentially a colon separated value (CSV!) data structure, but the data being provided is a) inconsistent, and b) structured.

Notably, the first output line says “there is a public key,” and the line after it says “here is a fingerprint.” Naively one might think that these are unrelated. But in fact, all of the lines from the one starting with pub up to the next one that starts with either pub or sec are actually details about the nature of the public key mentioned in the pub line – although to make things worse, the fpr lines after the sub lines refer to the sub line but not the pub line. Confused yet?

In reality, parsing this isn’t too terrible, but it can only be done in a reasonable way if you understand the structure of PGP keys and the output format of GnuPG. These are not reasonable assumptions for GnuPG to be making. Even armed with knowledge about the structure of keys and the handy DETAILS document, my first version of a parser was overly generic and terribly inefficient, because I kept trying to avoid inconsistencies.

Now, the inconsistencies start to get exciting around about here.

Notice these two lines:

1
2
pub:f:1024:17:6C7EE1B8621CC013:899817715:1055898235::m:::scESC:
fpr:::::::::ECAF7590EB3443B5C7CF3ACB6C7EE1B8621CC013:

These both follow the same output format, according to DETAILS. But look what happens when I add spaces to align the columns:

1
2
pub:f:1024:17:6C7EE1B8621CC013:899817715:1055898235::m:                                        ::scESC:
fpr: :    :  :                :         :          :: :ECAF7590EB3443B5C7CF3ACB6C7EE1B8621CC013:

Some of the columns are meaningless for some of the output lines, but more shockingly, some of the columns are MISSING sometimes. Three of the columns just simply evaporate if the line is an fpr-type line. On top of that, there’s no really good reason why the fingerprint needs to be a separate output line rather than just being added in at the right place. According to the DETAILS file, field 10 is for “User ID” – which is to say, the name, e-mail address, and comment associated with the key. Things that the fingerprint emphatically is not.

It this point you’ll notice that field 5 contains the Key ID. And for added pain, the key ID is variously the last 8 or the last 16 nibbles (hexadecimal digits) of the fingerprint.

Frustrated yet? Me too. But let’s just wave the rest of this category away, and move on to the next:

Inconsistent interfaces

So let’s imagine you want to generate a key. Sounds like a reasonable thing to do, right? So we’re all hip and cool and want to do so programatically with our shiny command line interface to GnuPG, so naturally we think it’ll look something like:

1
$ gpg --gen-key --name Smári McCarthy --email smari@mailpile.is --algorithm RSA --keysize 4096 --expires 2017-10-06

… or something to that effect. And have sensible defaults for any parameters that are skipped, or otherwise make them required. Right?

Wrong.

GnuPG does have a --gen-key flag, but when you call it you are dropped into an interactive interface where you are forced to answer questions, one at a time. In varying order, depending on the version, it seems.

The only sensible programmatic way to deal with this is to use “expect” style scripts, where your script captures the output and provides programmatic input depending on what the application last said. These used to be used a lot in the 80’s, but have fallen out of favour because: a) they make internationalization a nightmare, b) they make changing versions of software a nightmare, and c) they are almost never the right way to do anything.

They do work though. Kind of. Until they break, and it’ll be hell to debug them.

Now, avid users of GnuPG will at this point mention the --batch option, which allows in this case for providing options to the key generator in yet another format. Except, of course, that if you want to do something entirely reasonable like add more than one UID (for instance if you have multiple e-mail addresses) to a new key, you can’t. --batch just doesn’t support it.

So your options are to either painfully generate through using expect-style scripts, or use batch and then edit the key afterwards to add uids. Except that the --edit-key also relies on an interface which requires the use of expect-style scripts, so you just gained nothing.

Another thing that frequently happens when using encryption software with slow algorithms (such as secure pseudorandom number generation or RSA) is that you have to wait a long time for things to happen. When you’re making software with nice user interfaces, you sometimes start thinking that showing some kind of intermediate progress would be a nice thing to do. This is where we get to GnuPG’s wonderful status file descriptor.

Really, the status descriptor is awesome. It gives me lots of information that is valuable and can make life a lot better. There are however a few shortcomings. First, contrary to all other file descriptors that you may work with in GnuPG, the status descriptor is not guaranteed to give you a newline character at the end of a status, which renders a bunch of sensible methods of reading input from it unreliable and requires that I handle that descriptor with special magic. Nor are you strictly guaranteed to only get statuses. I have on occasion run into blank lines and other weirdness that needs to be stripped. Once those quirks are all managed, the status descriptor is actually invaluable and should not be overlooked — specially when mixed with the --enable-progress-filter flag.

The biggest complaint about the status descriptor is that it cannot be relied upon as a flow control mechanism. It does not always give output, or indicate the appropriate sequence of things, so an interface can use it for the purpose of increasing their information about the current situation, but not as a replacement for constant reading and parsing of the STDOUT and STDERR handles, and certainly not as a replacement for in-depth understanding of which order things happen in.

Actually, it should also be mentioned that as nice as it is to have all these descriptors, heavy use of descriptors turns into a world of problems on Windows. Windows is finicky enough as it is. Our solution was passing the status through to STDERR, which really works kind of fine.

Speaking of order, consider this handling of the passphrase descriptor — a special descriptor for accepting a passphrase sent by the user as part of a wrapper-mediated communication (because nobody ever uses pipes like that on the command line), from GnuPG’s gpg.c:

1
2
3
4
5
6
7
8
9
...
if( pwfd != -1 )  /* Read the passphrase now. */
read_passphrase_from_fd( pwfd );
...
switch (cmd)
  {
  case aPrimegen:
  case aPrintMD:
  ...

The interesting thing (aside from the annoying and dangerous lack of indentation on that if statement) is the way in which the passphrase is read from the password descriptor before the commands are managed. Which is to say, the passphrase must be sent, and, due to the way read_passphrase_from_fd is written, that descriptor closed on the sending end, before anything else happens. Which means that you need to know at the time of execution of the GnuPG binary that you need to send a passphrase, if you are going to do so programatically. This gives you three options: a) Send it every single time (requires storing the passphrase on the calling side, typically in insecure memory), b) Be willing to execute the same command twice, capturing potential errors on the first try and figuring out that they are due to a lack of passphrase — something the error message will not always be clear about, or c) keep track of the entirety of GnuPG’s internal state, which would be absolutely insane, even if it weren’t version dependent.

This behaviour is not obvious, or particularly reasonable, let alone documented. Figuring this out took a long time.

If you’ve seen Mailpile’s Windows and MacOS releases, you’ll have noticed that we are shipping slightly old versions of GnuPG. The reason for this is that we figured out pretty late that the passphrase-fd is not the correct way to do things and has been disabled in more recent versions of GnuPG in favour of expanded use of things that implement the gpg-agent mechanism. So Mailpile should be a gpg-agent.

(It is notable that several distributions still have GnuPG 1.4 as the default instead of GnuPG 2.x…)

The reason for this is that Mailpile provides a web interface, and in some of its use cases, it will do so from a server which is not necessarily capable of rendering a GTK window or provide a terminal prompt on the user’s device. So despite all of the reasons why people might not want to shift a PGP passphrase over a SSL connection, it might still be something people will want to do, and we need to be ready for that contingency. So we need to accept the passphrase through a web form, and pass it back to GnuPG one way or another. (Note: the generic case is Mailpile running on localhost, which is always a fine thing to do. Even over HTTP. Normal threat model limitations apply.)

All of this is weird and annoyingly inconsistent. This category of problems probably doubled our interface in size and complexity, and made error handling an absolute nightmare.

The Error Handling Issue

When writing a library like this, we need to be able to anticipate errors from GnuPG and respond appropriately. The number of different and confusing ways of receiving information also means that there are a number of different and confusing ways to receive error statuses and such. Sometimes the return value is useful, but frequently it is not. Sometimes there is something on the status descriptor, or on STDERR. Often both, sometimes neither. The entire thing is maddening.

The approach we’ve had to take is the opposite of what would be preferable. It is simply to check if the positive output we’re getting from GnuPG is roughly of the sort that we were expecting, and assume that if it isn’t, an error has occurred. As a general error handling strategy this is idiotic, we know, and we’d like it to stop.

What can be done?

The short answer is the same as Matt Green’s answer: It is time for PGP to die — or rather, RFC 4880 needs to be cleaned up, simplified, and replaced. PGP in its current form needs to evolve. There are a lot of very good reasons why, which Carlos has neatly catalogued. But realistically, PGP is what people use for e-mail, and until we have widespread adoption of crypto in e-mail at all, trying to replace PGP is just going to cause painful fragmentation. Since one of Mailpile’s goals is to get millions of people encrypting their e-mail by default, we can’t risk this fragmentation right now. If we round to the closest lakh, zero people currently encrypt their e-mail. This is scary and bad. The way forward is not to throw PGP out, but to start thinking seriously about what replaces RFC 4880.

But we’re stuck with RFC 4880. For now. A standard that is, for better or worse, being maintained entirely by one man.

Which gives us four options:

  1. Stick with GnuPG and improve it substantially.
  2. Fork GnuPG and improve it substantially.
  3. Replace GnuPG with something simpler and more consistent.
  4. Give up.

None of those approaches is good. I’m going to take option four off the table immediately because we’re not going to give up.

Option two is essentially the hostile version of option one, so I’ll write it off immediately. The people who’ve been developing GnuPG are great and we really like them. So we won’t be forking GnuPG anytime soon — heck, even if we did want to do that, we’d still not have any time to actually work on it.

Option three sounds most sensible long-term. Cruft is unavoidable, but Google’s End-to-End might potentially serve as the basis for “minimum viable PGP”. But End-to-End is also written in Javascript, and while people are entirely free to call me old-fashioned, I’d like the GnuPG replacement to be written in a compiled systems language.

But long term is long term. Short term, the only option is to stick with GnuPG.

I’d therefore like to propose the following:

GnuPG JSON Mode

As I mentioned, a lot of GnuPG’s output is actually structured a lot more than the output format supports. In our work so far, we’ve managed to build reasonable JSON structures out of that output for a lot of things. Completing that work and expanding on it, it would be possible to support something like this:

1
2
3
4
5
 $ gpg --json '{query}'
 {response1}
 {response2}
 ...
 {responseN}

This would be relatively easy to build atop of GnuPG’s current source code, making the --json flag preempt all else in the way --batch currently does. Then it uses a well supported library to parse the query, figure out what it is doing, call the appropriate internal functionality, and return the right data structures, also JSON encoded.

In order to support intermediate results, status descriptor style, an arbitrary number of results is allowed. They need not be comma separated, because we want our input parser to be able to pick them up one by one. Rather, just end each response block with a newline.

Have GnuPG exit after the last response.

With this, anybody implementing a GnuPG interface will be able to do all the magic relatively easily. The data structures can be well documented. Everything can become easy. I will stop losing my hair.

Somebody might ask, what about PGPME? Frankly, PGPME is great for a particular subset of GnuPG users. They can keep using it if they want. But if --json exists and is consistent and comprehensive, everybody will use that. Trust me.

Conclusion

GnuPG is important and great in many ways, but it is also deeply broken and downright dangerous. The sooner it becomes a consistent tool, the sooner it will become something other than a fool’s errand to attempt to interface with it. I’m happy to be on the caravan of fools for now, but only if there is something worthwhile at the end of this quest.

Software is hard. Security software is harder. Werner is doing great at managing a very shit situation, created by RFC 4880. I think there is a real possibility to make GnuPG way better. For now, we need JSON mode. I’m sure crowdfunding this work is possible, because we need it. I for one will put some cash down for this bounty. Join me?

Gluggað í Ríkisfjármálin

Ég asnaðist til að skoða ríkisfjármálin. Vitiði hvað kom í ljós?

Ríkisstjórnin sem boðaði aðhald í ríkisfjármálum hefur eytt umtalsvert meira en vonda fjárglæfrastjórnin sem sat áður.

Ríkisstjórnin sem boðaði lægri skatta hefur innheimt töluvert meira skattfé en vonda skattpíningarstjórnin sem sat áður.

Athugið að þetta er þrátt fyrir að ríkisstjórnin hafi afþakkað milljarðatugi í auðlindagjald frá handhöfum einokunarréttar á fiski.

Reyndar eru 25 milljarðar af tekjuaukningunni í formi sölu á hlutabréfum í eigu ríkisins.

Ég miða hér við árið 2013. Á fyrri hluta ársins 2013 var gamla ríkisstjórnin við lýði, fyrstu 5-6 mánuðina. Þetta er ekki sundurliðað eftir mánuðum, því miður. Því varð maður að bera þetta saman við nokkur fyrri ár til samanburðar, en ég á erfitt með að sjá annað en að það hafi mestmegnis verið nýja ríkisstjórnin sem breytti stefnunni.

Ríkisstjórnin kostaði 33.4% meira árið 2013 en árið 2012. Nákvæmlega tveir yfirflokkar kostuðu minna 2013 en 2012: Atvinnuvega- og nýsköpunarráðuneytið, og fjármagnskostnaður.

Atvinnuvega- og nýsköpunarráðuneytið lækkaði sig um 1%, að því er virðist aðallega með því að leggja niður fóðursjóð, húsbyggingasjóð, og minnka verulega framlög til byggðaáætlunar og iðju og iðnaðar (sem felur í sér átak til atvinnusköðunar og ýmis nýsköpunar- og markaðsmál).

Fjármagnskostnaður er peningur sem notaður er til að borga vexti (aðallega) af skuldum ríkisins, og lækkar helst ef vel gekk að borga af skuldum eða endurskipuleggja skuldirnar á árinu á undan.

Allt annað hækkaði.

Forsætisráðuneytið um 26.3%. Þar mátti helst nefna hækkun á fjárframlögum til aðalskrifstofu ráðuneytisins, töluverða hækkun á framlögum til óbyggðanefndar, þjóðminjasafnsins, og Þingvallaþjóðgarðs.

Fjármála- og efnahagsráðuneytið hækkaði um 19.7%. Þar mátti helst sjá 17 milljarða króna aukningu á afskriftum af skattakröfum, þar sem farið er um 11 milljarða umfram fjárheimild. Einnig er liður í því ráðuneyti sem ber titilinn “(óþekkt)”, þar sem tæplega 2 milljarðar hafa horfið.

Sumt eða allt af þessu kann að eiga sér ástæður, en erfitt er að sjá hvernig þetta telst vera aðhald í ríkisfjármálum.

Ég verð að viðurkenna að ég skil þetta illa. Það væri forvitnilegt að vita hvað formaður fjárlaganefndar hefur um þetta fyrirkomulag að segja, enda virðist ríkisstjórnin þrátt fyrir allt að vera að starfa vel innan fjárlaga á árinu 2013. Þetta voru auðvitað fjárlög sem voru sett 2012, þegar Björn Valur Gíslason var formaður fjárlaganefndar, en þetta er samt skrýtið og úr takti við árin á undan.

Það verður gaman að sjá hvernig þetta kemur út fyrir árið 2014 – en merkilegt nokk, þá hafa tölur fyrir árið ekki verið birtar, en það er ekki nein tæknileg ástæða fyrir því að það ætti ekki að vera hægt að birta útgjöld ríkisins jafnóðum.

Ríkisfjármál eru flókið mál. Hér er ég ekki að skammast í neinum, en mér finnst full ástæða til að draga athygli að þessu og spyrja spurninga.

Baul Bullukollanna

Það er ekki algengt í íslenskri stjórnmálaumræðu að hlutir séu sagðir með skýrum og afgerandi hætti svo ekki verði um villst. Því verður að teljast óþolandi þegar fullorðið fólk leikur sér að því að snúa út úr, þegar það er gert. Vandinn er að erfitt er að sanna að menn eins og Sigmundur Davíð Gunnlaugsson og Björn Bjarnason séu að snúa út úr, en séu ekki bara svona heimskir. Ýmislegt styður hvora tilgátuna.

Svo þetta sé gert alveg skýrt, enn og aftur:

Friðhelgi einkalífsins snýst um vernd hinna valdaminni frá misbeitingu hinna valdameiri.

Gagnsæi snýst um að opna hina valdameiri gagnvart eftirliti hinna valdaminni.

Upplýsingafrelsi snýst ekki um að allar upplýsingar séu opnar öllum alltaf, heldur að flestar upplýsingar séu opnar flestum alltaf, en sumar upplýsingar séu verndaðar, alltaf. Línan er dregin á skýrum stað: ef upplýsingar eiga erindi við almenning og það þjónar almannahagsmunum að þær séu opinberar, þá skulu þær vera opinberar. Ef upplýsingar eru persónulegar og koma engum við, þá skulu þær vera friðhelgar.

Það er ekki flókið að skilja þetta. Að forsætisráðherra landsins skuli eiga erfitt með að skilja einföld grunnatriði er grafalvarlegt. Neyðist maður til að spyrja sig hvaða önnur grunnatriði hann eigi í vandræðum með. Sem betur fer er Björn Bjarnason hættur að geta valdið skaða í íslensku samfélagi, nema með bauli sínu.

Crowdsourcing the Constitution - Lessons From Iceland

I was in Edinburgh some months ago visiting Bella Caledonia. I did this talk there, trying to give some history and background to the Icelandic constitutional process of 2010-2013, and putting it into a context of Scottish independence.

Suffice to say, I think Scotland should be independant. I say at least twice in this talk: EVERY reason that’s been given for people to vote “no” is invalid.

Maya Og óttinn

Í gær dó Maya Angelou, 86 ára gömul. Hún var kona sem barðist alla ævi sinni gegn mismunun. Vegna ótta annarra á hinu óþekkta fæddist hún, sem blökkukona í suðurríkjum Bandaríkjanna, inn í samfélag þar sem sumir máttu en aðrir ekki. Þessi aðgreining, sem var til komin vegna mannvonsku og fáfræði, ýtti undir fátækt, sem svo leiddi af sér glæpi.

Þegar hún var sjö ára gömul var henni nauðgað af kærasta móður sinnar. Hún sagði frá ódæðinu, sem varð til þess að æstur skríll drap nauðgarann. Hún öðlaðist við þetta sinn eiginn ótta – ótta við að orð hennar gætu haft alvarleg áhrif – og þagði hún því í sex ár þar á eftir.

Það er til fólk í öllum samfélögum sem nærist á ótta, eigin ótta eða ótta annarra. Þessi ótti er lamandi, hann tætir burt alla skynsemi og hamlar framförum.

Þessi ótti hefur ahrif á hegðun fólks. Hann veldur þröngsýni og fátækt í hugsun. Hann lætur fólk hverfa ofan í þjóðerni sitt, litarhaft eða trú. Lætur fólk reiðast þeim sem eru sér ólíkir, og spyrna gegn þeim. Í einhverjum tilfellum veldur það flóttahneigð: fólk skapar sér ímyndaðan heim þar sem það verður ekki vart við taugaveiklun sína gagnvart hinu óþekkta. Það var einmitt vegna þannig ótta sem Martin Heidegger kallaði eftir “rótfestu í hefðum sem tengjast stað og umhverfi sem eina öryggið sem býðst í pólitískum eða félagslegum aðgerðum í hættulegum heimi.”1

Aðgreining leyfir fáfræði um mismunandi félagslegar aðstæður og menningar að dafna, sem ýtir undir gróusögur, sögusagnir, og kolrangar staðalímyndir.

Popúlismi getur af sér popúlisma

Þegar fólk nærist á ótta annarra og hagnast á fordómum þess, þá kallast það popúlismi. Popúlistinn reynir að finna veikan blett, einhverja bólu í hugarfari náungans, og þrýsta á hann. Stundum kemur eitthvað slímugt út.

Popúlistinn er oft ekki að því vísvitandi: þeir eru sjaldan svo snjallir. Þeir athafna sig eftir sínum eigin ótta. Stundum er þessi ótti við fólk eins og Mayu Angelou, sem er öðruvísi á litinn en hinir hræddu. Stundum er þessi ótti við fólk eins og Harvey Milk, sem hefur aðra kynhneigð en hinir hræddu. Stundum er óttinn við fólk sem trúir á aðra guði, eða jafnvel sama guð undir öðru nafni. Eða fólk sem bara klæðir sig öðruvísi, eða talar annað tungumál.

En popúlistinn veit að hann getur ekki hagnast á sömu fordómum endalaust. Því þarf popúlistinn alltaf að víkka út. Bæði með því að víkka út eigin fordóma, en líka með því að skapa meiri ótta. Gera samfélagið beinlínis verra.

Þetta er gert með hólfun og skipulagningu. Allt á að vera á sínum stað, allt á að hegða sér rétt. Allt á að lúta stjórn. Eins og Vidler komst að orði eru nútímaborgir orðnar að “ímynd Taylorískrar framleiðslu”2. Edward T Relph sagði þessa hugmynd hafa leitt af sér samfélag sem var “afturhaldssamt, ljótt, sterílt, andfélagslegt, og almennt illa séð.”3

Popúlistinn hræðist það sem hann skilur ekki. Því gefur hann sér það hvernig allt virkar, og reynir að umraða heiminum í það líkan. Allt sem ekki passar er ýmist þröngvað inn í það, eða því er tortímt.

Einn daginn eru múslimar slæmir, og næsta dag eru það allir sem ekki eru kristnir. Næsta þar á eftir eru það einhverjir aðrir.

Frægt er ljóð Martins Niemöller: “first they came for the Socialists, but I did not speak out – because I was not a Socialist.” Muniði hvernig það endar?

Popúlistinn byrjar alltaf á einhverju einföldu, einhverju – eða einhverjum – sem öllum er sama um.

Það var enginn eftir til að tala fyrir þig

Það krefst hugrekkis að sigrast á ótta. Það krefst enn meiri hugrekkis að hafna popúlisma. Maya Angelou gerði hvort tveggja, og á langri ævi sinni sá hún heiminn breytast á ýmsa vegu, stundum til hins betra, stundum til hins verra.

Á þeirri tæpu öld sem hún lifði tók óttinn á sig margar birtingarmyndir. Maya Angelou byrjaði að tala á ný meðan seinni heimstyrjöld geisaði, á tíma þar sem milljónir létust vegna ótta. Stríðið kom til ekki síst vegna þess að fólk sem nærðist á ótta annarra náði yfirhöndinni yfir rökhyggju. Þetta er auðvitað einföldun, en skrefin voru þrjú:

  1. Hrun í fjármálakerfinu sem hafði alvarlegar afleiðingar fyrir afkomuöryggi fólks
  2. Vaxandi þjóðernishyggja, einangrunnarhyggja og annarskyns óttadrifin pólitík
  3. Heimsstyrjöld þegar það sauð upp úr milli nágrannaþjóða og þjóðarbrota

Við erum einu skrefi frá því að þurfa að horfa upp á annað blóðbað. Á vissan hátt er það þegar hafið: í Sýrlandi, í Úkraínu, í Tælandi. Eins í kosningunum í Evrópu um síðustu helgi, og kosningunum sem eru framundan á Íslandi, þá var óttadrifni popúlisminn aðal umræðuefnið. Það kemst ekkert að, nákvæmlega ekki neitt, nema hræðsluáróður, fordómar og viðbjóður.

Það kvarnast fljótt úr hugrekkinu þegar óttinn er allstaðar. En það er óskynsamt að óttast hið óþekkta, þegar hið þekkta er miklu verra: ef þessi óttadrifna alda popúlismans fær að halda áfram með sama hætti, þá er raunveruleg hætta á því að næsti umgangur verði ofbeldisfullur. Að samfélög sem höfðu öll heimsins tækifæri til að læra hvor af öðru og bæta sig taki sig í staðinn til og heyji stríð.

Það þarf ekki að gerast. Francis Fukuyama hafði rangt fyrir sér: sagan er ekki búin. Maya Angelou sigraðist á sínum ótta og varð ásamt Martin Luther King, Malcolm X og Nelson Mandela einn af risum mannréttindabaráttunnar. Þannig getur sagan okkar orðið. Hugrekkið getur tórað enn.


  1. Vitnað: Harvey, The Postmodern Condition, bls. 35.

  2. Anthony Vidler, The Third Typology.

  3. Edward T. Relph, The Modern Urban Landscape