Proxying Camlistore through nginx

I’ve went through this in order to secure access to Camlistore and delegate authorization to nginx. This doesn’t help, as I don’t yet have a securely stored password set up to protect it, but the first steps are there. Moving to something like LDAP-backed authorization for authentication on my machine and authentication of Camlistore is probably the way to go.

Without further ado, here is my partial nginx configuration, which should be useful if you already have a domain set up and you’d just like to direct toward Camlistore. I didn’t want to set up a subdomain, as I would have to get a new SSL certificate.

I’m bound to have made a mistake in configuration, so comments are welcome.

server {
        server_name ivan.vucica.net;
        access_log /var/log/nginx/ivan.vucica.net_access.log;
        error_log /var/log/nginx/ivan.vucica.net_error.log;
        root /somewhere/on/my/disk;
        listen 80;
        # skipped...

        location /camli {
                return 302 https://ivan.vucica.net:3180$request_uri;
        }
}
server {
        server_name ivan.vucica.net;
        access_log /var/log/nginx/ivan.vucica.net_access.log;
        error_log /var/log/nginx/ivan.vucica.net_error.log;
        root /somewhere/on/my/disk;
        listen 443 ssl;
        # skipped...

        location /camli {
                return 302 $scheme://ivan.vucica.net:3180$request_uri;
        }

        ssl_certificate /ssl/directory/on/my/disk/startssl-vucica.net.chained.crt;
        ssl_certificate_key /ssl/directory/on/my/disk/startssl-vucica.net.key;
        # ssl_protocols       SSLv3 TLSv1 TLSv1.1 TLSv1.2;
        # ssl_ciphers         HIGH:!aNULL:!MD5;

        # from: https://community.qualys.com/blogs/securitylabs/2013/08/05/configuring-apache-nginx-and-openssl-for-forward-secrecy#comment-3794
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_prefer_server_ciphers on;
        ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS";

        # for ssl cache - see http://nginx.org/en/docs/http/configuring_https_servers.html 
        keepalive_timeout   70;
}
server {
        server_name ivan.vucica.net;
        access_log /var/log/nginx/ivan.vucica.net_camli_access.log;
        error_log /var/log/nginx/ivan.vucica.net_camli_error.log;
        root /somewhere/on/my/disk;
        listen 3180 ssl;

        location /camli {
                rewrite ^/camli/(.*) /$1 redirect;
                rewrite ^/camli$ / redirect;
        }
        location / {
                proxy_pass http://127.0.0.1:3179;
                proxy_connect_timeout 5;

                # not using after all, as it would need access to /etc/shadow.
                # see http://web.iti.upv.es/~sto/nginx/ngx_http_auth_pam_module-1.3/README.html
                # auth_pam "Secured Camli";
                # auth_pam_service_name "nginx";

                auth_basic "Secured Camli";
                auth_basic_user_file /path/to/ivucica-camli-user_file;
        }

        ssl_certificate /ssl/directory/on/my/disk/startssl-vucica.net.chained.crt;
        ssl_certificate_key /ssl/directory/on/my/disk/startssl-vucica.net.key;
        # ssl_protocols       SSLv3 TLSv1 TLSv1.1 TLSv1.2;
        # ssl_ciphers         HIGH:!aNULL:!MD5;

        # from: https://community.qualys.com/blogs/securitylabs/2013/08/05/configuring-apache-nginx-and-openssl-for-forward-secrecy#comment-3794
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_prefer_server_ciphers on;
        ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS";

        # for ssl cache - see http://nginx.org/en/docs/http/configuring_https_servers.html 
        keepalive_timeout   70;
}

Camlistore itself is configured to listen only on 127.0.0.1. It doesn’t handle authentication, as using localhost authentication would require running nginx and camlistored under the same user. Adding a username+password authentication and using internal SSL would limit my long-term options for configuring authentication.

{
    "auth": "none",
    "listen": "localhost:3179",
    "identity": "AC5742DD",
    "identitySecretRing": "/path/to/camlistore/identity-secring.gpg",
    "blobPath": "/path/to/camlistore/blobs",
    "sqlite": "/path/to/camlistore/camli-index.db",
    "baseURL": "https://ivan.vucica.net:3180/",

    "shareHandler": true

}

To generate the .htpasswd file, refer to nginx documentation. Here follows an example; consider hard whether this is secure enough and appropriate for you.

printf "John:$(openssl passwd -crypt V3Ry)\n" >> .htpasswd # this example uses crypt encryption

Don’t forget that this will store the line in your .bash_history. (One way to avoid this specific issue is to prefix the command line with a space. Think hard whether this is enough for you.)

Mails appearing from d1stkfactory

If you have a Debian machine and emails appear to be coming from hostname @d1stkfactory, edit /etc/mailname and set this to your actual FQDN. Another instance where I found it locally is in /etc/exim4/update-exim4.conf.conf, which is almost certainly not read by my installation of Postfix (which does explicitly refer to /etc/mailname in one place).

This appeared on a machine running on DigitalOcean, so I presume “d” refers to DigitalOcean, and “factory” refers to their internal disk image building service. That’s just speculation though.

A European’s experience in [NOT] buying a MacBook Pro in US

I’m visiting NYC this week. I’ve come to US thinking hard whether or not I want to buy a MacBook Pro. I decided that yes, I do. Now I changed my mind. Why?

I wanted to do a couple of things. Use employee discount, pay with my Irish card, and I absolutely required a European-style keyboard layout. This last thing is the ultimate reason why I decided to skip it.

Apple is excellent if you are an average customer. Walk into one of the stores (there’s one 5 minutes of walk from me), pick up what you want, walk out happy. I could have done that. I can still do that. I don’t want to.

First, I want my discount. It saves me a non-insignificant an interesting amount of money. I doubt I can apply the discount from within the store. [UPDATE: Americans love to apply sales tax AFTER quoting you the price. So my total savings are a bit less than expected. Still, they exist. And my estimates include Irish VAT, because I'm nice like that. Smugglers would really save a lot.]

Second, I want a European-style keyboard (British at the very least). I have a US layout wireless keyboard, and if I switch to Croatian layout in software, I cannot type < and > on it.

Turns out that Apple does not stock laptops with different layouts in their stores. Turns out that I need to wait 1-3 business days for them to build the custom laptop (!!!) and then another 1-3 business days for them to expedite-ship it. Oh you want standard shipping? 3-5 days. You dared to desire to pick it up at the store (which should actually be easier for them)? Well, feelin’ bad for you buddy — it’s as if you picked standard shipping.

Maybe you wanted to pay with your Irish card? No go — they want your billing address phone number to be a US number, and your billing address zip code to be a US zip code and a US address.

Especially if you include the fact I’d need to deal with Irish customs people as well, that’s enough divine signs telling me I should skip this purchase. Maybe I change my mind, but I doubt it.

What kind of bullshit is not stocking some British-layout MBPs somewhere in NYC? What kind of bullshit is “it takes 1-3 days to replace a keyboard”? With minimum training, it could be done in-store, even if you couldn’t stock 2 variants in each store. They don’t sell often enough? Don’t keep 50 of them, but do keep 2 of them.

And I was so looking forward to a Mac with a decent GPU.

Ubuntu, touchscreen and dualscreen: HD Touchscreen on the left, laptop on the right

If you read about calibrating touchscreen with X11 and are capable of simple addition and division, you won’t have a problem building the transformation matrix and applying it. I won’t cover the same details on how to set the matrix using xinput set-prop since I just want to write down the resulting matrix and, perhaps, save people some time if they have a setup similar to mine.

To get the name of the device, open Terminal and punch in:

xinput

In the output I found that my “Acer T232HL”‘s input component is actually called “Advanced Silicon S.A CoolTouch(TM) System”. I can see all of its properties with:

xinput list-props Advanced Silicon S.A CoolTouch(TM) System

Since I’ve set up the screen to be on the left site, that means top-left corner of my touch area is simply offset by 0,0, and bottom-right is 1920,1080. Considering that the total width of the usable X area is 1920+1280, I get 1920/3600=0.6 as the c0 constant on the above link.

My usable area’s height is max(1080, 800) (the respective y component of the resolutions of the two displays). Considering that touch area’s height is also 1080, then the c2 component does not need to be scaled and should be exactly 1.

I also have absolutely no offsets, so c1 and c3 stay 0.

Result:

xinput set-prop "Advanced Silicon S.A CoolTouch(TM) System" --type=float "Coordinate Transformation Matrix" 0.6 0 0 0 1 0 0 0 1
# And if needed, to make the display configured as described above:
xrandr --output DP-0 --auto --left-of LVDS-0

From the times when I was trying to get Linux to run on iPaq H3800 devices using the kernel and distributions published at the now-defunct handhelds.org, I remember the “tslib” which included a touchscreen calibration tool. I wonder if that can and should be used to calibrate the modern multitouch displays.

Update (April 5th 2014): I just dug out package ‘xinput_calibrator’ by random googling, but have not tried it out yet.

Importing laptop into Ireland from outside EU

I am not a lawyer nor an accountant. This is not legal advice. I may be wrong.

Bureaucracy == exercise in exploration!

First, you want to read the general information for individuals, including PN 1878a – Customs and Excise Information for Travellers Arriving in Ireland from Countries Outside the European Union.

In PN 1878a there is an odd reference to digital cameras being 0% duty, and VAT 23%. Where does that come from?

In the section intended for businesses we can find the document A Guide to Customs Import Procedures (PDF) which mentions a system called TARIC. What is TARIC?

Apparently TARIC is an EU-wide system for determining duty on imports. Customs Directorate of Croatia (or whatever the name in English is) apparently also maintains their own TARIC query system with matching data.

Now we need the appropriate code. What is the code for laptops?

It’s 8471410000. If you enter this code and then drill down into ‘for use in civil aircraft’ and ‘other’, you’ll see that in both cases the duty is 0%, in most countries. (In Croatian mirror site, pick ‘Section XVI’ and ‘Chapter 84′. I find it curious that the Croatian system’s UI seems cleaner than the official EU system. More outdated, but nonetheless cleaner.)

Since USA is not on the list, I would presume it fits under the ‘Erga Omnes’ list.

Alright, what about VAT? There is a list of tax rates maintained by the Irish Tax and Customs. Laptops seem to fit under Computer Hardware – Supply, which has a Standard VAT rate. As of 1 January 2014, this rate is 23%.

My conclusion is that the duty is 0%, and that only VAT of 23% is paid.

Considering top-of-the-line 15″ Macbook Pro Retina costs $2.599 = 1877.21 EUR * 1.23 = approx 2309 EUR – versus 2649.00 EUR – we get the difference of 339 EUR that are still saved by buying the laptop from the US.

Fun. :-)

For the record, no, I did not choose to buy a laptop (nor this laptop in particular) — but exploring my options is good.

A few notes on Camlistore

(Useful as of April 7 2014, git commit 6842063f7c1ff6ae47e302b50b6a5030867cc2cd)

I was playing a bit with Camlistore and I like what I see. The presentation of Camlistore on FOSDEM 2014 done by Brad Fitzpatrick is quite insightful.

Here’s just a few notes so I don’t forget the caveats etc. I didn’t use a completely clean machine, so these are not instructions completely vetted up for public consumption. I presume a Debian Wheezy machine. Leave comments below, or mail me at blogのvucica.net.

Installing Go

Camlistore depends on Go language, at least Go 1.1. To install on Debian, as of early 2014 you need to use testing or unstable distributions. Here’s my /etc/apt/sources.list:

deb http://ftp.nl.debian.org/debian wheezy main
deb http://security.debian.org/ wheezy/updates main

deb http://ftp.nl.debian.org/debian testing main
deb http://ftp.nl.debian.org/debian unstable main

Here’s my /etc/apt/preferences.d/prioritize-stable (I prefer to use stable):

Package: *
Pin: release a=stable
Pin-Priority: 950

Package: *
Pin: release a=testing
Pin-Priority: 900

Package: *
Pin: release a=unstable
Pin-Priority: 800

Then use aptitude to choose golang version from testing. Hit e to use automated conflict resolver. With r, mark unsatisfactory solutions, then use , and . to browse different resolutions. At some point you’ll be offered correct installation of dependencies for golang. Hit ! to apply, then hit g twice to install.

Installing Camlistore

Nearly straight from the website:

git clone https://camlistore.googlesource.com/camlistore
cd camlistore/
go run make.go

If instructed to install any packages, do so. I know I was requested to install sqlite3 libraries.

Start camlistored:

./bin/camlistored

You’ll automatically get default config placed at ${HOME}/.config/camlistore/server-config.json and secret keyring at ${HOME}/.config/camlistore/identity-secring.gpg.

Interesting locations

  • ${HOME}/camlistore – you installed Camlistore here
  • ${HOME}/.config/camlistore – configuration files
  • ${HOME}/var/camlistore – blobstore

Preparing clients

Again taken from GettingStarted document and modified.

Upon startup, camlistored should have output the GPG key to use in the command below. For me, it was third line:

2014/03/04 23:26:24 Generated new identity with keyId "F300546B" in file /home/ivucica/.config/camlistore/identity-secring.gpg

Use this here:

./bin/camput init --gpgkey ${REPLACE_WITH_GPG_KEY}

Your new client configuration file should be at ${HOME}/.config/camlistore/client-config.json.

You’ll get output like this:

2014/03/04 23:29:29 Your Camlistore identity (your GPG public key's blobref) is: sha1-0b40618f0b2f6ff90ede6dc4ca7c5231eedf508b
2014/03/04 23:29:29 Wrote "/home/ivucica/.config/camlistore/client-config.json"; modify as necessary.

Web UI

Web UI will be available at http://${hostname}:3179. If accessing from local machine, use http://localhost:3179.

If you are visiting from a remote server, edit ${HOME}/.config/camlistore/server-config.json and under auth set this: userpass:alice:secret:+localhost (replace ‘alice’ and ‘secret’). This uses the completely insecure PLAIN authentication; use this only as a last resort. If you absolutely need this, you should consider turning on HTTPS, too, or reverse-proxying using nginx with HTTPS, so that you’re at least not leaking the password over the wire. Your password is also, obviously, stored locally in plaintext. This is all a BAD idea, but okay for testing.

Uploading blobs, files and directories

./bin/camput blob bin/README  # uploads a raw blob with no metadata; no filename, no nothing
sha1-4a8b6d44e3030bf36f39f0e0415209f8319ce019

./bin/camput blob bin/README  # note -- same output
sha1-4a8b6d44e3030bf36f39f0e0415209f8319ce019

BLOBHASH=$(./bin/camput blob bin/README)  # store hash in a variable...
./bin/camget ${BLOBHASH}  # ... then get this blob and print it out:
This is where Camlistore binaries go after running "go run make.go" in
the Camlistore root directory.

./bin/camtool list  # observe: just one blob, size 102
sha1-4a8b6d44e3030bf36f39f0e0415209f8319ce019 102

./bin/camput file bin/README  # uploads file data + creates metadata blob
sha1-129583b2870e357b17468508c4b8345faf228695

./bin/camtool list  # Note, no new content blob (content is the same) -- but there is the metadata blob
sha1-129583b2870e357b17468508c4b8345faf228695 356
sha1-4a8b6d44e3030bf36f39f0e0415209f8319ce019 102

./bin/camget sha1-129583b2870e357b17468508c4b8345faf228695
{"camliVersion": 1,
  "camliType": "file",
  "fileName": "README",
  "parts": [
    {
      "blobRef": "sha1-4a8b6d44e3030bf36f39f0e0415209f8319ce019",
      "size": 102
    }
  ],
  "unixGroup": "ivucica",
  "unixGroupId": 1000,
  "unixMtime": "2014-03-04T23:21:29.152341812Z",
  "unixOwner": "ivucica",
  "unixOwnerId": 1000,
  "unixPermission": "0644"
}

This file is not a permanode so it didn’t appear in the web UI.

./bin/camput file --permanode bin/README
sha1-129583b2870e357b17468508c4b8345faf228695
sha1-55e987dc3d81da6e2188b45320d7ee9e29c1366e
sha1-60a25fc96f7ee1830a9e95ecfc4c2fc80da65371

Now if you visit the WebUI, you’ll see the file appeared there. (You may also see it appear live if you had the web UI open.)

First hash is the file metadata, second is a claim, and the third one is the permanode itself, which you can use in the web UI to reach the content.

What is a claim? Quoting:

If you sign a schema blob, it’s now a “signed schema blob” or “claim”. The terms are used pretty interchangeably but generally it’s called a claim when the target of the schema blob is an object’s permanode (see below).

Signing involves claiming interest by a user over a blob. Signing is done with the previously created GPG key. It is my probably incorrect understanding that blobs without owners could be garbage collected. I can’t find the documentation to verify this; please leave a comment or email me whether this is correct or incorrect.

To upload a directory and simultaneously provide a title and some tags, as well as create a permanode:

./bin/camput file --permanode --title="Camlistore Documentation" --tag=documentation,camlistore ./doc/
sha1-ff227e6af6647d2c3958824a036d7a362a89d675
sha1-36483f956312bc82bfbca307739e1d2e225ccc08
sha1-6a57222b2a1a5338a11a02ba3a34fd3b43bb92ee
sha1-8cd8a87d82af1dd12fb7dce1c21510420ec8b419
sha1-3f22e0468104c908292d07271d744cd95f925d52
sha1-75d3db43c8b07bf82d601d65375ec64f7769a4bf

Search system’s camtool describe

Ask the search system about an object:

{
  "meta": {
    "sha1-75d3db43c8b07bf82d601d65375ec64f7769a4bf": {
      "blobRef": "sha1-75d3db43c8b07bf82d601d65375ec64f7769a4bf",
      "camliType": "permanode",
      "size": 562,
      "permanode": {
        "attr": {
          "camliContent": [
            "sha1-ff227e6af6647d2c3958824a036d7a362a89d675"
          ],
          "tag": [
            "documentation",
            "camlistore"
          ],
          "title": [
            "Camlistore Documentation"
          ]
        },
        "modtime": "2014-03-04T23:55:05.941276174Z"
      }
    }
  }
}

Mounting Camlistore filesystem

You won’t be mounting the Camlistore filesystem; you’ll be mounting a Camlistore filesystem. But first, we need to prepare you a bit.

cammount uses FUSE. Mounting (and FUSE) needs appropriate root-level permissions to, well, mount a filesystem. So, let’s use sudo -E (-E to maintain ${HOME} et al so Camlistore can find configuration)?

Not so fast. The localhost auth that is still used here is doing checks on several levels. First, you need to be on the local host (check!). Second, the socket connecting to camlistored needs to come from the same UID as the one that is running camlistored. Your sudo‘ed cammount would be running with UID=0; your camlistored runs under 1000 or more. (Presuming Debian Wheezy here.) So, camlistored rejects the auth. Don’t bother sudo‘ing camlistored; it doesn’t want to be run as root.

Solution?

sudo -E adduser ${USER} fuse

Yes, you just add the user to the fuse group. You now need to create a new login session (e.g. log out and log in is fine) so the appropriate subsystems notice you are a member of a new group. (“Hey, you never need to reboot Linux.”)

Alright, now we’re good to go. Find the hash of a permanode you want to mount. You can use web UI to do so. It’ll even have a handy instruction on what to do! Quoting from web ui:

./bin/cammount /some/mountpoint sha1-ff227e6af6647d2c3958824a036d7a362a89d675

Or more sanely:

mkdir -p mountpoint/
./bin/cammount mountpoint/ sha1-ff227e6af6647d2c3958824a036d7a362a89d675

Open a new session, and observe how there is actual content in the mountpoint/ directory!

After shutting down cammount with ctrl+c, you might need to unmount using:

fusermount -u mountpoint/

Use mount to see if you need to do that.

If you skip the permanode, you’ll get a special filesystem with virtual directories such as at, date, recent, roots, tag… and even though you can’t see that directory, you can cd sha1-ff227e6af6647d2c3958824a036d7a362a89d675 and see the same directory as before.

Closing notes

  • I haven’t looked at alternative indexing engines or alternative blobstores. For example, Amazon S3, Google Drive, Google Cloud Storage, …
  • I haven’t looked at blobstore sync.
  • I haven’t managed to correctly use ‘publishing’, especially for /pics/ (which is an example given in the documentation).
  • I haven’t shown you camput attr which sets custom attributes on permanodes.
  • I find the idea of importers, especially IMAP importers, very appealing. The Flickr one is already there. An IMAP importer (at 51:40) and an IMAP server (at 51:57) was mentioned in the FOSDEM 2014 talk.
  • I also find the idea of a built-in SMTP server quite appealing (mentioned at 32:12).

Given that the Camlistore filesystem is mountable and splits files into chunks a-la GFS (although in chunks that are far smaller in size than 64MB quoted in the GFS whitepaper), and given that the talk was given as recently as February 2014, and given that there is enthusiasm on author’s side, I’m really looking forward to something that could truly grow into a personalized distributed filesystem.