Author Archives: Ivan Vučica

About Ivan Vučica

Google+ - ivan.vucica.net/

A few notes on Camlistore

(Useful as of April 7 2014, git commit 6842063f7c1ff6ae47e302b50b6a5030867cc2cd)

I was playing a bit with Camlistore and I like what I see. The presentation of Camlistore on FOSDEM 2014 done by Brad Fitzpatrick is quite insightful.

Here’s just a few notes so I don’t forget the caveats etc. I didn’t use a completely clean machine, so these are not instructions completely vetted up for public consumption. I presume a Debian Wheezy machine. Leave comments below, or mail me at blogのvucica.net.

Installing Go

Camlistore depends on Go language, at least Go 1.1. To install on Debian, as of early 2014 you need to use testing or unstable distributions. Here’s my /etc/apt/sources.list:

deb http://ftp.nl.debian.org/debian wheezy main
deb http://security.debian.org/ wheezy/updates main

deb http://ftp.nl.debian.org/debian testing main
deb http://ftp.nl.debian.org/debian unstable main

Here’s my /etc/apt/preferences.d/prioritize-stable (I prefer to use stable):

Package: *
Pin: release a=stable
Pin-Priority: 950

Package: *
Pin: release a=testing
Pin-Priority: 900

Package: *
Pin: release a=unstable
Pin-Priority: 800

Then use aptitude to choose golang version from testing. Hit e to use automated conflict resolver. With r, mark unsatisfactory solutions, then use , and . to browse different resolutions. At some point you’ll be offered correct installation of dependencies for golang. Hit ! to apply, then hit g twice to install.

Installing Camlistore

Nearly straight from the website:

git clone https://camlistore.googlesource.com/camlistore
cd camlistore/
go run make.go

If instructed to install any packages, do so. I know I was requested to install sqlite3 libraries.

Start camlistored:

./bin/camlistored

You’ll automatically get default config placed at ${HOME}/.config/camlistore/server-config.json and secret keyring at ${HOME}/.config/camlistore/identity-secring.gpg.

Interesting locations

  • ${HOME}/camlistore – you installed Camlistore here
  • ${HOME}/.config/camlistore – configuration files
  • ${HOME}/var/camlistore – blobstore

Preparing clients

Again taken from GettingStarted document and modified.

Upon startup, camlistored should have output the GPG key to use in the command below. For me, it was third line:

2014/03/04 23:26:24 Generated new identity with keyId "F300546B" in file /home/ivucica/.config/camlistore/identity-secring.gpg

Use this here:

./bin/camput init --gpgkey ${REPLACE_WITH_GPG_KEY}

Your new client configuration file should be at ${HOME}/.config/camlistore/client-config.json.

You’ll get output like this:

2014/03/04 23:29:29 Your Camlistore identity (your GPG public key's blobref) is: sha1-0b40618f0b2f6ff90ede6dc4ca7c5231eedf508b
2014/03/04 23:29:29 Wrote "/home/ivucica/.config/camlistore/client-config.json"; modify as necessary.

Web UI

Web UI will be available at http://${hostname}:3179. If accessing from local machine, use http://localhost:3179.

If you are visiting from a remote server, edit ${HOME}/.config/camlistore/server-config.json and under auth set this: userpass:alice:secret:+localhost (replace ‘alice’ and ‘secret’). This uses the completely insecure PLAIN authentication; use this only as a last resort. If you absolutely need this, you should consider turning on HTTPS, too, or reverse-proxying using nginx with HTTPS, so that you’re at least not leaking the password over the wire. Your password is also, obviously, stored locally in plaintext. This is all a BAD idea, but okay for testing.

Uploading blobs, files and directories

./bin/camput blob bin/README  # uploads a raw blob with no metadata; no filename, no nothing
sha1-4a8b6d44e3030bf36f39f0e0415209f8319ce019

./bin/camput blob bin/README  # note -- same output
sha1-4a8b6d44e3030bf36f39f0e0415209f8319ce019

BLOBHASH=$(./bin/camput blob bin/README)  # store hash in a variable...
./bin/camget ${BLOBHASH}  # ... then get this blob and print it out:
This is where Camlistore binaries go after running "go run make.go" in
the Camlistore root directory.

./bin/camtool list  # observe: just one blob, size 102
sha1-4a8b6d44e3030bf36f39f0e0415209f8319ce019 102

./bin/camput file bin/README  # uploads file data + creates metadata blob
sha1-129583b2870e357b17468508c4b8345faf228695

./bin/camtool list  # Note, no new content blob (content is the same) -- but there is the metadata blob
sha1-129583b2870e357b17468508c4b8345faf228695 356
sha1-4a8b6d44e3030bf36f39f0e0415209f8319ce019 102

./bin/camget sha1-129583b2870e357b17468508c4b8345faf228695
{"camliVersion": 1,
  "camliType": "file",
  "fileName": "README",
  "parts": [
    {
      "blobRef": "sha1-4a8b6d44e3030bf36f39f0e0415209f8319ce019",
      "size": 102
    }
  ],
  "unixGroup": "ivucica",
  "unixGroupId": 1000,
  "unixMtime": "2014-03-04T23:21:29.152341812Z",
  "unixOwner": "ivucica",
  "unixOwnerId": 1000,
  "unixPermission": "0644"
}

This file is not a permanode so it didn’t appear in the web UI.

./bin/camput file --permanode bin/README
sha1-129583b2870e357b17468508c4b8345faf228695
sha1-55e987dc3d81da6e2188b45320d7ee9e29c1366e
sha1-60a25fc96f7ee1830a9e95ecfc4c2fc80da65371

Now if you visit the WebUI, you’ll see the file appeared there. (You may also see it appear live if you had the web UI open.)

First hash is the file metadata, second is a claim, and the third one is the permanode itself, which you can use in the web UI to reach the content.

What is a claim? Quoting:

If you sign a schema blob, it’s now a “signed schema blob” or “claim”. The terms are used pretty interchangeably but generally it’s called a claim when the target of the schema blob is an object’s permanode (see below).

Signing involves claiming interest by a user over a blob. Signing is done with the previously created GPG key. It is my probably incorrect understanding that blobs without owners could be garbage collected. I can’t find the documentation to verify this; please leave a comment or email me whether this is correct or incorrect.

To upload a directory and simultaneously provide a title and some tags, as well as create a permanode:

./bin/camput file --permanode --title="Camlistore Documentation" --tag=documentation,camlistore ./doc/
sha1-ff227e6af6647d2c3958824a036d7a362a89d675
sha1-36483f956312bc82bfbca307739e1d2e225ccc08
sha1-6a57222b2a1a5338a11a02ba3a34fd3b43bb92ee
sha1-8cd8a87d82af1dd12fb7dce1c21510420ec8b419
sha1-3f22e0468104c908292d07271d744cd95f925d52
sha1-75d3db43c8b07bf82d601d65375ec64f7769a4bf

Search system’s camtool describe

Ask the search system about an object:

{
  "meta": {
    "sha1-75d3db43c8b07bf82d601d65375ec64f7769a4bf": {
      "blobRef": "sha1-75d3db43c8b07bf82d601d65375ec64f7769a4bf",
      "camliType": "permanode",
      "size": 562,
      "permanode": {
        "attr": {
          "camliContent": [
            "sha1-ff227e6af6647d2c3958824a036d7a362a89d675"
          ],
          "tag": [
            "documentation",
            "camlistore"
          ],
          "title": [
            "Camlistore Documentation"
          ]
        },
        "modtime": "2014-03-04T23:55:05.941276174Z"
      }
    }
  }
}

Mounting Camlistore filesystem

You won’t be mounting the Camlistore filesystem; you’ll be mounting a Camlistore filesystem. But first, we need to prepare you a bit.

cammount uses FUSE. Mounting (and FUSE) needs appropriate root-level permissions to, well, mount a filesystem. So, let’s use sudo -E (-E to maintain ${HOME} et al so Camlistore can find configuration)?

Not so fast. The localhost auth that is still used here is doing checks on several levels. First, you need to be on the local host (check!). Second, the socket connecting to camlistored needs to come from the same UID as the one that is running camlistored. Your sudo‘ed cammount would be running with UID=0; your camlistored runs under 1000 or more. (Presuming Debian Wheezy here.) So, camlistored rejects the auth. Don’t bother sudo‘ing camlistored; it doesn’t want to be run as root.

Solution?

sudo -E adduser ${USER} fuse

Yes, you just add the user to the fuse group. You now need to create a new login session (e.g. log out and log in is fine) so the appropriate subsystems notice you are a member of a new group. (“Hey, you never need to reboot Linux.”)

Alright, now we’re good to go. Find the hash of a permanode you want to mount. You can use web UI to do so. It’ll even have a handy instruction on what to do! Quoting from web ui:

./bin/cammount /some/mountpoint sha1-ff227e6af6647d2c3958824a036d7a362a89d675

Or more sanely:

mkdir -p mountpoint/
./bin/cammount mountpoint/ sha1-ff227e6af6647d2c3958824a036d7a362a89d675

Open a new session, and observe how there is actual content in the mountpoint/ directory!

After shutting down cammount with ctrl+c, you might need to unmount using:

fusermount -u mountpoint/

Use mount to see if you need to do that.

If you skip the permanode, you’ll get a special filesystem with virtual directories such as at, date, recent, roots, tag… and even though you can’t see that directory, you can cd sha1-ff227e6af6647d2c3958824a036d7a362a89d675 and see the same directory as before.

Closing notes

  • I haven’t looked at alternative indexing engines or alternative blobstores. For example, Amazon S3, Google Drive, Google Cloud Storage, …
  • I haven’t looked at blobstore sync.
  • I haven’t managed to correctly use ‘publishing’, especially for /pics/ (which is an example given in the documentation).
  • I haven’t shown you camput attr which sets custom attributes on permanodes.
  • I find the idea of importers, especially IMAP importers, very appealing. The Flickr one is already there. An IMAP importer (at 51:40) and an IMAP server (at 51:57) was mentioned in the FOSDEM 2014 talk.
  • I also find the idea of a built-in SMTP server quite appealing (mentioned at 32:12).

Given that the Camlistore filesystem is mountable and splits files into chunks a-la GFS (although in chunks that are far smaller in size than 64MB quoted in the GFS whitepaper), and given that the talk was given as recently as February 2014, and given that there is enthusiasm on author’s side, I’m really looking forward to something that could truly grow into a personalized distributed filesystem.

On my final project for university

I’m completing my studies at the Polytechnic of Zagreb this semester, and my final project will be a compositing window manager (as a follow up to my text on the same subject written for the ‘Report’ class).

In discussion with my mentor, I wanted to double check the well-known fact that the university will be the owner of the resulting work. Yes, the university will claim ownership of the resulting work.

The follow up question was whether the resulting project could be GPLed. In that way, I would be able to continue working on the project later on. The answer is — no, I can’t GPL it. I might be able to wrestle with the bureaucracy and get a special exemption, but I’ve decided not to.

To the best of my knowledge, same policies exist at the Faculty of Computing and Electrical Engineering at the University of Zagreb — widely considered one of the best university-level schools in Croatia, if not the best.

I’m highly disappointed by the Croatian universities’ policy of appropriating work I am forced to do for purposes of acquiring a degree.

I have nothing against appropriating the accompanying paper, which is something that will not evolve further once written. I am highly frustrated by the application of same standards on potentially useful, potentially fast changing program code.

As a result, and as a form of protest, the software part of my project will be experimental, proof-of-concept research-quality code, and I will not try too hard to make it maintainable long term. The paper and the project will not be intentionally worse, but they will also not be intentionally better than they could be. This is because I want a clear road without obstacles whenever I decide to create a well-structured compositing window manager; I don’t want any obstacles to being able to modify my own code.

I am certain that numerous student developers in countries with similar practices do the same. This results in enormous waste of time that would, in the academic spirit of information sharing, better serve contributing to free software ecosystem. Instead, who knows how much code is either useless in the real world, or — worse — is actually useful in the real world, but claimed by the universities as their product?

I would suggest the responsible individuals in Croatia to review the history of Google, including the part where BackRub and PageRank Googol Google was a government-funded research project by two PhD students, yet they managed to take it out of the university and start a big company around it.

I would also suggest a review of Ayn Rand’s Atlas Shrugged which, while flawed and too narrow-focused, does offer some food for thought on government appropriations of works. And in cases of academic software projects that would be GPLed anyway, the situation is even worse: we’re seeing an appropriation of a work that I would gladly share with the world and previously intended to do so.

Ubuntu 13.10 live CD: Blank screen with EFI

Ubuntu 13.10 fails to start X11 on a Macbook Pro with retina display, and it fails to start X11 on VirtualBox when EFI mode is turned on. Even the failover mode fails. This has been tried with 64-bit version of Ubuntu 13.10. Machines: a 2013 Macbook Pro; and a virtual VirtualBox 4.3.6 machine configured for Ubuntu (64-bit) OS, with EFI turned on.

Ubuntu’s failover configuration tries to use vesa module, which is not available when running under native EFI or UEFI mode.

Let’s fix this by using fbdev module.

  1. Hit ctrl+alt+f1 to switch to console.
  2. Type sudo -i to become root.
  3. Now let’s fix the relevant files:
    cd /etc/X11/
    sed 's/Driver.*"vesa"/Driver "fbdev"/' xorg.conf.failsafe > xorg.conf
  1. Restart X11 et al: service lightdm restart
  2. If necessary, switch to the VT dedicated to X11: hit ctrl+alt+f7

Note that the screen will stay blank for a while longer; give the system some time to proceed.

Good luck!

Few notes on UPC Ireland’s Technicolor TC7200

Based on this thread which I ran into here. These are just notes; if you can learn something from them, do so; but please be careful and fully aware that these are not instructions — merely notes for my own personal use in the future. I do not recommend you follow the notes; quite the opposite.

Despite (according to their reps on Twitter) telnet being unsupported by UPC, you can easily telnet 192.168.100.1 and use username: webstar, password: webstar to log into the console. There isn’t much to do there, sadly. Apparently you may be able to use the set command to control SNMP settings.

Some configuration of hidden features can be done via SNMP. I highly advise you not to do that. This is unsupported by UPC, and you might be unable to restore settings to the previous state.

To switch the device into pure modem, non-routing mode, supposedly you use:

snmpset -v2c -c public 192.168.100.1 1.3.6.1.4.1.4413.2.2.2.1.7.1.1.0 i 1

This didn’t work for me immediately. It seems to work immediately after reboot. Also worth noting is the snmpwalk command:

snmpwalk -v2c -c public 192.168.0.1 1.3.6.1.4.1.4413.2.2.2.1.7.1.1.0

Note that the value of “1” means “modem mode” (also known as “bridge mode”), while “2” means “ordinary NAT+routing mode”.

Be very careful. Turning on “modem mode” breaks Wi-Fi. You may need to bring up a separate access point for Wi-Fi. You may have trouble restoring the setting. PUBLISHING OF THESE NOTES DOES NOT CONSTITUTE ENDORSEMENT TO ACTUALLY USE THEM.

Also, I was unable to restore the setting to “2” using SNMP. Hard reset by holding the reset button for 30 seconds worked. Unless you use it strictly as a modem, with another device that performs NAT and IPv6 routing, it isn’t worth switching to NAT mode. So please don’t do this unless you’re fully prepared for breakage of service and possibly annoying UPC (since they obviously don’t want people to play with this).

It’s interesting that UPC does assign a public, fully routable IPv6 /64 prefix. Too bad that it’s not exposed to regular users locked behind a NAT who cannot get direct access to the public Internet, and who have been locked out of it by a firmware that is seemingly arbitrarily restricted compared to the one used in Netherlands. UPDATE: Two sources have told me that they did not receive an IPv6 prefix or address. As I am not keeping my device in bridge mode, and I do not advise readers to do so either (unless they have very good reasons), I cannot and won’t verify this. You can opt to leave a comment on your success, but I want to be clear that neither my text, nor any comments below, represent any advice on my part.

It’s also interesting that the setting seems to have survived a factory reset through the web interface. Factory reset through web interface has, however, restored WiFi functionality, and it seems that WiFi and LAN are getting separate IP addresses. Factory reset by holding the reset button for 30 seconds is the actual hard reset and cleans up the setting for “modem mode”.

Switched to nginx

I’ve switched the server to nginx. I don’t have too many htaccess-based rules, and overall the Apache2 configuration was very simple to begin with. I installed php5-fpm, followed tutorials on configuring nginx (moving what’ll be common among 15ish virtual hosts into a separate file), configured proxying for XMPP BOSH and finally improving SSL security.

Apache2 now sleeps quietly and the RAM usage is down a bit. Hopefully the server won’t suffer any more hard crashes due to lack of RAM like it did immediately after I moved to the new host.

Overall, nginx seems like a much lighter and easier to understand solution for my very simple needs; if you’re running a simple server, I recommend it (as long as you read a bit about its security, if you intend to use it for serving PHP!).

On the referendum

For me, the issue was not (bluntly) whether the gay marriage is good or not.

The issue is whether stupidity and ignorance will prevail.

Well, now we see how damaging can changing referendum rules be. You see, some time before Croatia entered EU, constitution was amended to permit referendum to be valid without 50% voter turnout.

Croatia voted to enter EU with 43% turnout.

Today, with 37% turnout, Croatian voters demonstrated inability to separate their anxiety from good judgement.

And it’s simultaneously funny and sad; I’d love to see the reaction of people who said: “If 57% of voters can’t be bothered to turn out at the voting booths, then it’s okay to ignore their votes.” The same people have probably voted ‘against’, and are now widely disappointed.

I’m saddened by the fact that 24% of Croatian population has successfully demonstrated that a well-orchestrated brainwashing campaign of fear, uncertainty and doubt can succeed perfectly in Croatia. I’m saddened because this type of population is right for getting an oppressive but populist regime in place. 24% activists + 62% passives. Think of those numbers.

Reading articles on Croatian news portals from the distant Ireland, I see that there’s already talks about a referendum on “cyrillic”, a script whose presence in Vukovar, a city heavily damaged in the war during attacks by a cyrillic-using nation, offends a lot of people. So now not only are we taking away rights that didn’t exist in the first place, but we’ll have a referendum on a script, and waste 5 million more euros.

It’s sad that this thing managed to pass even with a disgusting brainwashing campaign run by certain large newspapers opposing the referendum question.

It’s sad that this thing managed to pass even among the international voters.

I’m happy that Croats in my new home, Ireland, voted ‘against’.

When leaving, I simply *knew* I’d be coming back some day. Now, I’m saddened that 24% of people manage to be misinformed and misled. I’m saddened that 62% find their vote to be irrelevant. And I’m questioning whether I can count on finding the 12% that has been both active and reasonable.

Oh scratch that; considering the campaigns against the referendum, the “reasonable” part is actually much smaller.

To the world: I’m sorry.

To my homeland: I hope you will heal some day.