The SOLR spatial community seems to be experiencing a flurry of activity, but as is always the way with solr, that activity seems to be focussed on the head revision, leaving anyone with deployed systems in the lurch really rather badly (Again). I know I know if it means that much I can fix it myself… But with so much effort going into the head, it seems like patching is wasted effort. Fortunately, the JTeam guys have done the work for us, and back ported some of the current development effort to a solr 1.4 spatial plugin. This posting is really just an Aide-mémoire for myself going through the process of replacing the old solr spatial libraries with the JTeam patch, to see if things are any better there.

First step is to get the jar from JTeam, this is (unfortunately, AFAIK) an email-register-download situation. I really understand JTeams desire to track usage and build up a user base, but I’m really not sure what they are doing is compatible with the letter of the original codes license (And I’m almost certain it’s not compatible with the spirit). Anyway, info about the plugin is here: http://www.jteam.nl/news/spatialsolr.html

Next up, our build process is maven based, so I need to import the jar:


mvn install:install-file -DgroupId=nl.jteam.search.solrext -DartifactId=spatial-solr -Dversion=1.0-RC5 -Dfile=spatial-solr-1.0-RC5.jar -Dpackaging=jar -DgeneratePom=true


Hats off to the JTeam guys for their documentation and installation notes… Here’s what I’ve done so far:

We build our own solr war from the apache source so we get a chance to pull in any extra plugins. You can see the source for the jteam branch here: http://developer.k-int.com/svn/default/solrwar/1.4.1-jteam-spatial This project simply removes the old localsolr plugin and all it’s dependencies and replaces it with the jteam jar. Specifically, the pom has changed: pom.xml

Next up, configuration changes in solrconfig.xml

I’ve updated the updateRequestProcessorChain which referenced com.pjaol.search.solr.update.LocalUpdateProcessorFactory as it’s processor class and changed the class to nl.jteam.search.solrext.spatial.SpatialTierUpdateProcessorFactory. There is also an additional str parameter _tier_ in this new update processor

The jteam plugin also adds a SpatialTierQueryParser and geodistance searchComponent. These are neatly documented in the pdf that comes with the JTeam download.

Finally, changes to the schema.xml

Add a dynamic field for the _tier_* fields, I’ve left the old _local* fields in, and will note later if it can be safely removed. For now, I’ve left my lat and lon fields as tdouble, even though the JTeam documentation seems different to that..

That represents the sum total of my mornings work… on to testing… more to follow.

After a couple of hours hacking on saturday I arrived at a point that needed some test data. What to reuse? I figured I could just throw together my usual collection of cybernetics books and use that. What data model? FRBR is getting some air-time at the moment. Wouldn’t it be great if you could just pick up a JPA definition of the FRBR model and drop it into your project? I thought so.. But a look around didn’t yield anything I could reuse. Since my test data really only needs work/creator at the moment, I’ve created some classes under org.frbr.datamodel, pasted a GNU affero license on it, and whacked it out there in the public domain. Think maybe this needs to live at sourceforge, but I don’t fancy going through the hell of that just for giggles. The sourcecode is here for now: http://developer.k-int.com/svn/default/sandbox/frbr_rel_model (Can be svn checked out from this address too). If this is something of interest, prod me and I’ll get it up on sourceforge and share out the permissions.

This could be something I work on over the coming months depending on interest, or it might just stay as a neat way to store test data for the generic JDBC<->OAI gateway.

People visiting my goodreads page recently might have noticed a bit of a theme – Understanding Gregory Bateson, Think Before you Think, etc. Anyone who knows me remotely well knows that I’ve always had a pretty substantial interest in cybernetics and information theory. This interest has been my saviour in a field of work so often permeated by mechanical “Systematic” thinking, allowing alternate perspectives and thinking to shed new light on what seem obvious (ly-wrong) solutions.

Sue Morton was, if memory serves, lead tutor for computer studies at Sheffield Poly when I set out on my degree waaaay back in 1989. Most importantly for me, Sue was the first person who I ever heard utter the magical words “General Systems Theory”. Until that moment, I’d been pretty much stuck on the hard, prickly, matter-of-fact, business-like and essentially ontological idea that a System is something out there – and really just a construct that helps us sell software. The idea that there are common qualities to what we now call viable systems, that those systems are permeable, and perhaps defined better by their boundaries and interfaces (In the bateson rather than the software sense) than their contents. This thinking led me to a path where critical thinking about our approach to so much more than just developing software is influenced and informed by this idea of “system”. Everything from software to the environment, relationships and social constructs were illuminated by this idea.

One of the best things about Finding Ada for me isn’t just about celebrating women in technology who have most influenced you, it’s about finding out why. When on placement, my line manager at the time was kind enough to share the reference that Sue had supplied for me. The actual text is long gone from my memory but the insight has stayed with me ever since. Sue, in reasonably direct terms, suggested that when I’m working in a team of peers who are motivated, enthusiastic and skilled my work could be quite good (OK, I toned that down a bit). But, Sue also warned that she’d noticed in me a tendency to try really hard to bring every group up to that level of engagement. If, however, at the end of that the team hasn’t lived up to my expectations, I had a tendency to, well, wander off on my own path. Perhaps harsh, but if I’m truly honest, one of the most insightful and useful bits of self knowledge I’ve ever been lucky enough to receive. I’m sooo lucky these days to work with friends and colleagues both in my day job, and in the wider grassroots scene that this bit of insight so very often works in my favour. Thanks to this I’m always mindful that when I feel that pull to wander off something deeper is going on. Thats the kind of self knowledge you just can’t buy or read from a book, it has to come from insight, empathy and skill. Three words that absolutely sum up my recollection of not only a great educator, but a great teacher, in every sense of the word!

I’m also reminded of the day when I got my degree results, when Sue handed me a bit of paper with the name and contact details of a professor looking for postgrads to do research. That moment and the decision that followed has remained one of the pivotal ones of my life. Not only was this a person I respected hugely, but one who had after all that help was still offering more assistance. Just recently, I’ve started a push on reading the latest works in and around general systems theory and more modern theories of complexity theory with the strongest intention of writing an proposal for the OU’s virtual MPhil. I don’t think many people could inspire a fresh graduate to an interest in a subject that would last 20 years, yet here I am, 20 years after first hearing Sue utter the words “General Systems Theory” pursuing learning in almost the self same subject.

If that doesn’t qualify as inspiring someone, I’ve no idea what does.

I’m sure I never said it at the time, Thankyou Sue!

Scripting and Data Import into Culture Grid – Culture Grid offers indexing of metadata, and search, OAI and preservation services for your metadata. If you can upload your metadata you can take advantage of these services and drive traffic to your resources as well as participate in the wider open/linked data community by sharing your records. To ingest data Culture Grid offers a SWORD based deposit service. The trouble is, many of our cultural data sets are in spreadsheets, bespoke file formats, custom XML schemas, databases, etc, etc. Culture Grid has an add on – a stand alone open source component called the Data Upload Tool. The data upload tool is an extensible toolbox that can be used to massage different input datasources into PNDS Dublin Core Application Profile (PNDS-DCAP) records and then deposit those records using the SWORD protocol. We can build in timestamp monitoring and other services at the tool level. It’s a handy way to get content out of existing silos and in to other systems. To be honest, there’s no reason to restrict the target to Culture Grid – any repository that supports SWORD could accept your data. So – the idea for this project would be instead of hacking together new code, if anyone out there has a dataset they would love to share why not work on some scripts / code for the data upload tool. This would give us a great output for the day – A bit of data that was previously in a silo thats now open. So, got the tools, really need some datasets now!

Idea 2 – Is along similar lines… I’ve been working (For a while now) on a web based drag/drop Relational Database -> OAI bridge. The idea is you put in your database connection details, wire together any additional relationships that cant be discerned from the database metadata, and then wire together that schema with a host of XML schemas to provider a bridge between any database that supports JDBC and OAI. I know there are a ton of Cultural datasets out there that are in relational databases – so this seems like a fun idea. In terms of bang-for-buck this is probably the most fun. Unfortunately, unless there are some other developers out there with an interest it’s probably also the least social as it’s down-and-dirty generic database code.

Idea 3 – Is something different. Crowdsourcing data cleansing. The idea here is blatantly inspired by the work Tom Morris did on reviewing data sets for Data.gov.uk. Take a cultural dataset and massage it down to a twitter URL. Whenever anyone visits the URL they are presented with the resource described and a form asking them some qualitative questions about the resource. The simple question seems to be “Does the metadata adequately describe the resource” (I know, it’s a question fraught with danger, but metadata quality is a real issue sometimes). This idea would require a fair amount of cross-working – From domain specific question design through to the nuts and bolts coding. Crowdsourcing data cleansing of cultural data seems like it should have some legs tho as an idea?

Will post more ideas as I have them

See you saturday!

So, long story short, I’m putting together three framework apps: (One that uses OpenURL and the dyu library for authentication (Great for deployment to google app engine), One that uses spring security, and one that uses a home grown identity service that we already use in loads of k-int open source projects). They are being “Incubated” here. feel free to copy/edit. They contain some app specific stuff at the moment, but my plan is to keep on cleaning up / adding default functionality to make the process of java-based-hackathon activity as easy as it is in python/ruby/etc.

On the road to this, I also decided it was time to get busy with jetty as a test tool and quick prototyping aid. All was wonderful (JDBC resources in JNDI context etc) right up to the point I tried to configure the jetty JDBCRealm. I’d configured the mysql connector jar as a dependency of the plugin as per the jetty docs, set up the etc/realm.properties yet the app consistently delivers java.lang.ClassNotFoundException: org.gjt.mm.mysql.Driver. 2 days of digging around, and a read through JETTY-574 made me realise that the jetty container filters some classes. Switching the jdbc driver to com.mysql.jdbc.Driver seems to make everything work just fine. I’m not so sure about the heritage of this particular driver class, but it’s a step forwards. I suspect that adding the right elements to the pom will enable the old style org.gjt driver to be found. Pushing on with the hacking for now.

Specifics of the JDBCRealm are in this project

During the course of this discussion we identified, not unexpectedly, that most interesting insights come at the intersection of multiple data sets. One idea that came to mind in this discussion was triggered by the example “Average age by postcode”. We then went on to think how this could be shown – one example was a temperature map – hotter = higher concentrations of the “Target age group”. I’d been re-reading stafford beer that day (Think before you think) and was reminded of the phrase “Only variety can absorb variety”. So, one interesting visualisation we came up with was trying to locate under-serviced regions for specific community groups. Specifically:

One slider increases the “Heat” level for one dimension of measurements. In this case, average age. A second slider indicates the provision of services targeted at that age group, EG “Home help provision”. As each slider moves up to its max, the “Cold” from the home help provision should “Balance out” the “Heat” from the average age index. If the map returns to a steady state, all is well, but if the map has blotchy red spots, maybe there are gaps in provision that would enable better targeting of services?

Not sure this example really stands up to harsh scrutiny, but I’d never heard of heat maps being used in quite this context before. Fun times.

I’ve been trying to prepare a project using localsolr for the impending (I hope) arrival of solr 1.4 and Lucene 2.9.1.. Here are some notes in case anyone else is suffering similarly. Obviously everything here may change by next week.

First problem has been the lack of any stable maven based lucene and solr artifacts to wire into my poms. Because of that, and as an interim only measure, I’ve built the latest branches of the lucene 2.9.1 and solr 1.4 trees and made them into a *temporary* maven repository at http://developer.k-int.com/m2snapshots/. in there you can find all the jar artifacts tagged as xxx-ki-rc1 (For solr) and xxx-ki-rc3 (For lucene) and finally, a localsolr.localsolr build tagged as 1.5.2-rc1. Sorry for the naming, but I don’t want these artifacts to clash with the real ones when they come along. This is really just for my own use, but I’ve seen messages and spoken to people who are really struggling to get their maven deps right, if this helps anyone, please feel free to use these until the real apache artifacts appear. I can’t take any responsibility for their quality. All the poms have been altered to look for the correct dependent artifacts in the same repository, adding the stanza

k-int-m2-snapshots
K-int M2 Snapshots
http://developer.k-int.com/m2snapshots

true

to your pom will let you use these deps temporarily until we see an official build. If you’re a maven developer and I’ve gone way around the houses with this, please tell me of an easier solution This repo *will* go away when the real builds turn up.

The localsolr in this repo also contains the patches I’ve submitted (A good while ago) to the localsolr project to make it build with the lucene 2.9.1 rc3 as the downloadable dist is currently built against an older 2.9 release that had a different API (IE won’t work with the new lucene and solr)

All this means that there is a working localsolr build.

Second up, I’ve also seen emails (And seen the exception myself) around asking about the following when trying to get all these revisions working together.

java.lang.NumberFormatException: Invalid shift value in prefixCoded string
(is encoded value really a LONG?)

There are some threads out there telling you that the Lucene indexes are not binary compatible between versions, but if you’re using localsolr, what you really need to know is:

1) Make sure that your schema.xml contains at least the following fieldType defs

2) Convert your old solr sdouble fields to tdoubles:

Pretty sure you would need to rebuild your indexes.

Ok, with those changes I managed to get a working spatial search.

My only problem now is that the radius param on the command line seems to need to be way bigger than it needs to be in order to find anything. Specifically, if I search with a radius of 220 I get a record back which marks it’s geo_distance as “83.76888211666025″. Shuffling the radius around ends up that a radius of 205 returns that doc, 204 and it’s filtered. I’m going to dig into this now, but if anyone knows about this I’d really appreciate any help.

Cheers all, hope this is of use to someone out there, if anyone has corrections/comments I’d really appreciate any info.

Best,
Ian.

1. An idea Aaron and I have mucked about with in the past – The arduino weather station. The idea here is to put together instructions for building an internet connected weather station capable of reporting wind speed and direction, direct sunlight (strengh), and possibly rainfall, maybe pressure etc. Thats sorta the #shacknet half of the project. The #geekup half would involve setting up some aggregator software capable of pulling data from all installed stations and presenting it in some meaningful manner.. like a real time weather network for sheffield.

I think there are a few practical applications for this.. my personal fave is as a tool for finding the best locations for wind turbines / solar power installations. I know @richardrixham has spoken with teachers about the problelms of getting this funded as an electronics project in schools, but I wonder if there is a way to just get it as a “Self assembly weather station” and do the electronics thing by stealth… anyone really interested could get the info and build one at home. I don’t doubt that if we had a quick brainstorm we could come up with oodles of add-on sensors and other instrumentation…. any thoughts?

2. Project 2 seems like a fun hardware/software/network mash-up too… inspired by the LETS scheme in sheffield. Richard and I were wondering if we could build a hardware terminal and some software to act as an epos device for the stones (The LETS unit of currency) used by all the current LETS schemes… basically a cash card for stones. We’d need to design and build a terminal with some security, and do the same job on the “Bank” server…. My exposure to LETS is only minimal at the moment, but I’m hoping to get more involved and am currently signing up… seems like an internet service for your LETS account would be quite a cool thing to have?

Just a brief update…. Any thoughts?

Download the zip package from googlecode at http://code.google.com/p/gdata-java-client/downloads/list

then cd to the lib directory and run

mvn install:install-file -DgroupId=gdata -DartifactId=base -Dversion=1.0 -Dfile=gdata-base-1.0.jar  -Dpackaging=jar -DgeneratePom=true
mvn install:install-file -DgroupId=gdata -DartifactId=blogger -Dversion=2.0 -Dfile=gdata-blogger-2.0.jar  -Dpackaging=jar -DgeneratePom=true
mvn install:install-file -DgroupId=gdata -DartifactId=blogger-meta -Dversion=2.0 -Dfile=gdata-blogger-meta-2.0.jar  -Dpackaging=jar -DgeneratePom=true
mvn install:install-file -DgroupId=gdata -DartifactId=books -Dversion=1.0 -Dfile=gdata-books-1.0.jar  -Dpackaging=jar -DgeneratePom=true
mvn install:install-file -DgroupId=gdata -DartifactId=books-meta -Dversion=1.0 -Dfile=gdata-books-meta-1.0.jar  -Dpackaging=jar -DgeneratePom=true
mvn install:install-file -DgroupId=gdata -DartifactId=calendar -Dversion=2.0 -Dfile=gdata-calendar-2.0.jar  -Dpackaging=jar -DgeneratePom=true
mvn install:install-file -DgroupId=gdata -DartifactId=calendar-meta -Dversion=2.0 -Dfile=gdata-calendar-meta-2.0.jar  -Dpackaging=jar -DgeneratePom=true
mvn install:install-file -DgroupId=gdata -DartifactId=client -Dversion=1.0 -Dfile=gdata-client-1.0.jar  -Dpackaging=jar -DgeneratePom=true
mvn install:install-file -DgroupId=gdata -DartifactId=client-meta -Dversion=1.0 -Dfile=gdata-client-meta-1.0.jar  -Dpackaging=jar -DgeneratePom=true
mvn install:install-file -DgroupId=gdata -DartifactId=codesearch -Dversion=2.0 -Dfile=gdata-codesearch-2.0.jar  -Dpackaging=jar -DgeneratePom=true
mvn install:install-file -DgroupId=gdata -DartifactId=codesearch-meta -Dversion=2.0 -Dfile=gdata-codesearch-meta-2.0.jar  -Dpackaging=jar -DgeneratePom=true
mvn install:install-file -DgroupId=gdata -DartifactId=contacts -Dversion=3.0 -Dfile=gdata-contacts-3.0.jar  -Dpackaging=jar -DgeneratePom=true
mvn install:install-file -DgroupId=gdata -DartifactId=contacts-meta -Dversion=3.0 -Dfile=gdata-contacts-meta-3.0.jar  -Dpackaging=jar -DgeneratePom=true
mvn install:install-file -DgroupId=gdata -DartifactId=core -Dversion=1.0 -Dfile=gdata-core-1.0.jar  -Dpackaging=jar -DgeneratePom=true
mvn install:install-file -DgroupId=gdata -DartifactId=docs -Dversion=2.0 -Dfile=gdata-docs-2.0.jar  -Dpackaging=jar -DgeneratePom=true
mvn install:install-file -DgroupId=gdata -DartifactId=docs-meta -Dversion=2.0 -Dfile=gdata-docs-meta-2.0.jar  -Dpackaging=jar -DgeneratePom=true
mvn install:install-file -DgroupId=gdata -DartifactId=finance -Dversion=2.0 -Dfile=gdata-finance-2.0.jar  -Dpackaging=jar -DgeneratePom=true
mvn install:install-file -DgroupId=gdata -DartifactId=finance-meta -Dversion=2.0 -Dfile=gdata-finance-meta-2.0.jar  -Dpackaging=jar -DgeneratePom=true
mvn install:install-file -DgroupId=gdata -DartifactId=health -Dversion=2.0 -Dfile=gdata-health-2.0.jar  -Dpackaging=jar -DgeneratePom=true
mvn install:install-file -DgroupId=gdata -DartifactId=health-meta -Dversion=2.0 -Dfile=gdata-health-meta-2.0.jar  -Dpackaging=jar -DgeneratePom=true
mvn install:install-file -DgroupId=gdata -DartifactId=media -Dversion=1.0 -Dfile=gdata-media-1.0.jar  -Dpackaging=jar -DgeneratePom=true
mvn install:install-file -DgroupId=gdata -DartifactId=photos -Dversion=2.0 -Dfile=gdata-photos-2.0.jar  -Dpackaging=jar -DgeneratePom=true
mvn install:install-file -DgroupId=gdata -DartifactId=photos-meta -Dversion=2.0 -Dfile=gdata-photos-meta-2.0.jar  -Dpackaging=jar -DgeneratePom=true
mvn install:install-file -DgroupId=gdata -DartifactId=spreadsheet -Dversion=3.0 -Dfile=gdata-spreadsheet-3.0.jar  -Dpackaging=jar -DgeneratePom=true
mvn install:install-file -DgroupId=gdata -DartifactId=spreadsheet-meta -Dversion=3.0 -Dfile=gdata-spreadsheet-meta-3.0.jar  -Dpackaging=jar -DgeneratePom=true
mvn install:install-file -DgroupId=gdata -DartifactId=youtube -Dversion=2.0 -Dfile=gdata-youtube-2.0.jar  -Dpackaging=jar -DgeneratePom=true
mvn install:install-file -DgroupId=gdata -DartifactId=youtube-meta -Dversion=2.0 -Dfile=gdata-youtube-meta-2.0.jar  -Dpackaging=jar -DgeneratePom=true

Edit - With the latest downloads you will also need to download the collections extensions at http://code.google.com/p/google-collections/ and install using

mvn install:install-file -DgroupId=gdata -DartifactId=youtube-meta -Dversion=2.0 -Dfile=gdata-youtube-meta-2.0.jar  -Dpackaging=jar -DgeneratePom=true