IAnnSolrMigration
Contents
Introduction
iAnn is a platform which collects and distributes the data about various events, like symposiums, presentations, workshops, conferences etc.
Problem description
While fetching the data from different sources, one needs to standardize it and name all properties in such way that name itself is descriptive and makes sense. Naming examples: submission_date, keywords, host_institution.
However, naming is part of agreement from the SASI document [sasi link] and it is straight forward. The challenge is to migrate all imported data from old to new schema. Solr does not provide automatized way and custom scripts need to be created. While working on customized scripts we considered several points; - scripts must be easy to modify - they must handle different types conversions while migrating - they must be able to handle different naming structures - they must provide a sort of feedback while migrating (process) - scripts must be configured on linode virtual server - migration from solr 4.1.0 to 4.10.2 should be done
Solutions required
Project 1: Creating/configuring SASI solr schema and upgrading solar service to new version
Project 2: Writing scripts for data reindexing
Project progress
Week number | Completed tasks | Status |
---|---|---|
Week 1 |
|
|
Week 2 |
|
|
Week 3 |
|
|
Week 4 |
|
|
Week 5 |
|
|
Week 6 |
Proposal for Schema changes
Current version of iAnn website is delivering a service by using custom schema for data attributes naming. However, it is not standardized and does not have default field naming scheme. Therefore, a community agreed on default schema layout, named SASI. In this document we will point out differences and describe new schema fields. As current iAnn collection runs on Apache Solr 4.1 and the latest one with some major bug fixes is 4.10.2, we decided to migrate and update our service as well.
Accomplished tasks:
- Understanding of existing scheme,
- Proposing structure changes and
- Highlighting differences
Old Scheme:
<field name="id" type="string" indexed="true" stored="true" required="true" /> <field name="title" type="text" indexed="true" stored="true"/> <field name="subtitle" type="text" indexed="true" stored="true"/> <field name="description" type="string" indexed="false" stored="true"/> <field name="provider" type="text_lowercase" indexed="true" stored="true"/> <field name="link" type="string" indexed="false" stored="true"/> <field name="start" type="tdate" indexed="true" stored="true"/> <field name="end" type="tdate" indexed="true" stored="true"/> <field name="venue" type="text_lowercase" indexed="true" stored="true"/> <field name="city" type="text_lowercase" indexed="true" stored="true"/> <field name="county" type="string" indexed="fasle" stored="true"/> <field name="country" type="text_lowercase" indexed="true" stored="true"/> <field name="postcode" type="string" indexed="false" stored="true"/> <field name="attachment" type="string" indexed="false" stored="true" multiValued="true"/> <field name="image" type="string" indexed="false" stored="true" multiValued="true"/> <field name="keyword" type="text_lowercase" indexed="true" stored="true" multiValued="true"/> <field name="category" type="text_lowercase" indexed="true" stored="true" multiValued="true"/> <field name="field" type="text_lowercase" indexed="true" stored="true" multiValued="true"/> <field name="submission_name" type="text_lowercase" indexed="true" stored="true" multiValued="true"/> <field name="submission_email" type="string" indexed="true" stored="true" multiValued="true"/> <field name="submission_date" type="tdate" indexed="true" stored="true" multiValued="true"/> <field name="submission_comment" type="text" indexed="false" stored="true" multiValued="true"/> <field name="submission_organization" type="text" indexed="true" stored="true" multiValued="true"/> <field name="latitude" type="double" indexed="true" stored="true"/> <field name="longitude" type="double" indexed="true" stored="true"/> <field name="sponsor" type="text_lowercase" indexed="true" stored="true"/> <field name="public" type="boolean" indexed="true" stored="true"/> <field name="text" type="text" indexed="true" stored="false" multiValued="true"/> <field name="_version_" type="long" indexed="true" stored="true"/>
New Scheme:
<field name="id" type="text" indexed="true" stored="true" required="true" /> <field name="title" type="text" indexed="true" stored="true"/> <field name="subtitle" type="text" indexed="true" stored="true"/> <field name="description" type="text" indexed="false" stored="true"/> <field name="prerequisites" type="text" indexed="false" stored="true"/> <field name="programme" type="programme" indexed="false" stored="true"/> <field name="comments" type="text" indexed="false" stored="true"/> <field name="fees" type="fees" indexed="false" stored="true" multiValued="true"/> <field name="discount" type="discount" indexed="false" stored="true"/> <field name="accreditation" type="text" indexed="false" stored="true"/> <field name="status" type="text" indexed="false" stored="true"/> <field name="eligibility" type="text" indexed="false" stored="true" multiValued="true"/> <field name="capacity" type="int" indexed="false" stored="true"/> <field name="contact" type="person" indexed="false" stored="true" multiValued="true"/> <field name="submitter" type="person" indexed="false" stored="true" multiValued="true"/> <field name="organizers" type="person" indexed="false" stored="true" multiValued="true"/> <field name="speakers" type="person" indexed="false" stored="true" multiValued="true"/> <field name="host_institution" type="organization" indexed="false" stored="true" multiValued="true"/> <field name="sponsor" type="organization" indexed="false" stored="true" multiValued="true"/> <field name="venue" type="text" indexed="false" stored="true"/> <field name="street_address" type="text" indexed="false" stored="true"/> <field name="city" type="text" indexed="false" stored="true"/> <field name="province" type="text" indexed="false" stored="true"/> <field name="country" type="text" indexed="false" stored="true"/> <field name="postcode" type="text" indexed="false" stored="true"/> <field name="post_office_box" type="text" indexed="false" stored="true"/> <field name="url" type="link" indexed="false" stored="true"/> <field name="attachment" type="link" indexed="false" stored="true" multiValued="true"/> <field name="social_media" type="link" indexed="false" stored="true" multiValued="true"/> <field name="starts" type="date" indexed="false" stored="true"/> <field name="ends" type="date" indexed="false" stored="true"/> <field name="time_zone" type="text" indexed="false" stored="true"/> <field name="last_update" type="date" indexed="false" stored="true" multiValued="true"/> <field name="deadlines" type="date" indexed="false" stored="true" multiValued="true"/> <field name="registration_opens_date" type="date" indexed="false" stored="true" multiValued="true"/> <field name="acceptance_notification_date" type="date" indexed="false" stored="true" multiValued="true"/> <field name="type" type="text" indexed="false" stored="true" multiValued="true"/> <field name="topic" type="text" indexed="false" stored="true" multiValued="true"/> <field name="public" type="boolean" indexed="false" stored="true"/> <field name="target_audience" type="text" indexed="false" stored="true" multiValued="true"/> <field name="spotlight" type="boolean" indexed="false" stored="true"/> <field name="latitude" type="double" indexed="true" stored="true"/> <field name="longitude" type="double" indexed="true" stored="true"/> <field name="_version_" type="long" indexed="true" stored="true"/>
Conclusion:
While using newly proposed scheme iAnn will have a possibility to serve even more diverse data. This will serve as a basis for new services on top of Solr search engine. On top of that, migration to newer version of Solr will make it safer.
GitHub Repository
GitHub : iANN_Solr