The wikipedia_solr system is a locally installed full text index of the wikipedia. It allows fast searches through the entire wikipedia.

This document has three main sections.

1. Installation details the steps for a basic install.
2. Running wikipedia-solr goes over the core commands to run the system and instructions for common scenarios. It also lists configuration options.
3. Interfacing details how other programs can query solr
4. Implementation notes describes how the system was built.

A running solr installation depends on a couple of things. It needs a wikipedia dump of articles, this dump is then indexed into solr and written to an index. Once solr has finished indexing, it is ready to respond to queries. The index is the data structure that solr stores documents in for fast retrieval. The solr server needs to be running to do the indexing and it needs to be running to query the index.

Java 6 is installed by default on OS X 10.6 and above.

The Apache Ant official installation guide Note: installing netbeans is a quick shortcut that assures that java and ant are installed properly

The git command line tools must be installed on your machine, along with ssh key access to the github repo. You can read more information about setting ssh keys here:

http://help.github.com/set-up-git-redirect/

http://en.wikipedia.org/wiki/CURL needs to be installed and working properly on your system. Mac OS X has a copy of curl by default.

You can test by running the following command in a shell.

curl http://google.com 

you should see the html of google’s landing page.

on ubuntu running

sudo apt-get install ant git openjdk-6-jdk curl

will install all the requirements

The source code and compiled files take about 700 Mb. a minimum index of only the first 1000 articles will take about 50-60 megs.

The complete index takes 16 Gb when finished, and up to 34 Gb while the indexing process is running.

The complete wikipedia data dump file is 8 Gb.

The greatest amount of space thaelseis system could take on your system is

43 Gb = 8 + 34 + 1

You can store all three components on separate drives. The repository root must be on a locally mounted drive, the system will fail if it is on a network share.

Run the one-step installer at a terminal by copying and pasting

nohup curl https://raw.github.com/paddymul/wikipedia_solr/master/apps/update/onestep_install.sh | sh > onestep.out &

You can read the one step installer commands here Here is aproximately the expected onestep installer output

After the installer has run take a look at I have successfully downloaded the data dump. What next?

If you want to customize install locations read how do I control where various files are stored.

search local solr for american

There are a set of apps which allow you to perform most regular maintenance tasks. These are the primitives upon which the common workflows section is based on. All of them are found in $GC_WIKIPEDIA_REPO_ROOT/apps , after the initial install steps.

Every app opens a terminal window and at the start outputs “$app_name:STARTING” when the app has finished running “$app_name:FINISHED” will be written to the terminal window.

This app starts the solr server and leaves a terminal window showing the current server log. Do not close this window because doing so will kill the solr server. Any code changes, or file location changes (the variables specified in ~/.profile) require restarting the server before the server recognizes the new code. The solr server needs to be running to index documents and to perform queries.

To verify that the solr server is running properly go to this url with your browser http://localhost:8983/solr/admin.

This app pulls all the latest code and recompiles every module necessary for the system.

This app downloads the latest wikipedia dump into your $GC_WIKIPEDIA_DL_DIR , it is necessary to have a wikipedia dump to index articles.

This script makes the smaller sections of the initial dump that aid faster testing of indexing, it makes a 1,000 10,000 and 100,000 article dump file. While it is running bunzip2 will take around 80-100% of a single core, 3 copies of tee and 3 copies of grep will also be running.

This script is run automatically by download_dump.app, but sometimes it is useful to run on its own if you downloaded the article dump separately.

This app takes about an hour to run.

there are 4 indexing apps, index_1_000.app, index_10_000.app, index_100_000.app and index_all.app. they index the first 1,000 10,000 100,000 articles from the data_dump. the index_all.app pulls in everything in the data_dump file.

Running any one of the Indexing apps blows away the current index. If you want to preserve the current index you might want to change your GC_WIKIPEDIA_SOLR_DATA_DIR location to keep an existing large index as a backup.

Last login: Sun Apr 29 22:01:11 on ttys003
/Users/patrickmullen/code/wikipedia_solr/apps/reindex/index_1_000 ; exit;
You have mail.
bash_profile
Macintosh-5:~ patrickmullen$ /Users/patrickmullen/code/wikipedia_solr/apps/reindex/index_1_000 ; exit;
index_1_000:STARTING
mkdir: /Users/patrickmullen/code/wikipedia_solr/named_pipes: File exists
 linking /Volumes/LaCie_1/data/data_wikipedia//first_1_000.xml to /Users/patrickmullen/code/wikipedia_solr/named_pipes/import.xml
/Volumes/LaCie_1/data/data_wikipedia//first_1_000.xml
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">1</int></lst><lst name="initArgs"><lst name="defaults"><str name="config">./data-config.xml</str></lst><lst name="invariants"><str name="solrHome">/Users/patrickmullen/code/wikipedia_solr/solr_home/solr</str><str name="dataFile">/Users/patrickmullen/code/wikipedia_solr/named_pipes/import.xml</str></lst></lst><str name="command">full-import</str><str name="status">idle</str><str name="importResponse"/><lst name="statusMessages"><str name="Total Requests made to DataSource">0</str><str name="Total Rows Fetched">1000</str><str name="Total Documents Skipped">198</str><str name="Full Dump Started">2012-04-29 22:01:11</str><str name="">Indexing completed. Added/Updated: 802 documents. Deleted 0 documents.</str><str name="Committed">2012-04-29 22:01:42</str><str name="Optimized">2012-04-29 22:01:42</str><str name="Total Documents Processed">802</str><str name="Time taken ">0:0:31.835</str></lst><str name="WARNING">This response format is experimental.  It is likely to change in the future.</str>
</response>
the index has been triggered, but not necessarily completed
read the indexing_apps section of the readme
---------
https://github.com/paddymul/wikipedia_solr/blob/master/readme.org#progress-tracking
---------
the solr_server should take aproximately 40 seconds to complete this index
index_1_000:FINISHED
logout

[Process completed]

the indexing apps only trigger a reindexing by the solr server. The indexing apps themselves finish very quickly. You can read more about

Here are the time estimates for the whole indexing: index_1_000: 40 seconds index_10_000: 4 minutes index_100_000: 40 minutes index_all: 12 hours

When a large import is running, your machine should be running at about 100% cpu for a java process, and 10-20% for a bunzip2 process.

You can track the import progress at this url http://localhost:8983/solr/admin/stats.jsp

Search in that page for “Rows Fetched” . Reload as necessary to watch this number increase, it counts the total number of documents that the import process has seen. It also counts “Documents Skipped”, documents are skipped as part of the import process because they would pollute the index with poor results, this is perfectly normal. The “Documents Processed” field lists the total number actually put into the index.

The full wikipedia dump has about 12 Million rows in it, at last run on 3/6/2012 5,785,453 documents were in the index after a full import.

Note: When running a full import the “Rows Fetched” will get to 12 million, and then the system will take approximately 1 more hour compacting and optimizing indexes before you can search against new data, this is normal. The index is not finished until status reads “IDLE” again.

Solr stores indexes in the path controlled by the GC_WIKIPEDIA_SOLR_DATA_DIR environment variable.

edit the line of your ~/.profile that looks like this

export GC_WIKIPEDIA_SOLR_DATA_DIR="/Volumes/LaCie_1/data/index_wikipedia"

to point to another location. Changes to this location require a restart of the server to be recognized.

First close the existing start_solr window, solr will have to be restarted to use the new code.

Run the following scripts or apps. The apps have the same name as the scripts, except they are followed by “.app” and the apps are not in the sub-directories, to run them, you can double-click on the .app from finder.

From the repo root, run the following scripts or apps

1: Pull the most recent code and compile it.

./update/update_all        

2: Start the server so that it is reading the most recent codebase.

./start_solr/start_solr

3: Next kickoff the reindex.

./reindex/index_all

1st, you need to start solr. Run the app or script apps/start_solr.app

2nd, go to this url with a browser http://localhost:8983/solr/admin/

3rd, kick off an indexing run. run apps/reindex/index_1000

4th, when the import has finished running run a query. Go to this url http://localhost:8983/solr/select/?q=*%3A*&version=2.2&start=0&rows=10&indent=on&wt=json that url returns the first 10 documents of everything in your index, there should be some results.

OS X doesn’t support creating named pipes on network mapped drives. http://caml.inria.fr/pub/docs/manual-ocaml/libref/Unix.html

Named pipes are key to the way the whole import system works. They let the system read the compressed dump into solr, without ever expanding the whole thing on disk.

It is necessary to have at least the $GC_WIKIPEDIA_REPO_ROOT on a locally mounted disk

Sometimes you will want to rerun the make_helper_file scripts that create smaller sections of the initial dump, without rerunning the whole download script.

In a terminal window, run these commands:

source ~/.profile
$GC_WIKIPEDIA_REPO_ROOT/apps/update/make_helper_files

This error shouldn’t occur anymore, it used to occur because solr had trouble creating lock files on network drives. Now solr is configured to use the NoLockFactory , so this shouldn’t be an issue. You can read more about lock factories here http://wiki.apache.org/lucene-java/AvailableLockFactories

these two queries are quite helpful markup ratio ascending markup ratio descending

average of markupPlainRatio

The wikipedia_solr system reads your ~/.profile for three locations.

This is the directory that the system expects to find the raw data dump files. This directory grows to about 9Gb with a full download and helper files.

export GC_WIKIPEDIA_DL_DIR="/Volumes/LaCie_1/data/data_wikipedia"

This is where solr looks for an index, if it is indexing, solr stores an index in this directory. For a full index of the wikipedia, it should be about 16Gb, it can grow to 34Gb during indexing.

export GC_WIKIPEDIA_SOLR_DATA_DIR="/Volumes/LaCie_1/data/index_wikipedia/"

This is the directory where source code and compiled files for the wikipedia_solr system are stored. This directory should take less than a Gb of storage.

export GC_WIKIPEDIA_REPO_ROOT="/Users/patrickmullen/code/wikipedia_solr"

The solr system is queried over http, results can be returned in json format or xml format. all examples are given using the json format.

http://wiki.apache.org/solr/CommonQueryParameters

http://localhost:8983/solr/select/?q=articlePlainText%3A%22american%22&version=2.2&start=0&rows=1000&indent=on&wt=json

The q parameter is the actual query, unurlescaped this query looks articlePlainText:”american” .

This tells solr to search the ‘articlePlainText’ field in the entire database for the term american.

The ‘version’ parmeter is of unknown consequence, use a value of 2.2 for continuity.

The start parameter controls the first row the result set to be returned.

The rows parameter controls how many documents (at most) to return after the start document.

The indent=on causes solr to pretty print the result.

The wt=json causes solr to return the result in json format.

Take a look at py/query_demo.py to see this as a running program.

Note qp takes the un-urlencoded q parameter as input, it executes the query and prints some simple stats about it, including the complete formed url, it returns the total number of documents found for that query.

Triple quotes are a python convention for encoding multiline strings or quote containing strings. the value of a triple quoted string is between the first triple quote and last triple quote.

A string preceded by a ‘u’ is a unicode string, for ascii only sequences it can be thought of as a string.

The leading and trailing space in the queries are there for readability.

assert is a python statement that throws an error when it is give a false value, none of the asserts in this tour throw an error.

american = qp(''' articlePlainText:"american" ''')

american_no_quote = qp(''' articlePlainText:american ''')

note - for single terms, we got the same number of documents back when we quoted “american” as we got back when we didn’t quote “american”

american_no_trail = qp('''articlePlainText:american''')

assert american == american_no_quote
assert american_no_trail == american_no_quote

samoa =qp(''' articlePlainText:samoa ''')

american_or_samoa = qp(''' articlePlainText:american OR _query_:"articlePlainText:samoa" ''')

american_and_samoa = qp(''' articlePlainText:american AND  _query_:"articlePlainText:samoa" ''')

samoa_not_american = qp(''' articlePlainText:samoa NOT _query_:"articlePlainText:american" ''')

american_samoa_phrase = qp(''' articlePlainText:"american samoa" ''')

assert american_or_samoa == (american + samoa_not_american)
assert 621927            == (619399   + 2528)

assert american_and_samoa >= american_samoa_phrase
assert 2227               >=     1397

a =qp(''' articlePlainText:"american samoa" AND  _query_:"articlePlainText:'manifest destiny'" ''')

Note: the following syntax query is unclear and I can’t decipher the results, don’t issue queries like this, the results are undecided and unsupported by me .

a =qp('''articlePlainText:american samoa''')

If you want more information about solr query syntax, try thes resources

nested queries in solr http://www.lucidimagination.com/blog/2009/03/31/nested-queries-in-solr/

the solr-wiki page, not actually that helpful http://wiki.apache.org/solr/SolrQuerySyntax

These notes are meant as a guide for a future maintainer of the solr/java search system. They assume a knowledge of solr, java, and common development practices.

This wikipedia search system uses solr http://wiki.apache.org/solr/ and the jwpl wikimedia markup parsing library http://code.google.com/p/jwpl/.

I used the DataImportHandler framework to import the XML wikipedia dump. I wrote a custom transformer that integrates into the DataImportHandlerFramework, this handler calls the jwpl parsing library to extract the article text from the wikimedia markup.

I modified solr in two places. First I changed the file reader so that it will read from a named pipe. This allows us to keep the article dump compressed on disk, allowing for faster I/O and less disk usage.

I also modified the xml reader so that it doesn’t kill an entire import if there is a missing xml tag. This extra fault tolerance ensures that hours of work aren’t lost. Wikipedia article dumps are of the format described in http://www.mediawiki.org/xml/export-0.5.xsd. The downloaded dumps seem to be missing the final closing </mediawiki> tag. We could compare md5sums if we are worried about integrity.

the custom code I wrote for this project can be found in

solr_home/Wikipedia_importer/wikipedia_solr/src/wikipedia_solr/WikimediaToTextTransformer.java

lib/solr/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/FileDataSource.java

lib/solr/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/XPathEntityProcessor.java

the solr schema.xml for this project can be found solr_home/solr/conf/schema.xml

http://wiki.apache.org/solr/SchemaXml#Fields

each field can have one of multiple flags applied to it:

An stored field has its original version saved by lucene.

An indexed field can be searched against.

This controls which fields are stored and indexed. We have a very simple schema, only three relevant fields, title, articlePlainText and sectionParsed.

articlePlainText is the field that is searched on, it is an indexed version of the plaintext of each wikipedia article. It isn’t stored since the plaintext on its own isn’t that useful.

This field is stored, but not indexed. it is a json-string http://www.json.org/ of article sections, in the form of

[{“section_name”:[“paragraph1”, “paragraph2”]}, {“another section title”: [“paragph1”, “p2”]}] .

the solr-config for this project can be found here solr_home/solr/conf/solrconfig.xml

It stays pretty close to the example config, except for additional java properties that it reads, which allow the system to be more easily configured.

There are a variety of maintenance scripts that can be found in apps/* , they are explained in the Running wikipedia_solr:Apps section of this document.

Core concepts fields have been added to make debugging parsing errors easier.

wikimediaMarkupCount stores the count of characters in the original version of the text. articlePlainTextCount stores the count of characters in the plaintext version of the text. markupPlainRatio stores articlePlainTextCount / wikimediaMarkupCount .

There are additional fields for java error debugging error When the parsing code detects an error this field will have a value of “true”, normally it will be “false”. exception When there is an exception this field will store the name of that exception. stackTrace This field stores the stacktrace of an exception.

The markup ratio ascending query is very useful for finding examples of documents that have a lot of wikimediaMarkup, and very little articlePlainText. Note that this query excludes that have errors, error’d articles rarely have any articlePlainText, so they are of no use.

markup ratio descending

The average of markupPlainRatio (you can read more about solr stats queries here solr stats component) is very useful for determining the affect of a change in the parsing code over a large quantity of documents. My assumption is that as parsing improves, the standard deviation of the markupPlainRatio will get smaller.

There are works in progress for python a python testing tool that pulls the wikimediaMarkup for an anomalous document from the index and stores it on the filesystem. This tool can be found in py/make_test_case.py.

The main class of the wikipediaTransformer jar now can accept a command line argument for a file of wikimediaMarkup, it reads this file and writes the parsed articlePlainText to stdout. These two tools allow quick iteration over problematic documents.