![]() You need to remember to expire your Lucene index' contents every now and then too, otherwise you'll never remove anything and it'll get bloated and eventually just dies because of its own internal complexity.Always remember to close the index too to avoid unnecessary LockObtainFailedExceptions to be thrown, as with all IO in Java such operation should of course be done in the finally block. Note that Lucene IndexWriter does have some cleanup methods which are handy to execute in a controlled manner, for example calling IndexWriter#commit() only after a bunch of documents have been added to index is good for performance and then calling IndexWriter#optimize() to make sure the index isn't getting hugely bloated over time is a good idea too. The five minute tutorial I already linked to basically does exactly that, look into the example addDoc(.) method and just replace the String with YourBean. ![]() Lucene-based automated indexerīeyond the obvious stuff with Lucene such as setting up a directory and understanding its threading model ( only one write operation is allowed at any time, multiple reads can exist even when the index is being updated), you of course want to feed your beans to the index. In terms of API calls, you should have something like HttpCrawler#getDocuments(String url) which returns a List to use in conjuction with the actual indexer. For simplicity's sake I'm assuming these are plaintext files with no fields or anything and won't go deeper into that but if you have multiple fields to store, I suggest you make your crawler to produce 1.n of specialized beans with accessors and mutators ( bonus points: Make the bean immutable, don't allow accessors to mutate the internal state of the bean, create a copy constructor for the bean) to be used in the other component. Once you get the actual resources you want such as bunch of text files, you need to identify the type of data to be able to know what to index and what you can safely ignore. If you decide to go the XPath route, consider using JDOM for DOM handling and Jaxen for the actual XPath. The actual implementation could be something like so: Use HttpClient to get the actual web pages/directory listings, parse them in the way you find most efficient such as using XPath to select all the links from the fetched document or just parsing it with regex using Java's Pattern and Matcher classes readily available. Assuming that it's any common web server which lists directory contents, making a web crawler is easy: Just point it to the root of the directory and define rules for collecting the actual files, such as "ends with. So you have a web site/directory you want to "crawl" through to collect specific resources. I'll tackle both of the components individually from the point of view of making your own since I don't believe that you could use Lucene to do something you've requested without really understanding what's going on underneath. First a word of couragement: Been there, done that.
0 Comments
Leave a Reply. |
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |