Lucene 應(yīng)用 WordNet 的同義詞典實(shí)現(xiàn)同義詞檢索(C#版)

1、從此處轉(zhuǎn)載（不知是否是原作者）感謝原作者！Lucene 應(yīng)用 WordNet 的同義詞典實(shí)現(xiàn)同義詞檢索(C#版) 12010-07-18 10:49同義詞檢索應(yīng)該很多時(shí)候會(huì)用得上的，舉個(gè)簡(jiǎn)單的例子，我們搜索關(guān)鍵字 good 的時(shí)候，與 well 和 fine 等的詞條也可能是你想要的結(jié)果。這里我們不自己建立同義詞庫(kù)，直接使用 WordNet 的同義詞庫(kù)，本篇介紹 C# 版的實(shí)現(xiàn)步驟，還會(huì)有續(xù)篇-Java 版。由于 Lucene 是發(fā)源于 Java，所以 C# 的應(yīng)用者就沒(méi)有 Java 的那么幸福了，Java 版已經(jīng)有 3.0.2 可下載，C# 的版本還必須從 SVN 庫(kù)里：https:/svn

2、./repos/asf/lucene/ 才能取到最新的 2.9.2 的源碼，二制包還只有 2.0 的。接下來(lái)就是用 VS 來(lái)編譯它的，不多說(shuō)。只是注意到在 contrib 目錄中有 WordNet.Net 解決方案，這是我們想要的，編譯 WordNet.Net 可得到三個(gè)可執(zhí)行文件：1. Syns2Index.exe 用來(lái)根據(jù) WordNet 的同義詞庫(kù)建立同義詞索引文件，同義詞本身也是通過(guò) Lucene 來(lái)查詢到的2. SynLookup.exe 從同義詞索引中查找某個(gè)詞有哪些同義詞3. SynExpand.exe 與 SynLookup 差不多，只是多了個(gè)權(quán)重值，大概就

3、是同義程度好啦，有了 Lucene.Net.dll 和上面那三個(gè)文件，我們下面來(lái)說(shuō)進(jìn)一步的步驟：二. 下載 WordNet 的同義詞庫(kù)可以從 /3.0/ 下載 WNprolog-3.0.tar.gz 文件。然后解壓到某個(gè)目錄，如 D:WNprolog-3.0，其中子目錄 prolog 中有許多的 pl 文件，下面要用到的就是 wn_s.pl三. 生成同義詞 Lucene 索引使用命令Syns2Index.exe d:WNprolog-3.0prologwn_s.pl syn_index第二個(gè)參數(shù)是生成索引的目錄，由它來(lái)幫你創(chuàng)建該目錄

4、，執(zhí)行時(shí)間大約 40 秒。這是順利的時(shí)候，也許你也會(huì)根本無(wú)法成功，執(zhí)行 Syns2Index.exe 的時(shí)候出現(xiàn)下面的錯(cuò)誤：Unhandled Exception: System.ArgumentException: maxBufferedDocs must at least be 2 when enabledat Lucene.Net.Index.IndexWriter.SetMaxBufferedDocs(Int32 maxBufferedDocs)at WorldNet.Net.Syns2Index.Index(String indexDir, IDictionary word2Nums

5、, IDictionary num2Words)at WorldNet.Net.Syns2Index.Main(String args)莫急，手中有源碼，心里不用慌，只要找到 Syns2Index 工程，改動(dòng) Syns2Index.cs 文件中的writer.SetMaxBufferedDocs(writer.GetMaxBufferedDocs() * 2*/); /GetMaxBufferedDocs() 本身就為 0，翻多少倍也是白搭為writer.SetMaxBufferedDocs(100); /所以直接改為 100 或大于 2 的數(shù)就行重新使用新編譯的 Syns2Index.exe

6、 執(zhí)行上一條命令即可。成功執(zhí)行后，可以看到新生成了一個(gè)索引目錄 syn_index, 約 3 M?，F(xiàn)在可以用另兩個(gè)命令來(lái)測(cè)試一下索引文件:D:wordnetSynLookup.exe syn_index hiSynonyms found for hi:hawaiihellohowdyhulloD:wordnetSynExpand.exe syn_index hiQuery: hi hawaii0.9 hello0.9 howdy0.9 hullo0.9也可以用 Luke - Lucene Index ToolBox 來(lái)查看索引，兩個(gè)字段，syn 和 word，通過(guò) word:hi 就可以搜索到

7、 syn:hawaii hello howdy hullo四. 使用同義詞分析器、過(guò)濾器進(jìn)行檢索相比，Java 程序員要輕松許多，有現(xiàn)成的 lucene-wordnet-3.0.2.jar，里面有一些現(xiàn)在的代碼可以用。C# 的那些分析器和過(guò)濾器就得自己寫了，或許我已走入了一個(gè)岔道，但也不算崎嶇。小步驟就不具體描述了，直接上代碼，大家從代碼中去理解：同義詞引擎接口view sourceprint?01.using System.Collections.Generic;02.03.namespace Com.Unmi.Searching04.05. / 06. / Summary descript

8、ion for ISynonymEngine07. / 08. public interface ISynonymEngine09. 10. IEnumerable GetSynonyms(string word);11. 12.同義詞引擎實(shí)現(xiàn)類view sourceprint?01.using System.IO;02.using System.Collections.Generic;03.using Lucene.Net.Analysis;04.using Lucene.Net.Analysis.Standard;05.using Lucene.Net.Documents;06.using

9、 Lucene.Net.QueryParsers;07.using Lucene.Net.Search;08.using Lucene.Net.Store;09.10.using LuceneDirectory = Lucene.Net.Store.Directory;11.using Version = Lucene.Net.Util.Version;12.13.namespace Com.Unmi.Searching14.15. / 16. / Summary description for WordNetSynonymEngine17. / 18. public class WordNe

10、tSynonymEngine : ISynonymEngine19. 20.21. private IndexSearcher searcher;22. private Analyzer analyzer = new StandardAnalyzer();23.24. /syn_index_directory 為前面用 Syns2Index 生成的同義詞索引目錄25. public WordNetSynonymEngine(string syn_index_directory)26. 27.28. LuceneDirectory indexDir = FSDirectory.Open(new

11、DirectoryInfo(syn_index_directory);29. searcher = new IndexSearcher(indexDir, true); 30. 31.32. public IEnumerable GetSynonyms(string word)33. 34. QueryParser parser = new QueryParser(Version.LUCENE_29, word, analyzer);35. Query query = parser.Parse(word);36. Hits hits = searcher.Search(query);37.38

12、. /this will contain a list, of lists of words that go together39. List Synonyms = new List();40.41. for (int i = 0; i hits.Length(); i+)42. 43. Field fields = hits.Doc(i).GetFields(syn);44. foreach (Field field in fields)45. 46. Synonyms.Add(field.StringValue();47. 48. 49.50. return Synonyms;51. 52

13、. 53.過(guò)濾器，下面的分析器要用到Lucene 應(yīng)用 WordNet 的同義詞典實(shí)現(xiàn)同義詞檢索(C#版) 22010-07-18 10:49view sourceprint?01.using System;02.using System.Collections.Generic;03.using Lucene.Net.Analysis;04.05.namespace Com.Unmi.Searching06.07. / 08. / Summary description for SynonymFilter09. / 10. public class SynonymFilter : TokenF

14、ilter11. 12. private Queue synonymTokenQueue = new Queue();13.14. public ISynonymEngine SynonymEngine get; private set; 15.16. public SynonymFilter(TokenStream input, ISynonymEngine synonymEngine)17. : base(input)18. 19. if (synonymEngine = null)20. throw new ArgumentNullException(synonymEngine);21.

15、22. SynonymEngine = synonymEngine;23. 24.25. public override Token Next()26. 27. / if our synonymTokens queue contains any tokens, return the next one.28. if (synonymTokenQueue.Count 0)29. 30. return synonymTokenQueue.Dequeue();31. 32.33. /get the next token from the input stream34. Token token = in

16、put.Next();35.36. /if the token is null, then it is the end of stream, so return null37. if (token = null)38. return null;39.40. /retrieve the synonyms41. IEnumerable synonyms = SynonymEngine.GetSynonyms(token.TermText();42.43. /if we dont have any synonyms just return the token44. if (synonyms = nu

17、ll)45. 46. return token;47. 48.49. /if we do have synonyms, add them to the synonymQueue,50. / and then return the original token51. foreach (string syn in synonyms)52. 53. /make sure we dont add the same word54. if (!token.TermText().Equals(syn)55. 56. /create the synonymToken57. Token synToken = n

18、ew Token(syn, token.StartOffset(),58. t.EndOffset(), );59.60. / set the position increment to zero61. / this tells lucene the synonym is62. / in the exact same location as the originating word63. synToken.SetPositionIncrement(0);64.65. /add the synToken to the synonyms queue66. synonymTokenQueue.Enq

19、ueue(synToken);67. 68. 69.70. /after adding the syn to the queue, return the original token71. return token;72. 73. 74.分析器，使用了多個(gè)過(guò)濾器，當(dāng)然最主要是用到了上面定義的同義詞過(guò)濾器view sourceprint?01.using Lucene.Net.Analysis;02.using Lucene.Net.Analysis.Standard;03.04.namespace Com.Unmi.Searching05.06. public class SynonymAna

20、lyzer : Analyzer07. 08. public ISynonymEngine SynonymEngine get; private set; 09.10. public SynonymAnalyzer(ISynonymEngine engine)11. 12. SynonymEngine = engine;13. 14.15. public override TokenStream TokenStream(string fieldName, System.IO.TextReader reader)16. 17. /create the tokenizer18. TokenStre

21、am result = new StandardTokenizer(reader);19.20. /add in filters21. / first normalize the StandardTokenizer22. result = new StandardFilter(result);23.24. / makes sure everything is lower case25. result = new LowerCaseFilter(result);26.27. / use the default list of Stop Words, provided by the StopAna

22、lyzer class.28. result = new StopFilter(result, StopAnalyzer.ENGLISH_STOP_WORDS);29.30. / injects the synonyms.31. result = new SynonymFilter(result, SynonymEngine);32.33. /return the built token stream.34. return result;35. 36. 37.最后，當(dāng)然是要應(yīng)用上面的同義詞引擎和過(guò)濾器，分析器了view sourceprint?01.using System.IO;02.usi

23、ng System.Web;03.using Lucene.Net.Index;04.using System;05.using Lucene.Net.Analysis.Standard;06.using Lucene.Net.Documents;07.using System.Collections.Generic;08.using Lucene.Net.Analysis;09.using Lucene.Net.Search;10.using Lucene.Net.QueryParsers;11.using Lucene.Net.Store;12.using Version = Lucene

24、.Net.Util.Version;13.using System.Collections;14.using Lucene.Net.Highlight;15.16.using LuceneDirectory = Lucene.Net.Store.Directory;17.18.namespace Com.Unmi.Searching19.20. public class Searcher21. 22. / 23. / 假定前面創(chuàng)建的同義詞索引目錄是 d:indexessyn_index，24. / 要搜索的內(nèi)容索引目錄是 d:indexesfile_index, 且索引中有兩字段 file 和

25、 content25. / IndexEntry 是你自己創(chuàng)建的一個(gè)搜索結(jié)果類，有兩屬性 file 和 fragment26. / 27. / queryString28. public static List Search(queryString)29. 30. /Now SynonymAnalyzer31. ISynonymEngine synonymEngine = new WordNetSynonymEngine(d:indexessyn_index);32. Analyzer analyzer = new SynonymAnalyzer(synonymEngine);33. 34.

26、LuceneDirectory indexDir = FSDirectory.Open(new DirectoryInfo(d:indexesfile_index);35. IndexSearcher searcher = new IndexSearcher(indexDir, true);36.37. QueryParser parser = new QueryParser(Version.LUCENE_29,content, analyzer);38. 39. Query query = parser.Parse(queryString);40.41. Hits hits = searcher.Search(query);42.43. /返回類型是一個(gè) IndexEntry 列表，它有兩個(gè)屬性 file 和 fragment44. List entries = new List();45.46. /這里還用到了 Contrib 里的另一個(gè) Lucene 輔助組件，高亮顯示搜索關(guān)鍵字47. SimpleHTMLFormatter simpleHTMLFormatter = new SimpleHTMLFormatter(, );48. Highlighter highlighter = new H