python正向最大匹配分词和逆向最大匹配分词的实例

yipeiwu_com5年前Python基础

正向最大匹配

# -*- coding:utf-8 -*-
 
CODEC='utf-8'
 
def u(s, encoding):
  'converted other encoding to unicode encoding'
  if isinstance(s, unicode):
    return s
  else:
    return unicode(s, encoding)
 
def fwd_mm_seg(wordDict, maxLen, str):
  'forward max match segment'
  wordList = []
  segStr = str
  segStrLen = len(segStr)
  for word in wordDict:
    print 'word: ', word
  print "\n"
  while segStrLen > 0:
    if segStrLen > maxLen:
      wordLen = maxLen
    else:
      wordLen = segStrLen
    subStr = segStr[0:wordLen]
    print "subStr: ", subStr
    while wordLen > 1:
      if subStr in wordDict:
        print "subStr1: %r" % subStr
        break
      else:
        print "subStr2: %r" % subStr
        wordLen = wordLen - 1
        subStr = subStr[0:wordLen]
#      print "subStr3: ", subStr
    wordList.append(subStr)
    segStr = segStr[wordLen:]
    segStrLen = segStrLen - wordLen
  for wordstr in wordList:
    print "wordstr: ", wordstr
  return wordList
    
      
def main():
  fp_dict = open('words.dic')
  wordDict = {}
  for eachWord in fp_dict:
    wordDict[u(eachWord.strip(), 'utf-8')] = 1
  segStr = u'你好世界hello world'
  print segStr
  wordList = fwd_mm_seg(wordDict, 10, segStr)
  print "==".join(wordList)
  
 
if __name__ == '__main__':
  main()
  

逆向最大匹配

# -*- coding:utf-8 -*-
 
 
def u(s, encoding):
  'converted other encoding to unicode encoding'
  if isinstance(s, unicode):
    return s
  else:
    return unicode(s, encoding)
 
CODEC='utf-8'
 
def bwd_mm_seg(wordDict, maxLen, str):
  'forward max match segment'
  wordList = []
  segStr = str
  segStrLen = len(segStr)
  for word in wordDict:
    print 'word: ', word
  print "\n"
  while segStrLen > 0:
    if segStrLen > maxLen:
      wordLen = maxLen
    else:
      wordLen = segStrLen
    subStr = segStr[-wordLen:None]
    print "subStr: ", subStr
    while wordLen > 1:
      if subStr in wordDict:
        print "subStr1: %r" % subStr
        break
      else:
        print "subStr2: %r" % subStr
        wordLen = wordLen - 1
        subStr = subStr[-wordLen:None]
#      print "subStr3: ", subStr
    wordList.append(subStr)
    segStr = segStr[0: -wordLen]
    segStrLen = segStrLen - wordLen
  wordList.reverse()
  for wordstr in wordList:
    print "wordstr: ", wordstr
  return wordList
    
      
def main():
  fp_dict = open('words.dic')
  wordDict = {}
  for eachWord in fp_dict:
    wordDict[u(eachWord.strip(), 'utf-8')] = 1
  segStr = ur'你好世界hello world'
  print segStr
  wordList = bwd_mm_seg(wordDict, 10, segStr)
  print "==".join(wordList)
 
if __name__ == '__main__':
  main()
  

以上这篇python正向最大匹配分词和逆向最大匹配分词的实例就是小编分享给大家的全部内容了,希望能给大家一个参考,也希望大家多多支持【听图阁-专注于Python设计】。

相关文章

pytorch cnn 识别手写的字实现自建图片数据

pytorch cnn 识别手写的字实现自建图片数据

本文主要介绍了pytorch cnn 识别手写的字实现自建图片数据,分享给大家,具体如下: # library # standard library import os # th...

Python获取好友地区分布及好友性别分布情况代码详解

Python获取好友地区分布及好友性别分布情况代码详解

利用Python + wxpy 可以快速的查询自己好友的地区分布情况,以及好友的性别分布数量。还可以批量下载好友的头像,拼接成大图。 本次教程是基于上次机器人后的,所有依赖模块都可以复用...

简单谈谈python中的语句和语法

python程序结构 python“一切皆对象”,这是接触python听到最多的总结了。在python中最基层的单位应该就是对象了,对象需要靠表达式建立处理,而表达式往往存在于语句中,多...

python字符串的方法与操作大全

一:字符串的方法与操作 *注意:首字母为l的为从左边操作,为r的方法为从右边操作 1.__contains__()判断是否包含 判断指定字符或字符串是否包含在一个字符串内,返回值为tru...

Django中数据库的数据关系:一对一,一对多,多对多

Django中数据库的数据关系:一对一,一对多,多对多

一对一: 一对一的关系极为一个数据仅对应一个数据,用下图的结构图可以帮助理解: 下面用代码实现一下,首先要创建工程项目如下: 接着,我们定义模型: 来到models.py文件,创建两...