Python制作数据导入导出工具

yipeiwu_com6年前Python基础

python 2.6编写,自己瞎写的,备用

'''
  Export and Import ElasticSearch Data.
  Simple Example At __main__
  @author: wgzh159@163.com
  @note: uncheck consistency of data, please do it by self
'''
 
import json
import os
import sys
import time
import urllib2
 
reload(sys)
sys.setdefaultencoding('utf-8') # @UndefinedVariable
 
class exportEsData():
  size = 10000
  def __init__(self, url,index,type):
    self.url = url+"/"+index+"/"+type+"/_search"
    self.index = index
    self.type = type
  def exportData(self):
    print("export data begin...")
    begin = time.time()
    try:
      os.remove(self.index+"_"+self.type+".json")
    except:
      os.mknod(self.index+"_"+self.type+".json")
    msg = urllib2.urlopen(self.url).read()
    print(msg)
    obj = json.loads(msg)
    num = obj["hits"]["total"]
    start = 0
    end = num/self.size+1
    while(start<end):
      msg = urllib2.urlopen(self.url+"?from="+str(start*self.size)+"&size="+str(self.size)).read()
      self.writeFile(msg)
      start=start+1
    print("export data end!!!\n\t total consuming time:"+str(time.time()-begin)+"s")
  def writeFile(self,msg):
    obj = json.loads(msg)
    vals = obj["hits"]["hits"]
    try:
      f = open(self.index+"_"+self.type+".json","a")
      for val in vals:
        a = json.dumps(val["_source"],ensure_ascii=False)
        f.write(a+"\n")
    finally:
      f.flush()
      f.close()
 
class importEsData():
  def __init__(self,url,index,type):
    self.url = url+"/"+index+"/"+type
    self.index = index
    self.type = type
     
  def importData(self):
    print("import data begin...")
    begin = time.time()
    try:
      f = open(self.index+"_"+self.type+".json","r")
      for line in f:
        self.post(line)
    finally:
      f.close()
    print("import data end!!!\n\t total consuming time:"+str(time.time()-begin)+"s")
  def post(self,data):
    req = urllib2.Request(self.url,data,{"Content-Type":"application/json; charset=UTF-8"})
    urllib2.urlopen(req)
 
if __name__ == '__main__':
  '''
    Export Data
    e.g.
              URL          index    type
    exportEsData("http://10.100.142.60:9200","watchdog","mexception").exportData()
     
    export file name: watchdog_mexception.json
  '''
  #exportEsData("http://10.100.142.60:9200","watchdog","mexception").exportData()
  exportEsData("http://10.100.142.60:9200","watchdog","mexception").exportData()
   
   
  '''
    Import Data
     
    *import file name:watchdog_test.json  (important)
          "_" front part represents the elasticsearch index
          "_" after part represents the elasticsearch type
    e.g.
              URL          index    type
    mportEsData("http://10.100.142.60:9200","watchdog","test").importData()
  '''
  #importEsData("http://10.100.142.60:9200","watchdog","test").importData()
  importEsData("http://10.100.142.60:9200","watchdog","test").importData()

以上所述就是本文的全部内容了,希望大家能够喜欢。

相关文章

python如何实现视频转代码视频

python如何实现视频转代码视频

本文实例为大家分享了python如何实现视频转代码视频的具体代码,供大家参考,具体内容如下 # -*- coding:utf-8 -*- #coding:utf-8 import...

详解Numpy中的数组拼接、合并操作(concatenate, append, stack, hstack, vstack, r_, c_等)

详解Numpy中的数组拼接、合并操作(concatenate, append, stack, hstack, vstack, r_, c_等)

Numpy中提供了concatenate,append, stack类(包括hsatck、vstack、dstack、row_stack、column_stack),r_和c_等类和函数...

python通过装饰器检查函数参数数据类型的方法

本文实例讲述了python通过装饰器检查函数参数数据类型的方法。分享给大家供大家参考。具体分析如下: 这段代码定义了一个python装饰器,通过此装饰器可以用来检查指定函数的参数是否是指...

Django上使用数据可视化利器Bokeh解析

Django上使用数据可视化利器Bokeh解析

前言 最近在实验室做的一个项目中,需要把大量的数据在 web 端进行可视化,需要绘制各类图表。数据都是以 csv 文件的形式保存在服务器中。本来是想使用 D3.js 这个数据可视化前端库...

利用Hyperic调用Python实现进程守护

利用Hyperic调用Python,实现进程守护,供大家参考,具体内容如下 调用操作系统方法获取进程信息,判断进程是否存在,Linux和Windows均支持,区别在于获取进程信息和启动...