基于h5py的使用及数据封装代码

yipeiwu_com5年前Python基础

1. h5py简单介绍

h5py文件是存放两类对象的容器,数据集(dataset)和组(group),dataset类似数组类的数据集合,和numpy的数组差不多。group是像文件夹一样的容器,它好比python中的字典,有键(key)和值(value)。group中可以存放dataset或者其他的group。”键”就是组成员的名称,”值”就是组成员对象本身(组或者数据集),下面来看下如何创建组和数据集。

1.1 创建一个h5py文件

import h5py
#要是读取文件的话,就把w换成r
f=h5py.File("myh5py.hdf5","w")

在当前目录下会生成一个myh5py.hdf5文件。

2. 创建dataset数据集

import h5py
f=h5py.File("myh5py.hdf5","w")
#deset1是数据集的name,(20,)代表数据集的shape,i代表的是数据集的元素类型
d1=f.create_dataset("dset1", (20,), 'i')
for key in f.keys():
 print(key)
 print(f[key].name)
 print(f[key].shape)
 print(f[key].value)

输出:

dset1
/dset1
(20,)
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
import h5py
import numpy as np
f=h5py.File("myh5py.hdf5","w")
a=np.arange(20)
d1=f.create_dataset("dset1",data=a)
for key in f.keys():
 print(f[key].name)
 print(f[key].value)

输出:

/dset1
[ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19]
2. hpf5用于封装训练集和测试集
#============================================================
# This prepare the hdf5 datasets of the DRIVE database
#============================================================
 
import os
import h5py
import numpy as np
from PIL import Image
 
def write_hdf5(arr,outfile):
 with h5py.File(outfile,"w") as f:
 f.create_dataset("image", data=arr, dtype=arr.dtype)
 
#------------Path of the images --------------------------------------------------------------
#train
original_imgs_train = "./DRIVE/training/images/"
groundTruth_imgs_train = "./DRIVE/training/1st_manual/"
borderMasks_imgs_train = "./DRIVE/training/mask/"
#test
original_imgs_test = "./DRIVE/test/images/"
groundTruth_imgs_test = "./DRIVE/test/1st_manual/"
borderMasks_imgs_test = "./DRIVE/test/mask/"
#---------------------------------------------------------------------------------------------
 
Nimgs = 20
channels = 3
height = 584
width = 565
dataset_path = "./DRIVE_datasets_training_testing/"
 
def get_datasets(imgs_dir,groundTruth_dir,borderMasks_dir,train_test="null"):
 imgs = np.empty((Nimgs,height,width,channels))
 groundTruth = np.empty((Nimgs,height,width))
 border_masks = np.empty((Nimgs,height,width))
 for path, subdirs, files in os.walk(imgs_dir): #list all files, directories in the path
  for i in range(len(files)):
   #original
   print "original image: " +files[i]
   img = Image.open(imgs_dir+files[i])
   imgs[i] = np.asarray(img)
   #corresponding ground truth
   groundTruth_name = files[i][0:2] + "_manual1.gif"
   print "ground truth name: " + groundTruth_name
   g_truth = Image.open(groundTruth_dir + groundTruth_name)
   groundTruth[i] = np.asarray(g_truth)
   #corresponding border masks
   border_masks_name = ""
   if train_test=="train":
    border_masks_name = files[i][0:2] + "_training_mask.gif"
   elif train_test=="test":
    border_masks_name = files[i][0:2] + "_test_mask.gif"
   else:
    print "specify if train or test!!"
    exit()
   print "border masks name: " + border_masks_name
   b_mask = Image.open(borderMasks_dir + border_masks_name)
   border_masks[i] = np.asarray(b_mask)
 
 print "imgs max: " +str(np.max(imgs))
 print "imgs min: " +str(np.min(imgs))
 assert(np.max(groundTruth)==255 and np.max(border_masks)==255)
 assert(np.min(groundTruth)==0 and np.min(border_masks)==0)
 print "ground truth and border masks are correctly withih pixel value range 0-255 (black-white)"
 #reshaping for my standard tensors
 imgs = np.transpose(imgs,(0,3,1,2))
 assert(imgs.shape == (Nimgs,channels,height,width))
 groundTruth = np.reshape(groundTruth,(Nimgs,1,height,width))
 border_masks = np.reshape(border_masks,(Nimgs,1,height,width))
 assert(groundTruth.shape == (Nimgs,1,height,width))
 assert(border_masks.shape == (Nimgs,1,height,width))
 return imgs, groundTruth, border_masks
 
if not os.path.exists(dataset_path):
 os.makedirs(dataset_path)
#getting the training datasets
imgs_train, groundTruth_train, border_masks_train = get_datasets(original_imgs_train,groundTruth_imgs_train,borderMasks_imgs_train,"train")
print "saving train datasets"
write_hdf5(imgs_train, dataset_path + "DRIVE_dataset_imgs_train.hdf5")
write_hdf5(groundTruth_train, dataset_path + "DRIVE_dataset_groundTruth_train.hdf5")
write_hdf5(border_masks_train,dataset_path + "DRIVE_dataset_borderMasks_train.hdf5")
 
#getting the testing datasets
imgs_test, groundTruth_test, border_masks_test = get_datasets(original_imgs_test,groundTruth_imgs_test,borderMasks_imgs_test,"test")
print "saving test datasets"
write_hdf5(imgs_test,dataset_path + "DRIVE_dataset_imgs_test.hdf5")
write_hdf5(groundTruth_test, dataset_path + "DRIVE_dataset_groundTruth_test.hdf5")
write_hdf5(border_masks_test,dataset_path + "DRIVE_dataset_borderMasks_test.hdf5")

遍历文件夹下的所有文件 os.walk( dir )

for parent, dir_names, file_names in os.walk(parent_dir): 
 for i in file_names: 
  print file_name 

parent: 父路径

dir_names: 子文件夹

file_names: 文件名

以上这篇基于h5py的使用及数据封装代码就是小编分享给大家的全部内容了,希望能给大家一个参考,也希望大家多多支持【听图阁-专注于Python设计】。

相关文章

python批量赋值操作实例

变量名和变量值俊发生规律的变化,利用for循环完成赋值操作。 举个例子: for i in range(1, 10): exec("t%d=i"%i) print(t1) pr...

部署Python的框架下的web app的详细教程

部署Python的框架下的web app的详细教程

作为一个合格的开发者,在本地环境下完成开发还远远不够,我们需要把Web App部署到远程服务器上,这样,广大用户才能访问到网站。 很多做开发的同学把部署这件事情看成是运维同学的工作,这种...

python获取array中指定元素的示例

对于array,如2-D的array,如何取指定元素 设array为3*10的shape s = array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],...

利用Python代码实现数据可视化的5种方法详解

利用Python代码实现数据可视化的5种方法详解

前言 数据科学家并不逊色于艺术家。他们用数据可视化的方式绘画,试图展现数据内隐藏的模式或表达对数据的见解。更有趣的是,一旦接触到任何可视化的内容、数据时,人类会有更强烈的知觉、认知和交流...

python实现批量nii文件转换为png图像

之前介绍过单个nii文件转换成png图像: /post/165693.htm 这里介绍将多个nii文件(保存在一个文件夹下)转换成png图像。且图像单个文件夹的名称与nii名字相同。...