Python中使用md5sum检查目录中相同文件代码分享

yipeiwu_com5年前Python基础

复制代码 代码如下:

"""This module contains code from
Think Python by Allen B. Downey

http://thinkpython.com

Copyright 2012 Allen B. Downey
License: GNU GPLv3 http://www.gnu.org/licenses/gpl.html

"""

import os

def walk(dirname):
    """Finds the names of all files in dirname and its subdirectories.

    dirname: string name of directory
    """
    names = []
    for name in os.listdir(dirname):
        path = os.path.join(dirname, name)

        if os.path.isfile(path):
            names.append(path)
        else:
            names.extend(walk(path))
    return names


def compute_checksum(filename):
    """Computes the MD5 checksum of the contents of a file.

    filename: string
    """
    cmd = 'md5sum ' + filename
    return pipe(cmd)


def check_diff(name1, name2):
    """Computes the difference between the contents of two files.

    name1, name2: string filenames
    """
    cmd = 'diff %s %s' % (name1, name2)
    return pipe(cmd)


def pipe(cmd):
    """Runs a command in a subprocess.

    cmd: string Unix command

    Returns (res, stat), the output of the subprocess and the exit status.
    """
    fp = os.popen(cmd)
    res = fp.read()
    stat = fp.close()
    assert stat is None
    return res, stat


def compute_checksums(dirname, suffix):
    """Computes checksums for all files with the given suffix.

    dirname: string name of directory to search
    suffix: string suffix to match

    Returns: map from checksum to list of files with that checksum
    """
    names = walk(dirname)

    d = {}
    for name in names:
        if name.endswith(suffix):
            res, stat = compute_checksum(name)
            checksum, _ = res.split()

            if checksum in d:
                d[checksum].append(name)
            else:
                d[checksum] = [name]

    return d


def check_pairs(names):
    """Checks whether any in a list of files differs from the others.

    names: list of string filenames
    """
    for name1 in names:
        for name2 in names:
            if name1 < name2:
                res, stat = check_diff(name1, name2)
                if res:
                    return False
    return True


def print_duplicates(d):
    """Checks for duplicate files.

    Reports any files with the same checksum and checks whether they
    are, in fact, identical.

    d: map from checksum to list of files with that checksum
    """
    for key, names in d.iteritems():
        if len(names) > 1:
            print 'The following files have the same checksum:'
            for name in names:
                print name

            if check_pairs(names):
                print 'And they are identical.'


if __name__ == '__main__':
    d = compute_checksums(dirname='.', suffix='.py')
    print_duplicates(d)

相关文章

几个提升Python运行效率的方法之间的对比

几个提升Python运行效率的方法之间的对比

在我看来,python社区分为了三个流派,分别是python 2.x组织,3.x组织和PyPy组织。这个分类基本上可以归根于类库的兼容性和速度。这篇文章将聚焦于一些通用代码的优化技巧以及...

使用 Django Highcharts 实现数据可视化过程解析

使用 Django Highcharts 实现数据可视化过程解析

概述 最近在一家公司实习,入职第一个大一点的需求是将公司开发的两个winstore app的排名信息进行可视化。大概挑选了下,排除了Flask和Echarts。最终选择使用Django...

Python定时执行之Timer用法示例

本文实例讲述了Python定时执行之Timer用法。分享给大家供大家参考。具体分析如下: java中Timer的作用亦是如此。python中的线程提供了java线程功能的子集。 #!...

详解Python中is和==的区别

在Python中一切都是对象。 Python中对象包含的三个基本要素,分别是: id(身份标识) type(数据类型) value(值) 对象之间比较是否相等可以用 ==...

Python+树莓派+YOLO打造一款人工智能照相机

Python+树莓派+YOLO打造一款人工智能照相机

不久之前,亚马逊刚刚推出了DeepLens 。这是一款专门面向开发人员的全球首个支持深度学习的摄像机,它所使用的机器学习算法不仅可以检测物体活动和面部表情,而且还可以检测类似弹吉他等复杂...