How does Hashing Process work and How to generate ETag(s) for uploaded objects?
    • 14 Dec 2023
    • 1 Minute to read
    • PDF

    How does Hashing Process work and How to generate ETag(s) for uploaded objects?

    • PDF

    Article summary

    There is way to determine whether a file on our local system has changed compared to the version hosted on Wasabi and therefore needs to be re-uploaded. You can generate an ETag locally and compare it with the ETag that's generated on the uploaded objects on Wasabi

    For Non-multipart: The ETag is simply the textual representation of the MD5 checksum of the file.

    For multipart uploads:
    Say you uploaded a 14MB file and your part size is 5MB. Calculate 3 MD5 checksums corresponding to each part, i.e. the checksum of the first 5MB, the second 5MB, and the last 4MB. Then take the checksum of their concatenation. Since MD5 checksums are hex representations of binary data, just make sure you take the MD5 of the decoded binary concatenation, not of the ASCII or UTF-8 encoded concatenation. When that's done, add a hyphen and the number of parts to get the ETag

    Many examples are given here to assist you with the same: 

    https://stackoverflow.com/questions/12186993/what-is-the-algorithm-to-compute-the-amazon-s3-etag-for-a-file-larger-than-5gb/26135975
    Note: A multi-part object can have one part. In that case, the ETag will NOT be the MD5 checksum of the file. It will instead be the MD5 checksum of the textual representation of the MD5 checksum of the file, followed by "-1".