Method for storing a data file

Abstract

A method for storing a data file (DF) on a storage entity (SE) includes receiving, by a proxy (PE) and from a computing entity (CE), a plurality of hash values corresponding to a plurality of blocks of the DF. The PE may check whether the plurality of blocks of the DF are stored in the SE based on the plurality of hash values. Based on determining that at least a subset of the plurality of blocks of the DF are not being stored in the SE, the PE may compute a secret associated with an encryption key. The PE may transmit, to the CE, the secret. The PE may receive, from the CE, information including storage locations of the subset of the plurality of blocks within the SE and one or more hash values, of the plurality of hash values, associated with the subset of the plurality of blocks.

Claims

1. A method for storing a data file (DF) on a storage entity (SE), the method comprising: receiving, by a proxy (PE) and from a computing entity (CE), a plurality of hash values corresponding to a plurality of blocks of the DF; checking, by the PE, whether the plurality of blocks of the DF are stored in the SE based on the plurality of hash values; based on determining that at least a subset of the plurality of blocks of the DF are not being stored in the SE, computing, by the PE, a secret associated with an encryption key; transmitting, by the PE and to the CE, the secret; receiving, by the PE and from the CE, information comprising: storage locations of the subset of the plurality of blocks within the SE; and one or more hash values, of the plurality of hash values, associated with the subset of the plurality of blocks; and based on a usage counter reaching zero, deleting, by the PE and for all blocks used by the DF, encryption keys and/or chunking keys associated with the DF.

2. The method according to claim 1, wherein computing the secret comprises computing the secret based on proxy information.

3. The method according to claim 1, wherein computing the secret comprises computing the secret based on CE information.

4. The method according to claim 1, wherein the encryption key is used by the CE to encrypt the subset of the plurality of blocks.

5. The method according to claim 1, the method further comprising: checking, by the PE and at a regular time interval, at least a portion of the DF that is stored on the SE to verify its correctness.

6. The method of claim 1, the method further comprising: receiving, by the PE and from the CE, a query request requesting information about the DF; and providing, by the PE and to the CE, the information about the DF.

7. The method according to claim 1, the method further comprising: prior to receiving the plurality of hash corresponding to the plurality of blocks of the DF, receiving, by the PE and from the CE, an initial hash value for the DF; and checking, by the PE and based on the initial hash value, whether the DF has been stored in its entirety in the SE.

8. The method according to claim 1, wherein the PE is provided in form of one or more client computing entities.

9. The method according to claim 1, wherein the PE is a keyserver.

10. The method according to claim 1, the method further comprising: receiving, by the PE and from the CE, a request to delete the DF; and deleting the DF in the SE based on the request wherein deleting the encryption keys and/or the chunking keys associated with the DF is based on the request to delete the DF.

11. The method according to claim 10, further comprising: decrementing, by the PE, the usage counter for all blocks used by the DF, and wherein deleting the DF in the SE comprises: informing, by the PE, the SE to delete the DF.

12. The method according to claim 1, wherein deleting the encryption keys and/or the chunking keys associated with the DF comprises: transmitting, by the PE and to the CE, a command to delete the encryption keys and/or the chunking keys associated with the DF on the CE.

13. The method according to claim 1, further comprising: distributing, by the PE and to the CE, metadata comprising a distributed hash table.

14. A proxy device (PE) that is used for storing a data file (DF) on a storage entity (SE), the PE comprising one or more hardware processors which, alone or in combination, are configured to provide for execution of the following: receiving, from a computing entity (CE), a plurality of hash values corresponding to a plurality of blocks of the DF; checking whether the plurality of blocks of the DF are stored in the SE based on the plurality of hash values; based on determining that at least a subset of the plurality of blocks of the DF are not being stored in the SE, computing a secret associated with an encryption key; transmitting, to the CE, the secret; receiving, from the CE, information comprising: storage locations of the subset of the plurality of blocks within the SE; and one or more hash values, of the plurality of hash values, associated with the subset of the plurality of blocks; and based on a usage counter reaching zero, deleting, for all blocks used by the DF, encryption keys and/or chunking keys associated with the DF.

15. The PE according to claim 14, wherein computing the secret comprises computing the secret based on proxy information.

16. The PE according to claim 14, wherein computing the secret comprises computing the secret based on CE information.

17. The PE according to claim 14, wherein the one or more hardware processors are further configured to provide for execution of the following steps: checking, at a regular time interval, at least a portion of the DF that is stored on the SE to verify its correctness.

18. The PE according to claim 14, wherein the one or more hardware processors are further configured to provide for execution of the following steps: receiving, from the CE, a request to delete the DF; and deleting the DF in the SE based on the request, wherein deleting the encryption keys and/or the chunking keys associated with the DF is based on the request to delete the DF.

19. The PE according to claim 18, wherein the one or more hardware processors are further configured to provide for execution of the following steps: decrementing the usage counter for all blocks used by the DF, and wherein deleting the DF in the SE comprises: informing the SE to delete the DF.

20. A tangible, non-transitory computer-readable medium having instructions thereon which, upon being executed by one or more processors, alone or in combination, provide for execution of the following: receiving, from a computing entity (CE), a plurality of hash values corresponding to a plurality of blocks of a data file (DF); checking whether the plurality of blocks of the DF are stored in a storage entity (SE) based on the plurality of hash values; based on determining that at least a subset of the plurality of blocks of the DF are not being stored in the SE, computing a secret associated with an encryption key; transmitting, to the CE, the secret; receiving, from the CE, information comprising: storage locations of the subset of the plurality of blocks within the SE; and one or more hash values, of the plurality of hash values, associated with the subset of the plurality of blocks; and based on a usage counter reaching zero, deleting, for all blocks used by the DF, encryption keys and/or chunking keys associated with the DF.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) The present invention will be described in even greater detail below based on the exemplary figures. The invention is not limited to the exemplary embodiments. All features described and/or illustrated herein can be used alone or combined in different combinations in embodiments of the invention. The features and advantages of various embodiments of the present invention will become apparent by reading the following detailed description with reference to the attached drawings which illustrate the following:

(2) FIG. 1 shows a conventional variable-sized block-based deduplication method;

(3) FIG. 2 shows part of a system according to an embodiment of the present invention;

(4) FIG. 3 shows part of the steps of a method according to an embodiment of the present invention;

(5) FIG. 4 shows part of the steps of a method according to an embodiment of the present invention;

(6) FIG. 5 shows part of the steps of a method according to an embodiment of the present invention;

(7) FIG. 6 shows part of the steps for a chunking key protection according to an embodiment of the present invention; and

(8) FIG. 7 shows part of the steps of a method according to an embodiment of the present invention.

DETAILED DESCRIPTION

(9) Although applicable in general, the present invention will be described with regard to deduplication.

(10) Although applicable in general to any kind of deduplication, the present invention will be described with regard to block-based deduplication.

(11) One of the problems addressed by embodiments of the present invention is that variable-size block-based deduplication techniques leaking significantly more information about file contents than file-based deduplication techniques. A further problem addressed by embodiments of the present invention is to enhance storage efficiency and data confidentiality.

(12) In an embodiment, the present invention provides a method for storing a data file, ‘DF’ on a storage entity, ‘SE’, including the steps of: a) Chunking the DF into a number of blocks using a one-way-function and a chunking key by a computing entity, ‘CE’, b) Computing a hash value for each of the blocks by the CE, c) Checking, by one or more proxies, ‘PE’, if the blocks are already stored resulting in a first number of already stored blocks and a second number of blocks not being stored, d) Encrypting, by the CE, the blocks not being stored using an encryption key, e) Transmitting, by the CE, the encrypted blocks to the SE for storing and f) Informing, by the CE, the PE about the hash values of the transmitted and corresponding storage location information of the transmitted blocks.

(13) In a further embodiment the present invention provides a method for reading a data file stored with a method according to embodiments of the present invention, that includes the steps of, A) Querying by the CE one or more of the PE for information about the DF, B) Providing the information about the DF by the queried proxies, C) Downloading the blocks of the DF by the CE, D) Decrypting the downloaded blocks, and E) Merging the decrypted blocks to the DF.

(14) In a further embodiment the present invention provides a method for deleting a data file stored with a method according to embodiments of the present invention, including the steps of: A1) Requesting, by the CE, the deletion of the DF from the PE, B1) Decrementing a usage counter for all blocks used by the DF, C1) Deleting by the PE for all blocks used by the DF, when the usage counter reaches zero, the corresponding encryption keys and/or chunking keys, D1) Informing the SE to delete the DF E1) Deleting the corresponding encryption keys and/or chunking keys on the CE.

(15) In a further embodiment the present invention provides a system for storing a data file, ‘DF’, including: At least one computing entity, ‘CE’, a proxy, ‘PE’, and a storage entity, ‘SE’ for storing the DF, the CE being adapted to a) Chunk the DF into a number of blocks using a one-way-function and a chunking key, b) Compute a hash value for each of the blocks, d) Encrypt the blocks not being stored using an encryption key, e) Transmit the encrypted blocks to the SE for storing and f) Inform, the PE about the hash values of the transmitted and corresponding storage location information of the transmitted blocks, and the PE being adapted to d) Check if the blocks are already stored resulting in a first number of already stored blocks and a second number of blocks not being stored.

(16) In a further embodiment the present invention provides a non-transitory computer readable medium storing a program causing one or more computing devices to execute a process for storing a data file, ‘DF’ on a storage entity, ‘SE’, including the steps of: a) Chunking the DF into a number of blocks using a one-way-function and a chunking key, b) Computing a hash value for each of the blocks, c) Checking if the blocks are already stored resulting in a first number of already stored blocks and a second number of blocks not being stored, d) Encrypting the blocks not being stored using an encryption key, e) Transmitting the encrypted blocks to the SE for storing, and f) Informing about the hash values of the transmitted and corresponding storage location information of the transmitted blocks.

(17) At least one of the embodiments has the advantage that security deduplication is enabled without leaking any information about the contents. A further advantage of at least one embodiment of the present invention is a protection against a strong attack which is common to all block-based deduplication techniques where an adversary can guess the files by simply of serving the size of stored blocks.

(18) The term “data file” refers to any kind of information or information parts, which may or may not be associated with each other in a data structure like a computer file or the like.

(19) The terms “storage entity”, “proxy entity”, “proxy”, “computing entity”, “client”, “client computing entity” refer each to a device adapted to perform computing like a personal computer, a tablet, a mobile phone, a server, or the like and includes one or more processors having one or more cores and may be connectable to a memory for storing an application which is adapted to perform corresponding steps of one or more of the embodiments of the present invention. Any application may be software based and/or hardware based installed in the memory on which the processor(s) can work on. The devices may be adapted in such a way that the corresponding steps to be computed are performed in an optimized way. For instance different steps may be performed in parallel with a single processor on different of its cores. Further one or more of the devices may be identical or combined forming a single computing device.

(20) The term “chunking key” refers to any kind of information or information parts, which may enable, include, etc. information to be used for dividing or chunking information like da to file, or the like into different parts.

(21) The term “encryption key” refers to any kind of information or information parts, which may enable, include, etc. information to be used for encryption of information.

(22) The term “hypervisor” refers to a virtual machine monitor or the like and may be instantiated as a piece of computer software, firmware or hardware that creates and runs virtual machines. These virtual machines run on one or more computing devices and may be adapted in such a way to perform certain steps or execute certain computer-implemented code.

(23) The term “computer readable medium” may refer to any kind of medium, which can be used together with a computation device or computer and on which information can be stored. The information may be any kind of data which can be read into a memory of a computer. For example the information may include program code for executing with the computer. Examples of a computer readable medium are tapes, CD-ROMs, DVD-ROMs, DVD-RAMs, DVD-RWs, BluRay, DAT, MiniDisk, solid state disks SSD, floppy disks, SD-cards, CF-cards, memory-sticks, USB-sticks, EPROM. EEPROM or the like.

(24) Prior to step a) a hash value over the DF may be computed by the CE and the hash value may be transmitted by the CE to the PE to check if the DF has already been stored. This allows in an efficient way to check if the complete DF has already been stored.

(25) If the DF has already been stored, the steps a)-f) may not performed. This allows to avoid unnecessary execution of steps a)-f) if the complete data file has already been stored. Thus, computational resources are saved.

(26) The encryption key may be computed based on at least one of: contents of the DF, a secret. This enables in an efficient way to generate an encryption key including information of the DF to be stored.

(27) The secret may be based on proxy information and/or CE information. This enhances the security in particular when for example the secret is based on proxy information and CE information.

(28) The secret may be computed by the PE and provided to the CE and/or is computed by the CE. This enables for example to enhance flexibility, since the secret may be computed by an entity, e.g. other than a CE.

(29) The one way function may be based on AES. AES is the Advanced Encryption Standard also known as Rijndael, and is a specification for the encryption of electronic data established by the U.S. National Institute of Standards and Technology (NIST) in 2001 as announced in Federal Information Processing Standards Publication 197, United States National Institute of Standards and Technology (NIST), Nov. 26, 2001, Retrieved Oct. 2, 2012. AES can be then used as a checksum function in order to achieve non-reversibility of the checksumming process and sufficient entropy for the checksumming parameters. Thus, AES enables to chunk a file based on the secret key without revealing the secret key or the plaintext.

(30) The chunking key is protected by at least one of: trusted computing, hardware dongle, secure multi-party computation. Since every client—assuming that there is a plurality of clients connected to the storage entity needs to use the chunking key and since this chunking key protects data confidentiality it is a potential weak point for an adversary. Using at least one of trusted computing, hardware dongle and secure multi-party computation enhances the security. Most commercially available systems support trusted computing and therefore an easy implementation can be provided.

(31) Hardware dongles, on the other hand, can in particular be used within a corporate scenario. Hardware dongles can be distributed among users. The chunking key could then be stored on these temper-proved dongles to prevent key leakage. The plaintext would be sent to the dongle which would execute the checking procedure and return the chunking boundaries. Since an adversary would only be able to pick inputs to the chunking procedure and observe their—for example AES-based—outputs, the adversary would not learn the chunking key.

(32) When employing secure multi-party computation (as for example disclosed in the non-patent literature of Wilko Henecka and Thomas Schneider, “Faster secure two-party computation with less memory,” in Proceedings of the 8th ACM SIGSAC Symposium on Information, Computer and Communications Security, ASIA CCS '13, pages 437-446, New York, N.Y., USA, 2013 ACM) this enables to limit the key exposure. The chunking key may then be stored and provided by a different entity. In the multi-party protocol the parties would not learn the other parties input. In this way a client cannot learn the chunking key while the key-storing party cannot learn the plaintext. For example if the proxy entity serves as second party, only a single entity would have to know the chunking key. If a trusted computing component or hardware dongle would serve as a second party, the key would be protected due to their tamper-proof nature.

(33) The trusted computing may be provided by a hypervisor entity as CE. Since already deployed TPMs usually do not offer AES support a hypervisor can be used to protect the chunking key. The hypervisor can either use static or dynamic route-of-trust techniques so that the trusted code would be able to access the chunking key. Additionally the hypervisor entity may be used in connection with further features or properties such as secure deletion or the like.

(34) The hypervisor entity may communicate with the PE by one or more secure channels. This enables a confidential integrity-protected exchange of information between the one or more PE and the hypervisor entity.

(35) The one-way function may be an oblivious one-way function. An oblivious one-way function further enhances the security in protecting the chunking key while being efficient.

(36) The CE may be a client computing entity. This enhances flexibility since the corresponding steps can be performed on a client itself.

(37) The PE may regularly check at least part of the stored information on the SE and verify its correctness. This enables to detect cheating clients and storage errors so that their impact is limited.

(38) The P are provided in form of client computing entities. This avoids a central bottleneck and a single point of failure since then the data transmitted to or from the proxies is distributed among the CE directly, for example through a distributed hash table. Additionally, a check for availability and reachability of the clients can be implemented. Thus security is enhanced.

(39) There are several ways how to design and further develop the teaching of the present invention in an advantageous way. To this end it is to be referred to the patent claims subordinate to the independent patent claims on the one hand and to the following explanation of further embodiments of the invention by way of example, illustrated by the figure on the other hand. In connection with the explanation of the further embodiments of the invention by the aid of the figures, generally further embodiments and further developments of the teaching will be explained.

(40) FIG. 1 shows a conventional variable-sized block-based deduplication method.

(41) In FIG. 1 a scheme of variable-sized block-based deduplication is shown. Across a file F, a so-called sliding window is moved and a checksum is computed at every byte-offset. If the checksum matches a specified so-called magic value, a chunk boundary is inserted. Otherwise the window keeps moving. The checksum function is used to achieve the non-reversibility of the checksumming process and sufficient entropy for checksumming parameters.

(42) FIG. 2 shows a system according to an embodiment of the present invention.

(43) In FIG. 2 an overall setup is shown with a plurality of clients C communicating with a keyserver K exchanging meta-data and keys with the corresponding clients C. The clients C exchange encrypted blocks with a storage cloud/provider S, wherein the storage provider S is a third-party storage provider. The clients C and the keyserver K are—in FIG. 2—part of a company network. The clients C are known and therefore trusted as they can be identified and punished in case of misbehavior. The keyserver K is run by the company and the clients C know the chunking key for chunking.

(44) FIG. 3 shows part of the steps of a method according to an embodiment of the present invention.

(45) In FIG. 3 steps for writing a file F assuming the scenario shown in FIG. 2 are shown. Communication with the keyserver K is not shown.

(46) In a first step C computes a cryptographic hash over F and sends it to K to check if F is already known. If F is known, the operation is complete.

(47) In a second step if F is not known C chunks the file using the AES approach and kc.

(48) In a third step C computes cryptographic hashes for the blocks and sends them to K to check if they are already known.

(49) In a fourth step C generates the encryption key k.sub.e. The generation can involve a second party and its secrets as well as C's secrets and the contents of F.

(50) In a fifth step C encrypts every unknown block using a key k.sub.e and uploads it to S.

(51) In a sixth step C informs K about the hashes, the used random keys k.sub.e and the storage locations for the previously unknown blocks.

(52) FIG. 4 shows part of the steps of a method according to a further embodiment of the present invention.

(53) In FIG. 4 steps for reading a file F are shown.

(54) In a first step C queries K for the metadata of F.

(55) In a second step K sends C the order of blocks within F, their encryption keys and storage locations.

(56) In a third step C downloads the blocks, decrypts them and merges them.

(57) FIG. 5 shows part of the steps of a method according to a further embodiment of the present invention.

(58) In FIG. 5 steps for deleting a file F are shown.

(59) In a first step C requests K to delete of F.

(60) In a second step K decrements the usage counter on all blocks that are used by F.

(61) In a third step for all blocks where the counter reaches 0, K securely deletes the block keys and informs the storage provider S to delete the data.

(62) In a fourth step K distributes commands to all the clients C to securely delete the matching keys.

(63) FIG. 6 shows part of the steps for a chunking key protection according to a further embodiment of the present invention.

(64) In FIG. 6 steps for a chunking key protection by trusted computing using a hypervisor H are shown.

(65) In a first step the hypervisor H reads F from the local storage and performs all the operations, such as hashing or chunking.

(66) In a second step H establishes a secure channel to K that allows the confidential and integrity-protected exchange of information.

(67) In a third step the hypervisor protects k.sub.c as well as intermediate computation from access through an operating system or applications.

(68) In a fourth step at the end, the hypervisor clears all relevant memory and ensures the secure deletion.

(69) FIG. 7 shows part of the steps of a method according to a further embodiment of the present invention.

(70) In a first step a client C computes a cryptographic hash over file F and sends it to the proxy to check if F has already been stored. If so, the operation is complete.

(71) In a second step the client C chunks the file using a one-way function and key k.sub.c.

(72) In a third step the client C computes a cryptographic hash over all the outputted blocks and sends them to the proxy K to check if they are already known.

(73) In a fourth step the client encrypts every unknown block using a key derived from the file contents and a secret derived by the proxy K (e.g. in an oblivious protocol) and uploads the blocks to the storage server S.

(74) In a fifth step the client C informs the proxy K about the hashes, and the storage locations of the newly stored blocks.

(75) The above steps provides secure deletion, data confidentiality and storage savings based on variable-sized block-based deduplication. It serves as a scheme and can be instantiated in different ways: The keyserver K can keep achieving secure deletion of encryption keys by storing them locally on a securely deleting medium. Either in this case the medium has to scale linearly with the overall storage size at the storage provider.

(76) Further to achieve secure deletion based on a fixed size medium the keyserver K can employ conventional techniques as disclosed for example in the non-patent literature of Joel Reardon, Hubert Ritzdorf, David Basin, and Srdjan Capkun, “Secure data deletion from persistent media, in Proceedings of the 2013 ACM SIGSAC Conference on Computer 38; Communications Security, CCS '13, pages 271-284, New York, N.Y., USA, 2013. ACM.

(77) In embodiment's of the present invention the keyserver K also distributes metadata. In order to avoid a central bottleneck in a single point-of-failure the metadata could also be distributed among the clients C directly, for example through a distributed hash table. Additional requirements about availability and reachability for the clients C may apply.

(78) To detect misbehaving clients or storage failures, a machine, for example the keyserver K may regularly check some of the stored data and verify its correctness. This enables to detect cheating clients and storage errors and limits their impact.

(79) To summarize the present invention does not reveal the data located at the boundaries or an identifiable size pattern, only—by their construction—block sizes and thus the rough overall file size. Therefore leakage is reduced through this component compared with conventional systems and methods. Further secure deletion is enabled as clients C and the key server K securely deletes keys and metadata for discarded data blocks. Finally as the keyserver K is able to observe access patterns and shared content between different files, the keyserver can also offer assisted deletion to notify user about files that should be deleted. Consequently this frees storage space further reducing storage costs and prevents inadvertent information leakage.

(80) The present invention enables a secure deduplication of blocks in a cloud without leaking any information about the content. Further, the present invention enables protection against a strong attack which is common to all block-based deduplication techniques where the adversary can guess the files by simply observing the size of the stored blocks. Thus, the present invention protects the confidentiality of deduplicated data even against side channel attacks where the adversary can guess content based on deduplicated block sizes.

(81) Further, the present invention enables 1) Chunking a file into variable size blocks using a trapdoor one-way function based on a secret which is not known to the adversary. 2) Performing the aforementioned operation in an oblivious manner by executing an oblivious protocol with the proxy who holds the secret key. 3) Performing the aforementioned operation without leaking the secret by leveraging support from secure hardware (e.g., trusted computing or secure dongle). 4) Achieving secure deletion over block-based deduplication.

(82) Many modifications and other embodiments of the invention set forth herein will come to mind to the one skilled in the art to which the invention pertains having the benefit of the teachings presented in the foregoing description and the associated drawings. Therefore, it is to be understood that the invention is not to be limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

(83) While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive. It will be understood that changes and modifications may be made by those of ordinary skill within the scope of the following claims. In particular, the present invention covers further embodiments with any combination of features from different embodiments described above and below. Additionally, statements made herein characterizing the invention refer to an embodiment of the invention and not necessarily all embodiments.

(84) The terms used in the claims should be construed to have the broadest reasonable interpretation consistent with the foregoing description. For example, the use of the article “a” or “the” in introducing an element should not be interpreted as being exclusive of a plurality of elements. Likewise, the recitation of “or” should be interpreted as being inclusive, such that the recitation of “A or B” is not exclusive of “A and B,” unless it is clear from the context or the foregoing description that only one of A and B is intended. Further, the recitation of “at least one of A, B and C” should be interpreted as one or more of a group of elements consisting of A, B and C, and should not be interpreted as requiring at least one of each of the listed elements A, B and C, regardless of whether A, B and C are related as categories or otherwise. Moreover, the recitation of “A, B and/or C” or “at least one of A, B or C” should be interpreted as including any singular entity from the listed elements, e.g., A, any subset from the listed elements, e.g., A and B, or the entire list of elements A, B and C.