TRANSPARENT DATA MOVEMENT BETWEEN A PRIVATE CLOUD AND STORAGE ECOSYSTEM AND ANOTHER STORAGE SYSTEM
20190205056 ยท 2019-07-04
Inventors
Cpc classification
G06F3/0646
PHYSICS
G06F16/16
PHYSICS
G06F3/0605
PHYSICS
H04L67/1097
ELECTRICITY
G06F3/067
PHYSICS
International classification
Abstract
The disclosure provides methods and systems for transparent data movement between a private cloud storage ecosystem and another storage system. The other storage system may be local storage or external storage separate and apart from the private cloud storage ecosystem. Originally, metadata and the data contents of a file are both stored on local storage in the private cloud storage ecosystem. The method separates the metadata from the data contents of a file such that the metadata and data contents are independently operable. After separation and based on policy, the data content is transparently moved between the private cloud storage system and the other storage system. The data is managed and tracked such that a user, e.g., a client or external program/entity, may access the data content using the original metadata stored on the private cloud storage ecosystem, despite the movement of the data contents to the other storage system.
Claims
1. A computer-implemented method for providing transparent movement of data between a private cloud storage ecosystem and another storage system, the method comprising separating metadata from data contents of one or more files stored in a private cloud storage ecosystem, the separating including storing the metadata and data contents for each of the one or more files separately such that the metadata and data contents are independently operable; and based on a policy, selectively moving the data contents of the one or more files transparently between the private cloud storage ecosystem and another storage system, the other storage system being a target storage system separate from the private cloud storage ecosystem.
2. The method of claim 1, wherein the respective metadata for each of the one or more files remains stored on the private cloud storage ecosystem after movement of the corresponding data contents to the other storage system.
3. The method of claim 1, wherein the other storage system is other, slower local storage in the private cloud storage ecosystem.
4. The method of claim 1, wherein the other storage system is external storage that is coupled via one or more communication networks to the private cloud storage ecosystem.
5. The method of claim 4, wherein the external storage is a public cloud storage system, a different private cloud storage system, or disk storage on an external server.
6. The method of claim 1, further comprising managing and tracking movement of the selectively moved data contents of each file of the one or more files.
7. The method of claim 6, wherein the managing and tracking comprises: storing in a database an identification of each of the one or more files for which the respective data contents is being or has been selectively moved and the new destination for the respective data contents that has been selectively moved; and tracking the movement of the data contents to the other storage, the corresponding metadata remaining stored in its original location on the private cloud storage ecosystem.
8. The method of claim 1, further comprising, in response to a request to the private cloud storage ecosystem from a client or an external program/entity, providing access to the moved data contents of the one or more files.
9. The method of claim 8, wherein the request is a read or write request from the client or the external program/entity to the private cloud storage system.
10. The method of claim 8, wherein the providing access to the moved data contents of the one or more files comprises: based on the respective metadata stored on the private cloud storage system for the moved data contents, obtaining a destination of the moved data contents from the database.
11. The method of claim 10, further comprising, based on the destination, providing read or write access for the respective data contents of the one or more files on the other storage system.
12. The method of claim 8, wherein the access to the moved data contents is provided to the client or an external program/entity via a network file sharing protocol, such that the client or an external program/entity accesses the requested file contents by accessing the respective metadata stored on the private cloud storage ecosystem using the network file sharing protocol.
13. The method of claim 12, wherein the standard network file sharing protocol includes Service Message Block (SMB), Network File System (NFS), or Apple File Protocol (AFP) such as utilized for a Network Attached Storage system (NAS).
14. The method of claim 12, wherein the data contents is accessible to the client or the external program/entity via a network file sharing protocol using the respective metadata such that workflow or client mount points are unchanged as a result of the movement of the data contents of the one or more files.
15. The method of claim 1, wherein the policy is a policy specifying Hierarchal Storage Management (HSM).
16. The method of claim 1, wherein the policy is a user-defined policy or combination of user-defined policies.
17. The method of claim 16, where the user-defined policy for selecting files to move may specify one or more of: selecting a folder and all its contents; selecting files that have not been accessed in a threshold period of time; setting a file size with a directory structure that should be moved; selecting particular file types; and using a metadata tag on files to override any other user-defined policy.
18. The method of claim 17, wherein the metatag is based on a geographical restriction so as to not permit transparent data movement out of the geographic location due to regulations of a geographic area.
19. A system for providing transparent movement of data, the system comprising at least one processor and a memory storing processor-executable codes, wherein the at least one processor is configured to implement the following operations upon executing the processor-executable codes: separate metadata from data contents of one or more files stored in a private cloud storage ecosystem, the separating including storing the metadata and data contents for each of the one or more files separately such that the metadata and data contents are independently operable; and based on a policy, selectively move the data contents of the one or more files transparently between the private cloud storage ecosystem and another storage system, the other storage system being a target storage system separate from the private cloud storage ecosystem.
20. A non-transitory processor-readable medium having instructions stored thereon, which when executed by one or more processors, cause the one or more processors to implement a method for providing transparent movement of data between a private cloud storage ecosystem and another storage system, the method comprising: separating metadata from data contents of one or more files stored in a private cloud storage ecosystem, the separating including storing the metadata and data contents for each of the one or more files separately such that the metadata and data contents are independently operable; and based on a policy, selectively moving the data contents of the one or more files transparently between the private cloud storage ecosystem and another storage system, the other storage system being a target storage system separate from the private cloud storage ecosystem.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Embodiments are illustrated by way of example, and not by limitation in the figures of the accompanying drawings, in which like references indicate similar elements.
[0014]
[0015]
[0016]
[0017]
[0018]
[0019]
[0020]
[0021]
DETAILED DESCRIPTION
[0022] The following detailed description includes references to the accompanying drawings, which form a part of the detailed description. The drawings show illustrations in accordance with example embodiments. These example embodiments, which are also referred to herein as examples, are described in enough detail to enable those skilled in the art to practice the present subject matter. The embodiments can be combined, other embodiments can be utilized, or structural, logical, and electrical changes can be made without departing from the scope of what is claimed. The following detailed description is therefore not to be taken in a limiting sense, and the scope is defined by the appended claims and their equivalents. In this document, the terms a and an are used, as is common in patent documents, to include one or more than one. In this document, the term or is used to refer to a nonexclusive or, such that A or B includes A but not B, B but not A, and A and B, unless otherwise indicated.
[0023] The techniques of the embodiments disclosed herein may be implemented using a variety of technologies. For example, the methods described herein may be implemented in software executing on a computer system or in hardware utilizing either a combination of microprocessors or other specially designed application-specific integrated circuits, programmable logic devices, or various combinations thereof. In particular, the methods described herein may be implemented by a series of computer-executable instructions residing on a storage medium, such as a disk drive or computer-readable medium. It should be noted that methods disclosed herein can be implemented by a computing device (e.g., a desktop computer, a tablet computer, a laptop computer, or other computing device).
[0024] Systems and methods are provided in various embodiments for transparent data movement between a private cloud and storage ecosystem and another storage system. Various embodiments include separating metadata from data (contents) stored in a private cloud and storage ecosystem, such that the metadata and the data can operate independently (i.e., are independently operable). Once there is the separation of metadata and data, various embodiments provide for transparent data movement between the private cloud and storage ecosystem and other storage systems. As used herein for various embodiments, the private cloud and storage ecosystem is private in the sense of not being openly available on the public Internet. The other storage system may be other local storage or external storage. For local storage, data contents may be moved, for example, from faster to slower, less costly disks in the same private cloud storage system. Data contents may be moved to less costly external storage. The external storage may be a public cloud storage system, another private cloud storage system, disk storage on an external server, or other external storage system.
[0025]
[0026] According to various embodiments, in the architecture in
[0027]
[0028] One of the other advantages of the TDM technology according to various embodiments, is that a client or external program/entity using the storage system never needs to change workflow or client mount points to account for the movement of the data contents since the data contents are accessible using the original metadata, i.e., the movement of the data has been made transparent to the user.
[0029] In various embodiments, a TDM server executing the method of the present technology can transparently move data to back-end servers or cloud (targets) without affecting the client or external program/entity, i.e., the movement is transparent to them. Various exemplary embodiments support other storage including back-end servers or the cloud (targets) that support Network File System (NFS) and Amazon Simple Storage Service (S3), to name a few. Example targets are shown in
[0030]
[0031] For the example in
[0032] As shown in the example in
[0033] According to various embodiments, the transparent data movement is based on policy. TDM can also eliminate data migrations, fork-lift upgrades, and enable 3rd party systems to take advantage of available cyber and compliance features.
[0034] In various embodiments, data can be selected for transparent data movement based on a policy specifying Hierarchal Storage Management (HSM) or may be based on a different user-defined policy or combination of policies. These policies can take into account several characteristics of the file in choosing whether the file should be selected for transparent data movement to other storage (e.g., another storage tier) or excluded. If selected for TDM, the content data would be moved to another tier of storage based on what tier is selected, e.g., by the policy. For example, according to one user-defined policy, a user can define a folder and all its contents to be selected for TDM and a target tier. In another example, a user can define a policy that selects files that have not been accessed in a threshold period of time (stale files), e.g., nobody has read/opened this file in a certain time period. A policy may be one in which the user can set a size of file within a directory structure that should be moved or a combination of size and last time accessed. According to another example policy, a user can choose file types that should be specifically selected for TDM or excluded from TDM. The user can use a metadata tag on files that would override and have precedence for what files are selected for TDM. Tags may be based on geographical restrictions, for example, not permitting transparent data movement out of the geography due to certain regulations. For instance, the data may be covered under the General Data Protection Regulation (GDPR) that addresses export of personal data outside the European Union.
[0035]
[0036] The example process 400 in
[0037] In the example
[0038]
[0039] Step 4 in the example in
[0040] The resultant storage of the data (contents) after the transparent data movement steps in
[0041]
[0042] The example in
[0043] Step 2 in
[0044] In Step 3, the TDM process 420 gets from the database 430 the file destination that is associated with movement of the data contents.
[0045] In Step 4, a read/write operation is performed by the TDM process 420 on external (NSF/S3) storage (e.g., for File2:Data read/writes for data that had been transparently moved to the external storage 470) or local storage 440 (e.g., for File1:Data read/writes for data that had been transparently moved to local storage 440).
[0046] It should be appreciated that the local storage 440, fast storage 450, slower storage 460, and external storage 470 refer to the physical storage devices in
[0047]
[0048] At operation 705 in the example in
[0049] At operation 710, based on a policy, the method includes selectively moving the data contents of the one or more files transparently between the private cloud storage ecosystem and another storage system. In various embodiments, the other storage system is a target storage system separate from the private cloud storage ecosystem. The other storage system may be local storage in the private cloud storage ecosystem. The other storage may be external storage that is separate and apart from the private cloud storage ecosystem, e.g., a public cloud storage system, a different private cloud storage system, or disk storage on an external server, to name a few.
[0050] In various embodiments, the selective data movement to other storage is transparent in the sense that the client or external program/entity can access the data contents in the same manner as before it was transparently moved (e.g., using the same workflow or client mount points and the original metadata). In various embodiments, the metadata for each of the one or more files remains stored in its original location on the private cloud storage ecosystem, even though the corresponding file contents has been moved to the other storage system. In some embodiments, pursuant to an applicable policy, there may be transparent movement of the data contents of a file from fast storage in the private cloud storage system to slow storage. In some embodiments, the end user may later indicate that the end user no longer wants management of that file. Based on this, the data contents may be transparently moved back to fast storage on the private cloud storage ecosystem or another private cloud system; and remain on the fast storage until an end user policy dictates that the data contents should be moved back to another storage system.
[0051]
[0052] In an example operation, a client or external program/entity creates a read/write request to the private storage ecosystem for a file. At operation 810, in response to the request, the example method provides access to the moved data contents of the one or more files. At operation 815, the providing access to the moved data contents of the one or more files comprises, based on the corresponding metadata stored on the private cloud storage system for the moved data contents, obtaining a destination of the moved file contents from the database. At operation 820, based on the destination, read or write access for the respective data contents (that has been transparently moved as described herein) of the one or more files on the other storage system is provided.
[0053] In various embodiments, the request from a client or the external program/entity is communicated to the private cloud storage system, upon which, access is provided for the moved data contents associated with the requested files. The client or the external program/entity can access moved data content using requests associated with the original metadata stored on the private cloud storage ecosystem. According to various embodiments, in creating the request, the client or the external program/entity need not be aware that the data contents has been independently and transparently moved to other storage.
[0054]
[0055] The example computer system 900 includes a processor or multiple processors 905 (e.g., a central processing units (CPU), a graphics processing unit (GPU), or both), a main memory 910, and a static memory 915, which communicate with each other via a bus 920. Computer system 900 can further include a video display 925 (e.g., a LCD, projector). Computer system 900 also includes at least one input device 930, such as an alphanumeric input device (e.g., a keyboard, keypad, remote control, graphical user interface, etc.), a cursor control device (e.g., a mouse), a microphone, a digital camera, a video camera, and so forth. Computer system 900 also includes a disk drive unit 935, a signal generation device 940 (e.g., a speaker), and a network interface device 945.
[0056] Drive unit 935 (also referred to as the disk drive unit 935) includes a machine-readable medium 950 (also referred to as a computer-readable medium 950), which stores one or more sets of instructions and data structures (e.g., instructions 955) embodying or utilized by any one or more of the methodologies or functions described herein. Instructions 955 can also reside, completely or at least partially, within the main memory 910 and/or the processors 905 during execution thereof by computer system 900. Main memory 910 and processors 905 also constitute machine-readable media.
[0057] Instructions 955 can further be transmitted or received over a communications network 980 via network interface device 945 utilizing one or more transfer protocols (e.g., Hyper Text Transfer Protocol (HTTP), Serial, and Modbus). Communications network 960 includes the Internet, television network, local intranet, Personal Area Network (PAN), Local Area Network (LAN), Wide Area Network (WAN), Metropolitan Area Network (MAN), virtual private network (VPN), storage area network (SAN), frame relay connection, Advanced Intelligent Network (AIN) connection, synchronous optical network (SONET) connection, Digital Data Service (DDS) connection, Digital Subscriber Line (DSL) connection, Ethernet connection, Integrated Services Digital Network (ISDN) line, cable modem, Asynchronous Transfer Mode (ATM) connection, or a Fiber Distributed Data Interface (FDDI) or Copper Distributed Data Interface (CDDI) connection. Furthermore, communications also includes links to any of a variety of wireless networks including Wireless Application Protocol (WAP), General Packet Radio Service (GPRS), Global System for Mobile Communication (GSM), Code Division Multiple Access (CDMA) or Time Division Multiple Access (TDMA), cellular phone networks, Global Positioning System (GPS), cellular digital packet data (CDPD), Research in Motion, Limited (RIM) duplex paging network, Bluetooth radio, or an IEEE 802.11-based radio frequency network.
[0058] While machine-readable medium 950 is shown in an example embodiment to be a single medium, the term computer-readable medium should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The term computer-readable medium shall also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by the machine and that causes the machine to perform any one or more of the methodologies of the present application, or that is capable of storing, encoding, or carrying data structures utilized by or associated with such a set of instructions. The term computer-readable medium shall accordingly be taken to include, but not be limited to, solid-state memories, optical and magnetic media. Such media can also include, without limitation, hard disks, floppy disks, flash memory cards, digital video disks, RAM, ROM, and the like.
[0059] The example embodiments described herein can be implemented in an operating environment comprising computer-executable instructions (e.g., software) installed on a computer, in hardware, or in a combination of software and hardware. The computer-executable instructions can be written in a computer programming language or can be embodied in firmware logic. If written in a programming language conforming to a recognized standard, such instructions can be executed on a variety of hardware platforms and for interfaces to a variety of operating systems. Although not limited thereto, computer software programs for implementing the present method can be written in any number of suitable programming languages such as, for example, Hypertext Markup Language (HTML), Dynamic HTML, XML, Extensible Stylesheet Language (XSL), Document Style Semantics and Specification Language (DSSSL), Cascading Style Sheets (CSS), Synchronized Multimedia Integration Language (SMIL), Wireless Markup Language (WML), Java, Jini, C, C++, C#, .NET, Adobe Flash, Perl, UNIX Shell, Visual Basic or Visual Basic Script, Virtual Reality Markup Language (VRML), ColdFusion or other compilers, assemblers, interpreters, or other computer languages or platforms.
[0060] Although embodiments have been described with reference to specific example embodiments, it will be evident that various modifications and changes can be made to these example embodiments without departing from the broader spirit and scope of the present application. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.