Storage and content delivery

Overview

10Duke SDK defines two main concepts related to storing data:

• file storage accessed via StorageProvider

• data storage accessed via DataProvider

Data storage relates to storing object data that can be relational and hierarchical, and that is typically (but not necessarily) stored in a relational database. Programmers see the stored data as object model objects.

This chapter describes the file storage, where programmers usually see the stored data as files. In the web, content delivery and Content Delivery Networks (CDN) are closely related to file storage.

Storage provider

10Duke SDK accesses and stores files using StorageProvider. StorageProvider offer uniform access and management of files, independent of the actual storage and content delivery. StorageProvider makes it easy to simply store files on the local disk, but it also allows storing files redundantly in a distributed system or using an external storage service like Amazon S3. In web applications, it is simple to deliver content to end users directly from the local disk, but also different kinds of Content Delivery Networks (CDN) can be easily hooked up.

Two main classes needed to use StorageProviders are:

• StorageProviders is a singleton class that manages and provides access to actual StorageProvider instances

• AbstractStorageProvider is abstract base class used by StorageProvider implementing classes. 10Duke SDK includes a basic storage provider implementation ready to use by applications. Developers can also implement own storage providers and configure them for the StorageProviders.

Usage of StorageProviders and AbstractStorageProvider is illustrated by the following example:

//
// Get default storage provider
AbstractStorageProvider storageProvider = StorageProviders.instance().getDefaultStorageProvider();
//
// Store a file using the default storage provider
File fileToStore = new File("MyVideo.mp4");
storageProvider.put(fileToStore, "MyVideo.mp4", "container_1");
//
// Retrieve the stored file
StoredObjectInformation storedObjectInformation = storageProvider.get("MyVideo.mp4", "container_1");
File retrievedFromStore = storedObjectInformation.getFile();

In this example, file name is used as identifier for the stored file, and "container_1" is used as container name. In actual applications key and container should be selected so that they uniquely identify each stored file.

Most applications can always use the default storage provider returned by StorageProviders.instance().getDefaultStorageProvider() as illustrated by the example above. StorageProviders also has more advanced features that may be needed when using or implementing redundant, distributed storage providers.

Configuration

StorageProviders supports defining several storage provider stacks for using or implementing redundant, distributed storage providers. These stacks are:

• Default storage provider, which is the main storage provider stack used by applications to access storage

• Internal local disk storage provider, which is used internally by storage provider implementations that need to access implementation internal access to local disk storage

• Internal LAN tier storage provider, which is used internally by storage provider implementations that need implementation internal access to LAN tier storage

• Internal WAN tier storage provider, which is used internally by storage provider implementations that need implementation internal access to WAN tier storage

Each storage provider stack uses internally ConfigurableChainedStorageProvider. ConfigurableChainedStorageProvider allows configuring stacks. Stack configurations allows different redundant and distributed storage implementations. For instance, a storage provider stack can work so that all files are primarily stored in a storage service like Amazon S3, and some files are selectively cached on local disk. Example storage provider stack configuration is given below:

<ObjectStorage>
   <StorageProvider className="configure fully qualified class name of StorageProvider 1 here"
                    configurationPath="configure path relative to base configuration path for provider 1 here">
   </StorageProvider>
   <StorageProvider className="configure fully qualified class name of StorageProvider 2 here"
                    configurationPath="configure path relative to base configuration path for provider 2 here">
   </StorageProvider>
   ...
   <StorageProvider className="configure fully qualified class name of StorageProvider N here"
                    configurationPath="configure path relative to base configuration path for provider N here">
   </StorageProvider>
</ObjectStorage>

<entry key="content.storage.localdisk.classname" >com.my.localdiskstorageprovider</entry>
<entry key="content.storage.localdisk.configurationpath" >./myconf/localdiskconf.xml</entry>
<entry key="content.storage.lantier.classname" >com.my.lanstorageprovider</entry>
<entry key="content.storage.lantier.configurationpath" >./myconf/lantierconf.xml</entry>
<entry key="content.storage.wantier.classname" >com.my.wanstorageprovider</entry>
<entry key="content.storage.wantier.configurationpath" >./myconf/wantierconf.xml</entry>

Distributed storage

Implementation for distributed storage are provided as extension modules for the 10Duke SDK. Cloud storage providers can be integrated with ease and the base SDK itself includes a StorageProvider implementation for using Amazon S3. Users of the SDK are also welcome to create own implementations of storage providers.

The 10Duke extension module for distributed storage is based on a concept for separation on three levels:

• Local disk storage
• LAN tier storage
• WAN tier storage

The motivation for this logical structure follows from efficiency and speed to access files on local disk vs. LAN and. WAN.