VirtualizingDataList – Displaying Large Datasets

In the last few posts I have introduced a number of features from the data framework included as part of the open-source Cocoon framework for Windows 8 Metro-style apps. Following an introduction to the framework I described the SimpleDataListSource and PagedDataListSource.
To recap, the IDataListSource implementations provide a simple way of accessing data from web APIs. These have been designed to reflect the typical structure of such APIs, generally resulting from a series of stateless HTTP requests to retrieve the data, often a page at a time.
In response to data provided from an IDataListSource, the DataList implementations are designed to request and display the information to the user. The aim here is to provide the user experience expected for a client application, with large scrolling lists of items and suitable support for background on-demand data retrieval.

Introducing The VirtualizingDataList

The first DataList implementation provided in the Cocoon data framework is the VirtualizingDataList<T> class. This can be used for situations where a large scrolling list only displays a small subset of the data at any one time. Only the data in view will be retrieved, with more data being fetched on-demand as the user scrolls through this list. In addition it supports the WinRT ISupportPlaceholder interface to indicate to the displaying UI to display a placeholder element whilst data is being retrieved in the background. This builds on top of Cocoon’s VirtualizingVector class so more information on data virtualization and placeholder support can be found in the associated post.
To use a VirtualizingDataList<T> you simply create a new instance, passing the IDataListSource via its constructor. It can then be bound to any of the many controls for displaying lists of information, most commonly the GridView and ListView controls.
If you are following the MVVM presentation pattern then you will expose the data list via a view model,

public class MyViewModel

{

    // *** Fields ***

    private IList<Person> employees;

    // *** Constructors ***

    public MyViewModel()

    {

        IDataListSource<Person> source = new EmployeesDataListSource();

        employees = new VirtualizingDataList<Person>(source);

    }

    // *** Properties ***

    public IList<Person> Employees

    {

        get

        {

            return employees;

        }

    }

}

Then in your XAML you can define a CollectionViewSource that binds to the data list, and connect that to your items control,

<UserControl.Resources>

    <CollectionViewSource x:Name="itemsViewSource" Source="{Binding Employees}"/>

</UserControl.Resources>

...

<GridView ItemsSource="{Binding Source={StaticResource itemsViewSource}}" ... />

In fact, this is exactly what the Visual Studio 11 Metro-style XAML templates will produce for you.

Summary

In the above discussing I have shown how easy it is to consume any IDataListSource and display it to the user through the fluid scrollable user experience expected of modern applications. When I introduced the Cocoon data framework I discussed “bridging the data divide” between the stateless HTTP calls of web APIs and the continuous scrollable lists displayed to the user. By coding the former as IDataListSource implementations, and the latter as DataLists, Cocoon provides the bridge to span these two worlds.

As usual the code is freely available for download from the Cocoon CodePlex site (to get the latest version go to the “Source Code” tab, select the first change set and use the “Download” link).

Notes

Please note that due to a known issue with the Visual Studio templates in the Consumer Preview (see this forum post) they disable virtualization support for the GridView. Therefore although the VirtualizingVector and VirtualizingDataList classes will still work, they unfortunately fetch all items in the collection rather than on demand. To re-enable the virtualization support you can remove the ScrollViewer that surrounds the GridView, however this will clip elements a little strangely. Hopefully this will be resolved in the final release of Visual Studio, although I am working on a temporary workaround.

Update: Cocoon for Windows Consumer Preview

Following the release of the Windows 8 Consumer preview I would like to announce that the Cocoon framework for Windows 8 Metro-style applications has been updated to support the new bits. As usual the full source code is available for download from the Cocoon CodePlex site (to get the latest version go to the “Source Code” tab, select the first change set and use the “Download” link).

The changes include,

• Code and project system migrated to the Visual Studio 11 Consumer Preview
• Virtualization support converted to use the new ISupportPlaceholder interface rather than IVirtualizingVector
• The VirtualizingVector class is now VirtualizingVector<T> for type-safe access
• Improvements to the internal resource management
• General tidying up of the codebase

Please note that due to a known issue with the Visual Studio templates in the Consumer Preview (see this forum post) they disable virtualization support for the GridView. Therefore although the VirtualizingVector and VirtualizingDataList classes will still work, they unfortunately fetch all items in the collection rather than on demand. To re-enable the virtualization support you can remove the ScrollViewer that surrounds the GridView, however this will clip elements a little strangely. Hopefully this will be resolved in the final release of Visual Studio, although I am working on a temporary workaround.

PagedDataListSource – Consuming Data From Common Web APIs

This is my third post in a series discussing the data framework that is included as part of the open-source Cocoon framework. Previously I have introduced the framework, and detailed the SimpleDataListSource base implementation.

In this post I will describe the more advanced PagedDataListSource base implementation included as part of the framework. This is designed to allow simple integration of data that is retrieved from web APIs as a series of pages. This is a common approach to enable access to large data sets in an efficient manner. In general there will be some way to retrieve the number of items in the data set, and a way to retrieve a single ‘page’ of data (often these can be combined in a single API call).

Consider for example a hypothetical web call,

http://www.example.com/api/getEmployees?page=1

This might return the following XML response,

<PersonResult TotalCount="450" Page="1" PageSize="50">

    <Person Name="Bob"/>

    <Person Name="Dave"/>

    <Person Name="Amy"/>

    ...

</PersonResult>

We can immediately see that there are a total of 450 employees, however the API returns only the first 50 entries. To retrieve subsequent employees you make the same call with the relevant ‘page’ query parameter.

From the point of view of a modern Windows 8 Metro style application, the user does not want to see separate pages of data. Instead they expect a continuous scrolling grid of items, with subsequent data retrieved on demand as it is required to be displayed. It is this mismatch between the paging web API and the desired UI that the PagedDataListSource and the Cocoon data framework sets out to address.

Implementing A PagedDataListSource

The PagedDataListSource<T> class in marked as abstract so you must derive a domain specific custom class for the type of data that you wish to retrieve. There are then three methods that you should override,

• FetchCountAsync – This should retrieve the number of items in the data set


• FetchPageSizeAsync – This should return the number of items per page


• FetchPageAsync – This should return the items for a specified page

Note that each of these returns a Task<DataListPageResult<T>> as their response. DataListPageResult<T> is defined as,

public struct DataListPageResult<T>

{

    // *** Constructors ***

    public DataListPageResult(int? totalItemCount, int? itemsPerPage,

                              int? pageNumber, IList<T> page)

        : this()

    {

        this.TotalItemCount = totalItemCount;

        this.ItemsPerPage = itemsPerPage;

        this.PageNumber = pageNumber;

        this.Page = page;

    }

    // *** Properties ***

    public int? TotalItemCount { get; private set; }

    public int? ItemsPerPage { get; private set;}

    public int? PageNumber { get; private set; }

    public IList<T> Page { get; private set; }

}

This contains all the data that could be returned from any of the above methods. Whilst the requested information must always be present (e.g. a call to FetchCountAsync() must always return a ‘TotalItemCount’) any of the other values may be ‘null’ if they are not known.

The reason for this approach is that in many cases several of these items are returned from a single web API call – for example in the ‘getEmployees’ example above an initial call to GetFetchAsync() will not only return the number of items, but also the page size and the contents of the first page. Hence the implementation can often write FetchCountAsync() and FetchPageSizeAsync() as a simple call to FetchPageAsync(…).

So for our hypothetical ‘getEmployees’ web call we could write,

public class EmployeesDataListSource : PagedDataListSource<Person>

{

    // *** Overriden Base Methods ***

    protected override Task<DataListPageResult<Person>>
                           FetchCountAsync()

    {

        return FetchPageAsync(1);

    }

    protected override Task<DataListPageResult<Person>>
                           FetchPageSizeAsync()

    {

        return FetchPageAsync(1);

    }

    protected async override Task<DataListPageResult<Person>>
                                 FetchPageAsync(int pageNumber)

    {

        PersonResult result = await ExampleEmployeesApi.
                                         GetEmployees(pageNumber);

        return new DataListPageResult<FlickrPhoto>
                      (result.TotalCount, result.PageSize,
                       result.Page, result.Items);

    }

}

The Cocoon framework will then ensure that the correct pages are retrieved as required and that each web API call is only made once, with the results cached for future use.

Summary

Here I have discussed the PagedDataListSource<T> implementation of the Cocoon data framework. As usual the code is freely available for download from the Cocoon CodePlex site (to get the latest version go to the “Source Code” tab, select the first change set and use the “Download” link).

Next time I will show how you link these IDataListSource implementations with a DataList that can then be bound to your Windows 8 Metro style app’s UI. Following this I will show an end-to-end sample of how to the data framework to access a real web API and display it to the end user (Note: If you are really eager, the sample application is available from the latest code drop on the Cocoon CodePlex site).

SimpleDataListSource – Consuming Data From Simple Services

In my last post I introduced the data framework exposed as part of the Cocoon framework project. Whilst the same principles could be applied to any presentation layer, this framework is designed from the ground up to work great with Windows 8 Metro applications written in managed .Net languages. This includes support for several of the advanced features of WinRT based UIs, such as data virtualization (which I discussed in this post).

As I discussed, Cocoon splits the passing of data from a data source into the UI with two parts. The DataListSource (represented by the IDataListSource<T> interface) describes how the data is retrieved and maps well onto the typical web service API calls to do so. At the other end of the pipeline is the DataList whose responsibility it is to determine when to retrieve the data and to present it in a format that is easily bound to the UI.

Whilst this system is designed to be easily extensible, there are a number of typical use-cases that are provided out of the box with the Cocoon framework. In this post I will describe the SimpleDataListSource.

The SimpleDataListSource Class

The SimpleDataListSource<T>, as its name suggests, is an implementation of the IDataListSource<T> interface for use with simple web APIs that return all items in a list with a single API call.

As an example consider the Flickr API method flickr.photos.people.getList that returns all the people that are tagged in a given photo. Since the number of people is likely to be relatively small the API simply takes a photo ID and returns all the people in one go, and is an ideal candidate for the SimpleDataListSource<T>. This is in contrast to a method such as flickr.interestingness.getList that returns all the interesting photos for the current day. In this case they may be many hundred items so the Flickr API splits these into several pages that are returned one at a time.

In code form we could consider this as,

public class FlickrApi

{

    public Task<IList<PeopleTag>> GetPeopleInPhoto(string photoId)

    {

        ...

    }

    public Task<IList<Photo>> GetInterestingPhotos(int page)

    {

        ...

    }

}

Implementing A Simple Data Source

Since SimpleDataSource<T> is an abstract class you must first derive a custom class for a specific data type. The only method you then are required to implement is the FetchItemsAsync() method that will initiate the call to the web API to retrieve all the items.

For our Flickr photo tagging example,

public class TaggedPeopleDataListSource : SimpleDataSource<PeopleTag>

{

    // *** Fields ***

    private FlickrApi flickrApi;

    private string photoId;

    // *** Constructors ***

    public TaggedPeopleDataListSource(string photoId, FlickrApi flickrApi)

    {

        this.flickrApi = flickrApi;

        this.photoId = photoId;

    }

    // *** Overriden Methods ***

    protected override Task<IList<PeopleTag>> FetchItemsAsync()

    {

        return flickrApi.GetPeopleInPhoto(string photoId);

    }

}

Here we firstly derive from ‘SimpleDataListSource<PeopleTag>’ (since every item in the list is of type ‘PeopleTag’). In the constructor we take the photo ID to retrieve the tagged people, and a reference to the ‘FlickrApi’ implementation. In the FetchItemsAsync() method we simply call the API method and return the result.

The underlying SimpleDataListSource<T> implementation will ensure that the web API is only called once, and will cache the results in memory for future use.

As usual the code is freely available for download from the Cocoon CodePlex site (to get the latest version go to the “Source Code” tab, select the first change set and use the “Download” link).

Next Time

Whilst simple scenarios such as those supported by the SimpleDataListSource are relatively basic to implement without the support provided by the Cocoon framework, many web APIs return paged results that are much more difficult to consume. For a fast and responsive UI these should be retrieved asynchronously and on demand as the user scrolls through a long list of items.

Next time I will introduce the PagedDataListSource that allows you to support this with minimal code. After that I will introduce the final piece in the puzzle – the DataList that can be connected to any of these DataListSources for binding to the UI. Finally I will describe a simple end-to-end example of how to use the data framework to bind to a real web service.

Bridging the Data Divide–An Introduction to Cocoon Data Lists

When I first introduced the Cocoon framework one of the targets was to improve the ease at which data retrieval APIs (primarily web APIs) could be integrated into modern desktop applications. In this post I will introduce two concepts, the “data list” and the “data list source”, that are used in the Cocoon framework to simplify this process when writing Windows 8 Metro-style applications, in particular those implementing the MVVM design pattern.

The Data Divide

Let us first consider a typical web API that will return a list of information in response to a web request from a desktop application. In many cases there may be a large number of items in the list, so the API will split the results into several “pages”. For example, the hypothetical web call,

public Task<PersonResult> GetEmployeesAsync(int pageNumber) {...}

Might return the resulting data for pageNumber=1,

<PersonResult TotalCount="450" Page="1" PageSize="50">

    <Person Name="Bob"/>

    <Person Name="Dave"/>

    <Person Name="Amy"/>

    ...

</PersonResult>

Here, whilst there are a total of 450 items in the data set, only the first 50 are returned. If you then require any further items then additional calls must be made for the second, third, etc. pages as required.

In contrast let us consider a modern Windows 8 Metro-style UI. Here the user generally expects to see a single list of all items that can be scrolled through as desired. For performance reasons this list should take advantage of data virtualisation to retrieve data on demand, filling in the UI smoothly as the results are retrieved (see my last post on VirtualizingVector for more information).

It is this “data divide” between the state-less paged result sets of web APIs and the fluid continuous lists of desktop applications that often require much code to implement correctly. This is the problem that “data lists” and “data list sources” in the Cocoon framework are desired to solve.

“Data Lists” and “Data List Sources”

The approach taken by the Cocoon framework is to separate the two concerns of the web API calls (or any other data access call) and the resulting lists to display in the UI into separate components. These can then be developed separately using their own conventions, with the Cocoon framework bridging the divide.

In this model a web API call (or calls) that return information on a list of information is represented as part of a “data list source”. This understands how to query the source API to retrieve the list of data as required. In addition it is your application’s own representation of what it knows about the list of data – the length of the data set, and any items that have already been downloaded. This can be reused amongst different parts of your application and in some ways acts as a local cache of the data. Multiple data list sources will be available, for simple cases where all data is returned at once and for data that is returned by pages, as well as the option for custom implementations.

At the other end of the chain is the “data list”. This is the IList<T> implementation that is bound to the UI, either via the code behind or through a view model. Multiple data lists can be attached to the same data list source so that the application’s view of the data is consistent. Different data list implementations can determine how the data is retrieved for display; for example,

• StaticDataList – This will retrieve all the information before display to the user
• VirtualizingDataList – This will use data virtualization to only retrieve the data items that are currently visible to the user
• IncrementalLoadingDataList – This will initially show a fixed number of items, but allow the user to expand the list if they wish to see more items
• DynamicDataList – This will start with an empty list, with items being added as they are retrieved from the data list source

Between the data list and the data list source are a number of optional processing steps. For example these could constrain the list to only the first 50 items for preview, they could filter the items based on a search term, or could group the items to display as part of a grouped grid view.

The IDataListSource Interface

All of the above is orchestrated by the Cocoon.Data.IDataListSource<T> interface.

public interface IDataListSource<T>

{

    Task<int> GetCountAsync();

    Task<T> GetItemAsync(int index);

}

You can think of this as an asynchronous version of IList<T>, albeit a minimal version. There are two methods:

• GetCountAsync() – Returns a task that results in the number of items in the list.


• GetItemAsync(int index) – Returns a task that results in the item at the specified index.

Note that the GetItemAsync(…) method should throw an exception if ‘index’ is less than zero, however if ‘index’ is greater than the last item it should return default(T) (for reference types this will be null).

Summary

To summarize, the Cocoon framework supplies an infrastructure designed to bridge the gap between state-less web APIs and modern fluid Windows 8 Metro style applications. Over the next few blog posts (and associated code drops) I will provide a number of generic implementations of data lists and data list sources to make consuming lists of data from the web quick and easy.

Data Virtualization in Metro-style apps with ISupportPlaceholder

Update 8th April 2012 : Modified to reflect changes upon release of Windows 8 Consumer Preview - References to IVirtualizingVector replaced with ISupportPlaceholder.

In my last post I introduced an implementation of the IObservableVector<T> interface that allows Windows 8 Metro-style applications to automatically update data bound XAML items controls as the underlying data source changes. In this post I am going to delve one stage deeper by demonstrating an implementation of the ISupportPlaceholder interface that enables data virtualization

A Background to Data Virtualization

To start I will provide some background to data virtualization and explain why this is becoming essential for modern applications to improve performance with large data sets. In a traditional, non-data-virtualized application, lists of data must be fully loaded into memory before they can be displayed to the user. When this data is held in a local database or file system this may mean loading many hundred items into memory before being shown in the user interface. With modern cloud-connected applications this becomes more of an issue as they must all be downloaded across what is a relatively slow connection.
In web applications this problem has been solved by the concept of paging – only a small number of items are shown at any one time, and the user can move between the pages by following links. In client based applications however the user expects a more seamless experience – a single list that can be scrolled through as desired.
Data virtualization solves this problem by only downloading and storing in memory the data that is required, deferring the download of further items until they are scrolled into view. The user sees a single list as if they are viewing the total data set, however as they move through the list any newly visible items are obtained on demand.

Data Virtualization for Metro-Style Apps

In Windows 8, Metro style applications support virtualized lists through the ISupportPlaceholder interface,

public interface ISupportPlaceholder

{

    bool IsPlaceholder(object item);

}

At first sight it is not immediately obvious how this can be used to virtualize a list, however it provides the vital link between a virtualization aware data set and a data bound user interface. An ISupportPlaceholder implementation is responsible for providing any data virtualization logic,

• In general the list should behave as you would expect for an IObservableVector<T> implementation
• When the items are requested the list should return the actual item if this is available, or a placeholder item if it has yet to be downloaded
• The IsPlaceholder(…) method may then be called with each item returned in the list, and the ISupportPlaceholder implementation should return true if this is a placeholder item, otherwise false
• It is also the responsibility of the  ISupportPlaceholder implementation to download any requested items in the background and update the list as these are available.

In return the standard Metro-style items controls will show any items for which IsPlaceholder(…) returns true as a grey rectangle,

An ISupportPlaceholder Implementation

To assist with data virtualization in .Net based Metro-style applications a base implementation is provided as part of the freely available Cocoon framework. This is provided as the Cocoon.Data.VirtualizingVector abstract class, into which you simply provide your custom data retrieval logic. You will either be requested for the number of items in the list or a specific item, and once obtained should then call the UpdateCount(…) or UpdateItem(…) methods

public class PersonCollection : VirtualizingVector

{

    protected override void FetchCount()

    {

        // Perform custom code here to retrieve

        //     the number of items in the dataset

        UpdateCount(itemCount);

    }

    protected override void FetchItem(int index)

    {

        // Perform custom code here to retrieve

        //     the specified item from the dataset

         UpdateItem(index, retrievedItem);

    }

}

The VirtualizingVector class then takes care of implementing IObservableVector<T> and ISupportPlaceholder, returning of suitable placeholder objects for items as they are requested, and updating items with real data as it is made available.

Of course this approach would still consume large amounts of memory when displaying a list with a large number of placeholder items. The VirtualizingVector class overcomes this by using the provided VirtualizingList<T> implementation. Without going into detail, this behaves as you would expect for a list with a large number of elements, whilst allocating memory efficiently to store only those items that are not virtualized.

Summary

In conclusion, the Cocoon framework provides an ISupportPlaceholder implementation designed to enable data virtualization for Metro-style applications. The code is freely available for download from the Cocoon CodePlex site (to get the latest version go to the “Source Code” tab, select the first change set and use the “Download” link).
In my introduction to the Cocoon framework I stated that one of my aims was to bridge the divide between the stateless paging model of most web APIs and the “fast and fluid” interfaces of Metro-style applications. Now that I have provided the fundamentals, next time I will try to bridge this gap by introducing “data lists” and “data list sources”.

ObservableVector as a replacement for ObservableCollection in Metro-style apps

For those of you who have had previous experience in making WPF and Silverlight applications you are probably familiar with the ObservableCollection<T> class. This represents a list of items (implementing IList<T>) whilst also raising the INotifyCollectionChanged.CollectionChanged event whenever items are added, removed or moved within the collection. When such collections are databound to the various list controls in WPF and Silverlight the UI will automatically update to reflect any changes.

ObservableCollection<T> and Metro-Style Apps

If you have followed the same technique when writing Windows 8 Metro-style apps you will have found that this no longer works. There are a couple of reasons for this behaviour,

1. In the Developer Preview version of Windows 8 the INotifyPropertyChanged interface is duplicated in both the System.ComponentModel and Windows.UI.Xaml.Data namespaces. Whilst ObservableCollection<T> implements the former, the new Metro XAML framework uses the latter.
2. Rather than INotifyCollectionChanged as implemented by ObservableCollection<T>, WinRT uses the IObservableVector<T> interface.

Observable Collections in Metro-Style Apps

Whilst this issue is likely to be addressed prior to release of Windows 8, for the time being there are a few workarounds.

1. The Windows 8 SDK Data Binding sample includes an ObservableVector class that takes an existing object that implements INotifyCollectionChanged, and wraps it in an IObservableVector<T> implementation.
2. Colin Eberhardt has posted a similar solution on his blog.
3. Avi Pilosof has taken a slightly different approach by deriving a new class from ObservableCollection<T>, implementing both INotifyCollectionChanged and IObservableVector<T>.

Whilst these solve the problem, they all involve wrapping the existing ObservableCollection<T> class, resulting in events being raised for both the old and new approaches. This is ideal for cases where you wish to write interoperable code for both existing and Metro applications, but is an added overhead when you are natively writing a new Windows 8 application.

A Metro-Designed Observable Collection Class

In response to this I have written a ground-up ObservableVector<T> class for use in Windows 8 Metro-style applications that does not have the overhead of an underlying ObservableCollection<T>. The code has been released as part of the freely available Cocoon framework. You can use this class by creating a new instance directly,

IList<Person> list = new ObservableVector<Person>();

list.Add(new Person("Andrew", "Wilkinson"));

or by deriving a custom class,

public class PersonCollection : ObservableVector<Person>

{

    ...

}

It is important to note however that in the current Windows Developer Preview there is a bug which means that databinding only works correctly to lists when the implement ObservableVector<object> rather than strongly typed collections. This is likely to change in the future.

In both cases the behaviour is designed to match that expected by WPF and Silverlight developers. There are a small number of subtle differences with respect to the previous ObservableCollection<T> class,

1. Since IObservableVector<T> does not have the same concept of moving items it does not implement the Move() method. Instead you should remove the item and then add it to its new place in the collection.


2. The constructor of the ObservableVector<T> class optionally accepts an IList<T> that is wrapped by the implementation rather than copied internally – the reason for this will become apparent in later posts.


3. Currently no attempts are made to stop VectorChanged event handlers from modifying the original collection.

Summary

In conclusion, the Cocoon framework provides an IObservableVector<T> implementation specifically designed from the ground-up for Metro applications. The code is freely available for download from the Cocoon CodePlex site (to get the latest version go to the “Source Code” tab, select the first change set and use the “Download” link).

Next time I will discuss data virtualisation and the IVirtualizingVector interface.