Handling List with Huge Amount of Data/Items ? -- the Concept of ContentIterator

How to Handle Large list data in SharePoint?

SPQuery Fails for Large Lists Items?

Unable to Retrieve list item from large lists? 

ContentIterator to the rescue...

so how to use it? 
 If indexed column condition return more value than List View Threshold, it handles by batch..

      Simply include the  Microsoft.Office.Server.dll which is available in 14/ISAPI/

2    Include namespace Microsoft.Office.Server.Utilities. as reference

So what it gives us??

· Fetches list items as a batch so it reduces the load.

· Batch Procesing can be stopped at anytime.

· Alternative to SPQuery as it fails for large data greater than list threshold

It works as follows

SPQuery.ListItemCollectionPosition in 2007 MOSS  helps to fetch large number of items efficiently in the batches. ContentIterator.ProcessListItems method make use of SPQuery.ListItemCollectionPosition internally in a such way that its value is less than the list threshold value .

ContentIterator will run through each item in the list, invoking the callback provided for list item processing—in this case, ProcessItem. If an error occurs while iterating the list, then the error function is invoked—in this case, ProcessError. Using this approach the ContentIterator processes the list in pieces and avoids any excessively large queries. This functionality is provided as part of Enterprise Content Management (ECM) in SharePoint Server 2010.

static int exceptions = 0;

static int items = 0;

protected void OnTestContentIterator(object sender, EventArgs args)

{

    items = 0;

    exceptions = 0;

    string query1 = @"<View>

        <Query>

            <Where>

                <And>

                    <BeginsWith>

                        <FieldRef Name='Title' />

                        <Value Type='Text'>A</Value>

                    </BeginsWith>

                </And>

            </Where>

        </Query>

    </View>";

    ContentIterator iterator = new ContentIterator();

    SPQuery listQuery = new SPQuery();

    listQuery.Query = query1;

    SPList list = SPContext.Current.Web.Lists["Parts"];

    iterator.ProcessListItems(list,listQuery,ProcessItem,ProcessError)

    );

}

public    bool ProcessError(SPListItem item, Exception e)

{

    // process the error

    exceptions++;

    return true;

}

public void ProcessItem(SPListItem item)

{

    items++;

    Do what you wish with the item

}