Android Paging 3 library with page and limit parameters

Also, one of the main cons was the lack of support for combined network and local queries, which is actually a very common case in our apps.

Luckily, Paging 3 came to the rescue and as it looks like it performs better and it’s actually not THAT complicated to implement. However, there’s always a but!

Oh RemoteMediator, the great source combinator

The new library introduces the RemoteMediator class. It helps you combine local and remote queries to provide consistent data flow to the user, regardless if the network is available or not.

Basically, everything you fetch from a remote source should be saved to your local database so users can quickly access it later at app startup until the data is refreshed from the remote source. Of course, this will also mean that your app supports offline mode, so users can load everything that was previously in the database without an actual network connection.

Paging using item keys seems defaultish

The idea of how the RemoteMediator should work and combine local and remote data is by using item keys (or IDs), not page number parameters.

For example, you have 5 items in your local database, the last one has an id = abcd.
In this type of API pagination, to query the next page of items, the API requires a parameter afterId, which means it would return a limit of items after the specified id.

GET BASE_URL/items?afterId=abcd&limit=5

This query will return the next 5 items, after the item with id = abcd.

This is actually demoed in Google’s Architecture Components repository with an example of Reddit and it works quite nicely, but…

You need to create an extra model and database table to pair up item IDs with remote keys and then write DAO for it.

It makes sense for this type of API, but what about if your API is using simple page & limit parameters?

Paging using page & limit parameters

To my disappointment, it looks like there is no easier way to paginate using just page number and limit. 

We tried to use AtomicInteger to increment page numbers between loads, but that was inconsistent. Also, the RemoteMediator reacts to how the user scrolls the list, so if you go back up, it will request the previous page, so you need to have smarter logic than just a plain integer variable.

Ultimately, we used the same solution as for pagination with item keys, only that our keys were not actually item IDs but the numbers of the next and previous page.

Models and database setup

Let’s create a model for the items we want to paginate. For example, let’s say we have a news feed, we can name the item NewsItem. This is your basic model that will correspond to your API specification, but it’s important to note the ID of the object.

@Entity
data class NewsItem(
        @PrimaryKey
        val id: Long,
        val  title: String,
        val imageUrl: String?,
        val createdAt: Long
)

For RemoteMediator to work, we will create another class that will actually hold the information about the previous and next page number for each item.

@Entity
data class NewsItemRemoteKeys(
        @PrimaryKey 
        val newsId: Long,
        val prevKey: Int?,
        val nextKey: Int?
)

In the case where you have paging with item keys (afterId, beforeId), the prevKey and nextKey would be the ID’s of the previous and next NewsItem.

But here, to achieve pagination using page number, prevKey and nextKey will represent the previous and next page number of the current item. 

If you’re loading in the batch of 20 items and the current page is 4, all these 20 items on page 4 will have a prevKey=3 and nextKey=5.

This will help RemoteMediator later to know which page to load next.

Create DAOs for your new models

Of course, to access the list of your items and show them to the user, you will need a DAO that will enable you to observe or to fetch them.

For your NewsItem, three DAO functions will be enough to achieve proper pagination.

@Dao
interface NewsDao {
      @Insert(onConflict = OnConflictStrategy.REPLACE)
      suspend fun insertNewsList(news: List<NewsItem>)
      @Query("SELECT * FROM NewsItem ORDER BY NewsItem.createdAt DESC")
      fun observeNewsPaginated(): PagingSource<Int, NewsItem>
      @Query("DELETE FROM NewsItem")
      fun deleteNewsItems(): Int
  }

Function insertNewsList() will enable you to save new pages of data, observeNewsPaginated() will create a PagingSource directly from the Room database and connect it with your PagingDataAdapter instance.

When refreshing data, you will use deleteNewsItems() to delete local copies and freshly save new batches fetched from the server.As for NewsRemoteKeyDao, you will also need three DAO functions.

@Dao
interface NewsRemoteKeyDao {
      @Insert(onConflict = OnConflictStrategy.REPLACE)
      fun insertAll(remoteKey: List<NewsRemoteKeys>)
      @Query("SELECT * FROM NewsRemoteKeys WHERE newsId = :newsId")
      fun remoteKeysByNewsId(newsId: Long): NewsRemoteKeys?
      @Query("DELETE FROM NewsRemoteKeys")
      fun clearRemoteKeys()
  }

We will need to save all of the NewsRemoteKeys objects after fetching a fresh batch of items from the server and assign them their proper prevKey and nextKey values. 

The remoteKeysByNewsId() function will help the RemoteMediator to get a specific NewsRemoteKeys object so it knows which page to load next.

Finally, same as for NewsItem DAO, you will need to clear all of the NewsRemoteKeys when the user performs a refresh.

How to RemoteMediator 

Here’s the hardest part. We have to load data from a remote source, save it to the database and somehow link the two into a one pageable flow.

Let’s break it down into several parts.
First, to create your RemoteMediator instance, you will create a new class which extends RemoteMediator by specifying some generic types.

@OptIn(ExperimentalPagingApi::class)
class NewsPageKeyedRemoteMediator(
      private val initialPage: Int = 1,
      private val db: AppDatabase,
      private val api: ApiInterface
) : RemoteMediator<Int, NewsItem>() {
      override suspend fun load(loadType: LoadType, state: PagingState<Int, NewsItem>): MediatorResult {
  }
}

By extending from RemoteMediator<Int, NewsItem>  I’m specifying that the pagination key will be integer (which will represent page number), and that the items that will be paginated are eof type NewsItem.

Notice that I’m passing a reference to the database instance and my API interface. We will  need this in a minute.

Now, all the magic is created in the load() function which should return some MediatorResult.

MediatorResult can be Success or Error, depending on if we successfully fetched and saved all the items and their corresponding RemoteKeys items or not.

Which page to load

First, inside the load() method, you need to calculate which page should be loaded.

// calculate the current page to load depending on the state
      val page = when (loadType) {
LoadType.REFRESH -> {
      val remoteKeys = getRemoteKeyClosestToCurrentPosition(state)
remoteKeys?.nextKey?.minus(1) ?: initialPage
}
LoadType.PREPEND -> {
      return MediatorResult.Success(true)
}
LoadType.APPEND -> {
      val remoteKeys = getRemoteKeyForLastItem(state)
?: throw InvalidObjectException("Result is empty")
remoteKeys.nextKey ?: return MediatorResult.Success(true)
  }
}
…
// outside of load function
private suspend fun getRemoteKeyForLastItem(state: PagingState<Int, NewsItem>): NewsRemoteKeys? {
      return state.lastItemOrNull()?.let { news ->
db.withTransaction { db.newsRemoteKeysDao().remoteKeysByNewsId(news.id) }
  }
}
private suspend fun getRemoteKeyClosestToCurrentPosition(state: PagingState<Int, NewsItem>): NewsRemoteKeys? {
      return state.anchorPosition?.let { position ->
state.closestItemToPosition(position)?.id?.let { id ->
db.withTransaction { db.newsRemoteKeysDao().remoteKeysByNewsId(id) }
    }
  }
}

If the loadType is REFRESH, then we need to check on which scroll position we are currently, which item it is and find its corresponding NewsRemoteKeys object so we can see which page is a current page.

Using the function getRemoteKeyClosestToCurrentPosition() we can get the NewsRemoteKeys which points to the current item at this scroll position, so the value of (nextKey – 1) will actually be current page that we need to refresh. 

If there is no corresponding NewsRemoteKeys object, that means that we don’t have anything in the database yet and we are starting from the beginning (initialPage).

If the loadType is APPEND, then we are gonna look for the last item in the list and see what it’s NewsRemoteKeys data specifies as the next page.

Using the function getRemoteKeyForLastItem() we can get the last NewsRemoteKeys and by so, we can get the appropriate next page number from its nextKey value.

If there is no such object, that means that we reached the end of the pagination and in this case we will return MediatorResult.Success with endOfPaginationReached = true .

Prepending the data is similar to the APPEND, it just goes in a different direction. For example, if your initialPage is 10 (let’s say you’re starting from the middle of your items list), then you can paginate towards the first page.

The function getRemoteKeyForFirstItem() would be similar to getRemotKeyForLastItem(), except you take the state.firstItemOrNull() value, instead of state.lastItemOrNull() value.

Here in this case, where we don’t have pagination in two directions, we just return for PREPEND type result Success with endOfPaginationReached = true in this direction.

Load remote data and update database

Now  when we know which  page  to   load, we can simply create a new network request and  fetch the data we need. 

Here you can see that we are using pageSize value from the pagination config object and the page number we previously calculated.

To see if we reached the end, for now we just compare if the number of items in the result is lower than the requested number, which would mean there are no more items on the server side. But this logic can be custom and you can do it your own way (if you have metadata in your API response which you can use for this, do it).

// load the list of items from API using calculated current page.
// make sure the sort of the remote data and local data is the same!
      val response = api.getNews(
limit = state.config.pageSize,
page = page
).data
// add custom logic, if you have some API metadata, you can use it as well
      val endOfPaginationReached = response.size < state.config.pageSize

The important thing here is to have the same sort locally (in your DAO) as it is on the remote source, otherwise the RemoteKeys will not be ordered properly and your next & previous page numbers might be wrong.

Now, finally, let’s save the data into the database so it can be rendered to the user.

At this point, it’s important to do these operations in a transaction so both items and their RemoteKeys objects are saved (or canceled) together.
First, you need to check if the loadType was REFRESH. If it was, then delete the items and their RemoteKeys so we store only the fresh new ones received from the server.

db.withTransaction {
// if refreshing, clear table and start over
      if (loadType == LoadType.REFRESH) {
db.newsRemoteKeysDao().clearRemoteKeys()
db.newsDao().deleteNewsItems()
}
      val prevKey = if (page == initialPage) null else page - 1
      val nextKey = if (endOfPaginationReached) null else page + 1
      val keys = response.map {
NewsRemoteKeys(newsId = it.id, prevKey = prevKey, nextKey = nextKey)
}
db.newsRemoteKeysDao().insertAll(keys)
db.newsDao().insertNewsList(response)
}
return MediatorResult.Success(endOfPaginationReached = endOfPaginationReached)

Here we also create new corresponding NewsRemoteKeys for each item, assigning their corresponding nextKey and prevKey values, which are actually our page numbers.

Finally, we can store all of this into the database and voila, this is your RemoteMediator.

You can find the entire NewsPageKeyedRemoteMediator implementation here.

Show me the data

Now we can finally make use of RemoteMediator to connect our data with PagingDataAdapter and magically achieve pagination.

If you’re using a Repository Pattern, then I suggest you add this block of code into your repository class, but if not, you can add it directly into your ViewModel (given that you have access to the database and API interface).

/**
* Simple example of data pagination.
* [NewsPageKeyedRemoteMediator] should observe the data from database and load the appropriate page of data
* for remote source.
*/
override fun observeNewsListPaginated(): Pager<Int, NewsItem> {
return Pager(
config = PagingConfig(NEWS_PAGE_SIZE, enablePlaceholders = true),
remoteMediator = NewsPageKeyedRemoteMediator(1, database, apiInterface)
) {
newsDao.observeNewsPaginated()
}
}

Create a new Pager object and pass some configuration parameters, like pagination batch size.

Also, if you’re experiencing glitches or jumps in position when scrolling & loading new pages, you can enable placeholders so it keeps the list steady.

For the remoteMediator parameter, pass the newly created NewsPageKeyedRemoteMediator which will do all the pagination magic, fetch the remote data and update the local source.

Finally, the pagingSourceFactory lambda will be a direct link to your database table.So as soon as RemoteMediator saves new data, this observer will be updated and the adapter will update as well. We will pass the observeNewsPaginated() function we previously created in our NewsDao as an observable source of data.

val newsPaginated = newsRepo.observeNewsListPaginated().flow 
// val newsPaginated = newsRepo.observeNewsListPaginated().flow.asLiveData()

Finally, this Pager object can provide a Flow object, or you can adapt it as a LiveData object.
This is what you observe in your fragment  /  activity.

// example of paginated list  listener
lifecycleScope.launchWhenCreated {
viewModel.newsPaginated.collect {
newsPagingAdapter.submitData(it)
}
}

Now, for each new update in the database, a newsPagingAdapter will be updated using the proprietary submitData(pagingData) method. 

Pager provides a PagingData object which contains information not only about the items in the list but also it contains metadata about scroll position and direction, so it can notify RemoteMediator if additional items need to be loaded.

We will not get into details on how to create a PagingDataAdapter as it is similar to a regular RecyclerView adapter, it just enforces you to create a DiffUtil which it will use to compare changes in the list of items.

Scroll, Forrest, Scroll

There, now you’ve got yourself a pagination that combines remote and local data to provide your users a seamless paging experience.

All the data that was stored in the database can later be accessed even without network and users can manually refresh the list if and when they want.

We would expect to have a simpler way of setting up pagination with page numbers and batch size, but after a couple of tries and combinations, the only working solution was using RemoteKeys pattern to store next and previous page numbers.

Maybe Paging 4 library will make it even simpler 😀

I hope you found this article helpful. Let us know if you have any questions about this, if you already tried a similar solution and how did you solve the issue.

We’re curious to hear about your experiences!