[272] Phoenix API Pagination with Scrivener

Using Scrivener.Headers to add pagination link information to the headers in our API responses, per an RFC Draft.

Subscribe now

Phoenix API Pagination with Scrivener [10.06.2016]

In another project, I'm building a Single-Page App with a Phoenix backend and an Elm frontend. I wanted to try to use a new standard for API link relationships in header fields, and Scrivener.Headers supports it, so let's check that out.

Project

I'm starting out with the time-tracker app, at tag before_episode_272.

Let's bring in scrivener, scrivener_ecto, and scrivener_headers:

vim mix.exs

defmodule TimeTrackerBackend.Mixfile do
  # ...
  def application do
    [
      mod: {TimeTrackerBackend, []},
      applications: [
        # ...
        :scrivener_ecto,
      ]
    ]
  end
  # ...
  defp deps do
    [
      # ...
      {:scrivener_ecto, "~> 1.0"},
      {:scrivener_headers, "~> 3.0"},
      # If we're testing pagination we # probably want a lot of test data, so
      # we'll bring in ex_machina
      {:ex_machina, "~> 1.0", only: :test}
    ]
  end

We can go ahead and fetch them:

mix deps.get

We also need to add Scrivener.Ecto to the repo:

vim lib/time_tracker_backend/repo.ex

defmodule TimeTrackerBackend.Repo do
  use Ecto.Repo, otp_app: :time_tracker_backend
  use Scrivener, page_size: 10
end

With that, we're ready to begin paginating. We're going to use ExMachina to generate test data so we'll want to start it up:

vim test/test_helper.exs

{:ok, _} = Application.ensure_all_started(:ex_machina)

We'll add the Factory:

vim test/support/factory.ex

defmodule TimeTrackerBackend.Factory do
  use ExMachina.Ecto, repo: TimeTrackerBackend.Repo
  alias TimeTrackerBackend.User

  def user_factory do
    %User{
      name: sequence(:name, &("Some Name - #{&1}"))
    }
  end
end

And now we can write the test:

vim test/controllers/user_controller_test.exs

We'll change the test we already have for the index action to test pagination:

defmodule TimeTrackerBackend.UserControllerTest do
  # ...
  import TimeTrackerBackend.Factory
  # ...
  describe "when authenticated" do
    setup [:set_auth_headers]

    test "paginates entries on index", %{conn: conn} do
      insert_list(12, :user)
      conn = get conn, user_path(conn, :index)
      response = json_response(conn, 200)["data"]
      assert length(response) == 10
    end
    # ...
  end
  # ...
end

If we run this test, we can see that it fails because there are 13 users returned rather than our page of 10 - that's the 12 we inserted, plus the user we inserted so we could authenticate. Let's add pagination to our API and see if we can get the test to pass:

defmodule TimeTrackerBackend.UserController do
  use TimeTrackerBackend.Web, :controller
  plug Guardian.Plug.EnsureAuthenticated, handler: __MODULE__

  def index(conn, params) do
    page =
      User
      |> Repo.paginate(params)

    conn
    |> Scrivener.Headers.paginate(page)
    |> render("index.json", users: page.entries)
  end

OK, so that passes now. We should also add some assertions for the headers that this sends back for the links - let's look at the headers for our response in the test real quickly:

    test "paginates entries on index", %{conn: conn} do
      insert_list(12, :user)
      conn = get conn, user_path(conn, :index)
      IO.inspect conn # <-
      response = json_response(conn, 200)["data"]
      assert length(response) == 10
    end

OK, we'll add assertions on the response headers:

    test "paginates entries on index", %{conn: conn} do
      insert_list(12, :user)
      conn = get conn, user_path(conn, :index)
      response = json_response(conn, 200)["data"]
      assert get_resp_header(conn, "total") == ["13"]
      assert get_resp_header(conn, "per-page") == ["10"]
      assert get_resp_header(conn, "total-pages") == ["2"]
      assert get_resp_header(conn, "page-number") == ["1"]
      # I really want to be able to add assertions on the link here!
      IO.inspect get_resp_header(conn, "link")
      assert length(response) == 10
    end

OK I really wanted to be able to write some assertions on that link. Luckily, Simon Rand already wrote a parser for this RFC in elixir, which is fantastic. We'll use that to write better tests.

vim mix.exs

  defp deps do
    [
      # ...
      {:ex_link_header, "~> 0.0.5"},
    ]
  end

mix deps.get

Now we can tweak our test a tiny bit:

    test "paginates entries on index", %{conn: conn} do
      insert_list(12, :user)
      conn = get conn, user_path(conn, :index)
      response = json_response(conn, 200)["data"]
      assert get_resp_header(conn, "total") == ["13"]
      assert get_resp_header(conn, "per-page") == ["10"]
      assert get_resp_header(conn, "total-pages") == ["2"]
      assert get_resp_header(conn, "page-number") == ["1"]
      link_header = get_resp_header(conn, "link") |> hd |> ExLinkHeader.parse!
      assert "2" == link_header.next.params.page
      assert "2" == link_header.last.params.page
      assert "1" == link_header.first.params.page
      assert length(response) == 10
    end

Now we know that we're sending out valid headers for pagination. This means that a consumer that knows how to deal with this RFC can move around our data nicely without having to know any other details. This is good!

Summary

OK, so today we introduced API pagination with Scrivener and we learned that there's an RFC for this sort of thing. Since we used a standard, I was able to find a parser for it too :) This suggests that we can build clients more easily. Obviously there are a lot of standards you could use for your API responses, but I think it's important to read the RFCs regardless. See you soon!

Resources