[286] Mechanical Turkdown

Building an entirely reasonable service for synchronous Markdown to HTML conversion, via Mechanical Turk

Subscribe now

Mechanical Turkdown [12.30.2016]

Today's episode is exciting for me. I had a stupid idea when talking to Adam Gamble a few years back to build a Mechanical Turk powered markdown library, for two reasons:

  • It is a really, really dumb idea.
  • Its name - Mechanical Turkdown - makes me giggle.

Today, we're going to build it. Let's get started.

Project

We'll kick off the project, pulling in a package to interact with Mechanical Turk.

Project Setup

mix new mechanical_turkdown --sup
cd mechanical_turkdown
vim mix.exs
defmodule MechanicalTurkdown.Mixfile do
  use Mix.Project
  # ...
  def application do
    [
      applications: [
        :logger,
        :erlcloud,
      ],
      mod: {MechanicalTurkdown, []}]
  end

  defp deps do
    [
      {:erlcloud, "~> 2.2.1"},

      # Dev + Test dependencies
      {:credo, "~> 0.5", only: [:dev, :test]},
    ]
  end
end
mix deps.get

If you set up the AWS environment variables, erlcloud will use them. I put them in a .envrc file that looks like this:

export AWS_ACCESS_KEY_ID=<Your AWS Access Key>
export AWS_SECRET_ACCESS_KEY=<Your AWS Secret Access Key>

I've already got one I'll copy in place:

cp ~/tmp/mturk.envrc .envrc

However, we want to use the Sandbox so that we can easily work our jobs and test things out without spending money. Consequently, we'll need to send a configuration option as the last argument to our erlcloud function calls.

Submitting a HIT (Human Intelligence Task) to Mechanical Turk

Let's submit a basic job to Mechanical Turk. Amazon calls these Human Intelligence Tasks, or HITs.

mkdir lib/mechanical_turkdown
vim lib/mechanical_turkdown/mechanical_turk.ex

To start with, we're going to have to extract a lot of records from erlcloud in order to construct a request:

defmodule MechanicalTurkdown.MechanicalTurk do
  @moduledoc """
  A module for interacting with MechanicalTurk for our specific tasks.
  """

  # We have to interact with the mturk_hit record from the erlcloud library, so
  # we'll extract it.
  require Record
  import Record, only: [defrecord: 2, extract: 2]

  defrecord :mturk_hit, extract(:mturk_hit, from_lib: "erlcloud/include/erlcloud_mturk.hrl")
  defrecord :mturk_money, extract(:mturk_money, from_lib: "erlcloud/include/erlcloud_mturk.hrl")
  defrecord :mturk_qualification_requirement, extract(:mturk_qualification_requirement, from_lib: "erlcloud/include/erlcloud_mturk.hrl")
  defrecord :mturk_question, extract(:mturk_question, from_lib: "erlcloud/include/erlcloud_mturk.hrl")
  defrecord :mturk_question_form, extract(:mturk_question_form, from_lib: "erlcloud/include/erlcloud_mturk.hrl")
  defrecord :mturk_free_text_answer, extract(:mturk_free_text_answer, from_lib: "erlcloud/include/erlcloud_mturk.hrl")
  defrecord :aws_config, extract(:aws_config, from_lib: "erlcloud/include/erlcloud_aws.hrl")
  # ...

Next, we'll build a submit_markdown function that accepts some markdown and creates a HIT via :erlcloud_mturk.create_hit:

defmodule MechanicalTurkdown.MechanicalTurk do
  # ...
  @doc """
  Submit a HIT for a markdown to html translation, because it's a totally
  reasonable thing to do.
  """
  def submit_markdown(markdown) do
    markdown
      |> markdown_conversion_hit
      |> :erlcloud_mturk.create_hit(mturk_config())
  end

  def mturk_config() do
    aws_config(
      # NOTE: We should really get these from Application.get_env ultimately
      access_key_id: System.get_env("AWS_ACCESS_KEY_ID"),
      secret_access_key: System.get_env("AWS_SECRET_ACCESS_KEY"),
      # NOTE: And we would put this in Application.get_env so we could switch
      # per-env configuration easily, as well as support changing it with
      # conform
      mturk_host: 'mechanicalturk.sandbox.amazonaws.com'
    )
  end
  # ...
end

Now all we need to do is create the function to create the mturk_hit record. This took a bit of reading through the code to figure out since there is no documentation or examples, but here's what I came up with:

defmodule MechanicalTurkdown.MechanicalTurk do
  # ...
  defp markdown_conversion_hit(markdown) do
    mturk_hit(
      reward: mturk_money(
        amount: '0.01',
        currency_code: 'USD',
        formatted_price: '$0.01'
      ),
      lifetime_in_seconds: 60 * 60, # 1 hour
      assignment_duration_in_seconds: 60 * 60, # 1 hour
      title: 'convert some markdown to html',
      description: 'convert some markdown to html',
      keywords: ['transcription'],
      question: mturk_question_form(
        content: [
          mturk_question(
            question_identifier: 'mturkdown1',
            display_name: 'some display name',
            question_content: [
              {:text, to_charlist(markdown)}
            ],
            answer_specification: mturk_free_text_answer()
          )
        ]
      )
    )
  end
end

With that, we can create a HIT from the REPL:

iex -S mix
iex(1)> hit_details = MechanicalTurkdown.MechanicalTurk.submit_markdown("## foo\n\n- bar\n- baz")
# => [hit_id: '3ZZAYRN1I6SBK2G1Z254JHHFUBIOTB',
#     hit_type_id: '3ICMLZI1INJH9IXNCIJGW1SMC14JTE']

Now there's a new HIT on Mechanical Turk. I'll log into my account and show it, at https://requestersandbox.mturk.com.

We can log in as a worker and work the job, at https://workersandbox.mturk.com. Search for 'markdown'.

As you can see, there's a little bit of an issue. Our answer form only supports a single line answer, which isn't sufficient for our Enterprise Markdown Conversion User Interface. We can modify the HIT configuration to specify that we need more lines in the worker interface:

  defp markdown_conversion_hit(markdown) do
    mturk_hit(
      # ...
      question: mturk_question_form(
        content: [
          mturk_question(
            # ...
            answer_specification: mturk_free_text_answer(
              number_of_lines_suggestion: 40 # <--
            )
          )
        ]
      )
    )
  end

Let's submit another HIT:

iex(1)> hit_details = MechanicalTurkdown.MechanicalTurk.submit_markdown("## foo\n\n- bar\n- baz")
# => [hit_id: '3ZZAYRN1I6SBK2G1Z254JHHFUBIOTC',
#     hit_type_id: '3ICMLZI1INJH9IXNCIJGW1SMC14JTE']

Now if we check as a worker again, we'll see that we have sufficient space to submit longer form HTML content.

Once we submit it, it's up to the requester to approve it. Let's just do it from the REPL:

iex(3)> assignments = hit_details[:hit_id] |> :erlcloud_mturk.get_assignments_for_hit(MechanicalTurkdown.MechanicalTurk.mturk_config)
# => [num_results: 1, page_number: 1, total_num_results: 1,
#     assignments: [[assignment_id: '31T4R4OBOSHPZZJLMXU4EO6ZJEX7CB',
#     worker_id: 'A2281LKWJAR55R', hit_id: '3D7VY91L65YNOD18M0VWY22RJPOMBS',
#     assignment_status: 'Submitted',
#     auto_approval_time: {{2017, 1, 2}, {0, 19, 0}},
#     accept_time: {{2016, 12, 30}, {0, 18, 36}},
#     submit_time: {{2016, 12, 30}, {0, 19, 0}},
#     answers: [{:mturk_answer, 'mturkdown1',
#       '<h2>Foo</h2>\n\n<ul>\n  <li>bar</li>\n  <li>baz</li>\n</ul>', :undefined,
#       :undefined, :undefined}]]]]
iex(4)> assignment = hd(assignments[:assignments])
[assignment_id: '31T4R4OBOSHPZZJLMXU4EO6ZJEX7CB', worker_id: 'A2281LKWJAR55R',
 hit_id: '3D7VY91L65YNOD18M0VWY22RJPOMBS', assignment_status: 'Submitted',
 auto_approval_time: {{2017, 1, 2}, {0, 19, 0}},
 accept_time: {{2016, 12, 30}, {0, 18, 36}},
 submit_time: {{2016, 12, 30}, {0, 19, 0}},
 answers: [{:mturk_answer, 'mturkdown1',
   '<h2>Foo</h2>\n\n<ul>\n  <li>bar</li>\n  <li>baz</li>\n</ul>', :undefined,
   :undefined, :undefined}]]
iex(5)> assignment[:assignment_id] |> :erlcloud_mturk.approve_assignment('', MechanicalTurkdown.MechanicalTurk.mturk_config)
# => :ok
# This next bit should really be handled by extracting the `mturk_answer` record
# and using it, but for the REPL we'll just pattern match.
iex(6)> {_, _, html, _, _, _} = hd(assignment[:answers])
{:mturk_answer, 'mturkdown1',
 '<h2>Foo</h2>\n\n<ul>\n  <li>bar</li>\n  <li>baz</li>\n</ul>', :undefined,
 :undefined, :undefined}
iex(7)> html
'<h2>Foo</h2>\n\n<ul>\n  <li>bar</li>\n  <li>baz</li>\n</ul>'

So that's the HTML that our worker provided. Let's add some functions to our module to do what we just did a bit more easily:

  @doc """
  Get the assignments for the HIT we submitted earlier.
  """
  # NOTE: We aren't handling "elixir-ifying" the API, but we probably should.
  @spec get_assignments(charlist()) :: list()
  def get_assignments(hit_id) do
    :erlcloud_mturk.get_assignments_for_hit(to_charlist(hit_id), mturk_config())
  end

  @doc """
  Approve the assignment with the given assignment_id
  """
  @spec approve(charlist()) :: :ok
  def approve(assignment_id) do
    assignment_id
      |> :erlcloud_mturk.approve_assignment('', mturk_config())
  end

Now we'd like to create our synchronous function to translate the markdown. We'll do this by introducing a GenServer with a weird API. Its state will embed a bit of a state machine:

  • :creating
  • :checking
  • :completed

There is almost certainly a better way to do this, but I haven't done this sort of thing too often and this works fine so I tend to fall back to it when I want to build an awkward polling-style API :)

defmodule MechanicalTurkdown.Worker do
  @moduledoc """
  A module to help provide a synchronous interface to Mechanical Turkdown jobs.
  """

  use GenServer
  alias MechanicalTurkdown.MechanicalTurk

  # Client
  def start_link(markdown) do
    # We'll start off in a `creating` state
    GenServer.start_link(__MODULE__, {:creating, markdown})
  end

  @doc """
  A synchronous call that waits indefinitely for the Mechanical Turk HIT to have
  an assignment, then returns the HTML that was sent as the response to the
  assignment.
  """
  def check(pid) do
    # We'll call `check` on the server, then use a case statement on the response.
    reply = GenServer.call(pid, :check)
    case reply do
      :waiting ->
        # If we're still waiting, we'll check again in a second
        IO.puts "waiting!"
        Process.sleep 1_000
        check(pid)
      {:finished, html} ->
        # If we finished, we'll return the html we got back
        GenServer.stop(pid)
        IO.puts "finished!"
        to_string(html)
      other -> IO.puts "Oops, got a weird response: #{inspect other}"
    end
  end

  @doc """
  Create a worker and check for a result in a single function.
  """
  def submit_markdown(markdown) do
    # This is just an API to make it trivial to use this GenServer as a very
    # long-running function
    {:ok, pid} = start_link(markdown)
    check(pid)
  end

  # Server
  def init(state) do
    # When we initialize the server, we'll send ourselves a message that we'll
    # handle in a `handle_info` later
    send(self, :after_init)
    {:ok, state}
  end

  # When we're creating, we'll submit the work and set a timer to check its
  # results.
  def handle_info(:after_init, {:creating, markdown}) do
    IO.puts "Submitting markdown"
    hit_details = MechanicalTurk.submit_markdown(markdown)
    Process.send_after(self(), :check_assignments, 1_000)
    {:noreply, {:checking, hit_details[:hit_id]}}
  end
  def handle_info(:check_assignments, {:checking, hit_id}) do
    # When we're checking the results, we'll use a case statement on the number
    # of results - once we have one, we'll auto-accept it and return its value
    IO.puts "checking assignments"
    assignments = MechanicalTurk.get_assignments(hit_id)
    case assignments[:num_results] do
      0 ->
        # If we have no results yet, we'll create a timer to check assignments
        # again in a bit.
        IO.puts "no results, still waiting!"
        Process.send_after(self(), :check_assignments, 1_000)
        {:noreply, {:checking, hit_id}}
      _ ->
        # If we had results, we'll approve the assignment and then switch to a
        # `completed` state, collecting the returned html.
        IO.puts "got a result!"
        assignment = hd(assignments[:assignments])
        assignment[:assignment_id] |> MechanicalTurk.approve
        IO.puts "approved the result!"
        {_, _, html, _, _, _} = hd(assignment[:answers])
        {:noreply, {:completed, html}}
    end
  end

  def handle_call(:check, _from, {:completed, html}) do
    # If we had completed the work, we'll reply with it.
    {:reply, {:finished, html}, :finished}
  end
  def handle_call(:check, _from, state) do
    # Otherwise, we'll reply that we're waiting and leave the state unmodified.
    {:reply, :waiting, state}
  end
end

This should cover it. We can test it out in the REPL:

iex(1)> MechanicalTurkdown.Worker.submit_markdown("ohai")

Now we can search for this HIT as a worker, submit our answer to it, and see it returned from this function call. Fantastic?

Summary

OK so this was an extremely contrived project, maybe. However, it was a lot of fun for me. Also, it shows a pretty good workflow for managing this sort of process - you can probably envision a Mechanical Turk flow that's less contrived and conceivably manage it this way.

Should you build a blocking function call that awaits market conditions to be favorable for your function to be manually completed? Probably not. But the other parts are really useful, and even this part is pretty fun. There are a lot of changes I'd like to introduce - using GenStage and GenStateMachine would both be nice modifications to this - but this works and it's still making me laugh, so I think it's a success!

I hope you never do this unless you're trolling. That's not the same thing as hoping you never do it though! Have fun!

Resources