Lessons learned

Christoph Kappel · · , , , · Reading time 2 minutes
In this article

A friend of mine approached me with a request to - let us say automate - a web request to a not-to-be-named raffle. I looked into it and was instantly hooked. Both price and challenge are interesting.

I wrote quite a few scraper in my time, so after a few lines good ol' Ruby, with the help of mechanize, the first shot was ready and failed miserably due to CSRF.

After a quick check, yes they really use a pseudo-random token, which is injected into the DOM and a hidden input field via JS.

I had two options now:

  1. Understand the code that writes the CSRF into the dom

  2. Find a scraper with a JS engine

In my day job, we always like to play with e2e-testing, which mostly involves scripts, that remote-control a web browser.

That said, I had a few glances at this stack again. And after some more reading, I settled on watir: and chromedriver.

Watir &

The API of watir is really amazing and easy to use:

require "watir"

browser = Watir::Browser::new

browser.goto "https://unexist.blog"

browser.close

I think the example is pretty self-explanatory, it opens up a remote session and points the browser to the given url. When you start that in e.g. irb, you can REPL your way to the desired outcome.

browser.link(visible_text: /GitHub/).when_present.click

The above example looks for a link with GitHub in its visible text and click it, when present. Easy as that.

Headless? &

One problem solved, this runs nicely on my local machine. Now it would be best, if I can just deploy it on a server without installing the whole docker stack.

Since we are targeting Linux, headless support is kind of built-in. And after a quick search I found headless.

This gem wraps the handling of a virtual framebuffer for you and, as it turns out, works pretty well with my stack:

require "watir"
require "headless"

Headless.ly do
    browser = Watir::Browser::new

    browser.goto "https://unexist.blog"

    browser.close
end

Conclusion &

Learnings from different scenarios can sometimes be applied to new problems!