Web automation with Groovy and Ruby
For a small private project I need to do some web automation.I wanted to use a scripting language and decided to give Ruby and Groovy a try.
In Ruby there is the Mechanize library. In Groovy there are different options.
The Ruby Mechanize library seems very intuitive:
require 'rubygems'
require 'mechanize'
a = WWW::Mechanize.new { |agent|
agent.user_agent_alias = 'Mac Safari'
}
a.get('http://google.com/') do |page|
search_result = page.form_with(:name => 'f') do |search|
search.q = 'Hello world'
end.submit
search_result.links.each do |link|
puts link.text
end
end
I like the DSLish way to both, scrape (eg:
earch_result.links.each) and manipulate (eg: search.q = 'Hello world') a web page.
In Groovy scraping is also pretty DSLish:
def page = new XmlSlurper(new org.cyberneko.html.parsers.SAXParser()).parse('http://groovy.codehaus.org/')
def data = page.depthFirst().grep{ it.name() == 'A' && it.@href.toString().endsWith('.html') }.'@href'
data.each { println it }
Manipulating a web page with groovy unfortunately is clumsier:
import com.gargoylesoftware.htmlunit.WebClient
def webClient = new WebClient()
def page = webClient.getPage('http://www.google.com')
// check page title
assert 'Google' == page.titleText
// fill in form and submit it
def form = page.getFormByName('f')
def field = form.getInputByName('q')
field.setValueAttribute('Groovy')
def button = form.getInputByName('btnG')
def result = button.click()
// check groovy home page appears in list (assumes it's on page 1)
assert result.anchors.any{ a -> a.hrefAttribute == 'http://groovy.codehaus.org/' }
Personal CV
