Screen Scrapping Guide for Java

I created and app about one year a go. The application need to screen scrap a website because it didn't give an API to access the data. When i started, I was confused by so many choice that available for screen scrapping in java. And at that time I've decided to go with NekoHTML, its very powerful but also complicated and a bit of nightmare to maintain. So today i decided to look for better alternative. It turn out, after one year I easily confused by the so many java library out there that i can use. So i decided to create this guide.

Regex

The simplest and probably the fastest way to do screen scrapping in Java. The regex class is already included in the standard java library. So you don't need to add another library and learn how they work.

The problem is, regex is very prone to error. There are many reason for the error. It could be because the html doesn't comply to standards, bad implementation, or even non matching tags. And also a simple change on the website could break the whole regex rule and you need to start from zero. 

To use regex, you need to be very diligent about white space character, or it's only a matter of time before it bites you.

NekoHTML

This is one of the best parser that i know. It often used on other framework such as HTMLUnit as the default html parser. It creates the DOM representation of the page, and then you can traverse through the DOM tree getting each node content. Since it such a low level access, its a bit of pain to maintain. And also you still need to traverse throug the node that isn't important for you. Such as empty text, series of newline, break tag, etc. 

To overcome this you can add a custom filter to remove the unnecessary tag, and uses XPath to query the tag that you are interested in. 

HTMLUnit

Beside as a framework for testing, you can use html Unit for screen scrapping. It can be used for screen scrapping. You just create a webclient and then you can access the tag that you are interested in using the getHtmlElementById or you can also uses XPath. Then you can access the data as an XML or Text. 

It gives a higher abstraction, you don't need to traverse the DOM tree manually. And it also gives build in cookie management, proxy access, JavaScript, CSS and Ajax Support. Everything that you ever need from a headless browsers.

The funny thing is, after looking for something new, i decided to stick with NekoHTML. Its fast, and implementing custom filter helps me a lot. And it also doesn't come with the unnecessary bagage such as Javascript, CSS and Ajax Support. I don't need it for the moment. And of course using XPath has become very valuable. It cuts my previous code to 1/3 even 1/4 in some case. No more manually navigating through the dome, and it's more intuitive. I think i finally found the right tool :)

Hope that helps , I'll re-write the rant some other time to give better explanation. :)

Crazy

I'm going crazy. There's only so much that one people can handle. Got to do something, or i will break down and crumble.

Unanswered prayer

If only someone know what I'm going through.

Be brave !

You know, there's many way that people choose how to live their lives.
There are the ignorants, the bastard, and there is the wanderer, the one that always wonder what to do and what would their live would have been.
But for most people, their live is govern by fears.

Fear of dying, fear of breaking the law, fear of god, fear of the uncertain future.
Fear of their position, fear of their place in the society, and the end fear for live it self.
Must've been sucks to live like that.
What ever the action, whatever the luckyness that some how happened to them,
they can never enjoy the things to the fullest.
Always worry what would happened next.

It's a choice, and maybe the one that is closest to the right answer is the ignorants. They don't seem to care about the future, the society, and maybe live it self. The only problem is, they don't care about anything. Because it takes a whole lot of courage to really care about something.

So...
Make a choice, don't be afraid.
Just live your life to the fullest.
Day by day, minute after minute.
Don't be afraid,
maybe society will treat you bad when you broke their rule,
and maybe , you're wrong. It just the way live is, everybody make mistake
so don't let anything stop you.
Just remember one rule, don't ever, ever ! Let your action based on the intention of hurting someone else.

In other word, be brave ! And don't give a fuck about what the world thinks !
And in the end, life will find you a path !

POM 2

- I treat my Friends like Family, and I treat my Family like my Enemy

- Politician, people that came when ever an opportunity arises, and leave in the wake of destruction. (AFF 2010)

- If we should lose, we should lose gracefully (AFF 2010)

Who we are

We work to satisfy our souls.

Money is important, but it should never be our first priority. 

It should be the means to keep us going.

Doing what we do, Doing what we loved.

Because what matter to us,

Is not how to get rich fast.

But to see the the joy, and appreciation.

When what we make, have a direct positive impact to the people that use it.

Its like sailing against the wind,

with barely enough, and torn out sail.

And for that, we need a lot of help,

and Friends.

But we are sure,

one day we'll make it. 

To be able to sail against the wind.

And when that time comes, 

please joins us with your spirit and courage.

Because after that, we can only go faster !

You

I just realize why she couldn't ever say yes.

It's because somehow it would make an end isn't it ? Your world would stop turning. Everything that is you, somehow would change. The problem is, whether you would like the change or not. I guess your action is the answer that you've found. Probably you're scared, very-very scared, that why you weren't able to believe. And I think you're probably right :)

It's such a shame that we weren't meant for each other :)