Perl scripting: Selenium on Debian wheezy

An urgent requirement came up where I needed to create an automatic script to perform data entry on a mass scale for a client. I gladly took to the Perl framework Test::WWW::Mechanize and happily busied myself with the job at hand. Using this great module, I started working on the script - logging in, navigating to the form, filling in details and submitting the form. After a day and a half of work, I finally had a script to read data from a CSV spreadsheet and import it by simulating the actions an actual user would do with a browser, though, of course much faster. Then came the moment of truth - testing the script on some real data - a spreadsheet with several rows of data. After a little fine tuning, bingo! The data suddenly appeared in the web application! I couldn't have been happier!!

Having written one script, there now was a second one to write. I went about as earlier simulating the actions with Test::WWW::Mechanize. All went well till I came upon a mandatory Javascript link without which the job would not be done. It is at this point, I figured I couldn't proceed with Test::WWW::Mechanize, since it does not implement Javascript. I therefore turned to Test::WWW::Selenium, because of it's Javascript support.

Selenium is a software which emulates browser actions. Supporting major browsers including Firefox, Google Chrome and Internet Explorer, it is an excellent tool for Web developers to run automated tests on web applications. So I installed the perl package Test::WWW::Selenium on my Debian Wheezy:

# apt-get install libtest-www-selenium-perl

I then installed the Selenium RC, also called the Selenium standalone server from http://seleniumhq.org/download/. I started the server using the command:

$ java -jar selenium-server-standalone-2.25.0.jar

INFO: Launching a standalone server

12:04:33.046 INFO - Java: Sun Microsystems Inc. 19.1-b02

12:04:33.074 INFO - OS: Linux 2.6.32-5-686 i386

12:04:33.184 INFO - v2.25.0, with Core v2.25.0. Built from revision 17482

12:04:34.105 INFO - RemoteWebDriver instances should connect to: http://127.0.0.1:4444/wd/hub

12:04:34.107 INFO - Version Jetty/5.1.x

12:04:34.109 INFO - Started HttpContext[/selenium-server/driver,/selenium-server/driver]

12:04:34.111 INFO - Started HttpContext[/selenium-server,/selenium-server]

12:04:34.111 INFO - Started HttpContext[/,/]

12:04:35.619 INFO - Started org.openqa.jetty.jetty.servlet.ServletHandler@1fae3c6

12:04:35.619 INFO - Started HttpContext[/wd,/wd]

12:04:35.624 INFO - Started SocketListener on 0.0.0.0:4444

I wrote a short and simple test script to visit the Google homepage and executed it:

$ perl sel-test.pl

However, I got some errors at this point:

java.lang.IllegalArgumentException: URI "file:selenium-server.jar" is not hierarchical".

After some searching, I found from a blog article http://tero.tilus.net/rutinat/2009/08/18/cucumber-running-selenium-on-debian-lenny/, that I needed to install sun-java6-jdk which is not available on Wheezy. The alternative is to install it from Oracle website. Instead, I used the alternative openjdk-6-jdk and icedtea-plugin:

# apt-get install openjdk-6-jdk icedtea-plugin

This got rid of the problem, but I ran into another:

... INFO - Preparing Firefox profile...

Could not read application.ini

... ERROR - Failed to start new browser session, shutdown browser and clear all session data

java.lang.RuntimeException: Timed out waiting for profile to be created!

I found that the solution to this was to add the firefox/iceweasel directory to the PATH:

$ export PATH=$PATH:/usr/lib/iceweasel

After doing this, I tried by test script and voila!

Why India needs FOSS

Expiring Software Licences

A friend of mine who purchased a laptop some months ago was concerned about her expiring software licences. It is not uncommon in India that one comes across laptops or PCs which have software installed by someone else - the proverbial "expert" who seems to excel at denying people the CDs and installation media needed to reinstall the software they supposedly own. In any case, often people buy software and somehow the installation media is not to be found when the system suffers from a virus attack or otherwise needs to be re-installed because the old version is not supported any longer.

Can one legally own Software?

One school of thought among computing device users is the legally clean hands approach. Such users will do everything in their power to use only legal software. They are people who think that "do not steal" means using only "authentic" software. Such users run around when their software licences expire - they rush to the nearest software dealer to get their "authentic" latest copy of their software. They feel that piracy is a crime just like murder, and that the perpetrators of this should be condemned to severe punishment. These users will suffer the inability to use their systems when their software expires.

Why is it that we seem to buy in to the above line of thought? I believe it is because we haven't delved deep enough into the essence of software. There is no doubt that computers and software have been very recent additions to human history. The average Indian has lived with computers and software for around two decades. Being a new "thing", we seem to ask the same questions of software that we ask of property, before analyzing it from a legal perspective. The question of ownership. Who owns "it"? If each copy of a software is "ownable" like land or a car, etc., surely it is a crime to get an "illegal" copy of the software, akin to stealing. No wonder software piracy begs to be regarded a crime. However, I believe software has certain properties which puts to question its comparison with "ownable" objects.

Distinguishing Feature of Software: Share-ability

Consider an object like an automobile, say the latest Ford car in the market. Before coming to the showroom, each car would have gone through an assembly line, where it's skeleton, it's interiors, it's engines, radiator, exhaust pipes, top cover, steering wheel, etc would have been assembled. All these parts need to be manufactured separately according to the size, specification and design required for the latest Ford model. The finished product therefore would be yours only if you pay the cost of ownership. For the company, the cost of manufacture will determine the profit they make on a sale. The company can streamline their systems, automate the manufacture of parts and use machines to assemble their cars to bring more efficiency and reduce their cost of manufacture and thereby increase their profit.

The truth is that software production does not work like this. Producing an extra copy of a piece software does not involve the kind of work that goes into the latest car or watch, for example. Your latest "legal" version of proprietary software did not require all those kilobytes or megabytes to be manually written onto the CD/DVD by hand - the cost of material and human effort involved is not comparable to other most commodities that you pay for. Your software was simply copied automatically from a master copy, with comparatively little extra cost or manpower. The only thing that is different between your CD/DVD and the other ones in the shop where you bought it is the "product key", a code which you need to key in to authenticate your copy of software. However, from a legal perspective, this product key does not add any value to the software itself, but is just a mechanism to ensure that only one person or party can use your copy of the CD/DVD you bought.

The point is that the possibility of sharing the proprietary software you bought with your friend is limited by the product key or a similar mechanism put in place by the software manufacturing company. Note here that I am not advocating "freeware", where you don't pay for the software. Rather, I am simply advocating that you should have the freedom to share the software you bought, and conversely, to borrow a software CD your friend bought! So if software cannot be compared with material objects, what can we compare it to and what should be our policy to determine the cost of software?

Of course, a software company needs to pay it's developers. After all, software programmers need to eat too! Though extra copies of software can be produced at minimal cost, the master copy of the software itself needed to be produced first. This surely needs a great deal of manpower, time and effort, which surely incurs a considerable cost. Human resources are the real assets of a software company. The methods, processes and methodologies used to develop the software determine the cost efficiency of software production.

Software akin to books?

If you compare software with books, extra copies of software can be produced somewhat like books, simply by machine printing/copying the text of the book onto paper and binding the paper with a cover around it. The original manuscript of the book took a good deal of effort from the author, and this is what an avid reader really wants to pay for - the content of the book, though of course the kind of paper, binding, etc also form part of the cost of the book. With software, the process of creation and copying is closer to how books are created and copied. Yet, there is a difference between the two. You buy books to read them, but you buy software to use it, rather than read it. You buy it because it performs a particular task, which it was designed to perform. A Word processing software for example, helps you create documents which you can share and print. However, the secret to how the software works is in the "source code". "Open Source" essentially gives you the privilege of reading the source code.

Q: Why should I care about the source code if I'm not a developer?
Ans: If you procure open source software, where you can access a copy of the source code, you are not at the mercy of your vendor or the developer in the future. You can take the source code and go to another developer/organization to maintain/customize the software.

FOSS / Free and Open Source Software - viable alternative to proprietary software

"Free software" means software that respects users' freedom and community. Roughly, the users have the freedom to run, copy, distribute, study, change and improve the software. Thus, "free software" is a matter of liberty, not price. It's free as in freedom, not cost. To understand the concept, think of "free" as in "free speech," not as in "free food" or "free beer".

A program is free software if the program's users have the four essential freedoms:

1. The freedom to run the program, for any purpose.
2. The freedom to study how the program works, and change it so it does your computing as you wish. Access to the source code is a precondition for this.
3. The freedom to redistribute copies so you can help your neighbor.
4. The freedom to distribute copies of your modified versions to others. By doing this you can give the whole community a chance to benefit from your changes. Access to the source code is a precondition for this.

Software for a Free India

India is a huge, diverse confluence of cultures, languages and religions. Leaving a rich history and tradition behind, we Indians will complete 65 years of Independence this coming August 15th, 2012. A lot has happened in all these years in our country, which is the world's largest democracy. India has always been complex country to advance and as solutions seem to be found, more questions are raised about the future. India has produced many great minds and continues to produce people who have made an impact, not just in the motherland, but all over the world. In this era of technology, India has made rapid strides in tapping technology for the development of the country. Today, the common man can stay connected by mobile, and be up-to-date with news, technology, etc via national TV and radio, which is found even in remote villages. In all major cities, there are several facilities like ATM's, bill payment kiosks (water / electricity), online payment facilities, online travel ticket booking (e.g IRCTC, KSRTC, Cleartrip, etc..). Even online e-filing of Income Tax returns, sales tax, etc. All these open up new possibilities for people.

However, for a developing country like India there is still a gap to be bridged when it comes to bringing technology to the people. Especially when it comes to public services, there are services which fail to cater to the larger spectrum of users. In this regard, there is a need for the Government to look at alternatives which serve all groups of people. Before investing tax-payers money in solutions, we need to ponder on whether the services provided are fair and reachable to all. Also when choosing a given technology, it is important to look at creating a level playing field where solutions on all platforms are placed on an equal footing. As a programmer with more than 14 years experience on both Windows and Linux, I'd insist that the Government should place Free and Open Source solutions on par with proprietary solutions from Microsoft or any other company. Whichever solution is fair to all citizens and is technologically strong and sustainable should be chosen.

For example, consider the website http://incometaxindiaefiling.gov.in for electronic filing of Income Tax returns online. If you visit this site and go to the "Return Preparation Software", you get a page full of MS Excel files, which you need to download and complete offline. The problem with this is that Excel is a product of Microsoft and runs only on Microsoft Windows. So users of other operating systems are excluded. There are some tools to run Windows applications on Linux, but not reliable or recommended. Even if you are using MS Windows, you don't get MS Excel by default. You need to get yourself a copy of Microsoft Office if you want to use MS Excel. This raises a few questions:

Q1: How many Indians can afford to get a legal copy of MS Excel?

Consider the student who has just been introduced to computers. He/she may be staying away from home and barely managing to pay education loan and house rent from his first job. There are many people who are first generation skilled workers while their parents generation were not even educated. Can they be expected to spend their hard-earned money on legal MS Office? If they have a computer, they have already spent on the hardware and maybe some software as well to run on their computers.

Q2: Is it fair to require Indians to use MS Excel (or any proprietary product) for e-filing of returns?

The Government requires every salaried person to file their IT returns. What about people who don't use Windows? They want to file returns on their own and do not have the time and transport to go to the Income Tax office and get themselves a form, fill it up and then submit it back at the IT office. Such people need to be able to use the e-filing facility with whatever computer they are using. Every operating system has software for spreadsheets similar to MS Excel. Why not provide option in alternative software also to file returns online?

Q3: Is it fair to cater only to users of MS Excel (or any proprietary product) for a public service to the citizens of the country?

Certainly, it's unfair! Any public service solution which is beyond the reach of one or more group of citizens of the country is a situation which needs immediate attention from the Government. No Government can claim to respect the rights of it's citizens if it gives denies services to users of any kind of software. Surely, there needs to be more awareness of the sectors which actually contribute to nation building - Open Source has made huge contributions to the country in different ways [1] [2] [3]. It is high time, Open Source users are treated on par with users of proprietary software.

Is there an alternative? Yes indeed!! Open Standards is the answer to this problem. OpenOffice is an office suite which has all the features needed to file income tax returns. This has been proved beyond doubt by the ITR migration project taken up to migrate the MS Excel files to OpenOffice.org. Surely, we need the concerned people in our Government to sit up and take notice of this pressing need. Surely, they can support this work if not ensure that there they provide software for OpenOffice.org . The reason this is a better choice in this scenario is that

1. OpenOffice is available for Windows, Linux and Mac OS and therefore, no one will be excluded if they use a non-Windows OS.
2. OpenOffice can be downloaded free of cost from the website and can be copied onto CD's and distributed at low cost even to those without Internet.

Rabindranath Tagore, legendary poet and thinker and the first non-European to win the Nobel Prize in one of his great and well remembered poems, wrote:

WHERE the mind is without fear and the head is held high

Where knowledge is free

Where the world has not been broken up into fragments

By narrow domestic walls

Where words come out from the depth of truth

Where tireless striving stretches its arms towards perfection

Where the clear stream of reason has not lost its way

Into the dreary desert sand of dead habit

Where the mind is led forward by thee

Into ever-widening thought and action

Into that heaven of freedom, my Father, let my country awake.

Political freedom was secured in 1947 but we can only dream of the day when there is freedom of technology for all Indians and we can really reap the benefits of Free and Open Source software while being able to use the public services like filing of Tax Returns which is a right of every Indian. That day is not today and we hope that this dream become a reality. Anyone listening?

UPDATE:

About a year after this article was written, the Income Tax e-filing website has added a facility for users to file income tax online if they need to file form ITR 1. This means, they do not need to depend on a tool like Microsoft Excel for this type of returns. This does improve the situation, though not all the forms are available with the online option. Yet, it is a step in the right direction - a step to put all citizens on an equal platform.

Rendering chess positions

Chess, 3-D graphics and FOSS are three areas I like. I have been using the POV-Ray program for sometime now and prepared files to render individual chess pieces, which can be included in a POV-Ray scene. I tested the include files with a board and a few lights. This way, I could include the shape definition for a piece and place it on a given position on the board by translating it.

Next, I studied the CPAN Chess libraries in Perl and figured how to read in a file saved in FEN with a Perl script. The script would then generate a POV-Ray file with necessary translate instructions to place different pieces in different positions, as specified in the input FEN file. Now, I simply used POV-Ray to render the file generated above and I have a 3-D view of the board position. I have included a sample image. I got this position by editing the start position in Xboard using the "Edit Game" feature. I saved the position as an FEN file and then ran the perl script and the povray program on my ubuntu powered laptop.

Varsity project on sourceforge.net

In the penultimate year of my bachelors course in Information Science engineering, my classmate, Vineeth and I developed a handy little piece of software. Back then, we were learning UML and were amused with how new applications could be designed using it's methodology. Why not develop software using which we could design UML diagrams, we thought. This idea stuck as our curricular mini-project for the 6th semester.

The UML design suite, which was the result of our efforts, finally had 7 modules. You could develop

• Use-case diagrams
• Sequence diagrams
• Activity diagrams
  
• Class diagrams
• Component diagrams
• Deployment diagrams
  

We implemented two options for saving work. First, an open XML based file format which we developed. Second, any of a set of well known image formats including JPEG and PNG.

The project was not publically available for long and we knew we could not leave something of it's kind stored somewhere on a CD or computer without any positives coming out of it. So recently, I submitted the online and requested sourceforge to have our project up on their archives.

On sourceforge, our project has a homepage, screenshots, a subversion repository, forums for development and open discussion. All these can be accessed by clicking:

http://sourceforge.net/projects/umldesignsuite

deepOfix public release

Many things are happening around the world. People are working for a living, children are studying in schools, youth are looking for employment, cities, especially here in south India are growing. Today is a very special day because I am part of something unique. If I were to read a newspaper and ask myself, what does this newspaper have in it for me to be happy about, for example, a fellow Indian - Ratan Tata's work through the years in bringing a grand old company to this age when it has acquired both quality and size, the events culminating in his TATA's takeover of the Corus group. This kind of news makes me take pride in a fellow countryman and in many ways, a man I look up to personally as one who, having had wealth, nevertheless worked and with genuine sincerity and constancy through the years, has brought us here in this massive subcontinent a smile.

Sometimes, days pass us by and we wonder what has changed. Today, I believe something that has been in the laboratory of deeprootlinux is being unveiled - released publicly. Today, DeepRoot Linux releases deepOfix Mail Server, a one-of-its-kind product. deepOfix has been developed without using any propreitary software development platform or software development tool. Our network runs completely on Free Software products. We use FOSS (Free and Open Source) products for testing, linking, interpreting and compiling code, version management, hosting of our servers, managing mail withing our company. I am pleased to know of the many institutions using our product, many of who have good things to say about it. The DeepOfix page on sourceforge.net is:

http://sourceforge.net/projects/deepofix

Update: The deepOfix project is now at http://deepofix.org

Men, Ideas, Computers and Society

Thomas Jefferson, third President of the United States of America wrote a letter to Isaac McPherson in 1813 regarding an 'idea'. He lived in a time of many intellectual inventions. Inventors often sought patents and Jefferson wanted to communicate to McPherson that an idea should be available to all, for mutual benefit. This is (part of) what he had to write:

"If nature has made any one thing less susceptible than all others of exclusive property, it is the action of the thinking power called an idea, which an individual may exclusively possess as long as he keeps it to himself; but the moment it is divulged, it forces itself into the possession of every one, and the receiver cannot dispossess himself of it. Its peculiar character, too, is that no one possesses the less, because every other possesses the whole of it. He who receives an idea from me, receives instruction himself without lessening mine; as he who lights his taper at mine, receives light without darkening me."

Today, the computer has been long since invented, streamlined and brought to many desks. People in countries all over the earth use computers - many have prepared documents using word processors, presentation software and so forth. Many programmers develop software and many organizations use software and also earn revenue from developing software.

In this article, I wish to write about my moral perspective on free and open source software and proprietary software. The human mind is the place where every invention starts. A person works very hard for a period of time and tries many methods, until finally (s)he comes upon a solution. Where does this solution come from and how is it shared fairly with all? This article is not a final word, or even one I expect the reader to accept. Yet, I hope it makes for interesting reading.

What makes us human? Many animals live on our planet and have many capabilities. But we are social by nature - we share and live together in families. What we own is not entirely ours. For example, if I wear a shirt, the idea of making thread was given to one person. He shared it and many know how to make thread. Similarly, many know how to weave thread into cloth. This idea too came from some person. Cloth making and computer manufacturing may have started not centuries, but millenia apart. Yet, we remain people in need of mutual help.

Now, if every person in the world believed in free software, believed that free software was going to solve many problems, this is positive, but not ultimate. From a moral perspective, we ask ourselves whether we have given to poorer people dignity and opportunity to grow. If we give complete liberty to our children, for example, would they learn and live by principle? How are we to know whether they would use freedom justly and fairly and be fair and just for others? People in a ravaged country would be conscious of the possible ills of a morally deficient society. People who have not felt the pain of certain kinds of deprivation may not grow up with this social sensitivity.

Ultimately, proprietary and open source software models are in the hands of people. If a software developer is sensitive to the poor man near his house, he wants to help him. The poor man has a great dignity as a man, he can have other treasures than wealth - virtue. When justice is not yet established in a land or society, its people suffer and there is need to change. If reform is initiated by people willing to take the risk and offer themselves for it, there may be improvement until possibly justice and peace reigns. If a land or a society enjoys justice and peace, it can be a nursery of mutual respect and growth. Yet, this can stagnate a society because the social interdependence which brought the people to this good time can be forgotten. People can become aloof to the needs of the underprivileged.

Virtues - charity, peace, desire to share, all stem from man's goodness. Social evil comes because of man's passion - greed, prejudice, selfishness. Yet, this is not only a societal problem, but a personal one too. Morality of a society is a necessary precursor of all good things which come later. History has shown that where there were greedy or power hungry people - those who caused many to die, these could not last the test of time. Men may have been feared because of their brutality - even if they were geniuses at military conflict, they did not leave a moral quality that can remain the strength of a society. Morality and ethics therefore are facets of human development that, though different from culture to culture, are a common need for all of us. Only in this way, I can trust you when we work together, or even play a game of football together. Only in this way can we build something that we cannot build individually.

Spiral using ncurses

I tried to write a program to draw a spiral using the ncurses library - one which draws slowly in a way which can be seen noticeably. I first analyzed the number of characters to be drawn in each direction - down, right, up and left.

I prepared a table having the number of characters in the outer spiral of the spiral design. I used a variable, m, to hold the number of characters in a particular direction. I programmed m to change according to the mathematical pattern which we can notice in the table.

+----------+----+-----+------+--------+
| n        |  1 |   2 |   3  |    4   |
| down     |  4 |   8 |   12 |   16   |
| right    |  3 |   7 |   11 |   15   |
| up       |  2 |   6 |   10 |   14   |
| left     |  1 |   5 |   9  |   13   |
+----------+----+-----+------+--------+

I leave the code for the interested reader here. I implemented this using C with the help of the ncurses library for moving the cursor in the relevant directions before printing the character.

#include <curses.h>
#include <math.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char* argv[])
{
 int x, y, n, i, m, dx, dy, j;
 int ay[] = {1, 0, -1, 0};
 int ax[] = {0, 1, 0, -1};
 int am[] = {1, 1, 1, 1};

 char s[80];

 n = atoi(argv[1]); 
 m = n * 4;

 initscr();
 cbreak();
 noecho();
 nonl();
 intrflush(stdscr, FALSE);
 keypad(stdscr, TRUE);

 x = 0;
 y = 0;

 while (m > 0)
 {
  for (i = 0; i < 4; ++i)
  {
   dx = ax[i];
   dy = ay[i];
   for (j = 0; j < m; ++j)
   {
    x += dx; y += dy;
    mvaddstr(y, x, "X");
   }
   system("sleep 1");
   refresh();
   --m;
  }
 }

 system("sleep 1");

 endwin();

 return 0;
}

The program, when run, should draw the pattern arm by arm or the spiral, i.e, first downward, then rightward, then upward and then leftward. The following is the output for n = 3:

X
X XXXXXXXXXX
X X        X
X X XXXXXX X
X X X    X X
X X X XX X X
X X X  X X X
X X XXXX X X
X X      X X
X XXXXXXXX X
X          X
XXXXXXXXXXXX