Seems a lot of high profile web services are starting to deliver mobile contents these days. This is great and I really believe in more people will use the web on their mobile phones than on a computer.
However, I have noticed some mobile web services are less rigorous in their testing. This is generally fine unless your service are directly dealing with real money. For example, EBay's mobile website http://m.ebay.com doesn't have anything to prevent automated brute force account cracking. I guess it wouldn't be too hard to get into xx active accounts in a few hours...
Friday, 26 October 2007
Tuesday, 29 May 2007
Making progressing on MBT
I've started implementing a model based testing framework for my project. The model I used is a finite state machine. The main structure is:
Each page of the web application represents a state and the links and buttons are the actions that lead to other states. The internal states in the server side are implemented as constraints. To generate test cases, just traverse through the state machine, randomize the order of traversing through each branch from a given node.
I've started with a simple model that contains around 10% of the features of the SUT with 6 states and 10 actions, 1 constraint. But I can already generate thousands of the test cases from it. I'm running one thousand of them tonight. I don't expect them to find any subtle bugs yet because the model they are based on is still very simplistic. The exciting thing is I was thinking about how to extend the model to enable the generation of more subtle test cases on my way home. Exactly as Harry Robinson said, test engineers don't enjoy writing test code, they enjoy thinking devious thoughts. It's exciting to think about what kind of test cases the tool will generate when I gradually add my knowledge about the application to the model. Also I know the knowledge is conveyed in a more abstract way than test scripts, and stored in a single place. No application specific details is repeated. This means much less maintenance cost than writing test scripts.
However, it's not all good news. One thing I realised is that MBT is not going to cover all the testing I want in the project. For example, details like numbers, text, layout are difficult to test with MBT, because the abstract model only contains the keyword of each page that are just enough to identify the page and the actions. Another problem is it's not easy to tell what has been tested because the large number of test cases are not really readable. I guess code coverage analysis can compensate this a bit, but not completely.
Will continue with this work and report here again!
Each page of the web application represents a state and the links and buttons are the actions that lead to other states. The internal states in the server side are implemented as constraints. To generate test cases, just traverse through the state machine, randomize the order of traversing through each branch from a given node.
I've started with a simple model that contains around 10% of the features of the SUT with 6 states and 10 actions, 1 constraint. But I can already generate thousands of the test cases from it. I'm running one thousand of them tonight. I don't expect them to find any subtle bugs yet because the model they are based on is still very simplistic. The exciting thing is I was thinking about how to extend the model to enable the generation of more subtle test cases on my way home. Exactly as Harry Robinson said, test engineers don't enjoy writing test code, they enjoy thinking devious thoughts. It's exciting to think about what kind of test cases the tool will generate when I gradually add my knowledge about the application to the model. Also I know the knowledge is conveyed in a more abstract way than test scripts, and stored in a single place. No application specific details is repeated. This means much less maintenance cost than writing test scripts.
However, it's not all good news. One thing I realised is that MBT is not going to cover all the testing I want in the project. For example, details like numbers, text, layout are difficult to test with MBT, because the abstract model only contains the keyword of each page that are just enough to identify the page and the actions. Another problem is it's not easy to tell what has been tested because the large number of test cases are not really readable. I guess code coverage analysis can compensate this a bit, but not completely.
Will continue with this work and report here again!
Thursday, 24 May 2007
Frustrated at maintaining test scripts
Test automation hasn't been much fun for me since I started doing it earlier this year. Coming up with new test cases and the first time automating them are alright or even enjoyable. But that's only the small portion of what I do everyday. A bigger chunk of my time has been spent in maintaining the test scripts. They break horribly every single week! Mostly due to various tweaks, changes, polishing ups the dev team check in. We have to be agile, we have to embrace change, and I have no complain with that. But I have to find a way to better adapt to the changes. There has to be a way. I can't believe the millions of test engineers are doing this everyday and still love their jobs.
So far I have been looking into Model based testing. It looks promising, at least for the logic and sequence part of testing. I'll try it out and see where I can go from there!
So far I have been looking into Model based testing. It looks promising, at least for the logic and sequence part of testing. I'll try it out and see where I can go from there!
Tuesday, 22 May 2007
Added Google analytics to blog
Finally I have started collecting data about "my users" too!
I've tried to do this sometime ago on blogger by editing the template HTML but without success. The new version of Blogger can supports a new page element "HTML/Java scripts", just add the analytics tracker snippet as such an element, and it works like a charm.
I've tried to do this sometime ago on blogger by editing the template HTML but without success. The new version of Blogger can supports a new page element "HTML/Java scripts", just add the analytics tracker snippet as such an element, and it works like a charm.
Subscribe to:
Posts (Atom)