Canoo WebTest vs Selenium

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Canoo WebTest vs Selenium

Dierk König
Hiall,

at the recent GrailsExchange conference I noticed quite some
interest in functional testing of Grails applications.

Marc Guillemot, committer to Groovy, WebTest, and HtmlUnit has
posted his thoughts on the different approaches to this topic
under
http://mguillem.wordpress.com/2007/10/29/webtest-vs-selenium-webtest-wins-13
-5/

Anybody concerned with functional testing of webapps will profit
from reading this thought-provoking post.

enjoy
Dierk


---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

Reply | Threaded
Open this post in threaded view
|

Re: Canoo WebTest vs Selenium

JamesPage
I think the summary is very unfair on selenium.

I have always been brought up with testing to eat your own dog food and Selenium enables one to do that...

The amount of problems that Selenium has found for me because of difference between IE and firefox have been amazing. It has saved me hours of work.

Also if you are working on a page that is deep into a process (like sign in) then the IDE can save you an amazing amount of time in just taking you there.

On running times I don't have a problem as I run the tests at night.

But as the article points out there are advantages to WebTest and I would propose that you make Selenium tests run on Webtest and vs vs. Then you can leverage the power of the Selenium IDE. People can run both Selenium and webtest and get the power of both.  Webtest and Selenium combined would be the most powerful testing enviroment.....

Selenium.open("ff/bar.gsp");
Selenium.type("your name","Mr foo bar");
Selenium.type("click","//input[@type=''submit"]);

is far easier than a bunch of xml.... and for the most part you don't even have to do that as the IDE will write the test for you.

Also there are advantages in being to run the test code on the native browser.

We also use Selenium for usability testing. You record peoples click stream and then can easily see where they have gone wrong.

All the best

James




On 10/29/07, Dierk Koenig <[hidden email]> wrote:
Hiall,

at the recent GrailsExchange conference I noticed quite some
interest in functional testing of Grails applications.

Marc Guillemot, committer to Groovy, WebTest, and HtmlUnit has
posted his thoughts on the different approaches to this topic
under
http://mguillem.wordpress.com/2007/10/29/webtest-vs-selenium-webtest-wins-13
-5/

Anybody concerned with functional testing of webapps will profit
from reading this thought-provoking post.

enjoy
Dierk


---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Canoo WebTest vs Selenium

hamishagain@dcs.shef.ac.uk
hi james

> is far easier than a bunch of xml.... and for the most part you don't
> even have to do that as the IDE will write the test for you.

that's the beauty of the grails webtest plugin:

       invoke(url:'admin') // will redirect to login
       verifyText(text:'Login ID', description:'Check this is login page')
       setInputField(value:"scott", name:"j_username")
       setInputField(value:"tiger", name:"j_password")
       clickButton(label:'Login')

no more XML :-)

best,

--
Hamish
http://www.dcs.shef.ac.uk/~hamish/

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

Reply | Threaded
Open this post in threaded view
|

Re: Canoo WebTest vs Selenium

Marc Palmer Local

On 31 Oct 2007, at 12:10, Hamish Cunningham wrote:

> hi james
>
>> is far easier than a bunch of xml.... and for the most part you  
>> don't even have to do that as the IDE will write the test for you.
>
> that's the beauty of the grails webtest plugin:
>
>      invoke(url:'admin') // will redirect to login
>      verifyText(text:'Login ID', description:'Check this is login  
> page')
>      setInputField(value:"scott", name:"j_username")
>      setInputField(value:"tiger", name:"j_password")
>      clickButton(label:'Login')
>
> no more XML :-)

I have to agree, but probably with pretty much everyone.

So far my analysis of this has been that Webtest is brilliant  
precisely because it does NOT use a browser. i.e. you test your code  
integrity on its own without any potential browser bugs.

Likewise, Selenium is brilliant because it lets you discover browser  
incompatibilities because it -does- use a real browser.

The two are entirely separate concerns. I would not want to bother  
running any Selenium tests until I know that my Webtests pass.  
Otherwise what has the bugs, the browser or your code?

Therefore, what I believe we need, is a standard DSL for simulating  
webbrowser interactions, much like the Webtest DSL. These need to be  
test scripts in the grails app, which can then be run through webtest  
OR selenium or any other testing tool you like.

This is just an issue of controlling what "browser" is used to run the  
tests. The report results should be unified also, so the format is the  
same and clearer than current webtest results.

I want to be able to do:

grails functional-test webtest
grails functional-test selenium

...to run the same tests twice, once with the webtest "driver" and  
once with "selenium" and for it to produce two branches of test reports.

Marc


---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

Reply | Threaded
Open this post in threaded view
|

Re: Canoo WebTest vs Selenium

JamesPage
In reply to this post by hamishagain@dcs.shef.ac.uk
Hamish, Marc....

I agree with Marc..... Except I want to use the Selerium IDE to generate the test.....

With the testing that we have done with the GrailsCache it has had to be done using Selerium.. As we need to test how real browsers cache the content... and how long they take to parse the page, the JS, and the CSS.  We are optimising both the JS and CSS on the server (which takes time), against faster render and download time on the browser (saves time). The  issue that we are trying to test at the moment is the time saved greater than the time spent on the server.
 
One massive advantage Selerium has is setting the test is very easy. Put Selerium and Webtest through a GORM test...  (GORM is a usability methodology that measures productivity... Apple and Rankin are big believers in  GORM) and Selerium would easily win...


webtest
= usertype(char) * 30

on Selenium
would be
= userclick()

So as comparison using GORM Selenium would easily win!

But if you enabled WebTest to be able run Selerium code then WebTest would be as fast to set up.

It would also be great if Selerium/Webtest could generate GORM metrics for usabiluity as well.

But there is more;-

As you can see what the Selenium IDE outputs is very similer to Webtest. But with Selenium you would have had to just click a bit on a webpage and it has generated the script for you.

But Selenium and are very smiler
                 
on Webtest
      invoke(url:'admin') // will redirect to login
Selenium Output
      Selenium.open("/admin")

                   

on Webtest
      verifyText(text:'Login ID', description:'Check this is login page')

on Selenium
      Selenium.verifyText("description:", "Login ID")


on Webtest
      setInputField(value:"scott", name:"j_username")

on Selenium
      Selenium.type("j_username", "scott")

on Webtest
      setInputField(value:"tiger", name:"j_password")

on Selenium
      Selenium.type("j_password", "tiger")


on Webtest

      clickButton(label:'Login')
on Selenium
      Selenium.clickAndWait('//login')

So it should not be hard to make the two work togther.....

J

On 10/31/07, Hamish Cunningham <[hidden email]> wrote:
hi james

> is far easier than a bunch of xml.... and for the most part you don't
> even have to do that as the IDE will write the test for you.

that's the beauty of the grails webtest plugin:

       invoke(url:'admin') // will redirect to login
       verifyText(text:'Login ID', description:'Check this is login page')
       setInputField(value:"scott", name:"j_username")
       setInputField(value:"tiger", name:"j_password")
       clickButton(label:'Login')

no more XML :-)

best,

--
Hamish
http://www.dcs.shef.ac.uk/~hamish/

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Canoo WebTest vs Selenium

Marc Palmer Local

On 31 Oct 2007, at 13:14, James Page wrote:

> Hamish, Marc....
>
> I agree with Marc..... Except I want to use the Selerium IDE to  
> generate the test.....
>
> With the testing that we have done with the GrailsCache it has had  
> to be done using Selerium.. As we need to test how real browsers  
> cache the content... and how long they take to parse the page, the  
> JS, and the CSS.  We are optimising both the JS and CSS on the  
> server (which takes time), against faster render and download time  
> on the browser (saves time). The  issue that we are trying to test  
> at the moment is the time saved greater than the time spent on the  
> server.
>
> One massive advantage Selerium has is setting the test is very easy.  
> Put Selerium and Webtest through a GORM test...  (GORM is a  
> usability methodology that measures productivity... Apple and Rankin  
> are big believers in  GORM) and Selerium would easily win...
>
>
> webtest
> = usertype(char) * 30
>
> on Selenium
> would be
> = userclick()
>
> So as comparison using GORM Selenium would easily win!
>

That can easily be improved though.

> But if you enabled WebTest to be able run Selerium code then WebTest  
> would be as fast to set up.
>
> It would also be great if Selerium/Webtest could generate GORM  
> metrics for usabiluity as well.
>
> But there is more;-
>
> As you can see what the Selenium IDE outputs is very similer to  
> Webtest. But with Selenium you would have had to just click a bit on  
> a webpage and it has generated the script for you.
>
> But Selenium and are very smiler
>
> on Webtest
>       invoke(url:'admin') // will redirect to login
> Selenium Output
>       Selenium.open("/admin")
>
>
> on Webtest
>       verifyText(text:'Login ID', description:'Check this is login  
> page')
> on Selenium
>       Selenium.verifyText("description:", "Login ID")
>
>
> on Webtest
>       setInputField(value:"scott", name:"j_username")
> on Selenium
>       Selenium.type("j_username", "scott")
>
> on Webtest
>       setInputField(value:"tiger", name:"j_password")
> on Selenium
>       Selenium.type("j_password", "tiger")
>
>
> on Webtest
>       clickButton(label:'Login')
> on Selenium
>       Selenium.clickAndWait('//login')
>
> So it should not be hard to make the two work togther.....


Again, this is not a webtest thing. It's an abstraction of functional  
testing. Webtest is a testing framework, not an abstraction of one. To  
support both webtest and selenium you need to abstract functional  
testing, as per my suggestion.

For the purposes of supporting other future testing mechanisms, it is  
therefore a requirement to abstract a functional testing DSL, based on  
the existing WebTest one (just because it is there) and have this able  
to plug in at execution time to a number of different back ends.

Then the final step is to make a Selenium -> "Grails functional  
testing DSL" converter.

Marc


---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

Reply | Threaded
Open this post in threaded view
|

Re: Canoo WebTest vs Selenium

JamesPage
Selerium works with about 10 different languages... So it should not be hard to get it to both output the f*Test DSl (functional (WebTest | Selerium | or whatever)) DSL.... and take it as input to convert it back to javascript.

I do want to look at the current DSL to see how compatible it is with GOMS user testing. GOMS measures how long it takes a user to perform a task. Most of everything generated for functional testing applies to GOMS User click/user text input/etc... There is the more complex GOMS models that try to model user thinking times as well. (They are accurate).. NASA has GOMS models for different tasks, like how somebody scans a page....

The great thing is adding a GOMS model to testing is that it would introduce usability a the start of a project and not at the end. The problem with most usability testing is that it is done at the end of a project. Normally produces a report that is highly critical of everybody... and the recommendations never get implemented.

GOMS is not ideal but it is better than nothing.

What does everybody think? Anybody else have any experience with GOMS and how easy it would be map from the DSL to the GOMS model?

 

On 10/31/07, Marc Palmer <[hidden email]> wrote:

On 31 Oct 2007, at 13:14, James Page wrote:

> Hamish, Marc....
>
> I agree with Marc..... Except I want to use the Selerium IDE to
> generate the test.....
>
> With the testing that we have done with the GrailsCache it has had
> to be done using Selerium.. As we need to test how real browsers
> cache the content... and how long they take to parse the page, the
> JS, and the CSS.  We are optimising both the JS and CSS on the
> server (which takes time), against faster render and download time

> on the browser (saves time). The  issue that we are trying to test
> at the moment is the time saved greater than the time spent on the
> server.
>
> One massive advantage Selerium has is setting the test is very easy.
> Put Selerium and Webtest through a GORM test...  (GORM is a
> usability methodology that measures productivity... Apple and Rankin
> are big believers in  GORM) and Selerium would easily win...
>
>
> webtest
> = usertype(char) * 30
>
> on Selenium
> would be
> = userclick()
>
> So as comparison using GORM Selenium would easily win!
>

That can easily be improved though.

> But if you enabled WebTest to be able run Selerium code then WebTest
> would be as fast to set up.
>
> It would also be great if Selerium/Webtest could generate GORM
> metrics for usabiluity as well.
>
> But there is more;-
>
> As you can see what the Selenium IDE outputs is very similer to
> Webtest. But with Selenium you would have had to just click a bit on
> a webpage and it has generated the script for you.
>
> But Selenium and are very smiler
>
> on Webtest
>       invoke(url:'admin') // will redirect to login
> Selenium Output
>       Selenium.open("/admin")
>
>
> on Webtest
>       verifyText(text:'Login ID', description:'Check this is login
> page')
> on Selenium
>       Selenium.verifyText("description:", "Login ID")
>
>
> on Webtest
>       setInputField(value:"scott", name:"j_username")
> on Selenium
>       Selenium.type ("j_username", "scott")
>
> on Webtest
>       setInputField(value:"tiger", name:"j_password")
> on Selenium
>       Selenium.type("j_password", "tiger")
>
>
> on Webtest
>       clickButton(label:'Login')
> on Selenium
>       Selenium.clickAndWait('//login')
>
> So it should not be hard to make the two work togther.....


Again, this is not a webtest thing. It's an abstraction of functional
testing. Webtest is a testing framework, not an abstraction of one. To
support both webtest and selenium you need to abstract functional
testing, as per my suggestion.

For the purposes of supporting other future testing mechanisms, it is
therefore a requirement to abstract a functional testing DSL, based on
the existing WebTest one (just because it is there) and have this able
to plug in at execution time to a number of different back ends.

Then the final step is to make a Selenium -> "Grails functional
testing DSL" converter.

Marc


---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

RE: Canoo WebTest vs Selenium

Dierk König
In reply to this post by Marc Palmer Local
making a functional testing DSL would require a common denominator.

webtest:

 clickElement xpath:"//div[@id='result']/a"
 not {
     verifyText text:'NullPointerException'
 }
 group(description:'go to cart'){
     goToShop()
     3.times { selectNextProduct() }
     selectCart()
 }

not even speaking about steps for verifying excel downloads,
PDF, email, applets, and so on.

Some of the above may also be possible with other tools, but
a DSL would be like a "standard", which can only encompass the
most basic features. I currently don't see much value in that.

Dierk


> -----Original Message-----
> From: Marc Palmer [mailto:[hidden email]]
> Sent: Mittwoch, 31. Oktober 2007 14:24
> To: [hidden email]
> Subject: Re: [grails-user] Canoo WebTest vs Selenium
>
>
>
> On 31 Oct 2007, at 13:14, James Page wrote:
>
> > Hamish, Marc....
> >
> > I agree with Marc..... Except I want to use the Selerium IDE to  
> > generate the test.....
> >
> > With the testing that we have done with the GrailsCache it has had  
> > to be done using Selerium.. As we need to test how real browsers  
> > cache the content... and how long they take to parse the page, the  
> > JS, and the CSS.  We are optimising both the JS and CSS on the  
> > server (which takes time), against faster render and download time  
> > on the browser (saves time). The  issue that we are trying to test  
> > at the moment is the time saved greater than the time spent on the  
> > server.
> >
> > One massive advantage Selerium has is setting the test is very easy.  
> > Put Selerium and Webtest through a GORM test...  (GORM is a  
> > usability methodology that measures productivity... Apple and Rankin  
> > are big believers in  GORM) and Selerium would easily win...
> >
> >
> > webtest
> > = usertype(char) * 30
> >
> > on Selenium
> > would be
> > = userclick()
> >
> > So as comparison using GORM Selenium would easily win!
> >
>
> That can easily be improved though.
>
> > But if you enabled WebTest to be able run Selerium code then WebTest  
> > would be as fast to set up.
> >
> > It would also be great if Selerium/Webtest could generate GORM  
> > metrics for usabiluity as well.
> >
> > But there is more;-
> >
> > As you can see what the Selenium IDE outputs is very similer to  
> > Webtest. But with Selenium you would have had to just click a bit on  
> > a webpage and it has generated the script for you.
> >
> > But Selenium and are very smiler
> >
> > on Webtest
> >       invoke(url:'admin') // will redirect to login
> > Selenium Output
> >       Selenium.open("/admin")
> >
> >
> > on Webtest
> >       verifyText(text:'Login ID', description:'Check this is login  
> > page')
> > on Selenium
> >       Selenium.verifyText("description:", "Login ID")
> >
> >
> > on Webtest
> >       setInputField(value:"scott", name:"j_username")
> > on Selenium
> >       Selenium.type("j_username", "scott")
> >
> > on Webtest
> >       setInputField(value:"tiger", name:"j_password")
> > on Selenium
> >       Selenium.type("j_password", "tiger")
> >
> >
> > on Webtest
> >       clickButton(label:'Login')
> > on Selenium
> >       Selenium.clickAndWait('//login')
> >
> > So it should not be hard to make the two work togther.....
>
>
> Again, this is not a webtest thing. It's an abstraction of functional  
> testing. Webtest is a testing framework, not an abstraction of one. To  
> support both webtest and selenium you need to abstract functional  
> testing, as per my suggestion.
>
> For the purposes of supporting other future testing mechanisms, it is  
> therefore a requirement to abstract a functional testing DSL, based on  
> the existing WebTest one (just because it is there) and have this able  
> to plug in at execution time to a number of different back ends.
>
> Then the final step is to make a Selenium -> "Grails functional  
> testing DSL" converter.
>
> Marc
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list please visit:
>
>     http://xircles.codehaus.org/manage_email

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

Reply | Threaded
Open this post in threaded view
|

Re: Canoo WebTest vs Selenium

JamesPage
Selerium:
Selenium.clickAndWait(//div[@id='result']/a" );
Selenium.assetNotBoddyText(.*NullPointerException'.*);
goToShop()


def goToShop()
{
   Selenium......
}

There is no differance there...

and for verifying excel downloads, PDF, email, applets, and so on is the reason that a combined WebTest/Selerium would be a good thing.

Eating dog food is a good practice, and Selerium allows me to do that.

But there is only so much that can be done with JS in the browser and therefore WebTest is important as well. As Marc said he wants to run WebTest while writing code, and Selerium at night... I think that is a good Practice.

Webtest is good for testing Server Functionality and Selerium is good for testing client Functionality.



On 10/31/07, Dierk Koenig < [hidden email]> wrote:
making a functional testing DSL would require a common denominator.

webtest:

clickElement xpath:"//div[@id='result']/a"
not {
     verifyText text:'NullPointerException'
}
group(description:'go to cart'){
     goToShop()
     3.times { selectNextProduct() }
     selectCart()
}

not even speaking about steps for verifying excel downloads,
PDF, email, applets, and so on.

Some of the above may also be possible with other tools, but
a DSL would be like a "standard", which can only encompass the
most basic features. I currently don't see much value in that.

Dierk


> -----Original Message-----
> From: Marc Palmer [mailto: [hidden email]]
> Sent: Mittwoch, 31. Oktober 2007 14:24
> To: [hidden email]
> Subject: Re: [grails-user] Canoo WebTest vs Selenium
>
>
>
> On 31 Oct 2007, at 13:14, James Page wrote:
>
> > Hamish, Marc....
> >
> > I agree with Marc..... Except I want to use the Selerium IDE to
> > generate the test.....
> >
> > With the testing that we have done with the GrailsCache it has had
> > to be done using Selerium.. As we need to test how real browsers
> > cache the content... and how long they take to parse the page, the
> > JS, and the CSS.  We are optimising both the JS and CSS on the
> > server (which takes time), against faster render and download time
> > on the browser (saves time). The  issue that we are trying to test
> > at the moment is the time saved greater than the time spent on the
> > server.
> >
> > One massive advantage Selerium has is setting the test is very easy.
> > Put Selerium and Webtest through a GORM test...  (GORM is a
> > usability methodology that measures productivity... Apple and Rankin
> > are big believers in  GORM) and Selerium would easily win...
> >
> >
> > webtest
> > = usertype(char) * 30
> >
> > on Selenium
> > would be
> > = userclick()
> >
> > So as comparison using GORM Selenium would easily win!
> >
>
> That can easily be improved though.
>
> > But if you enabled WebTest to be able run Selerium code then WebTest
> > would be as fast to set up.
> >
> > It would also be great if Selerium/Webtest could generate GORM
> > metrics for usabiluity as well.

> >
> > But there is more;-
> >
> > As you can see what the Selenium IDE outputs is very similer to
> > Webtest. But with Selenium you would have had to just click a bit on
> > a webpage and it has generated the script for you.
> >
> > But Selenium and are very smiler
> >
> > on Webtest
> >       invoke(url:'admin') // will redirect to login
> > Selenium Output
> >       Selenium.open("/admin")
> >
> >
> > on Webtest
> >       verifyText(text:'Login ID', description:'Check this is login
> > page')
> > on Selenium
> >       Selenium.verifyText("description:", "Login ID")
> >
> >
> > on Webtest
> >       setInputField(value:"scott", name:"j_username")
> > on Selenium
> >       Selenium.type("j_username", "scott")
> >
> > on Webtest
> >       setInputField(value:"tiger", name:"j_password")
> > on Selenium
> >       Selenium.type("j_password", "tiger")
> >
> >
> > on Webtest
> >       clickButton(label:'Login')
> > on Selenium
> >       Selenium.clickAndWait('//login')
> >
> > So it should not be hard to make the two work togther.....
>
>
> Again, this is not a webtest thing. It's an abstraction of functional
> testing. Webtest is a testing framework, not an abstraction of one. To
> support both webtest and selenium you need to abstract functional
> testing, as per my suggestion.
>
> For the purposes of supporting other future testing mechanisms, it is
> therefore a requirement to abstract a functional testing DSL, based on
> the existing WebTest one (just because it is there) and have this able
> to plug in at execution time to a number of different back ends.
>
> Then the final step is to make a Selenium -> "Grails functional
> testing DSL" converter.
>
> Marc
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list please visit:
>
>     http://xircles.codehaus.org/manage_email

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Canoo WebTest vs Selenium

Marc Palmer Local
In reply to this post by Dierk König

On 31 Oct 2007, at 14:31, Dierk Koenig wrote:

> making a functional testing DSL would require a common denominator.
>
> webtest:
>
> clickElement xpath:"//div[@id='result']/a"
> not {
>     verifyText text:'NullPointerException'
> }
> group(description:'go to cart'){
>     goToShop()
>     3.times { selectNextProduct() }
>     selectCart()
> }
>
> not even speaking about steps for verifying excel downloads,
> PDF, email, applets, and so on.
>
> Some of the above may also be possible with other tools, but
> a DSL would be like a "standard", which can only encompass the
> most basic features. I currently don't see much value in that.

The overwhelming amount of testing concepts is the same in these  
cases. The corner cases are just those, and PDF testing is not testing  
the browser - that can be in separate "webtest only" scripts.

Either that, or the Selenium "driver" for such DSL commands as  
"verifyPDF" could just pass flawlessly without doing any testing, but  
webtest would do what it does now.

Marc


---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

Reply | Threaded
Open this post in threaded view
|

RE: Canoo WebTest vs Selenium

Dierk König
In reply to this post by JamesPage
I don't have any experience with GOMS.
But I do have 7 years of experience with functional testing of webapps, having written well more than 10'000 tests myself and helped in maintaining test suites up to 100'000 tests. 
Here are some of my experiences:
- Execution speed matters. No way any locally installed browser can ever be fast enough for a reasonable test suite.
- Unsupervised execution matters. As long as the tests throw no error, you don't even want to know they are running. Think continuous integration.
- Scalability matters. You need to run tests in parallel or it will take forever. Throwing more hardware at the problem has a natural upper limit.
  If you really write tests for every feature of your application, you will reach that limit pretty soon (expect 2000 tests per developer-year). 
- Engineering matters. You need means to remove duplication, you need good naming, you need intention revealing tests. No way any click-tool will ever give you that.
- Everybody has an opinion on test automation - but very few have real-life experience beyond experimentation or toy-projects.  
 
sorry, but you got me started... ;-)
Dierk
 
P.S. what I count as 'test' is each server hit, together with the required setting of form fields (if any) and the trailing verifications.

What does everybody think? Anybody else have any experience with GOMS and how easy it would be map from the DSL to the GOMS model?
Reply | Threaded
Open this post in threaded view
|

RE: Canoo WebTest vs Selenium

Dierk König
In reply to this post by JamesPage
There _is_ a difference. Look into not{} and group{}.
The difference is in _structure_ and this difference would make it less
straightforward to come up with a common DSL.
Also, WebTest's StepContainer (not, group, and others) are reflected in the test report, which allows
showing test results at various levels of detail - a mandatory feature for all
test suites of real-life size.
 
Dierk
-----Original Message-----
From: James Page [mailto:[hidden email]]
Sent: Mittwoch, 31. Oktober 2007 15:46
To: [hidden email]
Subject: Re: [grails-user] Canoo WebTest vs Selenium

Selerium:
Selenium.clickAndWait(//div[@id='result']/a" );
Selenium.assetNotBoddyText(.*NullPointerException'.*);
goToShop()


def goToShop()
{
   Selenium......
}

There is no differance there...

and for verifying excel downloads, PDF, email, applets, and so on is the reason that a combined WebTest/Selerium would be a good thing.

Eating dog food is a good practice, and Selerium allows me to do that.

But there is only so much that can be done with JS in the browser and therefore WebTest is important as well. As Marc said he wants to run WebTest while writing code, and Selerium at night... I think that is a good Practice.

Webtest is good for testing Server Functionality and Selerium is good for testing client Functionality.



On 10/31/07, Dierk Koenig < [hidden email]> wrote:
making a functional testing DSL would require a common denominator.

webtest:

clickElement xpath:"//div[@id='result']/a"
not {
     verifyText text:'NullPointerException'
}
group(description:'go to cart'){
     goToShop()
     3.times { selectNextProduct() }
     selectCart()
}

not even speaking about steps for verifying excel downloads,
PDF, email, applets, and so on.

Some of the above may also be possible with other tools, but
a DSL would be like a "standard", which can only encompass the
most basic features. I currently don't see much value in that.

Dierk


> -----Original Message-----
> From: Marc Palmer [mailto: [hidden email]]
> Sent: Mittwoch, 31. Oktober 2007 14:24
> To: [hidden email]
> Subject: Re: [grails-user] Canoo WebTest vs Selenium
>
>
>
> On 31 Oct 2007, at 13:14, James Page wrote:
>
> > Hamish, Marc....
> >
> > I agree with Marc..... Except I want to use the Selerium IDE to
> > generate the test.....
> >
> > With the testing that we have done with the GrailsCache it has had
> > to be done using Selerium.. As we need to test how real browsers
> > cache the content... and how long they take to parse the page, the
> > JS, and the CSS.  We are optimising both the JS and CSS on the
> > server (which takes time), against faster render and download time
> > on the browser (saves time). The  issue that we are trying to test
> > at the moment is the time saved greater than the time spent on the
> > server.
> >
> > One massive advantage Selerium has is setting the test is very easy.
> > Put Selerium and Webtest through a GORM test...  (GORM is a
> > usability methodology that measures productivity... Apple and Rankin
> > are big believers in  GORM) and Selerium would easily win...
> >
> >
> > webtest
> > = usertype(char) * 30
> >
> > on Selenium
> > would be
> > = userclick()
> >
> > So as comparison using GORM Selenium would easily win!
> >
>
> That can easily be improved though.
>
> > But if you enabled WebTest to be able run Selerium code then WebTest
> > would be as fast to set up.
> >
> > It would also be great if Selerium/Webtest could generate GORM
> > metrics for usabiluity as well.
> >
> > But there is more;-
> >
> > As you can see what the Selenium IDE outputs is very similer to
> > Webtest. But with Selenium you would have had to just click a bit on
> > a webpage and it has generated the script for you.
> >
> > But Selenium and are very smiler
> >
> > on Webtest
> >       invoke(url:'admin') // will redirect to login
> > Selenium Output
> >       Selenium.open("/admin")
> >
> >
> > on Webtest
> >       verifyText(text:'Login ID', description:'Check this is login
> > page')
> > on Selenium
> >       Selenium.verifyText("description:", "Login ID")
> >
> >
> > on Webtest
> >       setInputField(value:"scott", name:"j_username")
> > on Selenium
> >       Selenium.type("j_username", "scott")
> >
> > on Webtest
> >       setInputField(value:"tiger", name:"j_password")
> > on Selenium
> >       Selenium.type("j_password", "tiger")
> >
> >
> > on Webtest
> >       clickButton(label:'Login')
> > on Selenium
> >       Selenium.clickAndWait('//login')
> >
> > So it should not be hard to make the two work togther.....
>
>
> Again, this is not a webtest thing. It's an abstraction of functional
> testing. Webtest is a testing framework, not an abstraction of one. To
> support both webtest and selenium you need to abstract functional
> testing, as per my suggestion.
>
> For the purposes of supporting other future testing mechanisms, it is
> therefore a requirement to abstract a functional testing DSL, based on
> the existing WebTest one (just because it is there) and have this able
> to plug in at execution time to a number of different back ends.
>
> Then the final step is to make a Selenium -> "Grails functional
> testing DSL" converter.
>
> Marc
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list please visit:
>
>     http://xircles.codehaus.org/manage_email

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

RE: Canoo WebTest vs Selenium

Dierk König
In reply to this post by Marc Palmer Local
> > clickElement xpath:"//div[@id='result']/a"
> > not {
> >     verifyText text:'NullPointerException'
> > }
> > group(description:'go to cart'){
> >     goToShop()
> >     3.times { selectNextProduct() }
> >     selectCart()
> > }
> >
> > not even speaking about steps for verifying excel downloads,
> > PDF, email, applets, and so on.
> >
> > Some of the above may also be possible with other tools, but
> > a DSL would be like a "standard", which can only encompass the
> > most basic features. I currently don't see much value in that.
>
> The overwhelming amount of testing concepts is the same in these  
> cases. The corner cases are just those, and PDF testing is not testing  
> the browser - that can be in separate "webtest only" scripts.

The concept of nesting and how it is reflected in the test
reports is not the same.
The concept of having the tests being executable Groovy code
is not the same.
The concept of using arbitrary ant tasks (e.g. for setting up data
in the DB, sharing resource bundles, cruise-control integration, etc.)
is not the same.

WebTest is Java - and thus Groovy.

Dierk

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

Reply | Threaded
Open this post in threaded view
|

Re: Canoo WebTest vs Selenium

Marc Palmer Local

On 31 Oct 2007, at 18:40, Dierk Koenig wrote:

>>> clickElement xpath:"//div[@id='result']/a"
>>> not {
>>>    verifyText text:'NullPointerException'
>>> }
>>> group(description:'go to cart'){
>>>    goToShop()
>>>    3.times { selectNextProduct() }
>>>    selectCart()
>>> }
>>>
>>> not even speaking about steps for verifying excel downloads,
>>> PDF, email, applets, and so on.
>>>
>>> Some of the above may also be possible with other tools, but
>>> a DSL would be like a "standard", which can only encompass the
>>> most basic features. I currently don't see much value in that.
>>
>> The overwhelming amount of testing concepts is the same in these
>> cases. The corner cases are just those, and PDF testing is not  
>> testing
>> the browser - that can be in separate "webtest only" scripts.
>
> The concept of nesting and how it is reflected in the test
> reports is not the same.
> The concept of having the tests being executable Groovy code
> is not the same.
> The concept of using arbitrary ant tasks (e.g. for setting up data
> in the DB, sharing resource bundles, cruise-control integration, etc.)
> is not the same.
>
> WebTest is Java - and thus Groovy.

This isn't a WebTest vs Selenium or a Java vs Groovy or JavaScript/
HTML vs Java vs Groovy issue.

The issue is that we as users have advanced needs. We need automated  
testing outside of a browser. Webtest meets this need. We need  
automated testing inside browsers to detect cross-browser  
incompatibilities.

I see no compelling reason at all why a custom DSL cannot be made to  
allow some scripts that would run in both environments.

I don't care about underlying implementation differences. That's what  
an abstracted driver would hide.

Surely this is a GOOD THING, if not please tell me why I am mad :)

Marc


---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

Reply | Threaded
Open this post in threaded view
|

RE: Canoo WebTest vs Selenium

Dierk König
> The issue is that we as users have advanced needs. We need automated  
> testing outside of a browser. Webtest meets this need.

agreed :-)

> We need  
> automated testing inside browsers to detect cross-browser  
> incompatibilities.

Hm, I have no experience in testing browsers, only webapps.
If you need to test whether browsers comply to a certain standard,
I have no idea about the best way to do that automatically.

One thing I have experience with is how to test whether your webapp
complies to the standards. If they do, chances are high that any
browser will show it correctly.

Of course, a margin of error remains. Some browsers may have internal
errors that prevent even the best webapps from rendering as expected.
That's the nature of the beast and simply cannot be solved by
automated testing. Some need for manual testing always remains, e.g.
around CSS, visual effects and such.

Automated testing of webapps is about testing the workflow,
error handling, persistency, availability, and sometimes load and
performance. Computers are very good at such tests.

People are very good at testing visual appearance.

> I see no compelling reason at all why a custom DSL cannot be made to  
> allow some scripts that would run in both environments.

I guess a custom DSL is possible to allow _some_ scripts to run in
both environment. But I feel that the feature set of those _some_
scripts is too limited. Analogy: SQL 92, a nice standard that every
vendor adds features to that you need to do any productive work. You
rely on the features and the value of the standard is zero.

> Surely this is a GOOD THING, if not please tell me why I am mad :)

You're certainly not mad.
I'm glad to see your rising interest in testing.

Dierk

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

Reply | Threaded
Open this post in threaded view
|

Re: Canoo WebTest vs Selenium

Marc Palmer Local
>
> Automated testing of webapps is about testing the workflow,
> error handling, persistency, availability, and sometimes load and
> performance. Computers are very good at such tests.
>
> People are very good at testing visual appearance.
>

...but people are not a cost effective choice for testing that some  
onXXX event fires as a result of some ajax action in 5 different  
browsers. Multiply this over 50 pieces of functionality and you start  
to see the scope of the problem.

>> I see no compelling reason at all why a custom DSL cannot be made to
>> allow some scripts that would run in both environments.
>
> I guess a custom DSL is possible to allow _some_ scripts to run in
> both environment. But I feel that the feature set of those _some_
> scripts is too limited. Analogy: SQL 92, a nice standard that every
> vendor adds features to that you need to do any productive work. You
> rely on the features and the value of the standard is zero.
>
>> Surely this is a GOOD THING, if not please tell me why I am mad :)
>
> You're certainly not mad.
> I'm glad to see your rising interest in testing.


My interest has always been there, but writing and debugging  
exhaustive webtest scripts is painful and the clients want the budget  
spent on things they can see. That's why products like Selenium are  
attractive as they can record what you do. This is MUCH faster and  
means you can have a relatively unskilled person doing it. They just  
need to know the trail of functions to perform and the desired  
outcome. This makes commercial sense.

So in the ideal world you have something like Selenium IDE to record  
scripts, and you can then run these scripts through webtest for your  
functional testing independent of browsers, and then in Selenium on  
every browser you can get your hands on, to make sure you have done  
all the workarounds you need to get things working on the major market  
share of browsers.

This is common sense surely, it's just not what people are doing just  
yet.

Marc


---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

Reply | Threaded
Open this post in threaded view
|

RE: Canoo WebTest vs Selenium

Dierk König
> So in the ideal world you have something like Selenium IDE to record  
> scripts, and you can then run these scripts through webtest for your  
> functional testing independent of browsers,

"Record-Replay is the least cost-effective way of test automation."
(Keith Zambelich,
Automated Testing Specialists Group and Mercury affiliate)

That's being said, you can use http://webtestrecorder.canoo.com
to have a jump-start for your webtests.

happy testing
Dierk

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

Reply | Threaded
Open this post in threaded view
|

Re: Canoo WebTest vs Selenium

JamesPage
Dierk,

I am not a Q&A guy... But I have done a fair bit of testing....

I have built a browser in the past for the ARM chip set. So I know a fair bit about testing on the client side.... I.e... it is hell....

With browser testing you have to eat dog food... For example on the site that I am working on at the moment it works fine on every browser except early versions of firefox... If I had to do a manual test on each of the browser that are out in the market I would not get any other work done......

>
Hm, I have no experience in testing browsers, only webapps.

Well the problem is that it is very hard to follow the standards for the browser.... There is so much badly formed code out there and you compromise and you compromise.... You end up breaking the standards to let in bad html. Just look at the parser for firefox, or look at the copyright notice for IE... to get an idea what a mess the code is in those two browsers.

>
complies to the standards.   If they do, chances are high that any browser will show it correctly.
Sadly that is not the case....  Just look at this http://www.quirksmode.org/dom/w3c_css.html for an example.

Webtest looks like it is allot more functional... But on the calling a browser action the two look smiler so why can't WebTest call Selenium? That would add to the fantastic tool that you have already got and we the users would have the ability to easily create tests (more tests are better), and also eating dog food (when we have to).

It would lead to better coverage and would that not be good?

All the best

James

On 10/31/07, Dierk Koenig <[hidden email]> wrote:
> So in the ideal world you have something like Selenium IDE to record
> scripts, and you can then run these scripts through webtest for your
> functional testing independent of browsers,

"Record-Replay is the least cost-effective way of test automation."
(Keith Zambelich,
Automated Testing Specialists Group and Mercury affiliate)

That's being said, you can use http://webtestrecorder.canoo.com
to have a jump-start for your webtests.

happy testing
Dierk

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

RE: Canoo WebTest vs Selenium

Dierk König
For example on the site that I am working on at the moment it works fine on every browser except early versions of firefox...  
Every browser except... How do you know?
 
 If I had to do a manual test on each of the browser that are out in the market I would not get any other work done......  
True. But if your application is build in such a way that it would require testing on so many browsers, than you 
are screwed no matter whether you test manually or whatever tool you use.  

>
Hm, I have no experience in testing browsers, only webapps.

Well the problem is that it is very hard to follow the standards for the browser....  
I personally don't think so.
 
 There is so much badly formed code out there and you compromise and you compromise....  
That's no excuse for letting badly formed code pass your tests.
 
 >complies to the standards.   If they do, chances are high that any browser will show it correctly.
Sadly that is not the case....  Just look at this http://www.quirksmode.org/dom/w3c_css.html for an example. 
If you have any tool that can test CSS, please tell me. WebTest can't and doesn't even try to do so.
verifyLooksSlick() will never be supported.
 
The WebTest approach is:
* distinguish core workflow from appearance
* test the core workflow automatically  
* test appearance visually 

Webtest looks like it is allot more functional... But on the calling a browser action the two look smiler so why can't WebTest call Selenium? That would add to the fantastic tool that you have already got and we the users would have the ability to easily create tests (more tests are better), and also eating dog food (when we have to).  
Because it is far too slow (around factor 10)
and requires too much resources (around factor 100).
In other words, WebTest scales better in three (!) orders of magnitude, which is relevant for any project that is really worth testing automatically.
 
WebTests are at least as easy to create as any other test. If you seek a jump-start and assistance, you can use http://webtestrecorder.canoo.com.
But as we all know, the real power comes from reusing common modules. This is what makes test suites grow exponentially without
introducing duplication.

It would lead to better coverage and would that not be good? 
 
Yes, that would be good. But we will only achieve good coverage and sustainable tests when we apply common
engineering knowledge to the test automation activity.
The unpleasant reality is, that this requires some work. Don't be fooled by the ones that tell the opposite.
Half-hearted attempts only produce costs since their artifacts are too unstable.
"Quality is for free - but only for those who are willing to invest heavily."
 
happy testing
Dierk
Reply | Threaded
Open this post in threaded view
|

Re: Canoo WebTest vs Selenium

JamesPage


On 10/31/07, Dierk Koenig <[hidden email]> wrote:
For example on the site that I am working on at the moment it works fine on every browser except early versions of firefox...  
Every browser except... How do you know?

Every browser that we test.... :) 

If I had to do a manual test on each of the browser that are out in the market I would not get any other work done......  
True. But if your application is build in such a way that it would require testing on so many browsers, than you 
are screwed no matter whether you test manually or whatever tool you use.  

>
Hm, I have no experience in testing browsers, only webapps.

Well the problem is that it is very hard to follow the standards for the browser....  
I personally don't think so.
 
 There is so much badly formed code out there and you compromise and you compromise....  
That's no excuse for letting badly formed code pass your tests.

Then the browser would not show a large percentage of the web sites that are out there. If IE/Opera/Firefox etc all followed the standards.... Hence quircks mode....

>complies to the standards.   If they do, chances are high that any browser will show it correctly.
Sadly that is not the case....  Just look at this <a href="http://www.quirksmode.org/dom/w3c_css.html" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://www.quirksmode.org/dom/w3c_css.html for an example. 
If you have any tool that can test CSS, please tell me. WebTest can't and doesn't even try to do so.
verifyLooksSlick() will never be supported.

Try this link in your browser to see if it meets the standards...
http://www.webstandards.org/files/acid2/test.html#top


Selenium (what an awful name)  will assert if something has been rendered at certain position etc...  Very important for Web 2.0 apps.

You could do the acid2 test and assert that results where correct on Selenium...

At the moment I am working with google maps... Because of Selenium I found out that you can not plot more than about 10 polylines on a map for it to work on ie6....

The WebTest approach is:
* distinguish core workflow from appearance
* test the core workflow automatically  
* test appearance visually 

Webtest looks like it is allot more functional... But on the calling a browser action the two look smiler so why can't WebTest call Selenium? That would add to the fantastic tool that you have already got and we the users would have the ability to easily create tests (more tests are better), and also eating dog food (when we have to).  
Because it is far too slow (around factor 10)
and requires too much resources (around factor 100).
In other words, WebTest scales better in three (!) orders of magnitude, which is relevant for any project that is really worth testing automatically.

Right... Are you saying that we should not eat dog food?

So far I would not have found the 3 major bugs that I mentioned on Web Test....

Also webtest will not find memory leaks in the client... and render times.... Which are very important again in Web 2.0 apps..

And we are saying use Webtest for the heavy lifting... and use Selenium for testing the client....

WebTests are at least as easy to create as any other test. If you seek a jump-start and assistance, you can use <a href="http://webtestrecorder.canoo.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://webtestrecorder.canoo.com.
But as we all know, the real power comes from reusing common modules. This is what makes test suites grow exponentially without
introducing duplication.

It would lead to better coverage and would that not be good? 
 
Yes, that would be good. But we will only achieve good coverage and sustainable tests when we apply common
engineering knowledge to the test automation activity.
The unpleasant reality is, that this requires some work. Don't be fooled by the ones that tell the opposite.
Half-hearted attempts only produce costs since their artifacts are too unstable.
"Quality is for free - but only for those who are willing to invest heavily."

Are you talking about getting  Selenium and WebTest  to work together or are you talking about testing in general?

What would be the problem?

happy testing
Dierk

All the best

James
 


12