All Posts by Abhimanyu Grover

When to stop testing or stop documenting?

As product managers, every now and then we have to make decision whether to continue testing that feature or move on? It doesn’t just apply to testing efforts, but also to test case coverage & documentation, i.e. to continue writing more test cases for a particular feature or move on to next?

How do you decide in such cases?

Maybe we can borrow a concept or two from behavioural psychology: maximizing and satisficing.

Maximizing is when you’re trying to do as much as possible within given means.
Strictly in psychology terms, Maximization is a style of decision-making characterized by seeking the best option through an exhaustive search through alternatives.

Satisfising is when you’re trying to do just good enough to be satisfied (MVP for ideas and decisions).
In psychology terms, Satisfising is a style of decision-making which would mean exploring through options until you find one just enough or acceptable enough.

So which one to use when?

Look how often your customer spend time on this feature. If it’s, say 80% – 90%, it’s certainly calls for maximizer behavior – and you want to put everything you have in your arsenal at this, in terms of testing and documenting.

If the feature is occasionally used, you can get away with satisfising. You just need to do enough so that you know it works as desired – nothing more!

Obviously this principle can be applied on all types of optimization problems: at work and at life in general.

 

Free test management plans launched for Test Collab

Today we have released pricing plans for Test Collab. These free forever plans will help development and quality assurance teams to improve their test management without worrying about any payments or subscription ever!

It has always been our mission to making quality assurance affordable but we realized  that there are not many free test management software available, especially cloud based. There are few open source test management software available which you have to download and host on your own server but it becomes quite cumbersome and complicated. Our free plans are available on cloud so your team can focus on testing rather than managing servers and software installation.

We have also made testing more affordable for bigger teams as we have launched $10/user pricing for large teams.

If you have an active free trial or expired account, you can find the option to downgrade/upgrade your account under ‘Update Plan’ page. There are however some limits in free plans which will be available on our pricing page. Signup for now for free plan.

 

 

GDPR Compliance update

We are in process of rolling out organization-wide update (internal and external) to comply with upcoming deadline of GDPR. Test Collab will be fully compliant with GDPR before the 25th May 2018 deadline.

If you have any questions or suggestions regarding GDPR compliance, reach us out from ‘Contact us’ link below.

New reports: one more step towards test intelligence

We’ve always been fascinated with extracting useful insights from the data - we’ve been reviewing a lot of our clients at Test Collab over last 9 months. Reviewing their problems, possible solutions and if we can positively impact the productivity given the data they have.


Each and every test case, its result of every execution ever, time spent and who assigned/executed a test - gave us a lot of data to work with.


There are potentially lot of useful insights that can be extracted with so much data. With this release we’ve attempted to scratch the surface - and there will be a lot more to come. For now, we believe you’re going to love these new reports.


You’ll find quite a few new charts on project overview page, milestone view page and new test case status report page (under Reports tab). It'll be unfair to not mention some of them here, given how useful they can be:

1. Test Case Last Run Statuses

Want to know overall health of your project? Just look at this chart and you'll have high-level understanding of how good your project is, testing-wise that is. This report shows you last state of all your test cases. 

Pro-tip: As you add new features, your unexecuted test cases will go up. When working on new project, keep an eye on 'unexecuted' test cases and schedule execution on such cases every few days. Alternatively, drill down suite-wise to see statuses in further details.

2. Time spent on test cases

This is one of my personal favourite. Over time, there are some test cases that take a lot of your developer's / tester's time. This chart will help you locate such outliers and let you further analyse why. You can plot multiple time metrics for time spent on each cases: average time, overall, maximum, minimum etc.

Pro-tip #1: Use this chart to locate outliers test cases, then run 5-whys analysis as to what took them so long.

Pro-tip #2: Alternatively, you can use this data to decide which test cases should be automated before the rest. ​

3. Error prone Test Cases

This is a distribution chart made of failure rates of your test cases. Highly useful when you want to pinpoint the troublesome cases of your project. If you think testing all cases all-the-time is good strategy, this chart will be an eye-opener. 

Sample use case: When developing new features, schedule testing of cases with overall high failure rates at relatively early stage. This will give your testers / developers more time to find cause of such failures resulting in less surprises on release date.

Pro-tip: We've observed cases with high failure rates are often a sign of either outdated test case documentation or some big underlying problem. Pay special attention to the cases above threshold failure rate.

4. Cases passed by suites

This heatmap chart shows you all the test suites of your project, color-coded as the percent of test cases passed. Green'ier the suite = Better passing % of test cases. The area of the block represents how many test cases there are in this suite relative to other suites. Larger area = Higher number of cases in the suite

Pro-tip: Sometimes a single module / set of features negatively impacts your project while other modules might be functioning as expected. This chart will help you locate such problem areas and act on them. Start paying attention to light green regions, find out what's lowering the score, is it development, testing or documentation?

5. Milestone burndown

This is useful when you're doing sprints or deadline-driven releases. Quickly see your tasks left as a burndown in timeline with ideal vs actual effort by your team. You'll also get instant feedback as a team of how your efforts are contributing towards the end goal and how fast.

Several other new reports which aren't mentioned here, but are released with this version: Test case assigned vs unassigned, defects reported over time, test run results over time and some new metrics.

As mentioned above, this is just beginning of large milestone - if you have some ideas for new reports / charts / widgets / metrics etc, please get in touch and let us know. 

Test Collab outage 5 December 2017: Issue rectified

As some of you may have noticed that we had an outage for quite a few hours. The issue was caused by manual error on production server at December 5, 22:00 PM. We were able to bring systems back up at December 6, 03:11:20 IST. We’ll be working on new measures to make sure such errors are not caused in future.

Test Collab’s new version: JIRA Cloud bi-directional integration, automatic screenshot upload and 4 new integrations

We are proud to announce the launch of Test Collab 1.16. This release aims to improve tester’s productivity and adds new integrations with your favorite tools.

 

JIRA Cloud bi-directional Integration

Test Collab can now be used inside your JIRA cloud instance. You can create/manage test cases and test executions directly from your JIRA cloud instance. JIRA cloud Plugin can be obtained from JIRA marketplace
If you are a JIRA self-hosted user, check this listing

 

Automatic Screenshot Upload

This feature is going to save a lot of time for testers. While testing when a bug is encountered, a tester usually has to attach a screenshot to a failed test case along with necessary information about the error. We’ve automated a part of that workflow and now attaching/annotating screenshot is done in single key-press. Forget about saving file in a directory and then attaching to your case, just click Print-Screen and it’s automatically uploaded.

 
Check out this video below to see it in action:

 

 

New Integrations

We’ve also release support for new integrations with these tools: Trello, Asana, YouTrack and Team Foundation server. You can see list of all our integrations.

Would you like to suggest a new ideas for our next release? Please get in touch.

Test Driven development for API-first apps with Postman & Newman

Test driven development as you may already know is a simple practice of having your dev workflow in such a manner that your tests determine the flow of development. In simple words, you’re writing tests first and then developing the actual code that passes those tests. Here are some benefits of this approach in case you’re wondering.

API-first involves developing your API first (preferably with a well-developed API framework) and any other UI or client later. There are immense benefits to this approach, since API endpoints are much easier to test. All you have to document is input and output in most cases while developing monoliths you know it’s not as simple.

Another benefit both of these strategies offers comes from separation of concerns principle, since you’re dealing with backend only – it eases up any future debugging and deployment.

If until now you’ve found yourself spending a lot of time on monoliths then you can understand how painful impractical it might be to use Test Driven development. However if you enter into API development, things change and wonderful things start to happen when you chose TDD.

With tools like Postman, it gets even easier to do so which we’ll explore shortly.

To Do

We’ll be combining Test Driven development principles with API-first strategy and focus on a workflow which would be much faster than traditional monolith-style development.

We’ll also look at implementing collaboration within team (scm), so everyone in a team can contribute.

About Postman & Newman

Postman is a simple tool which allows you to make HTTP requests with various parameters and save different set of collections which you can refer later or trigger again.

Newman is another tool from Postman developers which is a command line runner for API requests.

You’ll need to install both of these tools before moving forward.

Workflow

Our workflow is broken down in following steps:

1. Creating requests (equivalent of writing test cases)
2. Automating these requests for cli (so they can be used in build servers)
3. File structure for requests we create (to enable collaboration)
4. Bonus: Documentation generation from test cases

#1 Creating some requests using Postman (Test definition)

Since we’re talking test driven development here, let’s first create a request (or test) in Postman along with our expected result. We don’t have any API yet but let’s do it anyway since we need to define behavior first.

We’ll first create a collection in Postman so that API can be shared with team later. In that collection, we’ll make some requests.

test driven development - step 1

You can repeat these steps to document multiple end-points all at once or one-by-one.

#2 Automating requests with Newman

Now that we have some requests we’d like to execute, let’s see how we can automate them with our development environment. Right click on collection menu and export the whole collection to JSON file.

Your API collection can be exported to a json file.

It’ll enable you to maintain your source code control workflow and collaboration.

Alternatively, you can also generate a shareable link to your collection which is what we’re going to use for speed. Go to share collection from the context menu, then Collection Link to do so.

Now let’s run this collection with following command:

newman run <your collection json file or url>

image-3.png

As we have no API running right now, there are 2 failures which are clearly shown.

Now let’s develop our sample API and run this again:

We’ll use simple node/express code here:

const express = require('express')

const app = express()

 

app.get('/say-hello', function (req, res) {

 res.send('Hello World!')

})

 

app.listen(8080, function () {

 console.log('Example app listening on port 3000!')

})

And run it with

node server.js

And run newman again:

image-4.png

Note that I’m using URL as my JSON collection – for a typical development environment this would be a file in your local filesystem.

#3 Committing your work

Now you have API code + JSON collection, both are standard files which can be committed to any repository. You can decide any standard directory structure that works for you. For my example, I’ll choose:

/api (actual application)

/tests (for my collection file)

The good thing about having your collection file in source control is that all of your team members know how to call a particular API endpoint.

#4 Bonus: API doc generation from test cases

Another benefit is automatic documentation generation – Postman does a great job here. Each collection you upload to Postman server automatically gets a documentation page like this:

image-5-orig.png

Apart from that, each collection file can be fed into a monitoring agent when you deploy to production. These are available in Postman pro edition but you can do this yourself too.

Conclusion

Tools like Postman and Newman combine both GUI and command line interfaces so APIs can be developed easily. It also makes TDD easier along with team collaboration without leaving our standard source code control tools.

3

7 old tools teams need to dump in 2017 for better ones, to develop better

Today’s post is about migrations from old tools and technologies to new and better ones. One cannot ignore how quick our technology is progressing, one small change in workflow saves enormous amount of time which otherwise is overlooked. When I say technology I do not mean technology on conceptual level, but technology which empowers common users and teams like ours.

We often get comfortable with the set of tools we work with daily, even though we really need to be more transparent and more objective with such decisions. We at Test Collab are often guilty of the same thing. It’s crucial to get some external feedback to see things more objectively. So that’s why I’d like to share some tools you need to dump and why, along with their newer and better alternatives.

#1 CI System: Old self-hosted Jenkins to any new cloud hosted tools.

Out: Jenkins

In: Shippable

Why?

Jenkins is a mature project and while it was a great tool few years back, it no longer works with essential requirements teams need today. The absence of pipelines and branching as first class citizens, and stricter environment level provisioning – Jenkins no longer works for moving up the ladder of continuous deployment maturity.

Apart from getting old, other issue is cost. Computing costs have gone down and most of the cloud tools provide a lot of functionality and builds at free of cost – at a starter level. These new cloud tools are extremely scalable and provides great ROI.

Other alternatives:

Travis CI, Semaphore CI

 

#2 Version control: SVN to Git

Out: SVN

In: Git

Why?

In beginning, I was a bit skeptical about the gains it would offer compared to the cost it would require to migrate. So I set out to do some experimentation and was pleasantly surprised by source code push/pull speeds and merging. Git uses compressed network transfer mode which made pulling out 2500 or so files in seconds as opposed to minutes in Subversion.

Although SVN is still an active project but I doubt they’ll push something as innovative as Git – so this seems like a good move.

Alternative:

Mercurial

 

#3 Project management: JIRA to any Kanban tool

Out: JIRA

In: Trello

Why?

While JIRA and similar tools make good issue managers to dump all your bugs in, they make a terrible project management tool. Really, think about it, what project really is? First there’s your product and any iteration you do on it is a project. Now in order to be kickass with your iterations you need to see where everything is, along with their respective statuses and if something is being worked upon for too long to quickly identify issues. JIRA and such tools don’t offer that (at least they weren’t created for this purpose)

With new Kanban board tools, you get to see full picture every time you open them instead of dead index of issues. I don’t know about others but I’m urged to push forward when I see the bigger picture.

Alternatives:

Kanban Tool

 

#4 Test Management: Spreadsheets to Test Collab

Out: Spreadsheets

In: Test Collab

Why?

Okay, we maybe a little biased here but hey, we gotta pay our bills too. Spreadsheets doesn’t work well for test management tasks. Spreadsheets don’t delegate… Test Collab does. Spreadsheets doesn’t integrate with your issue managers, Test Collab does. Spreadsheets doesn’t track testing time and quality metrics, Test Collab does. So I guess that’s enough reason to at least try us. huh?

Alternatives:

HP Test Manager

 

#5 Development environment: Desktop to C9

Out: Desktop

In: C9

Why?

Why code on your laptop when you can code in browser on remote workspace? Wait did I say workspace? Try tens or hundreds of workspaces. With tools like C9 you can launch as many workspaces as you want with clean OS and nothing else. Finally, you can focus plainly on all things creative instead of resolving dependency hell and desktop issues.

You can experiment more on OS level and quickly install / uninstall so many packages. Desktop lovers might be thinking, so what? We’ve got VM’s and now Docker, what’s the big deal? Yes it is a big deal – I can open 10 parallel workspaces and play with them without watching my laptop hard-drive go crazy. And another lovable fact: your packages download super-fast without hurting your bandwidth.

Alternatives:

Code Anywhere

 

#6 Webhooks: Custom coded to Zapier

Out: Your hacky custom coded stuff

In: Zapier

Why?

There’s no reason to code several connectors among different web services anymore (well in at least 90% of the cases now). Because any service you can possibly imagine will most likely be found at Zapier and you can make your own integrations very easily. It has never been this easier to connect 2 or more services together and make data flow through them – it just works wonderfully.

Alternatives:

Automate.io

 

#7 Hosted servers to Server-less computing

Out: Hosted servers

In: AWS Lambda

Why?

I haven’t seen a similar company as Amazon. These guys are so disruptive that they disrupt their own products – Lambda is such a product. AWS Lambda removes the need of always-on servers altogether instead your code runs in the cloud and you just manipulate the result. Migrating an existing API to AWS Lambda has steep learning curve and also requires some kind of shift in team’s mentality but it’s totally worth it. You can cut down your costs as much as 50-60% in some cases.

Alternatives:

Google Cloud Functions

How to organize your test cases – lessons learned from some of our most successful customers

A small project can sometimes grow really fast in today’s marketplace. It can really be an unpleasant experience (instead of a positive one) to some teams who are less experienced or launched a start-up for first time. It helps to prepare upfront and this article is all about that: To equip you with the knowledge upfront which you can use now and in future.

Continue reading

1 2 3 6