Last year I started writing tests for the WhatTodo app and shared my experience in a talk called “Testing in Flutter” which was more focused on how to get started writing tests for the legacy codebase.

The idea was to start writing tests end-to-end first using the flutter driver. In that talk, I also mentioned some limitations of the flutter driver. Because of that, I was looking for a better library to test end-to-end flow with more flexibility.

Now that wait is over. Flutter team announced  Updates on Flutter Testing a few months back with a new integration test package which was built on widget testing API and supports firebase test lab.

So I could not help myself and started migrating my test. I must say it was a smooth migration compared to what I’ve done in the past.

So in this blog, I am going to share my experience and thoughts on this migration. This article is not going to deep dive into the testing concept.

Why do I need to migrate? What problem does it solve?

First of all, if the team is happy with the flutter driver tests and everything is working as expected then you might not need to migrate right away.

The migration cost would be low or high based on the number of tests you have in the project.

But there are few reasons in my opinion why you should migrate.

1. Limited flutter driver API.
   • Currently, the flutter driver is limited on text-based assertion. We cannot test color, size, visibility, and space matching on-screen.
2. Learn two APIs for testing.
   • We need to learn two different ways of doing the same thing which achieve the same result. For example, below is the code to match the text in both APIs.
     • Flutter Driver → expect(await driver.getText(finder),"MyText")
     • Widget test → expect("Mytext",findOneWidget)
3. Does not support Firebase Test lab.
   • Firebase test lab only allow Emulator UI and automator test to run. flutter_driver does not support running the test in espresso. Hence cannot test the apk in the firebase test lab.
4. Cannot access the Flutter framework code in Flutter Driver.
   • The problem described above in point 1 for not supporting color, size, and visibility matches are because of flutter driver runs on the different process and cannot access flutter framework code.
   • It’s become difficult to mock, fake, or stub a class in the test. This can partially be achieved by communication channels as I explained in my talk. But it’s a lot of boilerplate code and not an ideal way to do it in testing.

How to Migrate?

We will start the migration by setting up the new integration test first.

1. Setup

1. Add new integration test dependency in pubspec.yaml.

   # pubspec.yaml
   …
   dev_dependencies:
     flutter_test:
       sdk: flutter
     integration_test: ^1.0.2+2
   …

   Use the latest version of integration_test . The current version is ^1.0.2+2 as of this writing. In the beginning, I used an older version ^1.0.0 and got FacingTimer error when trying to run the test. I resolved the issue by upgrading to the latest version.
2. Create a dart file with the name integration_test_driver.dart in your existing test_driver folder with a code snippet given below.

   import 'package:integration_test/integration_test_driver.dart';
   Future<void> main() => integrationDriver();

3. Create a folder at the top-level with the name integration_test where all your new integration tests will reside. The structure will look like this.

   lib/
     ...
   integration_test/
     about_us_page_test.dart
   test/
     # Other unit tests go here.
   test_driver/
     integration_test.dart

4. We are done with setting up the new integration_test. The next step is to move the flutter driver test file into the new integration_test which is now using WidgetTesting API.

2. Refactoring: Move small and one test at a time

The second step is more of a refactoring technique. A simple rule in refactoring is to start small.

The first mistake I made was dragging and dropping all the files from test_driver to integration_test folder and got a whole bunch of errors.

I was trying to do a lot of stuff at once and failed miserably. I revert the changes and started back from moving one file at a time while fixing all the errors in that file.

This also helps me to understand what things I need to take care of when I move another file and also extract common functions or utility methods along the way.

3. Migrate Text based tests

So I start by moving a simple test about_us_page_test.dart which just simply matches the text. This how it looks

Before migration.

After the migration.

1. Add IntegrationTestWidgetsFlutterBinding.ensureInitialized(); which is imported from intregation_test in main()
2. We need to define a test entry point. In flutter driver we did this by keeping separate entry point file (about_us_page.dart) and test file (about_us_page_test.dart). We can directly do that into integration test using pumpWidget API which you can see in the above example done with AboutUsScreen() screen wrapped in a MaterialApp.
3. We can remove the flutter driver setup because we don’t need that anymore.
4. The flutter driver text match can be replaced by a widget testing API like this. This will save a bunch of boilerplate code and make it simple to use and understand.

   // flutter driver text match
    final titleAbout = find.byValueKey(AboutUsKeys.TITLE_ABOUT);
    expect(await driver.getText(titleAbout), "About");
    // is replaced with this.
    expect(find.text("About"), findsOneWidget);

5. Run a test using the following command and you will see “Test starting…” text in the emulator.

   flutter drive \
      --driver=test_driver/integration_test_driver.dart \
      --target=integration_test/about_us_page_test.dart

Note: A good practice would commit the changes after you successfully migrate one test file at a time.

4. Migrate Gesture based tests

The following is a test to add a task and showing on the home screen when successfully added. This is how it looks

Before migration.

After migration.

1. We removed the flutter driver, added integration initialization, and driver.getText is replaced by find.text as we learned from the previous test.
2. Here instead of running a specific widget/screen, we are running the whole app using app.main() which calls our home screen runsApp().
3. In the flutter driver, screen rebuild happens automatically. This is not the case with widget API. In widget API we need to explicitly call tester.pumpAndSettle() to rebuild the widget.
4. Keep extracting common methods as you go. In my case it’s tester.tapAndSettle() and byValueKey("MyKey")
5. Run the test and verify. Don’t forget to call app.main() in test else it will fail to start the test.

5. Migrate Mock based tests

This is how we seed fake data and then we clean it up.

1. In widget API we don’t need a communication channel to get access to the flutter framework. We directly inject mock values or fake them.
2. We just seed the data when we start the app. The seed function looks like this.

   Future seedAndStartApp(WidgetTester tester) async {
     app.main();
     await seedDataInDb();
     await tester.pumpAndSettle();
   }

Running Test with Code Magic

1. With Firebase Test lab

Integration test now allows us to create a separate android-test.apk which runs the espresso test. This will help us to test the apk in the firebase test lab. I am not going into details on how to do that in this article. They’re already great resources out there. If you are interested then please check this blog and video.

2. Without Firebase

You can also use the Code magic emulator to run your test without using the firebase test lab. Currently, I am running my test on the code magic emulator only.

Issues

Currently, the integration test does not allow running an entire folder of tests – just like normal unit tests, instead, it always running one test file at a time.

Conclusion

I’ve been part of few migrations in the past where the migration to a different library or a package leads to breaking API changes. Since we already knew how widget APIs work it was easy to migrate quickly. This was one of the smoothest migrations I ever did. Thanks to the flutter team.

If you currently have flutter_driver tests then it will good to migrate early to avoid more breaking API changes. This will give us more area to test instead of just text matching.

Thank you for taking the time to read my blog.

In case of any question, you can reach me on Twitter or Email me.

If you like this kind of content then you can subscribe to my blog and be the first one to get notified when any new content is published.

If you like this article, please like and share it with your friends/colleagues and help me to spread the word.

Resources

1. The official documentation on flutter website and Chris Sell blog on New Integration API.
2. Flutter integration test with Firebase Test Lab & Codemagic CI/CD
3. Flutter Integration Test Tutorial + Firebase Test Lab & Codemagic

Photo by Handiwork NYC on Unsplash