6.194
Automated Unit and Integrity Testing in R

Introduction

Testing is an integral part of the software development lifecycle, ensuring that software applications and their individual components function correctly and meet specified requirements. Various types of testing are used to validate different aspects of the application, from individual units of code, such as functions or classes, to the integrated system and external components.

Test-to-Pass vs Test-to-Fail

Two fundamental approaches to testing are test-to-pass and test-to-fail. Understanding these approaches helps in designing more robust and comprehensive test cases.

Test-to-Pass confirms that the software works as expected under normal conditions. The general approach is to write tests with valid inputs to verify that the software performs the desired operations correctly. The focus is on typical usage scenarios to ensures that the software meets its functional requirements which helps verify the correctness of the main functionality. However, these tests may not uncover edge cases or rare conditions and thus are primarily focused on confirming positive scenarios.

On the other hand, Test-to-Fail testing identifies scenarios where the software might fail and ensure it handles them gracefully. The idea is to write tests with invalid or unexpected inputs to check how the software handles errors and edge cases and whether the system’s or application’s performance gracefully degrades – or simply crashes. Thus the focus is on robustness and error handling to ensure that the software fails in a controlled manner and provides meaningful error messages or handles exceptions appropriately. The main benefits of test-to-fail include helping to identify and fix potential weaknesses in the software, ensure robustness by handling edge cases and unexpected inputs, and improve the reliability of the software by testing how it fails. Naturally, it may not cover all typical usage scenarios and requires substantial effort to identify and create edge case tests.

Both test-to-pass and test-to-fail are essential components of a robust testing strategy. While test-to-pass ensures that the software meets its functional requirements, test-to-fail identifies potential weaknesses and improves the software’s robustness. By integrating both approaches, the software developer can develop more reliable and resilient software, ultimately enhancing user satisfaction and reducing the risk of failures in production.

Testing Workflow

A comprehensive testing workflow should include both test-to-pass and test-to-fail approaches and generally requires the following process steps:

1. Define Clear Requirements: Clearly define the expected behavior of the software, including both normal operations and error handling.

2. Develop a Test Plan: Create a test plan that outlines which scenarios to test, including both typical use cases and edge cases.

3. Automate Testing: Use automated testing tools to regularly run both test-to-pass and test-to-fail scenarios, ensuring continuous validation of the software.

4. Prioritize Critical Functions: Focus on critical functions and high-risk areas first, ensuring that they are thoroughly tested under both normal and exceptional conditions.

5. Review and Refactor: Continuously review and refactor your test cases based on new insights and changing requirements, ensuring comprehensive coverage.

Types of Testing

Most software development organizations perform seven different levels of testing, each ensuring a distinct degree of functionality.

1. Unit Testing
2. Integration Testing
3. System Testing
4. Acceptance Testing
5. Performance Testing
6. Security Testing
7. Usability Testing

Unit Testing

Objective: Verify that individual units or components of the software work as intended.

How to Perform: - Write test cases for each function or method in isolation. - Use assertions to check if the function returns expected results for given inputs. - Tools: JUnit for Java, pytest for Python, testthat for R.

Integration Testing

Objective: Ensure that different modules or services in a system work together as expected.

How to Perform: - Test the interaction between integrated components. - Identify and test critical integration points. - Tools: JUnit with Spring for Java, pytest for Python, testthat for R.

System Testing

Objective: Validate the complete and fully integrated software product to ensure it meets the requirements. Create user stories and acceptance criteria to validate if the arithmetic operations fulfill the business needs.

How to Perform: - Conduct end-to-end testing of the application. - Test scenarios that reflect real-world usage. - Tools: Selenium for web applications, QTP/UFT for desktop applications.

Acceptance Testing

Objective: Verify that the software meets the business requirements and is ready for delivery.

How to Perform: - Conducted by the end-users or testers. - Use predefined acceptance criteria. - Tools: Cucumber, FitNesse.

Performance Testing

Objective: Ensure the software performs well under expected load conditions and that throughput and response time meet non-functional quality-of-service requirements.

How to Perform: - Measure response time, throughput, and resource utilization. - Conduct stress, load, and scalability tests. - Tools: JMeter, LoadRunner.

Security Testing

Objective: Identify vulnerabilities and ensure the software is secure, for example checking for common vulnerabilities such as SQL Injection Attacks, Denial of Service Attacks, and Buffer Overflow Attacks. Also, verifies that information is kept secure, encrypted, and confidential.

How to Perform: - Perform penetration testing and vulnerability scanning. - Test for common security issues like SQL injection, XSS, etc. - Tools: OWASP ZAP, Burp Suite.

Usability Testing

Objective: Verify that the software is user-friendly and meets user experience standards. Commonly uses observation and task completion studies.

How to Perform: - Conduct tests with real users. - Gather feedback on ease of use, interface design, etc. - Tools: UserTesting.com, Morae.

Best Practices

1. Automate Where Possible: Use automated testing tools to save time and reduce human error.
2. Test Early and Often: Integrate testing into your development process from the beginning.
3. Use a Test Plan: Develop a comprehensive test plan that outlines testing strategies, scope, resources, and schedule.
4. Prioritize Tests: Focus on critical functionalities and high-risk areas.
5. Review and Refactor: Continuously review and improve your test cases to adapt to changing requirements.

Test Automation with testthat

Testing is a crucial part of software development that ensures your code works correctly. In R, the testthat package provides a robust framework for writing and running tests. This tutorial will guide you through the concepts of unit and system testing using the framework of the testthat package.

Setting Up the Environment

First, you need to install the testthat package if you haven’t already, using either the R code fragment below, or by installing it through your IDE (RStudio, Posit, or VS Code, etc.):

Installation is only necessary once, but you need to load the package into your R session everytime you wish to use it:

library(testthat)

Unit Testing

Unit testing focuses on testing individual functions or components of your code. The goal is to verify that each function performs as expected.

Writing Unit Tests

Consider a simple function that adds two numbers:

add <- function(x, y) {
  return(x + y)
}

To write a unit test for this function, create a separate test file. By convention, test files are placed in a “tests/testthat” directory within your project folder. Let’s name our test file test-add.R.

In test-add.R, you would write the following tests:

# TEST SCRIPT: test-add.R

library(testthat)

test_that("addition works", {
  expect_equal(add(1, 1), 2)
  expect_equal(add(-1, -1), -2)
  expect_equal(add(1, -1), 0)
})

## Test passed 🎊

In the above code, the function test_that defines a test block, and expect_equal checks if the function output matches the expected result.

Running Unit Tests

You can run the tests using the following command (assuming that you have your tests defined in the folder tests/testthat:

test_dir("tests/testthat")

Alternatively, if you are using RStudio/Posit, you can use the “Build” pane to run all tests. In addition, you can run the above command from the R Console within RStudio/Posit which would allow you to keep the testing mechanism outside your program/script code.

System Testing

System testing, also sometimes referred to as integration testing (even though it is not quite the same), involves testing the complete system or application. It ensures that different parts of the application work together as expected.

Writing System Tests

Let’s expand our example to a small system that performs basic arithmetic operations. We’ll create functions for addition, subtraction, multiplication, and division:

add <- function(x, y) {
  return(x + y)
}

subtract <- function(x, y) {
  return(x - y)
}

multiply <- function(x, y) {
  return(x * y)
}

divide <- function(x, y) {
  if (y == 0) {
    stop("Division by zero error")
  }
  return(x / y)
}

Next, we will create a new test file in the folder tests/testthat named test-arithmetic.R for system tests:

# TEST FILE: test-arithmetic.R

library(testthat)

test_that("arithmetic operations work correctly", {
  expect_equal(add(2, 3), 5)
  expect_equal(subtract(5, 3), 2)
  expect_equal(multiply(2, 3), 6)
  expect_equal(divide(6, 3), 2)
  
  # Testing edge cases
  expect_error(divide(1, 0), "Division by zero error")
})

In this test, we are checking the correctness of all arithmetic functions together, including handling edge cases like division by zero.

Running System Tests

Run the tests using the same command as for unit tests (again, assuming that the above test code exists in the folder):

test_dir("tests/testthat")

## ✔ | F W  S  OK | Context
## ⠏ |          0 | add                                                            ✔ |          3 | add
## ⠏ |          0 | arithmetic                                                     ✔ |          5 | arithmetic
## 
## ══ Results ═════════════════════════════════════════════════════════════════════
## [ FAIL 0 | WARN 0 | SKIP 0 | PASS 8 ]

Note that this command runs all tests in the folder tests/testthat. This allows tests to be segmented and more easily managed.

Best Practices for Testing in R

1. Write Tests Early: Develop your tests alongside your code to catch issues early. Some software development frameworks require tests to be written before the code is created to ensure that the code matches the requirements: write against tests.
2. Test Edge Cases: Always consider and test for edge cases and unusual inputs, especially checking boundary conditions. Most errors occur on boundaries, e.g., if a number is supposed to be 10 or more, then test for 9 (fail) and 10 (pass).
3. Keep Tests Small and Focused: Each test should focus on a single aspect of the function’s behavior.
4. Use Descriptive Names: Use descriptive names for your test cases to make it clear what is being tested.
5. Automate Testing: Integrate your tests into your development workflow to run them automatically.

Testing Side Effects

Side Effects in Programming

In programming, a side effect occurs when a function or expression modifies some state or interacts with the external world in ways beyond returning a value. Side effects can include altering a variable, modifying a data structure, writing to a file, writing to a database, sending a message to another process, or changing the state of the user interface.

There are several characteristics of side effects, including

1. Modifying Variables: Changing the value of variables that exist outside the function’s local scope.
2. I/O Operations: Reading from or writing to external sources like files, databases, or network sockets.
3. State Changes: Modifying the state of objects or data structures in ways that persist beyond the function call.
4. Interaction with External Systems: Calling external APIs or interacting with hardware components.

Consider a function that increments a global variable:

counter <- 0

increment_counter <- function() {
  counter <<- counter + 1
}

Here, increment_counter modifies the global variable counter, causing a side effect.

Another example is when a function writes to a file:

write_to_file <- function(text) {
  write(text, file = "output.txt")
}

Writing to a file is a side effect because it changes the state of the file system.

Functional Programming

In functional programming, side effects are minimized or eliminated to ensure functions are pure. A pure function is one that, given the same input, always returns the same output and does not cause any side effects.

Example of a Pure Function:

add <- function(x, y) {
  return(x + y)
}

This function is pure because it does not modify any state or interact with the outside world.

Practical Side Effects

Interactions with the outside world (e.g., user input, file operations) are inherently side effects and are essential for real-world applications. They would not be possible to build without side effects. Side effects can be useful for managing state and controlling program flow.

However, functions with side effects are more challenging to test because their behavior can depend on external state. In addition, side effects can introduce defects (bugs) that are difficult to trace, especially in large codebases. Consequently, functions with side effects are less predictable, as they may behave differently based on external conditions.

In practice, while we often need side effect, it is advisable to manage side effects effectively and to minimize them as much as feasible. Use pure functions wherever possible and limit side effects to specific parts of the code. Encapsulate side effects within well-defined boundaries, such as specific modules, classes, or layers of the application. Prefer immutable data structures that do not change state after creation. And, finally, isolate side effects in tests, using techniques like mocking to simulate interactions with external systems.

Testing Database Code

Modifications of databases through SQL statements or triggers is a type of side effect. To test SQL code and triggers using the testthat package, we need to write functions that interact with the database and perform the required checks.

Let’s walk through an example using SQLite. The steps are shown below

1. Creating a Test Environment

Create a temporary SQLite database, set up the necessary tables and triggers, and populate the database with synthetic sample data. In the example below, we use the special database name “:memory:” which creates a SQLite database in memory rather than in the file system. The function dbExecute() returns a status of 0 when the SQL code was executed correctly. We are omitting the checks in the code below for brevity.

library(RSQLite)

# Create a temporary in-memopry SQLite database
con <- dbConnect(RSQLite::SQLite(), ":memory:")

# Create tables
s <- dbExecute(con, paste0("CREATE TABLE users (",
                           "  id INTEGER PRIMARY KEY, ",
                           "  name TEXT, ",
                           "  last_updated TEXT)"))
s <- dbExecute(con, paste0("CREATE TABLE orders (",
                           "  order_id INTEGER PRIMARY KEY, ",
                           "  user_id INTEGER, ",
                           "  status TEXT, ",
                           "  FOREIGN KEY (user_id) REFERENCES users(id))"))
s <- dbExecute(con, "CREATE TABLE audit_log (log_id INTEGER PRIMARY KEY, user_id INTEGER, action TEXT)")

# Create a trigger
s <- dbExecute(con, "
  CREATE TRIGGER update_timestamp
  AFTER UPDATE ON users
  FOR EACH ROW
  BEGIN
    UPDATE users SET last_updated = CURRENT_TIMESTAMP WHERE id = NEW.id;
    INSERT INTO audit_log (user_id, action) VALUES (NEW.id, 'updated');
  END
")

2. Writing Tests Using testthat**

Next, we write test cases for unit testing, integration testing, and data integrity testing.

Unit Testing

library(testthat)

test_that("trigger updates timestamp", {
  # Insert a test record
  dbExecute(con, "INSERT INTO users (id, name, last_updated) VALUES (1, 'Alice', '2023-01-01 00:00:00')")

  # Update the test record
  dbExecute(con, "UPDATE users SET name = 'Alice Updated' WHERE id = 1")

  # Check if the trigger updated the timestamp
  result <- dbGetQuery(con, "SELECT last_updated FROM users WHERE id = 1")
  expect_false(result$last_updated == '2023-01-01 00:00:00')
})

## Test passed 😸

Integration Testing

test_that("integration of users and audit_log", {
  # Insert a test record
  dbExecute(con, "INSERT INTO users (id, name) VALUES (2, 'Bob')")

  # Update the test record
  dbExecute(con, "UPDATE users SET name = 'Bob Updated' WHERE id = 2")

  # Check if the trigger inserted into the audit_log
  result <- dbGetQuery(con, "SELECT * FROM audit_log WHERE user_id = 2")
  expect_equal(nrow(result), 1)
  expect_equal(result$action, 'updated')
})

## Test passed 🥇

Data Integrity Testing

test_that("foreign key constraint prevents deletion", {
  # Insert test records
  dbExecute(con, "INSERT INTO users (id, name) VALUES (3, 'Charlie')")
  dbExecute(con, "INSERT INTO orders (order_id, user_id, status) VALUES (1, 3, 'pending')")

  # Attempt to delete the user (should fail due to foreign key constraint)
  expect_error(dbExecute(con, "DELETE FROM users WHERE id = 3"))

  # Verify that the user is still present
  result <- dbGetQuery(con, "SELECT * FROM users WHERE id = 3")
  expect_equal(nrow(result), 1)
})

3. Running the Tests

As shown before, to run the tests, use the test_dir or test_file functions from the testthat package. If your tests are in a directory called tests/testthat, you can run them all with:

testthat::test_dir("tests/testthat")

Or, if you have your tests in a single file, you can run:

testthat::test_file("tests/testthat/test_sql.R")

Conclusion

Effective testing is vital for delivering high-quality software. By understanding and implementing different types of testing, you can ensure that your software not only meets requirements but also performs well in real-world scenarios. Adopting a comprehensive testing strategy helps in identifying issues early, reducing costs, and enhancing the overall reliability and user satisfaction of your software product.

Using the testthat package in R, you can easily create and run both unit and system tests to ensure your code functions as intended. By adopting a comprehensive testing strategy, you can improve the reliability and maintainability of your R projects.

Side effects are a fundamental concept in programming, representing any interaction with or modification of external state. While they are necessary for real-world applications, managing them effectively is crucial for writing reliable, maintainable, and testable code. Understanding and controlling side effects can significantly improve the quality and predictability of software.

By using testthat, you can effectively test SQL code and triggers within R. This approach allows you to write comprehensive unit, integration, and data integrity tests, ensuring your SQL operations perform as expected and maintain data integrity. Integrating these tests into your development workflow helps identify issues early and improves the reliability of your database operations.

---

Files & Resources

All Files for Lesson 6.194

---

References

None yet.

Errata

Let us know.