Tuesday, December 1, 2015

Automated Testing Aid: Manually Running a Quartz Job

I work in a project where testing is a first class citizen. We do unit tests, security tests, integration tests, end-to-ends api tests (SBE), and end-to-end functional (interface based) tests also by using SBE.
All right, everything's fine, until I need to throw in some asserts at the end of my integration/sbe test where I should check if the whole process performed well. Ok, only that this part of the process is accomplished by quartz jobs that run asynchronously in their specific setup, beyond our control.

Note: The assumption for this post is that your Quartz scheduler can be run within the same application with your tested classes. For distributed quartz jobs there is another story.

Some could say, ok, by you have the possibility to get a job by it's name and call triggerJob(JobKey) on a quartz scheduler instance, that should trigger the job immediately. But, be careful is about triggering a job, and not running the job. That means that:

  • the command is asynchronous and return immediately;
  • the job could actually start later, depending on the schedule's config and
  • you don't actually know when the job shall finish so that you can test your assumptions about the outcome of it.

Two quick solutions:
  1. after test's execution finished, before asserting on data, sleep the test thread for a while to give quartz time to do it's work. But sleep for how long? Some manual tries could give us some empirical idea of how long should we wait before the jos is usually executed, but we are never going to be 100% sure it actually was. And then, this approach could bring the execution of our test suite to last forever, imagine running hundreds of tests of this type that each are sleeping for few seconds... It doesn't sound very appealing.
  2. add a JobListener listener to the scheduler, then trigger the job and then put your main thread in wait until the listener is triggered on execution finished, and notifies your main thread so it can resume it's testing task. But, again, there might be many jobs already triggered and running until ours get's it's change to run. And after all, would you really want to get into unexpected threading issues? I think not.

So, after trying the aforementioned approaches and not really being happy with them I thought, why not directly run the jobs I am directly interested in?
Well this is not that trivial, because I'd like to run the jobs as they are, without having to know what other stuff is injected in each of my classes extending QuartzJob in order to make it work. So, after some research and study of how quartz works in collaboration with spring, that is what came out:


import org.quartz.Job;
import org.quartz.JobExecutionContext;
import org.quartz.Scheduler;
import org.springframework.beans.BeanWrapper;
import org.springframework.beans.BeansException;
import org.springframework.beans.MutablePropertyValues;
import org.springframework.beans.PropertyAccessorFactory;
import org.springframework.context.ApplicationContext;
import org.springframework.context.ApplicationContextAware;

import java.lang.reflect.Method;
import java.util.Map;

public class ManualJobExecutor implements ApplicationContextAware {

    private ApplicationContext applicationContext;

    public void executeJob(final Class<Job> jobClass) {

        try {
            //create job instance
            final Job quartzJob = jobClass.newInstance();
            // For the created job instance, search all services that are injected by quartz.
            // Those service instances are kept inside each scheduler context as a map
            final BeanWrapper beanWrapper = PropertyAccessorFactory.forBeanPropertyAccess(quartzJob);
            final MutablePropertyValues propertyValues = new MutablePropertyValues();
            //get all schedulers defined across all spring configurations for this application
            final Map<String, Scheduler> schedulers = applicationContext.getBeansOfType(Scheduler.class);
            for (final Scheduler scheduler : schedulers.values()) {
                // Populate the possible properties with service instances found
                propertyValues.addPropertyValues(scheduler.getContext());
            }
            //set the properties of the job (injected dependencies) with the matching services
            //the other services in the list that have no matching properties shall be ignored 
            beanWrapper.setPropertyValues(propertyValues, true);

            //get method executeInternal(JobExecutionContext) from job class extending QuartzJobBean 
            final Method executeJobMethod = quartzJob.getClass().getDeclaredMethod("executeInternal", (JobExecutionContext.class));
            executeJobMethod.setAccessible(true);
            //call the processItems method on the Job class instance
            executeJobMethod.invoke(quartzJob);
        } catch (final Exception e) {
            throw new RuntimeException(String.format("Exception while retrieving and executing job for name=%s", jobClass.getName()), e);
        }
    }

    @Override
    public void setApplicationContext(final ApplicationContext applicationContext) throws BeansException {
        this.applicationContext = applicationContext;
    }
}

That's it!
Of course there are also other aspects, i.e checking if other job of the same class is already executing so that it won't overlap with your execution. Usually in Quarz, @DisableConcurrentExecution takes care of this but here you need to check it yourself.
You could also make your method accept a job by its name instead of class so you can get the names from your database instead of looking into project classes.

I hope this is going to ease your testing.
Please share your thoughts.


Have a nice day,
Dikran

Wednesday, May 27, 2015

Spring Boot & Jasypt easy: Keep your sensitive properties encrypted


Goal


I want to store my database password encrypted in the application properties file and provide the property encryption password at runtime as java system property or environment variable.

Context:


Java 7, Spring Boot 1.2.3.RELEASE
Currently Spring Boot does not offer native property encryption support.

Solution


Use jasypt encryption library and integrate it into Spring Boot's configuration flow.

How?
Here is a quick and dirty example:

1. Download jasypt and unzip the contents in a folder;
2. Choose a password for encrypting your sensitive properties; for the purpose of this example we choose "my-encryption-password";
3. Choose the property you want encrypted; here we choose to encrypt the database password "my-database-password";
4. Encrypt the database password ("my-database-password") using jasypt and the encryption password ("my-encryption-password"); go into the jasypt bin folder and run:

$ encrypt.sh  input=my-database-password password=my-encryption-password

----ENVIRONMENT-----------------

Runtime: Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 24.60-b09

----ARGUMENTS-------------------

input: my-database-password

password: my-encryption-password

----OUTPUT----------------------

TJ1vA+DLWFrwEmbZKmGmawEonbJw4DxhkFf53JzKfvY=

The output is the encrypted password.
To configure the database in the SpringBoot's application.properties we add:

#for this example we use H2 database
spring.datasource.driver-class-name=org.h2.Driver
spring.datasource.url=jdbc:h2:mem:my-schema
spring.datasource.username=test-user

#here we provide the database encrypted password by enclosing in ENC()
#so that jasypt can detect and decrypt it
spring.datasource.password=ENC(TJ1vA+DLWFrwEmbZKmGmawEonbJw4DxhkFf53JzKfvY=)


Integrating Spring Boot and Jasypt


In order to instruct Spring Boot to transparently interpret our property file and extract and decrypt the encrypted properties we need to:

1. Create a PropertySourceLoader implementation that knows how to parse property files, identify encrypted properties and decrypt them before making them available to other components. Also the class knows to get the encryption password from system properties (provided at command line by -Dproperty.encryption.password=my-encryption-password) or as an environment variable in the operating system (export PROPERTY_ENCRYPTION_PASSWORD="my-encryption-password"). Listing follows:
package com.myexample;

import org.jasypt.encryption.pbe.StandardPBEStringEncryptor;
import org.jasypt.spring31.properties.EncryptablePropertiesPropertySource;
import org.springframework.boot.env.PropertySourceLoader;
import org.springframework.core.PriorityOrdered;
import org.springframework.core.env.PropertySource;
import org.springframework.core.io.Resource;
import org.springframework.core.io.support.PropertiesLoaderUtils;

import java.io.IOException;
import java.util.Properties;


/**
 * This class is a replacement for the default Spring PropertySourceLoader. It has the capability of detecting
 * and decrypting encrypted properties via Jasypt Encryption Library.
 * The decryption password must be provided via an environment variable or via a System property. The name of the property can be {@code PROPERTY_ENCRYPTION_PASSWORD} or {@code property.encryption.password}.
 * For more information see http://www.jasypt.org/ and http://www.jasypt.org/spring31.html
 * For Spring Boot integration the default {@link PropertySourceLoader} configuration was overriden by
 * META-INF/spring.factories file.
 *
 * @see org.springframework.boot.env.PropertySourceLoader
 */

public class EncryptedPropertySourceLoader implements PropertySourceLoader, PriorityOrdered {

    private static final String ENCRYPTION_PASSWORD_ENVIRONMENT_VAR_NAME_UNDERSCORE = "PROPERTY_ENCRYPTION_PASSWORD";
    private static final String ENCRYPTION_PASSWORD_ENVIRONMENT_VAR_NAME_DOT = "property.encryption.password";
    private static final String ENCRYPTION_PASSWORD_NOT_SET = "ENCRYPTION_PASSWORD_NOT_SET";

    private final StandardPBEStringEncryptor encryptor = new StandardPBEStringEncryptor();

    public EncryptedPropertySourceLoader() {
        this.encryptor.setPassword(getPasswordFromEnvAndSystemProperties());
    }

    private String getPasswordFromEnvAndSystemProperties() {
        String password = System.getenv(ENCRYPTION_PASSWORD_ENVIRONMENT_VAR_NAME_UNDERSCORE);
        if (password == null) {
            password = System.getenv(ENCRYPTION_PASSWORD_ENVIRONMENT_VAR_NAME_DOT);
            if (password == null) {
                password = System.getProperty(ENCRYPTION_PASSWORD_ENVIRONMENT_VAR_NAME_UNDERSCORE);
                if (password == null) {
                    password = System.getProperty(ENCRYPTION_PASSWORD_ENVIRONMENT_VAR_NAME_DOT);
                    if (password == null) {
                        password = ENCRYPTION_PASSWORD_NOT_SET;
                    }
                }
            }
        }
        return password;
    }

    @Override
    public String[] getFileExtensions() {
        return new String[]{"properties"};
    }

    @Override
    public PropertySource load(final String name, final Resource resource, final String profile) throws
            IOException {
        if (profile == null) {
            //load the properties
            final Properties props = PropertiesLoaderUtils.loadProperties(resource);

            if (!props.isEmpty()) {
                //create the encryptable properties property source
                return new EncryptablePropertiesPropertySource(name, props, this.encryptor);
            }
        }

        return null;
    }

    @Override
    public int getOrder() {
        return HIGHEST_PRECEDENCE;
    }
}

2. Create a com/myexample/META_INF/spring.factories file to override the default PropertyResurceLoader (org.springframework.boot.env.PropertiesPropertySourceLoader) which is provided with the Spring Boot distribution in META-INF/spring.factories. Our file should contain one line as follows:
org.springframework.boot.env.PropertySourceLoader=com.myexample.EncryptedPropertySourceLoader

That's it! Now your application should be able to use encrypted properties.

Thanks for reading!
Dikran

To give the right credits, info that helped me solving the problem and writing this post were gathered from this Stackoverflow post.