Typically you want to use end-to-end (e2e) tests to prove that everything works as intended in a realistic environment. In the Juice Shop application that idea is changed to the contrary. Here the main purpose of the e2e test suite is to prove that the application is as broken as intended!
"WTF?" you might ask, and rightfully so. Juice Shop is a special kind of application. It is an intentionally insecure Javascript web application designed to be used during security trainings, classes, workshops or awareness demos. It contains over 25 vulnerabilities that an aspiring hacker can exploit in order to fulfil challenges that are tracked on a score-board.
The job of the e2e test suite is twofold:
When does Juice Shop pass its e2e test suite? When it is working fine for the average nice user and all challenges are solvable, so an attacker can get a 100% on the score-board!
Juice Shop is created entirely in Javascript, with a Single-Page-Application frontend (using AngularJS with Bootstrap) and a RESTful backend using Express on top of NodeJS. The underlying database is a simple file-based SQLite with Sequelize as a OR-mapper and sequelize-restful to generate the simple (but not necessarily secure) parts of the API dynamically.
There three different types of of tests to make sure Juice Shop is not released in an unintendedly broken state:
If all stages pass and the application survives a quick monkey-test by yours truly it will be released on GitHub and SourceForge.
There are two reasons to run Juice Shop tests on Sauce Labs:
Having laid out the context the rest of the article will explain how both these goals could be achieved by integrating with Sauce Labs.
Juice Shop builds on Travis-CI which Sauce Labs integrates nicely with out of the box. The following snippet from the .travis.yml shows the necessary configuration and the two commands being called to excecute unit and e2e tests.
addons:
sauce_connect: true
after_success:
- karma start karma.conf-ci.js
- node test/e2eTests.js
env:
global:
- secure: <your encrypted SAUCE_USERNAME>
- secure: <your encrypted SAUCE_ACCESS_KEY>
The karma.conf-ci.js contains the configuration for the frontend unit tests. Juice Shop uses six different OS/Browser configurations:
var customLaunchers = {
sl_chrome: {
base: 'SauceLabs',
browserName: 'chrome',
platform : 'Linux',
version: '37'
},
sl_firefox: {
base: 'SauceLabs',
browserName: 'firefox',
platform: 'Linux',
version: '33'
},
sl_ie_11: {
base: 'SauceLabs',
browserName: 'internet explorer',
platform: 'Windows 8.1',
version: '11'
},
sl_ie_10: {
base: 'SauceLabs',
browserName: 'internet explorer',
platform: 'Windows 8',
version: '10'
},
sl_ie_9: {
base: 'SauceLabs',
browserName: 'internet explorer',
platform: 'Windows 7',
version: '9'
},
sl_safari: {
base: 'SauceLabs',
browserName: 'safari',
platform: 'OS X 10.9',
version: '7'
}
};
In order associate the test executions with the Travis-CI build that triggered them, some extra configuration is necessary:
sauceLabs: {
testName: 'Juice-Shop Unit Tests (Karma)',
username: process.env.SAUCE_USERNAME,
accessKey: process.env.SAUCE_ACCESS_KEY,
connectOptions: {
tunnelIdentifier: process.env.TRAVIS_JOB_NUMBER,
port: 4446
},
build: process.env.TRAVIS_BUILD_NUMBER,
tags: [process.env.TRAVIS_BRANCH, process.env.TRAVIS_BUILD_NUMBER, 'unit'],
recordScreenshots: false
}
reporters: ['dots', 'saucelabs']
Thanks to the existing karma-sauce-launcher
module the tests are executed and their result is reported back to Sauce Labs out of the box. Nice. The e2e suite was a tougher nut to crack.
For the Protractor e2e tests there are no separate configuration files for local and CI, just one protractor.conf.js with some extra settings then running on Travis-CI to pass necessary data to Sauce Labs:
if (process.env.TRAVIS_BUILD_NUMBER) {
exports.config.seleniumAddress = 'http://localhost:4445/wd/hub';
exports.config.capabilities = {
'name': 'Juice-Shop e2e Tests (Protractor)',
'browserName': 'chrome',
'platform': 'Windows 7',
'screen-resolution': '1920x1200',
'username': process.env.SAUCE_USERNAME,
'accessKey': process.env.SAUCE_ACCESS_KEY,
'tunnel-identifier': process.env.TRAVIS_JOB_NUMBER,
'build': process.env.TRAVIS_BUILD_NUMBER,
'tags': [process.env.TRAVIS_BRANCH, process.env.TRAVIS_BUILD_NUMBER, 'e2e']
};
}
The e2e tests are launched via e2eTests.js which spawns a separate process for Protractor after launching the Juice Shop server:
var spawn = require('win-spawn'),
SauceLabs = require('saucelabs'),
colors = require('colors/safe'),
server = require('./../server.js');
server.start({ port: 3000 }, function () {
var protractor = spawn('protractor', [ 'protractor.conf.js' ]);
function logToConsole(data) {
console.log(String(data));
}
protractor.stdout.on('data', logToConsole);
protractor.stderr.on('data', logToConsole);
protractor.on('exit', function (exitCode) {
console.log('Protractor exited with code ' + exitCode + ' (' + (exitCode === 0 ? colors.green('SUCCESS') : colors.red('FAILED')) + ')');
if (process.env.TRAVIS_BUILD_NUMBER && process.env.SAUCE_USERNAME && process.env.SAUCE_ACCESS_KEY) {
setSaucelabJobResult(exitCode);
} else {
server.close(exitCode);
}
});
});
The interesting part regarding Sauce Labs is the call to setSaucelabJobResult(exitCode)
in case the test is run on Travis-CI with Sauce Labs credentials which are passed in by the extra config part in protractor.conf.js.
This function passes the test result from Protractor on to Sauce Lab's REST API:
function setSaucelabJobResult(exitCode) {
var sauceLabs = new SauceLabs({ username: process.env.SAUCE_USERNAME, password: process.env.SAUCE_ACCESS_KEY });
sauceLabs.getJobs(function (err, jobs) {
for (var j in jobs) {
if (jobs.hasOwnProperty(j)) {
sauceLabs.showJob(jobs[j].id, function (err, job) {
var tags = job.tags;
if (tags.indexOf(process.env.TRAVIS_BUILD_NUMBER) > -1 && tags.indexOf('e2e') > -1) {
sauceLabs.updateJob(job.id, { passed : exitCode === 0 }, function(err, res) {
console.log('Marked job ' + job.id + ' for build #' + process.env.TRAVIS_BUILD_NUMBER + ' as ' + (exitCode === 0 ? colors.green('PASSED') : colors.red('FAILED')) + '.');
server.close(exitCode);
});
}
});
}
}
});
}
This was necessary because there was no launcher available at the time that would do this out-of-the-box.
How does Protractor get its test result in the first place? It must be able to determine if all challenges were solved on the score board and cannot access the database directly to do that. But: It can access the score board in the application:
As solved challenges are highlighted green instead of red some simple generic function was used to assert this:
protractor.expect = {
challengeSolved: function (context) {
describe("(shared)", function () {
beforeEach(function () {
browser.get('/#/score-board');
});
it("challenge '" + context.challenge + "' should be solved on score board", function () {
expect(element(by.id(context.challenge + '.solved')).getAttribute('class')).not.toMatch('ng-hide');
expect(element(by.id(context.challenge + '.notSolved')).getAttribute('class')).toMatch('ng-hide');
});
});
}
}
When watching the e2e suite run Protractor will constantly visit the score board to check each challenge. This is quite interesting to watch as the progress bar on top moves closer to 100% with every test. But be warned: If you plan on trying to hack away on Juice Shop to solve all the challenges yourself, you will find the following screencast to be quite a spoiler! ;-)
<iframe title="juice-shop hacking challenges solved by Protractor e2e tests (v1.5.4)" width="560" height="315" src="https://www.youtube.com/embed/CXsWFkzS8Ag" frameborder="0" allowfullscreen></iframe>