Testing

iniscan: a security scanner for php.ini

I do a lot of talking at conferences and in the articles I write about application security, mostly in PHP. There’s been a resurgence of interest in creating secure applications and some of the best practices around it. There’s lots of examples of bad code out there but thankfully there’s a growing base of good reference materials that show how to Do Things the Right Way too. Most of these focus on good code practices, though, and don’t think too much about something fundamental to the PHP language and key to any installation – the php.ini configuration.

PHP’s configuration has quite a few settings (and is getting more all the time) so it can be confusing to have to keep it all straight and be sure you’re “thinking secure” in your app’s setup. There’s lots of sites out there that offer recommendations about how you should set up your config file to follow along with the best security practices, but they’re either all the same thing (copy and pasted?) or refer to settings that are now deprecated. So, in an effort to make it easier for developers (and sysadmins) to set up a php.ini file with a more secure configuration, I created the “iniscan” tool (catchy name, huh?).

I want to thank Ed Finkler for the foundational work he did on the PHPSecInfo project a few years back. PHPSecInfo was a web-based tool that you could load in a browser and report back similar information as what the iniscan tool reports. A lot of the rules in the iniscan tool are inspired by the ones he defined.

The tool runs from the command line and evaluates the php.ini file you’ve given it and evaluates it based on a set of pre-defined rules to provide a pass/fail grade on your current configuration. The goal behind it was to create something that was easy to install and easy to use to reduce the barrier for adoption and lead to an increased awareness about what a hardened php.ini looks like.

You can install it using Composer and call the command line “scan” command to get the results. They look something like:

== Executing INI Scan [11.09.2013 01:51:13] ==

Results for /private/etc/php.ini:
============
Status | Severity | Key                      | Description
----------------------------------------------------------------------
PASS   | ERROR    | session.use_cookies      | Must use cookies to manage sessions
FAIL   | WARNING  | session.cookie_domain    | It is recommended that you set the default domain for cookies.
PASS   | ERROR    | session.cookie_httponly  | Setting session cookies to 'http only' makes them only readable by the browser
FAIL   | WARNING  | session.save_path        | Path /tmp is world writeable

[...more test results...]

15 passing
6 failure(s)

Each of the tests gets a pass/fail grade with a bit of color-coding to help the really bad issues stick out a bit more. It looks at settings like:

  • session.cookie_httponly
  • register_globals
  • safe_mode
  • allow_url_fopen
  • expose_php
  • register_long_arrays

…and more. The tool, by default, reports back the information in a “table” output directly back to the command line. There’s options you can pass in that will change the output format into something a bit more machine-friendly (like JSON and XML).

If this sounds interesting to you, check out the github repo for it and try out the latest version (v2.5 at the time of this post). I definitely welcome feedback so I can help make this an even better tool for the PHP community overall. Many thanks to the PHP community members that have already contributed back to the project – Eric Hogue, Jeremy Cook, Hari KT and Alexandru G!

iniscan

Advertisement

Behat + FuelPHP = RESTful Testing Happiness

If you’ve been following my recent posts, you know I’ve been working more lately with Behat for resting some REST services. In this post I showed you how to get things set up for some testing. In this post, I’ll show you how to use a custom class that I’ve put together to make a reusable system for testing REST.

For those that want to cut to the chase, I’ve posted some example code to github showing the code for the two different sides of the equation – the Behat testing and the PHP framework side (I went with FuelPHP because I was already familiar with it and it makes RESTful interfaces dead simple). The key is in the FeatureContextRest.php file that uses the Guzzle client to make the HTTP requests over to the framework. It provides some handy contexts that make it easy to create Scenarios.

So, still with me? Good – I’ll show you how to get my sample scripts installed so you can see my example in action. In the repository you’ll find two different directories – “fuel” and “behat”. In each you’ll find some files:

behat

  • features/index.features:
    An example set of Scenarios that use the REST testing contexts to create, find and deleting a “User” from the system
  • features/bootstrap/FeaturesContextRest.php:
    The key to the puzzle, the implementation of some context methods for use in Behat

fuelphp

  • fuelphp/classes:
    These are the example models and controllers you can drop into your Fuel install to have them “magically” work (with any recent version of the framework).

This is assuming you already have Behat installed and working and have set up FuelPHP on a host somwhere. For our example, we’ll just use http://behat-test.localhost:8080” Here’s what to do:

  1. Start by copying over the FeaturesContextRest.php file to the “features/bootstrap” directory of your Behat installation alongside the default FeaturesContext.php.
  2. Copy over the behat.yml configuration file into your testing root (the same level as the “features” directory) and update the base_url value for your hostname.
  3. Take the “classes” directory and copy over its contents to the location of your FuelPHP installation – if you’re familiar with the structure of the framework, this is simple…it just goes in “app/classes”.
  4. You’ll need a database to get this working, so you’ll need to get the FuelPHP ORM stuff configured and working in your install. Be sure to update your config.php to automatically load the “orm” module.
  5. Make a database for the project and use the init.sql to create the “users” table.
  6. To test and be sure our models/controllers are working right, hit your host in a browser, calling the “User” action like: “http://behat-test.localhost:8080/user”. This should call what’s in Controller_User::get_index(). It’ll probably just return an empty result though, since there’s nothing for it to pull.
  7. To use the FeaturesContextRest.php, be sure that you include that file into your default FeaturesContext.php file and that your class “extends FeaturesContextRest”.

So, if all has gone well, you have all the pieces in place to get started. Let’s start with a sample Scenario to show you how it all works:

Scenario: Creating a new User
	Given that I want to make a new "User"
	And that its "name" is "Chris"
	When I request "/user/index.json"
	Then the response is JSON
	And the response has a "userId" property
	And the type of the "userId" property is numeric
	Then the response status code should be 200

If you’ve used Mink in the past, so of this will seem familiar. This doesn’t use Mink, however, and I’ve opted to use Guzzle as the client for the backend to replace some of the browser interactions. Really, we don’t need all of the fancy interface interaction Mink gives, so this streamlined interface is more useful.

In this example, we’re creating a user with a property “name” of “Chris” and sending that off to the “/user/index.json” endpoint. A few tests are included like: checking the response code, seeing if the return value has a “userId” property, etc. If all went well and the user was created (this is a POST, so it calls “post_index” in the controller), this test should pass with flying colors….green, hopefully. 🙂

The key is in the phrases “to make a new” (POST), “to find a” (GET) and “to delete a” (DELETE) to tell the tests which verb to use.

Hopefully this library will be useful to some folks out there – I’m going to be improving it as I work on more tests for some current projects to ensure better quality of the software.

From the Start: Setting up Behat testing

In a previous post I showed you how to use HTTP Auth with the Goutte driver in Behat testing. I want to take a step back and show you how I even got to that point with a simple guide to installing the tool and creating your first tests. In a future post, I’ll show the (super simple) integration you can do with Jenkins to execute the tests and plug in the results.

First off, a definition for those not sure what Behat is:

Behat was inspired by Ruby’s Cucumber project and especially its syntax part (Gherkin). It tries to be like Cucumber with input (Feature files) and output (console formatters), but in core, it has been built from the ground on pure php with Symfony2 components.

And, for those not sure what Cucumber is:

Cucumber lets software development teams describe how software should behave in plain text. The text is written in a business-readable domain-specific language and serves as documentation, automated tests and development-aid – all rolled into one format.

These are both tools that implement something called behavior-driven development, the practice of approaching testing from a more natural angle, allowing tests to be written in “plain English” (or the language of your choosing with Behat) and executed at a domain level rather than a code level (as unit tests would). Behat makes this super simple and it really only takes a few minutes to get it up and running.

For my examples, I’m going to be working on a Linux-based system (CentOS actually) but the same kinds of ideas apply to other platforms too, just modified slightly for things like pathing. So, let’s get started…first off, the installation. Personally, I recommend the Composer method for installation – it just makes it easier to get everything you need without having to think hardly at all.

Here’s a set of commands (linux command line) to get everything set up for a basic project structure:

mkdir behat-testing-ftw; cd behat-testing-ftw;
curl -s http://getcomposer.org/installer | php
vi composer.json

Inside the composer.json file, enter the stuff from the Behat instructions here (the JSON format), then we can run composer on it:

php composer.phar install

At this point you should see a bunch of things being installed in the “Installing dependencies” section (at the time of this post, that includes things like symfony/finder, symfony/dependency-injection and behat/gherkin). Once this is finished, you should have a directory structure similar to:

Chriss-MacBook-Pro:behat-testing-ftw chris$ ls
bin		composer.json	composer.lock	composer.phar	vendor

Behat should be magically installed in the bin/ directory – to test it out, try this:

Chriss-MacBook-Pro:behat-testing-ftw chris$ bin/behat

If all goes well, you should see a big red error talking about “features” not existing. This is good – we’ll fix this now by making the testing base for Behat. Thankfully, it makes this easy too:

Chriss-MacBook-Pro:behat-testing-ftw chris$ bin/behat --init
+d features - place your *.feature files here
+d features/bootstrap - place bootstrap scripts and static files here
+f features/bootstrap/FeatureContext.php - place your feature related code here

Your directory should now look like this:

Chriss-MacBook-Pro:behat-testing-ftw chris$ ls
bin		composer.json	composer.lock	composer.phar	features	vendor

The new features directory is where your Behat tests will live. If you look in there now, there’s not much – just a bootstrap directory with a FeatureContext.php file in it. This file is where you’ll define any custom rules you might need to use outside of the pre-defined ones Behat (and later Mink) come with. If you re-run Behat again, you should see:

Chriss-MacBook-Pro:behat-testing-ftw chris$ bin/behat
No scenarios
No steps
0m0.001s

So, let’s give Behat something to do now – Behat tests are organized in “feature” files. Let’s make a simple one for illustration and call it “sample.features” and put some content in it:

cd features
vi sample.feature

And put the following inside it:

Feature: Testing Behat install
        This is a testing feature to see if everything
        is working correctly

        Scenario: My first test
                When I run "ls"
                Then I should see the file "composer.json"

By default, Behat doesn’t come with any contexts pre-defined (you can see this by executing /bin/behat -dl) so we’ll need to make some for our new feature. Thankfully, Behat makes this simple too (sensing a theme here?) – if you run behat again, you should get something similar to:

eature: Testing Behat install
  This is a testing feature to see if everything
  is working correctly

  Scenario: My first test                      # features/sample.feature:5
    When I run "ls"
    Then I should see the file "composer.json"

1 scenario (1 undefined)
2 steps (2 undefined)
0m0.051s

You can implement step definitions for undefined steps with these snippets:

    /**
     * @When /^I run "([^"]*)"$/
     */
    public function iRun($argument1)
    {
        throw new PendingException();
    }

    /**
     * @Then /^I should see the file "([^"]*)"$/
     */
    public function iShouldSeeTheFile($argument1)
    {
        throw new PendingException();
    }

Se those last few lines? That tells you exactly what you’ll need to drop into the FeaturesContext.php file to make things work. The key to the matching is in the DocBlock comments – there’s regular expression matches in there that Behat tries to use when it’s executing the tests. So, let’s update our FeaturesContext with some new code:

vi features/bootstrap/FeatureContext.php

And insert into the class the following:

[php]
/**
* @When /^I run "([^"]*)"$/
*/
public function iRun($command)
{
exec($command,$result);
$this->output = $result;
}

/**
* @Then /^I should see the file "([^"]*)"$/
*/
public function iShouldSeeTheFile($fileName)
{
if (!in_array($fileName,$this->output)) {
throw new Exception(‘File named ‘.$fileName.’ not found!’);
}
}
[/php]

This code tells Behat to execute the $command (via exec), return the results to the “output” property and then check to see if the $fileName is in the list. As long as you were in the main “behat-testing-ftw” directory when you executed this, all the tests should come up green:

Feature: Testing Behat install
  This is a testing feature to see if everything
  is working correctly

  Scenario: My first test                      # features/sample.feature:5
    When I run "ls"                            # FeatureContext::iRun()
    Then I should see the file "composer.json" # FeatureContext::iShouldSeeTheFile()

1 scenario (1 passed)
2 steps (2 passed)
0m0.016s

To see how it looks when tests fail, try changing the filename in the sample.feature to something like “composer.json-foo” and you’ll see that step turn red.

So, that’s pretty much it – that’s, in a nutshell how to get Behat testing up and running (via the Composer option). There’s a few minor things that I came across when I was trying to execute the Composer and Behat tools, but those usually stemmed from missing PHP functionality/packages not being installed on the server. Those were solved with a quick “yum install” call for the needed functionality.

I hope to continue this series and talk next about getting Mink up and working to add some really useful features to Behat. I’ve been using it for testing a REST service, so that’s one of the end goals.

Behat and HTTP Auth (and Goutte)

So I’ve been trying out Behat for some REST testing recently and came across an interesting situation – I needed to send HTTP Auth credentials as a part of my test. Of course, there’s a method for this on the Session object (setBasicAuth) but it’s not implemented as a default phrase. It took me a bit to make it to the (now logcial) point that I needed to set these before I used the “Given I am on…” phrase to go to the page.

So, to accomplish this, I set up a custom “Given” phrase to set them: “Given that I log in with ‘username’ and ‘password'” so that my full Scenario looks like:
[php]
Scenario: Get the main Index view
Given that I log in with "myuser" and "mypass"
And I am on "/index/view"
Then the response should contain "TextFromPageTitle"
[/php]

And the code is super simple:

[php]
/**
* Features context.
*/
class FeatureContext extends BehatMinkBehatContextMinkContext
{
/**
* @Given /^that I log in with "([^"]*)" and "([^"]*)"$/
*/
public function thatILogInWithAnd($username, $password)
{
$this->getSession()->setBasicAuth($username,$password);
}
}
[/php]

Hope this helps someone else out there trying to send HTTP Auth pre-connection. I’m sure there’s probably an easier way that I’m missing, but this seemed the simplest to me.

How long is too long (for unit test method names)?

As I ws working on some new development in our codebase this morning, I started updating some
unit tests to match. I was looking through the code for one of them and came across a test name that was easily 50 characters long and
it got me wondering – what was the popular opinion on naming conventions and commenting in unit tests. Obviously, there’s some things
(like annotations) that have to be in comments, but do you sacrifice clarity of a comment for a method name that you can basically read
like a novel?

I mentioned something on Twitter about it and here’s some of the responses I got:

——————————–

Rafael Dohms (rdohms)
@enygma @elazar @bdeshong taking into consideration PHPUnit and its use of method names (like testdox) i would focus more on name not doc
2 hours ago

weierophinney (weierophinney)
@enygma I don’t document my tests; the test name and test content *are* documentation. /cc @elazar @bdeshong
2 hours ago

weierophinney (weierophinney)
@enygma I do those all the time. For use with #PHPUnit’s testdox output. 🙂
2 hours ago

Ralph Schindler (ralphschindler)
@bdeshong @elazar @enygma if you are using PHPUnit, run the tests with –testdox, you’ll see the benefit of long test names 🙂
2 hours ago

Chris Cornutt (enygma)
@mwillbanks ah, good point…I can see how that’d be handy heh

Mike Willbanks (mwillbanks)
@enygma putting it in a docblock is nice, however, when running it on a CI server it is nice to see what broke by method name and test name
2 hours ago

Mike Willbanks (mwillbanks)
@enygma I think we have one that is almost 100 characters 🙂 here is 1 example: testEditActionRedirectWhenJournalEntryNotFound
2 hours ago

Brian DeShong (bdeshong)
@enygma @elazar 120 characters? 🙂
2 hours ago

Michael Maclean (mgdm)
@enygma around the 100 character mark? 🙂

2 hours ago

Chris Cornutt (enygma)
@elazar @bdeshong ah, but where should method names end and docblocks begin? 😉

Brian DeShong (bdeshong)
@enygma If the method name is long, but descriptive, then that’s good. Descriptive test method names are better than short ones.
2 hours ago

Matthew Turland (elazar)
@enygma Better to be verbose and accurately descriptive than overly concise and obscure? 😛
2 hours ago

jcarouth (jcarouth)
@enygma Hey, I’ll take that over testApp1() and testApp2() any day.
3 hours ago

Chris Cornutt (enygma)
well hello entirelyTooLongUnitTestMethodName, nice to meet you
3 hours ago

——————————–

That last one was my not-so-innocent comment that started the whole thing off. I’m still interested, though, in what developers
out there think is a good point to switch from documenting via the test functions name and moving it up to a docblock comment. Seems
like there’s arguments for both sides and I’m not sure if there’s a “winner”, but it’s always fun to get opinions.

What are your preferences in unit test naming versus commenting?

A Partial Review of “Test-Driven JavaScript Development”

So ever since the fine folks at Addison Wesley sent over a copy of Test-Driven JavaScript Development (by Christian Johansen) I’ve been trying to wrap my head around a new sort of testing when it comes to web apps. I’m not even half-way through the book and I’ve already had my mind blown by advanced javascript and testing methods that I just never thought about when I work on my frontends.

I’m a big fan of unit testing and promote it whenever I can – not only does it help create a code base that a better quality, but it also lets you keep things consistent. No more of the “cross your fingers” kind of coding. A good, well-written set of unit tests can save you a whole world of hassle in the long run. Taking it one step further, you get to TDD (test-driven development) where you write your tests before you even write your code. You start with the usual “W”s – why, where, when, how (yes, I know that last one’s an “H”s) and develop the test for those rather than looking at a current chunk of code and testing what it already does. There’s some people that swear by it, one of them being the author of this book.

He starts with an introduction to TDD and a general how-it-works with javascript quickly followed by an introduction to his tool of choice – JsTestDriver. All of the test examples in the following chapters are run with this tool. Thankfully it’s simple to set up and even easier to write tests for. He shows some introductory tests to help you get familiar with things before diving into the next section – testing javascript itself.

I have to admit, this section left me at a loss for words. I thought I knew a good bit about how javascript works, both out in front and behind the scenes. I was surprised, however, just how much of the functionality he tested I had no idea worked that way. It’s pretty in-depth, so if you’re a beginning javascript dev, you might go read some more tutorials before diving right in.

I’m just now getting to the “Unobtrusive Javascript” part of the book (Part 2, Chapter 9) so I still have a ways to go, but each page is enlightening, especially for some one like myself that’s has a primary focus on backend work and unit testing. I’d definitely recommend this book to anyone looking to get into testing their frontends – it’s a great resource and can probably teach you a thing or two.

I’ll try to put up another post when I get closer to the end – it’s hard to find time to sit and read these days, but just little nibbles at a time work just fine.

API Testing with Frisk (and Joind.in!)

Two great tastes taste great together, right? As I’ve been working more on Frisk in my spare time (yes, there’s a little in there between PHPDeveloper.org and Joind.in) I’ve been trying to develop features for it that would be helpful not only for testing web site interaction, but also in working with other kinds of web-based interfaces. Naturally, my thoughts wandered over to the Joind.in API and how I could use Frisk to work with and test the responses it gives.

I’ve added a few more handy things lately – like matching values inside of JSON objects and different output options – but one of the latest things is an improvement to the POST and GET handlers to allow for more verbose options on requests. This comes in handy when we want to test something a bit more technical than just your average web site.

First, if you just want to check out the test, here’s the code in a handy .phps file. It’s the sample test I’ll be describing.

[php]
public function testSiteStatus()
{
// This is our object to send
$postData = json_encode(array(
‘request’=>array(‘action’=>array(
‘type’=>’status’,’data’=>array(
‘test_string’=>’my test’))
)));

// This is what the object that comes back should look like
$response = json_encode(array(
‘dt’=>date(‘r’),’test_string’=>’my test’
));

// Build the settings for the connection
$settings=array(
‘location’ => ‘/api/site’,
‘host’ => ‘ji-enygma.localhost’,
‘postData’ => $postData,
‘outputFormat’ => ‘json’,
‘headers’ => array(
‘Content-Type’=>’text/json’
)
);

// Make the request and check the response!
$this->post($settings)
->assertEquals($response);
}
[/php]

The large part of the example test is made up of the “setup” to even make the test happen. In $postData I’ve created a JSON message based on what the Joind.in API is expecting for the Site/Status request and pushed it into a string. The second JSON object, $response, mimics what the API’s response will look like. The $settings array is the more verbose way of specifying settings for the ActionPost request (including the additional headers). The Joind.in API requires that you give it a Content-Type of the message you’re giving it – it’s not a very good guesser.

The next two lines do all of the work – make the post request to the remote API, return the message and check to see if it’s the same as $response. Simple, right?

If you wanted to get a little more fancy with checking, you can use the “paths” in the assertContains to look inside of a JSON object. So, you could do something like:

[php]
<?php

$this->post($settings)
->assertContains(‘look for me’,’message_list/message/message1′);

?>
[/php]

That will look at “message1” in this JSON to see if it contains that “look for me” string:

{“message_list”:{“message”:{“message1″:”I think you should look for me”,”message2″:”but not here”}}}

You can find out more about the Joind.in API and all it has to offer on the Joind.in site and for more information on Frisk, check out it’s project on github.

  • Joind.in API – Interface directly with Community events & feedback
  • Frisk – automated functional unit testing

Test with Frisk!

For a while now, I’ve had a renewed interesting testing applications – unit, functional, performance, etc – and have been learning all I can about the software you can use to run these tests. Obviously, one of the mainstays of the PHP world is PHPUnit for unit testing. There’s a few others out there that are good for other kinds of testing – too many to name (and lots that aren’t written in PHP).

Lately, though, I’ve been looking more at testing the frontend of applications and their flow. I started looking around at the current testing tools and found some I wanted to use, but I wanted more to know how they worked. So, as an exercise, I started on Frisk, a light-weight simple testing framework that has some of the basics of functionally testing, nothing fancy though. Oh, and of course it’s open sourced so you can take a look at the work behind it too.

It’s not a complete testing tool, but it does work. I’ve thrown together a simple example of how I’ve used it to test two things on the Joind.in site – the homepage content and the result of a login:

JoindinTest.phps

When this file is dropped in the “tests” directory and the “frisk” executable is run, it’ll execute these tests and check the results of those requests. The “ji-enygma.localhost” is a reference to my local Joind.in instance. The results are spit back out as a PHP array that reports the results of each part of the test (pass/fail) split out by test function name and action/assert result:

[php]
Start!

=====================
ending status: Array
(
[JoindinTest] => Array (
[testHomepageContent] => Array (
[ActionGet] => Array (
[0] => pass
[1] =>
)
[AssertContains] => Array (
[0] => pass
[1] =>
)
)
[testValidLogin] => Array (
[ActionGet] => Array (
[0] => pass
[1] =>
)
[ActionSubmitform] => Array (
[0] => pass
[1] =>
)
[AssertContains] => Array (
[0] => fail
[1] => AssertContains: Term not found
)
)
)
)
[/php]

The project is hosted on Github so you can grab the source, fork it and play with the assertions and actions (or helpers!) and check out the project wiki for complete details on what the project has to offer.

Working with WebTests (Help?)

Well, I wish I could say that this was going to be a guide to getting WebTests (an automated front-end testing tool that sits on top of Ant) but it seems as though I’m having a bit of a Java issue to content with myself.

Here’s what I’ve done so far:

  • First, I noted happily that Ant was already installed on my MacBook’s OS X install (seems a strange choice to have bundled, but who am I to judge), making for one less thing that I’d have to get up and running
  • Next I grabbed the latest install of WebTest from the Canoo website and unpacked it to a testing directory. It comes as a nice, neat binary so there’s no messy compiling to worry about.
  • I wrote up a simple test and PHP page to try it all out:
    [php]

    [/php]

  • Finally, I gave it all a whirl: “bin/webtest -buildfile /my/path/mybuild.xml

This is where I hit the first snag…apparently since OS X’s Java install is just a bit different than some of the normal ones, a few of the files aren’t where they need to be. In my case, I was getting an error:

BUILD FAILED
/www/tests/mytest.xml:5: 
java.lang.NoSuchMethodError: 
org.apache.xpath.compiler.FunctionTable.installFunction
(Ljava/lang/String;Ljava/lang/Class;)I

After poking around a bit on the web, I happily came across this that recommends copying a jar file over to the Java extensions directory. A simple “cp” later and the build made it past that point. Unfortunately, not much past it, though – another Java error popped up that seems a bit more difficult to find information on:

BUILD FAILED
/dev_tmp/webtest/webtest.xml:265: 
java.lang.NoClassDefFoundError: 
org/apache/xml/serializer/ExtendedContentHandler

So far I haven’t been able to find much that’d help me solve this one, so if there’s anyone out there that’s seen it before, help would be appreciated. It seems like it actually runs the test just fine (according to the output reports) but it’s not very useful if the build always fails.