SSH/SFTP unit testing using JUnit

Working on the sfnet-mvnrepo-plugin in Infix there was a need to unit test piece of code which connected to a SFTP site and parsed the directory tree. The easy option of course was to use a specific server with SSH/SFTP enabled, but didn’t want the tests to depend on external environment or resources. So the obvious solution was to use an embedded SSH server, or at least something that can be started and managed from Java code.

Looking for a solution on google (with a bias towards stuff written in Java and embeddable) two obvious candidates emerged:

• Apache Mina SSHD: This nice little under-development tool originally by Guillaume Nodet, now part of the Apache Mina project.
• J2SSH: This is the grand old Java SSH library. The original library is no longer developed, but is available as a commercial product J2SSH Maverick, there is also a fork of the original code base with some bug fixes at J2SSH Fork on Google code hosting.

SSHD being an in development project and being designed as a core server was the preferred library to use, but after an hour of fooling around with the tool, it was obvious that SFTP support is not available or mature in the tool yet, and looking for a quick library solution, I did not want to go around hacking SSHD and trying to implement SFTP support in there. I should mention here that the verification mechanism used here was to start the server in Java and then try to connect to it from WinSCP running on the same machine.

The next step was to look at J2SSH, I went with the J2SSH fork, as it did have some bugs fixed. Working from the Quick Start guide, I did have a server up and connected from WinSCP in no time, but it would not let me browse the directories through WinSCP. In any case I was further along with J2SSH than I was with SSHD, so decided to pursue J2SSH as a potential solution. Counting out the issues that needed to be handled:

1. Getting directory browsing to work
2. Configuring the server
3. Controlling the server from JUnit
4. Using temporary directories and on demand directory structures in the SFTP ‘view’ rather than depending on strict directory structures
5. Securing the server so as not to open up the machine for attacks through SSH
6. Making the library Maven accessible

The biggest challenge with the library (other than being a bit buggy) is that it has poor to no documentation! Anyways, thanks to Eclipse and Java debugging, it was actually not that hard to get it to work. So here goes…

Mavenizing J2SSH Fork

This section only applies if the project is being set up as a Maven project. If other dependency management mechanisms are being used, feel free to get the latest version of J2SSH or J2SSH Fork and skip this section.

Two versions of J2SSH are currently available on public Maven repositories: 0.2.2 on the Central Repo (sshtools:j2ssh-daemon:jar:0.2.2) and 0.2.7 on JBoss third party releases repository (sshtools:j2ssh-daemon:jar:0.2.7). The latest release of J2SSH from the original Sourceforge project is 0.2.9 from September 2007 and the latest from J2SSH fork is 0.3.1 (the jars are in the SVN repository, not in the downloads page).

The POMs of the two versions in the public subversion repositories are incomplete – they do not have the dependencies set up properly. J2SSH daemon depends on J2SSH Core and J2SSH Core. One way is to manually add all three as (test) dependencies in your Maven project and not have to worry about the latest version. Though not tested 0.2.2 or 0.2.7 should work fine for this test.

I wanted to use 0.3.1, but it was not in Maven, and I went ahead and mavenized the project. A core j2ssh-fork pom project with 4 modules generating the 4 jars similar to the 0.3.1 distribution but with correct pom dependencies and source/javadoc jars. I have not decided if or where I will publish it. Might have to put the distribution repo up somewhere for Infix use though.

J2SSH Daemon Configuration

From the available documentation and code inspection, the easiest way to configure J2SSH was using XML configuration files. The configuration files are not documented, but their structure can be inferred from the J2SSH daemon configuration beans. The sample configuration is available here.

I wanted to be able to configure the server daemon dynamically and not have them read configuration from the launch directory (as the example code did). For my requirements, I only needed to configure the server and the platform. And since I needed to generate XML files in the end, I decided to use JAXB beans to represent the server and platform configuration. The code is in the infix SVN repository, see ServerConfiguration and PlatformConfiguration. The other configurations are used as is from the J2SSH fork project.

SSH Server Method Rule

I decided to have the SSH server and control as a JUnit Rule, so it can be easily managed from JUnit tests. For a complete listing of the code see SSHServerResource. Parts of the code are discussed in this post.

The SSHServerResource class extends JUnit ExternalResource, and uses two TemporaryFolders – one for the configuration files for the SSHDaemon and the other as the root directory for SFTP. The resource takes in 3 parameters:

• userId: The temporary user id which is allowed access to the server
• port: The port on which SSH will listen
• bindAddress: The address on which the server should listen. Can be set to 127.0.0.1 for limiting to the current host.

Configuring and starting the server

The server is configured and started from the before method of the ExternalResource:

/* (non-Javadoc)
 * @see org.junit.rules.ExternalResource#before()
 */
@Override
protected void before () throws Throwable {
    // Setup the temporary folder and copy configuration files
    baseDir.create();
    configDir.create();
    setupConfiguration();

    // Run it in a separate thread
    Executors.newSingleThreadExecutor().submit(new Callable() {

        @Override
        public Object call () throws Exception {
            start();
            return null;
        }
    });
}

The configuration involves 5 steps, to set up the server and platform configuration xmls in the configuration directory, copy other configurations to the configuration directory, set up the home directory structure for the user and tell SSH daemon about the configurations.

The following methods setup the server and platform configuration xmls and tell SSHD about the configurations:

private void createPlatformConfig () throws IOException {
    PlatformConfiguration platformConfig = new PlatformConfiguration();
    platformConfig.setNativeProcessProvider("com.sshtools.daemon.platform.UnsupportedShellProcessProvider");
//        platformConfig.setNativeAuthenticationProvider("com.sshtools.daemon.platform.DummyAuthenticationProvider");
    platformConfig.setNativeAuthenticationProvider("com.mindtree.techworks.infix.pluginscommon.test.ssh.JunitDummyAuthenticationProvider");
    platformConfig.setNativeFileSystemProvider("com.sshtools.daemon.vfs.VirtualFileSystem");
    PlatformConfiguration.VFSRoot root = new PlatformConfiguration.VFSRoot();
    root.setPath(baseDir.getRoot().getAbsolutePath().replace('\\', '/'));
    platformConfig.setVfsRoot(root);

    marshall(platformConfig, "platform.xml");
}

private void createServerConfig () throws IOException {
    ServerConfiguration serverConfig = new ServerConfiguration();

    ServerConfiguration.ServerHostKey hostKey = new ServerConfiguration.ServerHostKey();
    hostKey.setPrivateKeyFile(configDir.getRoot().getAbsolutePath().replace('\\', '/') + "/test-dsa.key");
    serverConfig.addServerHostKey(hostKey);

    serverConfig.setPort(port);
    serverConfig.setListenAddress(bindAddress);
    serverConfig.setMaxConnections(3);
    serverConfig.addAllowedAuthentication("password");
    serverConfig.addAllowedAuthentication("keyboard-interactive");

    ServerConfiguration.Subsystem subsystem = new ServerConfiguration.Subsystem();
    subsystem.setName("sftp");
    subsystem.setType("class");
    subsystem.setProvider("com.sshtools.daemon.sftp.SftpSubsystemServer");
    serverConfig.addSubsystem(subsystem);

    marshall(serverConfig, "server.xml");
}

private void configureServer () throws ConfigurationException {

	String configBase = configDir.getRoot().getAbsolutePath().replace('\\', '/') + '/';

	// We store serverPlatformConfiguration as a static variable so we can
	// reinitialize it. This is required if the server is started and stopped
	// from multiple test executions - the design of J2SSH daemon does not
	// allow re-configuration (configs stored in static variables).
	if (null == serverPlatformConfiguration) {
		serverPlatformConfiguration = new XmlServerConfigurationContext();
		serverPlatformConfiguration.setServerConfigurationResource(ConfigurationLoader
			.checkAndGetProperty("sshtools.server", configBase + "server.xml"));
		serverPlatformConfiguration.setPlatformConfigurationResource(System.getProperty(
			"sshtools.platform", configBase + "platform.xml"));
		ConfigurationLoader.initialize(false, serverPlatformConfiguration);
	} else {
		serverPlatformConfiguration.setServerConfigurationResource(ConfigurationLoader
			.checkAndGetProperty("sshtools.server", configBase + "server.xml"));
		serverPlatformConfiguration.setPlatformConfigurationResource(System.getProperty(
			"sshtools.platform", configBase + "platform.xml"));
		serverPlatformConfiguration.initialize();
	}

	XmlConfigurationContext context2 = new XmlConfigurationContext();
	context2.setFailOnError(false);
	context2.setAPIConfigurationResource(ConfigurationLoader.checkAndGetProperty(
		"sshtools.config", configBase + "sshtools.xml"));
	context2.setAutomationConfigurationResource(ConfigurationLoader.checkAndGetProperty(
		"sshtools.automate", configBase + "automation.xml"));
	ConfigurationLoader.initialize(false, context2);
}

Finally start the server:

private void start () throws IOException {

    SshServer server = new SshServer() {

        public void configureServices (ConnectionProtocol connection) throws IOException {
            connection.addChannelFactory(SessionChannelFactory.SESSION_CHANNEL,
                new SessionChannelFactory());

            if (ConfigurationLoader.isConfigurationAvailable(ServerConfiguration.class)) {
                if (((com.sshtools.daemon.configuration.ServerConfiguration) ConfigurationLoader
                    .getConfiguration(com.sshtools.daemon.configuration.ServerConfiguration.class)).getAllowTcpForwarding()) {
                    new ForwardingServer(connection);
                }
            }
        }

        public void shutdown (String msg) {
            // Disconnect all sessions
        }

        @Override
        protected boolean isAcceptConnectionFrom (Socket socket) {
            return true;
        }
    };

    server.startServer();
}

Shutting down the server

The server is shutdown using the after method of the ExternalResource:

/* (non-Javadoc)
 * @see org.junit.rules.ExternalResource#after()
 */
@Override
protected void after () {

    try {
        stop();
    } catch (Throwable e) { /* Ignore */ }

    // Delete stuff
    try {
        configDir.delete();
    } catch (Throwable e) { /* Ignore */ }

    try {
        baseDir.delete();
    } catch (Throwable e) { /* Ignore */ }
}

private void stop () throws ConfigurationException, UnknownHostException, IOException {
    Socket socket = new Socket(InetAddress.getLocalHost(),
        ((com.sshtools.daemon.configuration.ServerConfiguration) ConfigurationLoader
            .getConfiguration(com.sshtools.daemon.configuration.ServerConfiguration.class))
            .getCommandPort());

    // Write the command id
    socket.getOutputStream().write(0x3a);

    // Write the length of the message (max 255)
    String msg = "bye";
    int len = (msg.length() <= 255) ? msg.length() : 255;     socket.getOutputStream().write(len);     // Write the message     if (len > 0) {
        socket.getOutputStream().write(msg.substring(0, len).getBytes());
    }

    socket.close();
}

Overriding the Authentication Provider

The authentication and home directory structure of the SSH Daemon Server is controlled using an implementation of the NativeAuthenticationProvider. Out of the box J2SSH comes with a dummy authentication provider which authenticates any user with the same password as the user. This worked fine for JUnit tests, but the problem is that the authentication provider is also responsible for getting the user’s home directory, and the dummy authentication provider presumes that a real directory exists at the path /home/${userid}. This of course does not work for either test users, or on Windows machines. So we create an extension of the DummyAuthenticationProvider overriding the getHomeDirectory(String) method:

public class JunitDummyAuthenticationProvider extends DummyAuthenticationProvider {

	@Override
	public String getHomeDirectory (String username) throws IOException {
		VFSMount vfsroot = ((PlatformConfiguration) ConfigurationLoader.getConfiguration(PlatformConfiguration.class)).getVFSRoot();
		String base = vfsroot.getPath();
		File homeDir = new File (base + "/home/" + username);
		return homeDir.getAbsolutePath().replace('\\', '/');
	}
}

Using SSHServerResource in Tests

To use the SSHServer resource in a Junit test, simply add the resource as a public field in the test class annotated with @Rule:

@Rule
public SSHServerResource sshServer = new SSHServerResource("test", 22, "127.0.0.1");

The directory references from the server can be extracted from the server using the getter methods, other than that the server will be listening on the requested IP and port and tests can be executed.

Gotchas

A few things to note:

1. The server by default has a connection limit of 3 (set in the server configuration above). You should not need more connections. If you do, change the server resource.
2. Make sure to close and clean up all connections and sessions to the server (even for failed tests), or it will not shut down and temporary directories will be left in the system, it might also affect other tests following.

Hope this post is helpful!