Click or type ~ to show Console
Github rate limit:
Type help or hit TAB for a list of commands. Press Ctrl-C to hide the console.

Github Console

This tutorial expands on the Quake Console tutorial by using the GitHub REST API instead of a faked filesystem. The purpose of the tutorial is to show how to wire up Josh.PathHandler and custom commands to a remote API.

Try out the Console

Type ~ to activate the shell we will be discussed belowl.

You can explore the current repository's file system via the standard ls and cd commands as well as take advantage of path TAB completion. Additional commands are:

You will be limited to 60 requests/hour by the API (where each console command may use multiple requests). If you authenticate via GitHub, you will have a more flexible 5000 requests/hour to play with.

Annotated Source

The approach of this tutorial is to walk through the pieces required to wire up Josh.Shell to a remote REST API via asynchronous calls to produce an interactive command line interface by explaining the flow. While some code will be shown inline, the primary code reference is the annotated source code with links to specific, mentioned functions throughout the tutorial.

Application State

The console is designed to always be in the context of a repository, so that there is no state in which commands like ls are not available. It initializes with the sdether/josh.js repository and after this the user can switch users, repositories and branches, while always staying in the context of some repository. That means that at any point in time, we will have a current user object, a list of all that user's repositories, and a current directory on the current branch as state. Changing users picks a default repository and branch to keep that state populated. Branches and current directory information are loaded on demand.

The available state looks like this:

{
  api: "http://josh.claassen.net/github/", // the proxy we use for the GitHub API
  shell: $instance_of_Josh.Shell,
  pathhandler: $instance_of_Josh.PathHandler,
  user: $current_user_object,
  repo: $current_repository,
  repos: $list_of_user_repos,
  branch: $current_branch,
  branches: $lazy_initialized_list_of_branches_for_current_repo
}

The GitHub API

GitHub provides a REST API, giving access to most of its data and functionality. We're just going to worry about read capability around repositories. Each repo is basically a file system which plays nicely into Josh.PathHandler's area of applicability.

The user object comes from:

GET /users/:user

We never fetch an individual repository, instead opting to fetch all at user initialization via:

GET /users/:user/repos

We do the same for branches, fetching all, lazily, once we need to auto-complete or list them:

GET /repos/:owner/:repo/branches

Finally, we fetch the current directory via

GET /repos/:owner/:repo/contents/:path

This is done for the root during repo initialization, and for path completion, cd, ls, etc. on demand.

The API returns json, which is perfect for us as well, but it is rather drastically rate limited without authentication. I.e. without authentication you will be limited to 60 requests per hour, while with authentication the limit is 5000 per hour. For this reason, we proxy all github calls through a simple node.js application that can handle oauth to optionally authenticate the use of this console. This application is outside the scope of the tutorial, but the code can be found on the josh.js github-authentication-backend branch.

Initializing the Console

In order for us to show the console, we have to have initialized a user, a repository and retrieved it's root directory. This is done after document.ready

$(document).ready(function() {
      setUser("sdether", "josh.js",
        function(msg) {
          initializationError("default", msg);
        },
        initializeUI
      );
    });

We call setUser(user_name, repo_name, err, callback) for sdether and josh.js, before setting the authenticated user as the current user and initializing the UI of the shell.

function setUser(user_name, repo_name, err, callback) {
  if(_self.user && _self.user.login === user_name) {
    return callback(_self.user);
  }
  return get("users/" + user_name, null, function(user) {
    if(!user) {
      return err("no such user");
    }
    return initializeRepos(user, repo_name, err, function(repo) {
      _self.user = user;
      return callback(_self.user);
    });
  });
}

This function follows the pattern of providing both an error and success callback, since once the shell is initialized we need to make sure that any action we take on its behalf does result in its callback being called with some value, lest the shell stop functioning. Unlike previous tutorials, we're now doing network requests and those will fail sooner or later. For this reason we need to make sure we always have a quick err callback to stop the current operation and call the callback provided by Josh.Shell on command execution. We also need to make sure that we do not mutate the application state until we are done with all operations that can fail, so that the worst case is us reporting to the shell that the operation failed while our current, known good state is preserved.

All API access goes through a helper function get(resource, args, callback), which is responsible for constructing the json call and inspecting the response. For simplicity, all error conditions just result in callback being called with null instead of a json payload.

function get(resource, args, callback) {
  var url = _self.api + resource;
  if(args) {
    url += "?" + _.map(args,function(v, k) { return k + "=" + v; }).join("&");
  }
  var request = {
    url: url,
    dataType: 'json',
    xhrFields: {
      withCredentials: true
    }
  };
  $.ajax(request).done(function(response, status, xhr) {

    // Every response from the API includes rate limiting headers, as well as an
    // indicator injected by the API proxy whether the request was done with
    // authentication. Both are used to display request rate information and a
    // link to authenticate, if required.
    var ratelimit = {
      remaining: parseInt(xhr.getResponseHeader("X-RateLimit-Remaining")),
      limit: parseInt(xhr.getResponseHeader("X-RateLimit-Limit")),
      authenticated: xhr.getResponseHeader('Authenticated') === 'true'
    };
    $('#ratelimit').html(_self.shell.templates.rateLimitTemplate(ratelimit));
    if(ratelimit.remaining == 0) {
      alert("Whoops, you've hit the github rate limit. You'll need to authenticate to continue");
      _self.shell.deactivate();
      return null;
    }

    // For simplicity, this tutorial trivially deals with request failures by
    //just returning null from this function via the callback.
    if(status !== 'success') {
      return callback();
    }
    return callback(response);
  })
}

Most of this function is actually devoted to CORS and rate limiting handling, which is unique to us calling the GitHub API via a proxy located on another server. When dealing with your own API, calls will likely just be $.getJSON()

Also for simplicity, any initialization failures, just bail out via initializationError(). Once we have a user, we can call initializeRepos(user, repo_name, err, callback).

Adding Commands

Commands are added via Josh.Shell.SetCommandHandler(cmd,handler) where handler is an object with two properties, exec and completion.

Josh.Shell.SetCommandHandler($cmd, {
  exec: function(cmd, args, callback) {
    ...
  },
  completion: function(cmd, arg, line, callback) {
    ...
  });
  }
});

Unlike the callback pattern we used for setUser, Josh functions do not have a separate error handler. Since Josh interacts with the UI, it has no concept of failure -- it has to execute the callback to continue executing. It is up to the caller to deal with errors and transform them into the appropriate UI response. But it still gives us the flexibility to undertake asynchronous actions, such as calling a remote API and complete the function execution upon asynchronous callback from the remote call.

user [username]

The user command does not have TAB completion, since doing efficient tab completion against the full set of GitHub users is beyond this tutorial. Instead it expects a valid username for setUser(user_name, repo_name, err, callback) and renders the user template with the new current user.

If called without a username, we simply render the user template with the current user.

repo [-l | reponame]

The repo command can show information about the current repository, change the current repository or list all repositories belonging to the user. It also provides TAB completion of partial repository names against the repositories of the current user.

Given no argument, we simply render the repository template with the current repository.

If the argument is -l, we render the repository list template with the repositories we fetched on user initialization.

Finally, the argument is used to try and look up the repository from the known repositories list. If that succeeds, we call setRepo(repo, err, callback), which fetches the root directory to initialize the current node of Josh.PathHandler before changing the current repository to the one specified. Upon switching we again render the repository template with the now current repository.

The completion handler for the command simply calls Josh.Shell.bestMatch with the partial argument and a list of all repository names. bestMatch takes care of creating the completion object with the appropriate argument completion and list of possible choices.

branch [-l | branchname]

The branch command either displays the current branch name, changes the current branch or list all branches for the current repository. It also provides TAB completion of partial branch names against the lazy initialized list of all branches for the current repository.

Given no argument, the command simply prints the current branch name. The -l argument renders a list of all known branches for the current repository, while an actual branchname as argument will cause the console to change its current branch.

Showing the list of branches uses ensureBranches(err, callback) to lazily initialize the list of branches from the API.

The completion handler for the command calls Josh.Shell.bestMatch -- just like repo completion -- with the partial argument and the list of all branches.

Wiring up Josh.PathHandler

PathHandler provides unix filepath handling. This works by abstracting the filesystem into two operations, getNode(path, callback) and getChildNodes(node, callback). The former returns a pathnode given a path string while the latter returns pathnodes for all children of a given node. With these two all tree operations including TAB completion can be accomplished by PathHandler.

A pathnode is an opaque object in which we can track any node state we want but has to have two properties:

{
  name: 'localname',
  path: '/full/path/to/localname'
}

getNode is responsible for fetching the appropriate directory from the API either by relative or absolute path. A path is considered relative if it lacks a leading /. PathHandler tracks the current directory/file node in PathHandler.current which is used to convert a relative path to an absolutish path. Since we also support the standard file system . and .. symbols and the github API does not, we then need to take the absolutish path and resolve these symbols before passing the resulting absolute path to getDir(repo_full_name, branch, path, callback).

It is the job of getDir to fetch a directory node via GET /repos/:owner/:repo/contents/:path. This API call returns either an array of file objects for a directory or a file object in case the path points directly at a file. We only care about directories for completion and cd, ls, etc. so we ignore file results and build a pathnode for the directory like this:

var node = {
  name: _.last(_.filter(path.split("/"), function(x) { return x; })) || "",
  path: path,
  children: data
};

where name is set to the last segment in the path and children stores the actual API results (which are lazily converted to childNodes for completion by function makeNodes(children)

Easy remote integration

The use of the github console is fairly limited, but it illustrates that wiring commands and file system behavior to a remote API is fairly simple. While we chose to create a custom vocabulary, we could have just as easily proxied calls to mimic git itself. Either way, Josh.js is an easy way to add a Command Line interface for your existing API or for an API custom tailored to your CLI, allowing you to create powerful admin tools without providing access to the servers themselves.