SuperPumpup (dot com)

General awesomeness may be found here.

19 April 2017

Build Your Own Serverless mint.com Using Firebase & Cloud Functions

Why am I doing this?

As part of a project I'm exploring with a couple friends, I recently decided to try to do "account aggregation" - that is, pull in transactions at several of my "bank accounts" into some coherent portal. I decided I'm a good use case (my favorite guinea pig). Tracking my personal expenses/spending is hard for a few reasons (not just because I don't really want to do it), including: we use credit cards a lot, some transactions go through my main checking account, we change "primary" credit cards a couple times a year, and my wife has her own cards.

Fortunately, we don't have to deal with multiple checking accounts, so all the "income" comes into one place, but it's still a total mess tracking the transactions and how they should be debited or credited against a real cash flow system. Maybe we should be using something like QuickBooks, but that's a question for another day.

I'm trying to do this in this "serverless" way because on the heels of a project that was incredibly complex and involved us building an entire systems framework, I wanted to see if i could do this with a bare minimum of technology "infrastructure development".

Why would you want to do this?

I have no idea. Probably you don't want to do this just for yourself. Getting a live account at Finicity or one of their competitors (Plaid, Yodlee) requires a bit of explanation, underwriting on their end, and a long contract that's fairly expensive. Maybe there is a use case for them for a "hacker package" that lets you create one customer (yourself) to explore with. But if you're interested in the FinTech space, it's amazing what these tools can do for pretty inexpensively (like less than your laptop cost).

What tools will we use?

Firebase is pretty neat and can be wired up directly to ReactJS (how I would/will build a frontend for this), and google recently announced their Cloud Functions which will let us actually avoid spinning up and deploying software to servers (though if you want to geek out about packaging and deploying software, I have some good posts about that).

Finicity is the account aggregation API that I was able to get working with the least amount of interaction with "account executives". I know software sales is super important (and the guys at Finicity are SUPER NICE), but for an exploratory project like this, talking to a human is off the "hot path" to success.

Walking Skeleton

So basically all we need for this is to go to the https://console.firebase.google.com, and sign up with whatever google account you want, and create a new project.

Welcome to Firebase

If you then open up the "Database" tab you will see an empty database. We'll track our "progress" here by seeing that database fill up.

If you click on the "Functions" tab, you will see a "Get Started" link that will walk you through setting up your firebase tools locally (npm install -g firebase-tools, firebase init, firebase deploy).

Firebase Menu

It takes a few minutes to "provision" a function when you do the first deploy with a new function, but after that it's pretty quick. Think of it like Heroku for just bits of code.

After initializing, our directory structure will look a bit like this:

Your initialized project

For now, I want to invoke these via a curl and so my first function could look something like this:

var functions = require('firebase-functions');
var xml = require('xml');
var request = require('request-promise');

const admin = require('firebase-admin');
admin.initializeApp(functions.config().firebase);

exports.testFinicity = functions.https.onRequest((req, res) => {
  const text = req.query.text;

  admin.database().ref('/some-test-path').push({query: text}).then(snapshot => {
    // Redirect with 303 SEE OTHER to the URL of the pushed object in the Firebase console.
    res.redirect(303, snapshot.ref);
  });
});

When I firebase deploy this, and then curl -v the url they return, I can see that I got a path that I can then either query directly in curl, or see in the web console of firebase. Awesome.

Getting transactions

Ok, now we get to the meat and potatoes. I'm warning you though - this is a bit of a mess. What's great about these cloud functions, and building this in general, is you can extract whatever other kind of functions/objects/whatever and invoke them during this request/response cycle. The big thing to look out for is that you are working within an "asynchronous" context, so your functions are all returning promises. It's kind of hairy, and I'm still developing expertise in building in that context, so won't give you MY thoughts, but there are great resources out there.

Here is just the authentication component:

exports.testFinicity = functions.https.onRequest((req, res) => {
  const original = req.query.text;

  finicityAuthenticate().then((response) => {
    console.log("Finicity Auth returned")
    // Push it into the Realtime Database then send a response
    const token = response.token;
    admin.database().ref('/finicity/authentication').push({status: 'ok', response: response}).then(snapshot => {
      // Redirect with 303 SEE OTHER to the URL of the pushed object in the Firebase console.
      res.redirect(303, snapshot.ref);
    });
  }).catch((error) => {
    console.log("Finicity Auth errored")
    // Push it into the Realtime Database then send a response
    admin.database().ref('/finicity/authentication').push({status: 'error', error: error}).then(snapshot => {
      // Redirect with 303 SEE OTHER to the URL of the pushed object in the Firebase console.
      res.redirect(303, snapshot.ref);
    });
  });
});

function finicityAuthenticate() {
  console.log("Authenticating");

  credentials = {
    partnerId: "MY_ID",
    partnerSecret: "MY_SECRET"
  }

  var options = {
    method: 'POST',
    url: 'https://api.finicity.com/aggregation/v2/partners/authentication',
    json: true,
    headers: {
      'Finicity-App-Key': 'MY_APP_KEY'
    },
    body: credentials
  };

  return request(options).then(function (response) {
    console.log('Found a token', response);
    return response;
  }).catch((error) => {
    console.log("Error", error)
    return error;
  });
}

So you'll see that that will just write an Auth token to my database. Finicity wants me to reuse my auth token for awhile, and cycle them periodically. That's a generally good practice, and so I'll just stash it in the database for now. Actually, to be honest, I was really just writing it there to make sure I could parse things right - maybe it's not a great thing to do in production.

Here is a complete, "Authenticate, create new customer, add their accounts, get transactions" flow:

exports.testFinicity = functions.https.onRequest((req, res) => {
  const original = req.query.text;

  finicityAuthenticate().then((response) => {
    console.log("Finicity Auth returned")
    // Push it into the Realtime Database then send a response
    const token = response.token;
    admin.database().ref('/finicity/authentication').push({status: 'ok', response: response}).then(snapshot => {
      // Redirect with 303 SEE OTHER to the URL of the pushed object in the Firebase console.
      // res.redirect(303, snapshot.ref);
      admin.database().ref('/finicity/customers').push({something: 'else'}).then(snapshot => {
        console.log("Customer Created:", snapshot.key);
        var customerId = snapshot.key;
        finicityCreateCustomer(token, customerId).then((response) => {
          snapshot.ref.child('finicityId').set(response.id);
          snapshot.ref.child('finicityCreatedDate').set(response.createdDate);
          var institutionId = 101732;
          var finicityCustomerId = response.id;
          var loginForm = {
            "loginForm": [
             {
              "id": "101732001",
              "name": "Banking Userid",
              "value": "",
              "description": "Banking Userid",
              "displayOrder": 1,
              "mask": "false",
              "instructions": ""
             },
             {
              "id": "101732002",
              "name": "Banking Password",
              "value": "",
              "description": "Banking Password",
              "displayOrder": 2,
              "mask": "true",
              "instructions": ""
             }
            ]
          }
          var credentials = {
            "credentials": [
              {
                "id": "101732001",
                "name": "Banking Userid",
                "value": "demo"
              },
              {
                "id": "101732002",
                "name": "Banking Password",
                "value": "go"
              }
            ]
          }
          finicityAddAllAccounts(token, finicityCustomerId, institutionId, credentials).then((response) => {
            snapshot.ref.child('accountRetrievals').push(response);
            var checkingAccount = response.accounts.find((account) => {
              console.log("LOoking at this account: ", account.type, account)

              return account.type === 'checking'
            });

            console.log("Checking account: ", checkingAccount)
            finicityLoadHistoricalAccountTransactions(token,
              finicityCustomerId,
              checkingAccount.id).then((response) => {
                console.log("Historical TXNs loaded");
                finicityGetAccountTransactions(token,
                  finicityCustomerId,
                  checkingAccount.id).then((response) => {
                    console.log("Get transactions success")
                    snapshot.ref.child('accountTransactions').push(response);
                  })
                })
            res.redirect(303, snapshot.ref);
          })
        })
      })
    });
  }).catch((error) => {
    console.log("Finicity Auth errored")
    // Push it into the Realtime Database then send a response
    admin.database().ref('/finicity/authentication').push({status: 'error', error: error}).then(snapshot => {
      // Redirect with 303 SEE OTHER to the URL of the pushed object in the Firebase console.
      res.redirect(303, snapshot.ref);
    });
  });
});

function finicityGetAccountTransactions(token, finicityCustomerId, accountId) {
  console.log("Getting transactions", token, finicityCustomerId, accountId);

  var options = {
    method: 'GET',
    url: 'https://api.finicity.com/aggregation/v3/customers/' + finicityCustomerId + '/accounts/' + accountId + '/transactions',
    json: true,
    headers: {
      'Finicity-App-Key': 'MY_APP_KEY',
      'Finicity-App-Token': token
    },
    qs: {
      fromDate: '1428159047', // I used https://www.epochconverter.com/ to generate these
      toDate: '1459781447'
    }
  };

  return request(options).then(function (response) {
    console.log('Account transactions lookup successful', response);
    return response;
  }).catch((error) => {
    console.log("Error", error)
    return error;
  });
}

function finicityLoadHistoricalAccountTransactions(token, finicityCustomerId, accountId) {
  console.log("Getting transactions", token, finicityCustomerId, accountId);

  var options = {
    method: 'POST',
    url: 'https://api.finicity.com/aggregation/v1/customers/' + finicityCustomerId + '/accounts/' + accountId + '/transactions/historic',
    json: true,
    headers: {
      'Finicity-App-Key': 'MY_APP_KEY',
      'Finicity-App-Token': token
    }
  };

  return request(options).then(function (response) {
    console.log('account transactions lookup successful', response);
    return response;
  }).catch((error) => {
    console.log("Error", error)
    return error;
  });
}

function finicityLookupInstitutionLoginForm(token, institutionId) {

  console.log("Getting Institution Login Form", customerId, token, institutionId);

  var options = {
    method: 'GET',
    url: 'https://api.finicity.com/aggregation/v1/institutions/{institutionId}/loginForm',
    json: true,
    headers: {
      'Finicity-App-Key': 'MY_APP_KEY',
      'Finicity-App-Token': token
    }
  };

  return request(options).then(function (response) {
    console.log('Institution lookup successful', response);
    return response;
  }).catch((error) => {
    console.log("Error", error)
    return error;
  });
}

function finicityAddAllAccounts(token, customerId, institutionId, customerCredentials) {
  console.log("Adding All Accounts", customerId, token, institutionId);

  var options = {
    method: 'POST',
    url: 'https://api.finicity.com/aggregation/v1/customers/' + customerId +'/institutions/' +institutionId +'/accounts/addall',
    json: true,
    headers: {
      'Finicity-App-Key': 'MY_APP_KEY',
      'Finicity-App-Token': token
    },
    body: customerCredentials
  };

  return request(options).then(function (response) {
    console.log('Created accounts added', response);
    return response;
  }).catch((error) => {
    console.log("Error", error)
    return error;
  });

}


function finicityCreateCustomer(token, customerId) {
  console.log("Creating customer", customerId, token);

  var options = {
    method: 'POST',
    url: 'https://api.finicity.com/aggregation/v1/customers/testing',
    json: true,
    headers: {
      'Finicity-App-Key': 'MY_APP_KEY',
      'Finicity-App-Token': token
    },
    body: {
      username: customerId
    }
  };

  return request(options).then(function (response) {
    console.log('Created customer successfully', response);
    return response;
  }).catch((error) => {
    console.log("Error", error)
    return error;
  });

}

function finicityAuthenticate() {
  console.log("Authenticating");

  credentials = {
    partnerId: "MY_ID",
    partnerSecret: "MY_SECRET"
  }

  var options = {
    method: 'POST',
    url: 'https://api.finicity.com/aggregation/v2/partners/authentication',
    json: true,
    headers: {
      'Finicity-App-Key': 'MY_APP_KEY'
    },
    body: credentials
  };

  return request(options).then(function (response) {
    console.log('Found a token', response);
    return response;
  }).catch((error) => {
    console.log("Error", error)
    return error;
  });
}

Again, it's a bit of a mess, but it got us a list of transactions!

One trick I wound up doing while I was developing this was using the following deploy "snippet".

firebase deploy --only functions && sleep 30 && curl https://us-central1-test-<project>.cloudfunctions.net/testFinicity && say "done"

So that does the deployment, sleeps for a bit in order for the functions to finish "setup", then makes the request, and says "Done" out my speakers, so I don't have to watch the terminal window for a few minutes.

After all that, I get something like this:

A list of customer accounts

Transactions (this guy likes Wal-Mart)

And that's something from which we can work! I'm still thinking about how to structure this data better for it to be more useful (NoSQL is still something I'm adapting to), but it's pretty informative.