All Your Certificates Are Belong To Us

OVERVIEW:

  • Spinning up a Google Cloud Compute Engine VM
  • Setting up Certbot to help with Let'sEncrypt Certificates.
  • Hosting a SSL node server on Google Cloud

PREREQUISITES:

SETTING UP COMPUTE ENGINE:

To get started lets create a new node.js vm in google compute engine. Navigate to the google cloud launcher (https://cloud.google.com/launcher/). Then click on Explore Launcher. Once there click on the View All for developer tools. Scroll down to Node.js and click on it.

Click Launch On Compute Engine

Configuration:

Configuration for the Node.js VM

Configuration for the Node.js VM

Now you can navigate to your Compute Engine / VM Instances page and get the gloud command for connecting to your VM via SSH (just use the "View gcloud command" option).

Now SSH into your VM and lets get started.

CertBot Installation:

I'll give you an overview of the certbot installation, but if you want more info you can go here.

First Install Certbot and then generate the certs with the following commands:

sudo apt-get install certbot -t jessie-backports

sudo certbot certonly --standalone -d <YOUR DOMAIN HERE>

If you do not have a domain you can use for this, you can go to GoDaddy.com and register one to use.

Now that we have the SSL set up its time to get our express server serving up SSL pages.

Setting up Node server:

First thing we will need is a couple npm packages (express,body-parser, and compression) by running the following command.

npm install body-parser compression express

Then we will create App.js with the following code. Be sure to fill in the code with you domain name on the sslPath variable.

App.js

var express = require('express');
var path = require('path');
var https = require('https');
var request = require('request');
var bodyParser = require('body-parser');
var compression = require('compression');
var fs = require('fs');

var app = express();
var users = {};
app.use(compression());
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({extended: false}));

app.get('/', function (req, res, next) {
  res.send('Welcome to Your SSL Server. This is root endpoint');
});
// catch 404 and forward to error handler
app.use(function (req, res, next) {
    var err = new Error('Not Found');
    err.status = 404;
    next(err);
});

var sslPath = '/etc/letsencrypt/live/<YOUR_DOMAIN_HERE>/';

var https_options = {
  key: fs.readFileSync(sslPath + 'privkey.pem'),
  cert: fs.readFileSync(sslPath + 'fullchain.pem')
};

// Start listening
var port = process.env.port || process.env.PORT || 80;
https.createServer(https_options, app).listen(443, function(){
  console.log('Web Server listening on port %s', 443);
});

Now just run just start up the node server and hit your Web server (https://<domain>)

sudo node app.js

That's all there is to it. Now go have fun securing the world!

File Format Load Speeds: Parquet vs. Avro

We do a lot of work using HDFS as our file system. It provides us with a fault tolerant distribute environment to store our files. The two main file formats used with HDFS are Avro and Parquet. Since Parquet is a compressed format, I wondered how the performance of Parquet writes compare to Avro writes. So I setup some tests to figure this out.

Since Impala cannot write to Avro, I will be needing to perform these inserts in Hive (which will make the inserts longer). So I fire up my ole' Hive console and get started. First I need to create a table to insert into. I created one that is Avro backed and one that is Parquet backed. Then I insert into each of these tables 8 times with an "insert overwrite table avro select * from insertTable". The insertTable has 913304 Rows and 63 Columns with a partition column on the month. The resulting size of the Avro and Parquet directories on HDFS are 673.6M and 76.8M respectively. The following table displays the results.

Avro Parquet
128.121 114.786
99.132 112.18
97.644 125.045
97.627 97.107
97.17 96.553
98.832 104.718
97.14 108.456
97.104 98.2
AVERAGE AVERAGE
101.59 107.13

So I am seeing Parquet being almost 1/10th the size of Avro but only taking ~6% more time to write. If the application use pattern is one that is write once read many times, it makes sense to go with Parquet. But, depending on your data profile, even if you are writing many times, it may still make sense to use Parquet.  There may be other sets of data which widen the gap between these two formats. If anyone has any insight on how to do this, I could rerun these tests with updated data. Just leave a comment.

Next step is to Compare the write performance of Parquet vs Kudu. 

Excel Functionality in Javascript

At Venerite we are working with a client that requires a web app to display data in a grid. In the interest of not reinventing the wheel we started evaluating third party libraries to help us with this functionality. One such library we found was Handsontable. This library is MIT licensed and really easy to get started with.

We have a requirement to add a checkbox to a cell to control some functionality. Here is a little insight on how we accomplished this.

Fist thing we need to do is define our data model for our grid. 

var myData = [{
        Goal: '100',
        PercGoal: .08,
        Locked: true
    }, {
        Goal: '200',
        PercGoal: .08,
        Locked: true
    }, {
        Goal: '1000',
        PercGoal: .08,
        Locked: false
    }, {
        Goal: '34',
        PercGoal: .08,
        Locked: false
    }, ];

Next we instantiate an instance of the handsontable grid and pass the required options. 

The data field will associate the table with the dataset we previously defined.

The colHeaders field will define what headers we want associated with this grid.

The columns field will define how our dataset is associated with the datagrid columns.

var container = document.getElementById('example');
var hot = new Handsontable(container, {
    data: myData,
    colHeaders: ["Goal", "%Growth Goal", "Locked"],
    columns: [
        {data: "Goal"},
        {data: "PercGoal", type:'numeric', format: '0%'},
        {data: "Locked",type: 'checkbox'
            ,checkedTemplate: 'true'
            ,uncheckedTemplate: 'false'}
    ]
 });

The key to adding a checkbox to a cell is to add a "type" to the column you are wanting to have this functionality. If you do not have a dataset that uses the default values of "true" and "false" you will need to add some params ... namely checkedTemplate and uncheckedTemplate. These attributes will let you define what values you want to use for a checked checkbox and an unchecked one. 

Here is a JSFiddle project where you can play around with this functionality.

 

Meteor Client Error Logging to the Server

As convenient as it is to develop mobile apps in Meteor using Cordova (PhoneGap), it is difficult if not impossible to view errors, logging, or other things happening in real time on real mobile hardware. These errors and logs are output to the user's console. Because of this, they will most likely never be reported. So, taking inspiration from Loggly, we implemented a few Meteor server methods which take a message as an argument and logs that message to the server and the console. This technique will also be helpful in finding client error on Meteor webapps.

Lets run through what we did.

First things first: adding a Meteor method to deal with different types and levels of logging. In our case, we opted for INFO,WARN, and ERROR level logging. Upon establishing these, we implemented a simple method signature to include both the logging level as well as the message needing to be logged via our new method. We also want to make sure this method does not block other DDP messages from being processed. We use this.unblock() to accomplish this (to learn more about the meteor unblock read about it on meteorhacks.com). 

clientLog: function(level,message) {
      this.unblock();
      var user = Meteor.call("getUserEmail");
      switch (level) {
        case LEVEL_WARN:
          console.log("WARN-CLIENT:",user,":",message); 
          break;
        case LEVEL_ERROR:
          console.log("ERROR-CLIENT:",user,":",message);
          break;
        default:
          console.log("INFO-CLIENT:",user,":",message);   
      }
    },
  });

We then make a call to get the user name of the logged in user which we can associate this log. We are using the accounts-password package so we just look in the database to find the corresponding user record and then grab the email address.

getUserEmail: function() {
  if(this.userId) {
    var userRecord = Meteor.users.findOne(this.userId);
    return userRecord["emails"][0]["address"];
  } else {
    return "";
  }
}

Now to use these Method we just make a call from the client. To make it easier we just monkey patch console.log, console.warn, and window.error to perform the proper server call. So all you need to do is add console.log (or console.warn) in your client code and it will automatically be logged on the server as well as the client.

//remap console.log to send to the server
  var clientLogger = console.log;
  var clientWarnLogger = console.warn;
  console.log = function() {
    clientLogger.apply(this,arguments);
    return Meteor.call('clientLog', LEVEL_LOG, arguments);
  };
  console.warn = function() {
    clientWarnLogger.apply(this,arguments);
    return Meteor.call('clientLog', LEVEL_WARN, arguments);
  };
  window.onerror = function() {
    return Meteor.call('clientLog', LEVEL_ERROR, arguments);
  };

Here is the final product.

var LEVEL_LOG = "log";
var LEVEL_WARN = "warn";
var LEVEL_ERROR = "error";
if (Meteor.isServer) {
  Meteor.methods({
    getUserEmail: function() {
      if(this.userId) {
        var userRecord = Meteor.users.findOne(this.userId);
        return userRecord["emails"][0]["address"];
      } else {
        return "";
      }
    },
    clientLog: function(level,message) {
      this.unblock();
      var user = Meteor.call("getUserEmail");
      switch (level) {
        case LEVEL_INFO:
          console.log("INFO-CLIENT:",user,":",message);   
          break;
        case LEVEL_WARN:
          console.log("WARN-CLIENT:",user,":",message); 
          break;
        case LEVEL_ERROR:
          console.log("ERROR-CLIENT:",user,":",message);
          break;
      }
    },
  });
}

Meteor.startup(function() {
  //remap console.log to send to the server
  var clientLogger = console.log;
  console.log = function() {
    clientLogger.apply(this,arguments);
    return Meteor.call('clientLog', LEVEL_LOG, arguments);
  };
  console.warn = function() {
    return Meteor.call('clientLog', LEVEL_WARN, arguments);
  };
  window.onerror = function() {
    return Meteor.call('clientLog', LEVEL_ERROR, arguments);
  };
});

Hope that helps in debugging any future client issues.