This will hopefully be a short post about how to listen to changes from your CosmosDB SQL API using the Node SDK. For some reasons, it reminds me of the oplog tailing mechanism in MongoDB that Meteor utilized in the past to achieve real-time, hence this article's title.

I do not have much time right now, but let us talk about a bit of my use case for this. I am currently working on an IoT project that uses several Azure technologies as its infrastructure. As you might have guessed, it pipes data from the IoT sensors into Azure Event Hubs. The data then gets ingested in Azure Databricks; ultimately ending up in an intermediate data warehouse (for dashboards and further analysis) and a NoSQL application data store (coming up next).

This NoSQL application data store is the CosmosDB instance that we are going to tail. The objective was to push changes from this data store to a mobile application in real-time or at least real-time (vs long polling or querying in set intervals). To make the long story short, I ended up tailing the CosmosDB's Change Feed in a GraphQL service application to make it easier for the client application to implement a PubSub system. More about this in an older post that I published early this year.

Before we dig into the code, let me just say that we are not going to go through how to initialize your CosmosDB collection or how to write a facade / repository class in NodeJS. We will go straight to the Change Feed part but first, make sure you have the correct Node SDK in your project:

npm install --save @azure/cosmos

Once you have set up your CosmosDB boilerplate code in Node, you can access the change feed iterator using the code below:

const changeFeed = container.items.readChangeFeed(partitionKey, options);

If you need to brush up on iterators in JavaScript, you can check this post that I have written last year).

And that's it! You now have a change feed iterable where you can listen to changes from CosmosDB, but let us see what options we have in doing so. The Node SDK currently gives us four ways of listening to the change feed and they are:

1. Start from now

const options = {};

2. Start from a continuation

const { headers } = await container.items.create({ ... });

const lsn = headers["lsn"];

const options = { continuation: lsn };

// I have not used this specific method yet, this example is from here.

3. Start from a specific point in time

const options = { startTime: new Date() }; // any Date object will do

4. Start from the beginning

const options = { startFromBeginning: true };

As all of these are self-explanatory, we will go ahead and use our changeFeed iterator. To retrieve the next value, we can use await changeFeed.executeNext() or we can loop through the next values like this:

while (changeFeed.hasMoreResults) {

  const { result } = await changeFeed.executeNext();

  // do what you want with the result / changes

}

Reading the source code of the Node SDK revealead that it is also exposing a real generator (the function signature being public async *getAsyncIterator(): AsyncIterable<ChangeFeedResponse<Array<T & Resource>>>). This would have allowed a more elegant for of construct, but unfortunately I bumped into a few issues regarding Symbols when I tried it. If you have used it in the past, please feel free to share in the comments!

That will be all for now, and I hope you learned something in this post.

In this post, we will not go through GraphQL from the ground up. My assumption is that you are here to learn how to implement real-time through the use of the graphql-subscription package and that you already have a basic understanding of GraphQL types, queries and mutations.

We are going to use Apollo for this tutorial. Apollo was made by the same guys who made Meteor and is a bit opinionated but is also arguably one of the most popular full-featured GraphQL libraries around. We will also use React and create-react-app to bootstrap our client application on the second part of this tutorial. That being said, some knowledge of higher order components is also assumed (in Part 2).

Server Boilerplate

Let's start outlining our backend. Initialize a Node project by issuing npm init on your preferred folder, and then install dependencies like so:

npm i --save express body-parser cors graphql graphql-tools apollo-server-express

Next, create the three files that we will use for this short tutorial:

touch index.js resolvers.js schema.js

We will then define a type, a root query and a mutation that we will use for our subscription:

const { makeExecutableSchema } = require('graphql-tools');

const resolvers = require('./resolvers');

const typeDefs = `

  type Message {

    message: String

  }

  type Query {

    getMessages: [Message]

  }

  type Mutation {

    addMessage(message: String!): [Message]

  }

  schema {

    query: Query

    mutation: Mutation

  }

`;

module.exports = makeExecutableSchema({ typeDefs, resolvers });

Okay, so at this point, we have the schema for a GraphQL server that allows you to send a mutation to add a Message, and a query that allows you fetch all messages in the server. Let's implement resolvers.js so we can start using our schema:

const messages = [];

const resolvers = {

  Query: {

    getMessages(parentValue, params) {

      return messages;

    }

  },

  Mutation: {

    addMessage(parentValue, { message }) {

      messages.push({ message });

      return messages;

    }

  }

};

module.exports = resolvers;

Oh shoot. We have defined a schema and the functions that will resolve their return values, but we have not set our server up. At least not yet. We are going to use express and apollo-server-express to serve our GraphQL implementation through HTTP:

const express = require('express');

const bodyParser = require('body-parser');

const cors = require('cors');

const { graphqlExpress, graphiqlExpress } = require('apollo-server-express');

const { createServer } = require('http');

const schema = require('./schema');

const app = express();

app.use(bodyParser.json());

app.use(bodyParser.urlencoded({ extended: true }));

app.use(cors());

app.use(

  '/graphql',

  graphqlExpress({

    schema

  })

);

app.use(

  '/graphiql',

  graphiqlExpress({

    endpointURL: '/graphql'

  })

);

const PORT = process.env.PORT || 3030;

const server = createServer(app);

server.listen(PORT, () => {

  console.log(`Server now running at port ${PORT}`);

});

We can now have a working GraphQL server running on http://localhost:3030/graphql by issuing node index.js. Since we have configured the interactive Graphiql as well, we can explore our schema, and issue some sample queries and mutations on http://localhost:3030/graphiql:

Adding real-time through Subscriptions and PubSub

Server Configuration

Our simple GraphQL server is running. That means we can now proceed to the interesting part: implementing real-time through Apollo PubSub. As with all modern real-time frameworks, the implementation is often done on top of WebSockets. We need to install additional dependencies to make use of this transport layer:

npm i --save graphql-subscriptions subscriptions-transport-ws

We then need to make use of these libraries to enable WebSockets support on index.js:

const { execute, subscribe } = require('graphql');

const { SubscriptionServer } = require('subscriptions-transport-ws');

. . .

const server = createServer(app);

server.listen(PORT, () => {

    console.log(`Server now running at port ${PORT}`);

    new SubscriptionServer(

        {

            execute,

            subscribe,

            schema

        },

        {

            server,

            path: '/subscriptions'

        }

    );

});

Let's modify our /graphiql endpoint as well to make use of our new transport layer, so we can demonstrate that this is working through Graphiql once we are done:

app.use(

    '/graphiql',

    graphiqlExpress({

        endpointURL: '/graphql',

        subscriptionsEndpoint: 'ws://localhost:3030/subscriptions'

    })

);

That's it for the server setup! Let's proceed on fleshing out the subscription implementation.

Defining Subscriptions

In GraphQL, a subscription is just a type, pretty much like query and mutation. Go ahead and define our subscription on our schema.js:

  const typeDefs = `

  . . .

  type Subscription {

    newMessageAdded: Message

  }

  schema {

    query: Query

    mutation: Mutation

    subscription: Subscription

  }

`;

We have just defined our first subscription. It will allow applications or clients to subscribe and receive updates whenever new messages are added (through a mutation). Just to make sure everything is working correctly, visit Documentation Explorer on Graphiql and you should now be able to see Subscription and newMessageAdded:

If there are no errors and you can see the definition above, then we are ready to make this work by, you guessed it, implementing the resolver function for newMessageAdded.

Implementing the Subscription and Publishing Messages

With the transport configuration and the type definitions done, the only thing we need to do now is to implement newMessageAdded and the actual message publication. The flow will be like this:

1. A client will subscribe to `newMessageAdded`
2. Every time our `addMessage` mutation is queried, we will publish a message to `newMessageAdded`, using the new `message` as the payload.

We need to tweak our resolvers.js to import helpers from graphql-subscriptions. We will use them to implement our newMessageAdded subscription query:

const { PubSub, withFilter } = require('graphql-subscriptions');

const pubsub = new PubSub();

. . .

const resolvers = {

  Query: {

    . . .

  },

  Mutation: {

    . . .

  },

  Subscription: {

    newMessageAdded: {

      subscribe: withFilter(

        () => pubsub.asyncIterator('newMessageAdded'),

        (params, variables) => true

      )

    }

  }

};

module.exports = resolvers;

We just implemented our first subscription query! Every time our server publishes a message to newMessageAdded, clients that are subscribed will get the published payload.

As an aside, the helper function withFilter is not actually required nor used in our example here (just subscribe: () => pubsub.asyncIterator('newMessageAdded') will do for this tutorial), but I figured that this will be helpful if you want to try something useful with this whole pubsub ordeal, like say, a classic chat app.
The second function that you pass as an argument to withFilter will allow you to filter out the subscribers who will receive the message. This is done by using the field in the actual payload that is about to get published (params) and the GraphQL query variables from the subscription (variables). All you need to do is to return a truthy value if you want it sent to this particular subscriber. It will look roughly similar to this: return params.receiverId === variables.userId. Of course, that is assuming that a query variable called userId was sent along with the subscription.

Since we do not have an application that will subscribe to our server yet, why don't we try this out with Graphiql?

If you can see the same message above, great! Everything is working awesome. But if we do not publish anything anywhere on our server, nothing will happen. Yep, we are about to do just that.

In fact, we just need to add one line to our addMessage resolver:

  Mutation: {

    addMessage(parentValue, { message }) {

      messages.push({ message });

      // blame prettier for not making this a one-liner 

      pubsub.publish('newMessageAdded', {

        newMessageAdded: { message }

      });

      return messages;

    }

  }

We can now test this using Graphiql on two browser windows. The first browser will be the subscriber, and the second one will send the mutations:

As soon as you send a addMessage mutation on the second browser, the first browser receives the message, and displays it instantly! How cool is that? Let's wrap up what we learned in this short tutorial.

Wrap up

In this tutorial, we learned how to set up subscriptions and publish message across subscribers using graphql-subscriptions. On the next part of this tutorial, we will use apollo-client with react-apollo to see how this will work with a real application as the subscriber.

The complete source code for this tutorial can be found here

If you encountered any errors or have any questions about this, let me know in the comments section below!

Hey everyone. I know it has been a while since I wrote something. I have been busy with multiple, large scale projects during the past few months, so I was almost always too tired at the end of the day to compose a new entry. I also had to relocate; I think the adjustment phase also took a lot of my time and energy. Anyway, what I am going to try to do now is to write short, straight to the point tutorials about how to do specific tasks (as opposed to going into more detailed, wordy posts). I will still write the elaborate ones, but I will be focusing on consistency for now. I have been working on a lot of interesting problems and relevant technologies at work, and I just feel guilty that I do not have enough strength left at the end of the day to document them all.

Let us start with this simple topic just to get back to the habit of writing publicly. I have been configuring Linux VMs for a while now, but I have not really written anything about it, aside from my series of Raspberry Pi posts. Also, it is my first time to work with the Azure platform, so I thought that it might be interesting to write about this today.

This tutorial will assume that the Ubuntu 16.04 VM is already running and you can SSH properly into the box with a sudoer account.

The Basics: Installing MongoDB

You can read about the official steps here. If you prefer looking at just one post to copy and paste code in sequence, I will still provide the instructions below.

Add the MongoDB public key

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 0C49F3730359A14518585931BC711F9BA15703C6

Add MongoDB to apt's sources list

echo "deb [ arch=amd64,arm64 ] http://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.4.list

Update apt repository and install MongoDB

sudo apt-get update && sudo apt-get install -y mongodb-org

Run and check if MongoDB is running properly

sudo service mongod start

tail -f /var/log/mongodb/mongod.log

If everything went well, you should see something like this:

2017-10-04T01:18:51.854+0000 I NETWORK [thread1] waiting for connections on port 27017

If so, let's continue with the next steps!

Create a root user

I will not get into the details of how to create and manage MongoDB databases and collections here, but let us go into the processes of creating a root user so we manage our database installation remotely through this user.

Connect to MongoDB CLI

mongo

Use the admin database

use admin

Create admin user

db.createUser(

    {

      user: "superadmin",

      pwd: "password123",

      roles: [ "root" ]

    }

)

SSL and some network-related configuration

Now that we have MongoDB installed and running, we need to make some changes with the mongod.conf file to enable SSL and to make our MongoDB installation accessible on our VM's public IP and chosen port.

SSL Certificates

Creating a self-signed certificate

If you already have a certificate or you just bought one for your database for production use, feel free to skip this step. I am just adding this for people who are still experimenting and want SSL enabled from the start. More information regarding this can be found here.

This self-signed certificate will be valid for one year.

sudo openssl req -newkey rsa:2048 -new -x509 -days 365 -nodes -out mongodb-cert.crt -keyout mongodb-cert.key

Create .pem certificate

This .pem certificate is the one that we will use on our mongod.conf configuration file. This command will save it on your home directory (/home/<username>/mongodb.pem or ~/mongodb.pem).

cat mongodb-cert.key mongodb-cert.crt > ~/mongodb.pem

MongoDB Configuration

Now that we have our self-signed certificate and admin user ready, we can go ahead and tweak our MongoDB configuration file to bind our IP, change the port our database will use (if you want to), enable SSL and to enable authorization.

I use vim whenever I am dealing with config files via SSH; you can use your favorite text editor for this one.

sudo vim /etc/mongod.conf

Make sure to change the following lines to look like this:

net:

  port: 27017

  bindIp: 0.0.0.0

  ssl:

    mode: requireSSL

    PEMKeyFile: /home/<username>/mongodb.pem

security:

  authorization: enabled

Restart the MongoDB service:

sudo service mongod restart

If we go ahead and print the MongoDB logs like we did earlier, we should be able to see something that looks like this (notice that there's an SSL now):

2017-10-04T01:18:51.854+0000 I NETWORK [thread1] waiting for connections on port 27017 ssl

If you got that, it means that everything is working fine. We just need to add one more command to make sure that our MongoDB service will restart across VM reboots. systemctl will take care of that for us:

sudo systemctl enable mongod.service

Azure Firewall

Now, if you try to connect to your database using your favorite MongoDB database viewer or by using the Mongo CLI on your local machine, you might notice that you will not be able connect. That's because we need to add an Inbound security rule on the Azure portal first.

Once on the Dashboard, click on All Resources.

Click on the Network Security Group associated with your VM.

From here, you can see a summary of all the security rules you have for your virtual network. Click on Inbound security rules under Settings on the left pane.

Click Add. You should be able to see a form with a lot of fields. We are used MongoDB's default port, so we can just click on Basic at the top so we can select from a list of preset ports.

Just click on OK, and we are done! You can start connecting to your MongoDB installation using your tool of choice.

eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE0ODk5OTEyNjI3NTIsImlzcyI6IkpvaG4gQ3Jpc29zdG9tbyIsIm5hbWUiOiJjcmlzb3N0b21vIiwiZW1haWwiOiJjcmlzb3N0b21vQGpvaG4uY29tIn0._CP8KU_AX4XNJKyxD561LTiFbY0HcPFKRgI1AztGMsI

const users = [

  { _id: 1, name: "john", email: "john@crisostomo.com", password: "john12345" },

  { _id: 2, name: "crisostomo", email: "crisostomo@john.com", password: "crisostomo12345" },

];

function validateUser(username, password) {

  const user = users.find((user) => {

    return user.name === username && user.password === password;

  });

  return user;

}

module.exports = { validateUser };

const tokens = [];

module.exports = {

  add: function(token, payload) {

    tokens[token] = payload;

  },

  isValid: function(token) {

    if (!tokens[token]) {

      return false; 

    }

    if (tokens[token].exp <= new Date()) {

      const index = tokens.indexOf(token);

      tokens.splice(index, 1);

      return false;

    } else {

      return true;

    }

  }

}

const express = require('express');

const bodyParser = require('body-parser');

const jwt = require('jwt-simple');

const moment = require('moment');

const users = require('./users');

const tokens = require('./tokens');

const app = express();

app.use(bodyParser.json());

const jwtAttributes = {

  SECRET: 'this_will_be_used_for_hashing_signature',

  ISSUER: 'John Crisostomo', 

  HEADER: 'x-jc-token', 

  EXPIRY: 120,

};

app.listen(8080);

console.log('JWT Example is now listening on :8080');

// AUTH MIDDLEWARE FOR /token ENDPOINT

const auth = function (req, res) {

  const { EXPIRY, ISSUER, SECRET } = jwtAttributes;

  if (req.body) {

    const user = users.validateUser(req.body.name, req.body.password);

    if (user) {

      const expires = moment().add(EXPIRY, 'seconds')

        .valueOf();

      const payload = {

        exp: expires,

        iss: ISSUER,

        name: user.name,

        email: user.email, 

      };

      const token = jwt.encode(payload, SECRET);

      tokens.add(token, payload);

      res.json({ token });

    } else {

      res.sendStatus(401);

    }

  } else {

    res.sendStatus(401);

  }

};

app.post('/token', auth, (req, res) => {

  res.send('token');

});

// VALIDATE MIDDLEWARE FOR /secretInfo

const validate = function (req, res, next) {

  const { HEADER, SECRET } = jwtAttributes;

  const token = req.headers[HEADER];

  if (!token) {

    res.statusMessage = 'Unauthorized: Token not found';

    res.sendStatus('401').end();

  } else {

    try {

      const decodedToken = jwt.decode(token, SECRET);

    } catch(e) {

      res.statusMessage = 'Unauthorized: Invalid token';

      res.sendStatus('401');

      return;

    }

    if (!tokens.isValid(token)) {

      res.statusMessage = 'Unauthorized : Token is either invalid or expired';

      res.sendStatus('401');

      return;

    }

    next(); 

  }

};

app.get('/secretInfo', validate, (req, res) => {

  res.send('The secret of life is 42.');

});

I have been watching a movie last night when my mind spun on a different thread and remembered a JavaScript language feature that have existed for some time now, but I have never had the chance to use. At least, directly.

We do bleeding edge JavaScript at the office. That means we have all these new language features at our disposal as early as possible through the use of Babel. We write JavaScript code using the newest language specification (ECMAScript 6/7) and our code gets transpiled into ECMAScript 5. We have been using all the nifty features such as import, async/await, spread/rest operators and destructuring as early as last year. These are just the new ES6 features that I can think of off the top, maybe because they are the most practical ones.

There is one feature, however, that can be really powerful but I have not really been able to leverage. They are generators. Prior to V8 v5.5 and Node v7.6.0, Babel's async/await and other asynchronous libraries around has been using generatorsunder the hood to implement this feature.

But what are generators? According to the venerable MDN page:

A generator is a special type of function that works as a factory for iterators. A function becomes a generator if it contains one or more yield expressions and if it uses the function* syntax.

MDN's definition is clear and straightforward, but let me rephrase it from what I have understood. Aside from producing iterables, think of a generator as a function that you can play and pause. This characteristic enables it to implement asynchronous programming, and when used with promises, you can come up with all sorts of things- including your own async library if you want to make one for learning purposes.

Let's dig into some basic code examples to solidify our understanding of generators:

function* counter() {

  for (let i = 0; i < 5; i++) {

    yield i

  }

}

This function was declared using function* and has a yield inside the function body, so this must be a generator. When we invoke it and assign the result to a variable like so, let c = counter(), we get back an iterable object that we can use to iterate over the values of i. An iterable object in JavaScript must have a next()method. This method returns an object that contains a value and a done property. Let's see that in action:

/***************************************************

  Using next() to step through the values explicitly

****************************************************/

let c1 = counter();

console.log(c1.next().value);

// 1

console.log(c1.next().value);

// 2

console.log(c1.next().value);

// 3

console.log(c1.next().value);

// 4

console.log(c1.next().value);

// 5

/***************************************************

  Using a for-of loop

****************************************************/

let c2 = counter();

for (const num of c) {

  console.log(c);

}

// 1

// 2

// 3

// 4

// 5

/***************************************************

  Using the done property explicitly

****************************************************/

let c3 = counter();

let i = c3.next();

while (!i.done) {

  console.log(i.value);

  i = c3.next();

}

// 1

// 2

// 3

// 4

// 5

We went through three different ways on how to iterate over the iterable that was returned by our counter generator. On the first example, we manually stepped through the iterator by using next(). We know that next() returns an object with a valueand a done property, and so we were able to chain .value every time we log the iteration to the console. This shows us one of the concepts that we have discussed earlier: we were able to play and pause the generator's execution by using the next() method. Another interesting thing is that it remembers its internal statethrough its iterations.

It works this way: the generator function stops immediately at every yield statement, and passes the value on its right to the object being returned by next(). We used a loop on our example, and by doing so, the loop gets suspended every time it encounters a yield statement.

Another thing worth knowing is that we can alter the generator's internal state from outside the generator by passing in an argument to next():

function* counter (limit) {

  for (let i = 1; i <= limit; i++) {

    let j = yield i;

    if (j) limit = j;

  }

}

/***************************************************

  Passing a value to next() to alter internal state

****************************************************/

const c1 = counter(2)

console.log(c1.next().value); // 1

console.log(c1.next().value); // 2

console.log(c1.next().value); // undefined

/***************************************************

  Passing a value to next() to alter internal state

****************************************************/

const a2 = counter(2)

console.log(c2.next().value); // 1

console.log(c2.next().value); // 2

console.log(c2.next(5).value); // 3

console.log(c2.next().value); // 4

console.log(c2.next().value); // 5

The example above is yet another contrived modification to our earlier example. This counter generator accepts an argument as the limit to the number of values it can generate. It has the same loop as the above example, except that the control is now dependent on the limit parameter that was passed to it.

Inside the loop body, we have declared a variable j that is being assigned to the value of yield. This expression is being followed by another control structure: an if statement that checks the value of j. The value of j will replace the value of limit if it has a truthy value.

As I have mentioned prior to showing the examples, we can control the internal state of generators by passing an argument to the next() method. This argument will become the value of yield inside the generator, and as such we can assign it to control its behavior.

This can be seen above where we both declared a generator with an initial limit of 2 values. On the first one, we did not pass an argument to next() and so we were only able to iterate through two values. On the second example, we did the same thing, but we passed in a value of 5 as an argument to next(). This altered the generator's internal limit from two to five values, enabling us to get three more values out of it.

On this post, we have learned about the basics of ES6's generators. We went through the basic implementation and usage through some simple examples. We found out that generator functions are declared using the function* keyword, and contains at least one yield statement/expression. We also found out that a generator produces and iterable with a next() method. Since this post is getting long, I have decided to split this post into two. On my next post, we will explore how to implement basic async/await functionality through the use of generators and promises.

You can try the library out by visiting the demo page.

package main

import "fmt"

type person struct {

  name string

  age  int

}

func (p person) talk() {

  fmt.Printf("Hi, my name is %s and I am %d years old.\n", p.name, p.age)

}

func main() {

  p1 := person{"John Crisostomo", 25}

  p1.talk()

  // prints: "Hi, my name is John Crisostomo and I am 25 years old."

}

Run this code

package main

import (

	"fmt"

)

type creature struct {}

func (c creature) walk() {

  fmt.Println("The creature is walking.")

}

type human struct {

  creature

}

func main() {

  h := human{

    creature{},

  }

  h.walk()

  // prints: "The creature is walking."

}

Run this code

package main

import (

	"fmt"

)

type creature struct {}

func (c creature) walk() {

  fmt.Println("The creature is walking.")

}

type human struct {

  creature

}

func (h human) walk() {

  fmt.Println("The human is walking.")

}

func main() {

  h := human{

    creature{},

  }

  h.walk()

  // prints: "The human is walking."

  h.creature.walk()

  // prints: "The creature is walking."

}

Run this code

package main

import (

	"fmt"

)

type lifeForm interface {

   walk()

}

type creature struct {}

func (c creature) walk() {

  fmt.Println("The creature is walking.")

}

type human struct {

  creature

}

func (h human) walk() {

  fmt.Println("The human is walking.")

}

func performAction(lf lifeForm) {

  lf.walk()

}

func main() {

  c := creature{}

  h := human{

    creature{},

  }

  performAction(c)

  // prints: "The creature is walking."

  performAction(h)

  // prints: "The human is walking."

}

Run this code

This post is not sponsored by Cloudflare; it is an update on my self-hosting journey with the Raspberry Pi.

I am happy with the result of the script that I shared on my last post because I no longer have to manually reboot the Pi every time the Internet connection goes down. However, it is still suboptimal; if the Internet connection goes down for an extended period of time, the blog goes down with it. Not only is it bad for would be readers, it was also frustrating on my end. The thought of moving this blog to a cheap cloud instance crossed my mind during the first few days, but I had to think of something more pragmatic. That was when I decided to check Cloudflare out. When I found out that they are offering a free plan that has more features than what I would need for this blog, I was sold.

Cloudflare is a security company that gained notoriety for stopping DDoS attacks through their Content Delivery Network (CDN)-like feature. It can help your site become more performant by caching your static content in their data centers around the world. This enables your site to load faster and allows more concurrency by serving cached content first before hitting your server. Cloudflare offers this and more for free; including three page rules, analytics, free SSL through their network and even enabling security measures like HTTP Strict Transport Security (HSTS). All of these can be easily configured in their nice looking dashboard. If you want to read more about the company's history, here is a good article about their humble beginning.

Getting a Cloudflare account is straightforward. A walkthrough video of the initial setup process is available on their landing page. In a nutshell, the process only has three steps:

• Signing up with your email address and password
• Adding your domain
• Pointing your domain's nameservers to Cloudflare's own nameservers

After going through those steps quickly, you will be presented with a modern, easy to use admin interface:

It will be impossible to discuss all of what Cloudflare has to offer in a single post, so I will just write about the tweaks that I did to suit my current self-hosted Raspberry Pi setup.

Cypto

I obtained my domain's SSL certificate through Let's Encrypt, a trusted certificate authority that issues certificates for free. Since I have my own certificate configured on NGINX, I do not need to use Cloudflare's free SSL. I just selected Full (Strict) mode under SSL and enabled HSTS, Opportunistic Encryption and Automatic HTTPS Rewrites.

Speed

I enabled Auto Minify for both Javascript and CSS to optimize load times and save on bandwidth. I decided against minifying the HTML to preserve the blog's markup, which in my opinion is important for search engine optimization. I also enabled the Accelerated Mobile Links support for a better mobile reading experience. They also have a Beta feature called Rocket Loader™ (improves the load time of pages with JavaScript), this is off by default, but I decided to give it a try.

Caching

This is the feature that I needed the most. I clicked on this menu before I even explored the other settings above. I made sure Always Online™ is on, and made some minor adjustments with the Browser Cache Expiration.

Page Rules

Cloudflare gives you three page rules for free, and you can subscribe should you need more. Here's how I made use of my free page rules:

Dynamic DNS Configuration

My blog's DNS records are now being handled by Cloudflare so I need to make sure that they are updated automatically if my ISP gives me a new IP address.

The easiest way to achieve this is to install ddclient from Raspbian's default repository, along with the Perl dependencies:

sudo apt-get install ddclient libjson-any-perl

Unfortunately, this version of ddclient does not support Cloudflare's Dynamic DNS API. We need to download the current version here, and overwrite the executable that has been installed by the previous command:

$ wget http://downloads.sourceforge.net/project/ddclient/ddclient/ddclient-3.8.3.tar.bz2

$ tar -jxvf ddclient-3.8.3.tar.bz2

$ cp -f ddclient-3.8.3/ddclient /usr/sbin/ddclient

We installed the old version first to benefit from the daemon that comes with it. This daemon keeps ddclient running in the background and spawns it automatically after each reboot.

This new version of ddclient looks for the configuration file in a different directory so we need to create that directory and move our old configuration file:

$ sudo mkdir /etc/ddclient

$ sudo mv /etc/ddclient.conf /etc/ddclient

Here's my ddclient.conf for reference:

# Configuration file for ddclient generated by debconf

#

# /etc/ddclient.conf

protocol=cloudflare

zone=johncrisostomo.com

use=web

server=www.cloudflare.com

login=*Enter your cloudflare email address here*

password=*Enter your API key here*

blog.johncrisostomo.com

We can now restart ddclient and check its status to make sure that everything is working as expected:

$ sudo service ddclient restart

$ sudo service ddclient status -l

The last command should give you the current status of the daemon along with the latest event logs. Check the event logs for any error messages or warnings, and if everything turned out to be okay, you should see something similar to this: 

SUCCESS: blog.johncrisostomo.com -- Updated Successfully to xxx.xxx.xxx.xxx.

So far this setup works well and I am happy with the blog's performance. It is a shame that I have not gathered data before Cloudflare to objectively compare the performance boost I am getting out of it. However, the blog's initial loading time has become noticeably faster, at least on my end. I guess we will have to see in the next couple of days.

It has been almost a week since I decided to self-host my Ghost blog. It was a fun experience and most importantly, I knew a lot of new things that I would not otherwise know. On the less technical side, it inspired me to write more about my learning journey because not only does it solidify what I already know, it also drives me to learn more.

There is a little problem though. My Internet connection is flaky and it causes my blog to be sporadically down throughout the day. This is not intended to be a for-profit blog, however, seeing people share some of my posts while my blog is down was frustrating. I just had to do something about it. I observed the Pi's behavior by writing several BASH scripts and cron jobs that makes sure these events are logged. Sifting through the logs after work, I found out that aside from the ISP problem, there is another queer phenomenon that was happening. Whenever my home router loses Internet connection, the Raspberry Pi will lose its default gateway; it persists even after rebooting the router.

My initial attempts to fix this issue was to mess with the resolve.conf and /etc/network/interfaces configuration files. I tried everything from manual, dhcpand even static. Nothing really fixed the issue and it was still losing the default gateway route whenever the Internet connection goes down. I finally solved this problem by writing a small BASH script:

#!/bin/bash

ping -c1 google.com > /dev/null

if [ $? != 0 ]

then

  echo `date` "No network connection, restarting wlan0" >> /home/uplogs.txt

  /sbin/ifdown 'wlan0'

  sleep 5

  /sbin/ifup --force 'wlan0'

else

  echo `date` "Internet seems to be up" >> /home/uplogs.txt

fi 

The script pings google.com and then checks the exit code. If the ping exited with an error, the Pi restarts the wireless LAN interface. It also logs all these events so that I can check how reliable my Internet connection was throughout the day. It was a quick and dirty fix. Nothing fancy, but it works.

tmux new -s [session name]

tmux ls

tmux attach -t [session name]

Ctrl - b % 

Ctrl - b "

Ctrl - arrow keys

Ctrl - b z

Ctrl - d

Ctrl - b c

Ctrl - b ,

Ctrl - b n

or

Ctrl -b p

Ctrl - b &

Ctrl - b d

I received a Raspberry Pi 3 Model B last Christmas, but I did not know what to do with. Or at least not yet. The problem has little to do with the Pi and more of the fact that most of the projects that I do can easily be solved with an Arduino.

When I stumbled upon these series of posts by the Docker Captain Alex Ellis, I figured out that this is a perfect opportunity to learn a tool I have always wanted to use. I know virtual machines well, but I had a hard time understanding how to make Docker fit into my workflow. The idea of containers that I cannot simply SSH into (I now know that you can exec bash to peek inside them, but that's not the point), just seemed absurd when I was first trying to use it. To be honest it felt too complex and cumbersome that I just dismissed it as something that was not worth it. Well, it turned out that I did not understand the philosophy behind it. I would like to talk about it and discuss images and containers in depth, but I decided that it will be better to have a dedicated post for that. After getting my hands dirty with Docker last weekend, I can say that I have attained a working proficiency with it and I can comfortably use it for my projects from here on.

After three days, I finally got it to work. The blog that you are reading right now is hosted on a Raspberry Pi with Docker Engine installed. I have two Docker containers running: the Ghost blog and the NGINX server that handles the caching. It took me a lot of trial and errors before I finally got it to work; I do not have any prior knowledge of NGINX when I embarked on this weekend project. The Pi's limited hardware made building images painstakingly slow. Building SQLite3 from source for the ARM architecture was excruciating.

I will be sharing my Dockerfiles and some configuration below. I won't go into more detail right now, but I am hoping that I will have the time to do so in my next post. Some of these are directly forked/copied from [Alex]((http://blog.alexellis.io)'s GitHub repositories; I could have pulled the images from Docker Hub or cloned the Dockerfiles but I decided to train my muscle memory by typing the Dockerfiles manually. I still have a lot to learn about NGINX and Docker in particular, but I consider this blog as a milestone.

Ghost Dockerfile

FROM alexellis2/node4.x-arm:latest

USER root

WORKDIR /var/www/

RUN mkdir -p ghost

RUN apt-get update && \

    apt-get -qy install wget unzip && \

    wget https://github.com/TryGhost/Ghost/releases/download/0.11.4/Ghost-0.11.4.zip && \

    unzip Ghost-*.zip -d ghost && \

    apt-get -y remove wget unzip && \

    rm -rf /var/lib/apt/lists/*

RUN useradd ghost -m -G www-data -s /bin/bash

RUN chown ghost:www-data .

RUN chown ghost:www-data ghost

RUN chown ghost:www-data -R ghost/*

RUN npm install -g pm2

USER ghost

WORKDIR /var/www/ghost

RUN /bin/bash -c "time (npm install sqlite3)"

RUN npm install

EXPOSE 2368

EXPOSE 2369

RUN ls && pwd

ENV NODE_ENV production

RUN sed -e s/127.0.0.1/0.0.0.0/g ./config.example.js > ./config.js

CMD ["pm2", "start", "index.js", "--name", "blog", "--no-daemon"]

Blog Dockerfile

FROM johncrisostomo/ghost-on-docker-arm:0.11.4

ADD Vapor /var/www/ghost/content/themes/Vapor

RUN sed -i s/my-ghost-blog.com/blog.johncrisostomo.com/g config.js

NGINX Dockerfile

FROM resin/rpi-raspbian:latest

RUN apt-get update && apt-get install -qy nginx

WORKDIR /etc/nginx/

RUN rm /var/www/html/index.nginx-debian.html && \

    rm sites-available/default && \

    rm sites-enabled/default && \

    rm nginx.conf

COPY nginx.conf /etc/nginx/

COPY johncrisostomo.com.conf conf.d/

EXPOSE 80

CMD ["nginx", "-g", "daemon off;"]

JOHNCRISOSTOMO.COM.CONF

server {

  listen 80;

  server_name blog.johncrisostomo.com;

  access_log /var/log/nginx/blog.access.log;

  error_log /var/log/nginx/blog.error.log;

  location / {

    proxy_cache              blog_cache;

    add_header X-Proxy-Cache $upstream_cache_status;

    proxy_ignore_headers     Cache-Control;

    proxy_cache_valid any    10m;

    proxy_cache_use_stale    error timeout http_500 http_502 http_503 http_504;

    proxy_set_header  X-Real-IP $remote_addr;

    proxy_set_header  Host      $http_host;

    proxy_pass        http://blog:2368;

  }

}

docker-compose.yml

version: "2.0"

services:

  nginx:

    ports:

      - "80:80"

    build: "./nginx/"

    restart: always

  blog:

    ports:

      - "2368:2368"

    build: "./blog.johncrisostomo.com/"

    volumes:

      - ghost_apps:/var/www/ghost/content/apps

      - ghost_data:/var/www/ghost/content/data

      - ghost_images:/var/www/ghost/content/images

      - ghost_themes:/var/www/ghost/content/themes

    restart: always

volumes:

   ghost_apps:

      driver: local

   ghost_data:

      driver: local

   ghost_images:

      driver: local

   ghost_themes:

      driver: local

I have written several follow up posts about this project. Feel free to check them out as most of them are troubleshooting issues and optimizations that are built on top of this project.

• Troubleshooting my Raspberry Pi's Wireless Issue
• Enhancing my self-hosted blog with Cloudflare

Gagarin is a mocha based testing framework designed to be used with Meteor. It can spawn multiple instances of your meteor application and run the tests by executing commands on both server and client in realtime. In other words, Gagarin allows you to automate everything you could achieve manually with meteor shell, browser console and a lot of free time. There’s no magic. It just works.

npm install -g gagarin

gagarin

const Categories = new Mongo.Collection(‘categories’);

Meteor.publish(‘categoriesList’, () => {

  return Categories.find()

});

Meteor.publish(‘categoriesOwnedBy’, (owner) => {

  check(owner, String);

  return Categories.find({owner: owner});

});

Meteor.methods({

  ‘categoriesAdd’(data) {

     check(data, Object);

     Categories.insert({

       name : data.name,

       owner: data.owner,

       createdAt : new Date(),

     });

   },

   ‘categoriesUpdate’(data) {

     check(data, Object);

     Categories.update({_id:data._id},{$set:{

       name : data.name, 

       owner: data.owner,

       modifiedAt : new Date(),

     }});

   },

});

describe(‘Categories’, function() {

  var app = meteor({flavor: “fiber”});

  var client = ddp(app, {flavor: “fiber”});

  it(‘should be able to add’, function() {

    client.call(‘categoriesAdd’, [{name: ‘First category’}]);

    client.sleep(200);

    client.subscribe(‘categoriesList’);

    var categories = client.collection(“categories”);

    expect(Object.keys(categories).length).to.equal(1);

 }); 

}

{ Hpu6Z4h7ZFtC6Q77m: 

 { _id: ‘Hpu6Z4h7ZFtC6Q77m’,

 name: ‘First category’,

 owner: null,

 modifiedAt: 2016–06–07T08:29:06.026Z,

 createdAt: 2016–06–07T08:29:06.026Z } }

it(‘should be able to update’, function() {

  client.subscribe(‘categoriesList’);

  var categories = client.collection(“categories”);

  var id = Object.keys(categories)[0];

  client.call(‘categoriesUpdate’, [{_id:id, name: ‘updated    category’}]);

 client.sleep(200);

  categories = client.collection(“categories”);

  expect(categories[id].name).to.equal(‘updated category’);

});

describe(‘categoriesOwnedBy publication’, function() {

  var app = meteor({flavor: “fiber”});

  var client = ddp(app, {flavor: “fiber”});

  it(‘should only publish a specific users category’, function() {

    app.execute(function() {

      var categories = [

        { name: ‘Category 1’, owner: ‘John’ },

        { name: ‘Category 2’, owner: ‘Jessica’ },

        { name: ‘Category 3’, owner: ‘John’ }

      ];

      Meteor.call(‘categoriesAdd’, categories[0]);

      Meteor.call(‘categoriesAdd’, categories[1]);

      Meteor.call(‘categoriesAdd’, categories[2]);

   });

  client.subscribe(‘categoriesOwnedBy’, [‘John’]);

  var johnsCategories = client.collection(‘categories’);

  expect(Object.keys(johnsCategories).length).to.equal(2);

});

With the release of Meteor 1.3, unit testing has never been easier in Meteor. Our team recently decided to adopt Arunoda’s Mantra spec for developing Meteor applications. It is an application architecture which allows for a more modular approach with clear separation of concerns with the client and the server side. It has been a rough month for us since the spec is new and there are limited learning resources available. There were also a lot of things to learn since adopting the Mantra spec meant that we have to learn React for the presentation logic, instead of sticking with Blaze.

Unit testing is something that can be easily accomplished with the Mantra spec. Since it is modular and it clearly separates the presentation logic from the business logic through containers, components and actions, everything can be unit tested. Meteor 1.3 also introduced native NPM support, which means that using familiar tools such as Mocha, Chai and Sinon can be imported in a straightforward manner. This is going to be my first blog post and so I am only going to explain Sinon’s spy and stub methods as they were used in Arunoda’s mantra-sample-blog application.

Spies

People who are new to unit testing tools in JavaScript (with me included) were initially confused about the difference between Sinon’s spy and stub. It turns out that a spy is a basic function that you can use in Sinon, and that stubs and mocks were built on top of it.

According to the Sinon.JS documentation, a spy is

a function that records arguments, return value, the value of this and exception thrown (if any) for all its calls. A test spy can be an anonymous function or it can wrap an existing function

What this means is that a spy can be used as a replacement for an anonymous callback function or you can wrap an existing function into it so that you can spy on its behavior. For example, if you have a function that accepts another function as an argument to be called back later after a certain condition, you can instead pass Sinon’s spy() function as the callback. You can then assert or use Chai’s expect() to see if that function was called by using spy()’s calledOnce(). Additionally, you can check if the correct arguments were passed to it by using its calledWith(). You can check the different spy functions that are available here.

Now let’s look at how spies are used in the Mantra sample blog application. We are going to use the tests that were written for the posts action. Let’s check this code snippet out:

it('should call Meteor.call to save the post', () => {

  const Meteor = {uuid: () => 'id', call: spy()};

  const LocalState = {set: spy()};

  const FlowRouter = {go: spy()};

  actions.create({LocalState, Meteor, FlowRouter}, 't', 'c');

  const methodArgs = Meteor.call.args[0];

  expect(methodArgs.slice(0, 4)).to.deep.equal([

    'posts.create', 'id', 't', 'c'

  ]);

  expect(methodArgs[4]).to.be.a('function');

});

In the test block above, we are testing if our action will correctly invoke a Meteor Method in the server through Meteor.call(). The first three lines are creating local objects for Meteor, LocalState and FlowRouter to be used exclusively in this test case. In the Mantra spec, these are exported as the application context inside client/configs/context.js.

Actions in Mantra receive this application context as the first parameter. We are creating local objects in lieu of the real app context and by doing so, we can trick the action into thinking that it is receiving its first expected argument (which is the app context).

Next, we spy on how these objects are used inside the action that we are testing. See how the Meteor object that we have passed contains a call property, which is a Sinon’s spy() function? When the action gets invoked on Line 6, it will go ahead and invoke Meteor.call() inside it. The Meteor object that it will receive is something that we have just created for spying purposes, so we have access to the arguments that were passed into it when it was invoked (Lines 7 — 12). We used the arguments that we have obtained through spying to verify that our function is invoking Meteor.call() with the correct arguments.

This is what the create action looks like, for reference:

  create({Meteor, LocalState, FlowRouter}, title, content) {

    if (!title || !content) {

      return LocalState.set('SAVING_ERROR', 'Title & Content are required!');

    }

    LocalState.set('SAVING_ERROR', null);

    const id = Meteor.uuid();

    Meteor.call('posts.create', id, title, content, (err) => {

      if (err) {

        return LocalState.set('SAVING_ERROR', err.message);

      }

    });

    FlowRouter.go(`/post/${id}`);

  }

Stubs

Now that we are done with spies, and have a basic understanding on how they work, let’s work with stubs. Stubs are just like spies, and in fact they have the entire spy() API inside them, but they can do more than just observe a function’s behavior. According to the Sinon API, stubs are:

functions (spies) with pre-programmed behavior. They support the full test spy API in addition to methods which can be used to alter the stub’s behavior.

and they should be used when you want to:

Control a method’s behavior from a test to force the code down a specific path. Examples include forcing a method to throw an error in order to test error handling.

or

When you want to prevent a specific method from being called directly (possibly because it triggers undesired behavior, such as a XMLHttpRequest or similar).

Okay, so where spies can observe and spy on how a function is going to be called, the number of times it’s going to be called and the arguments that were sent with it, stubs can do all these, plus you can also programmatically control its behavior.

Let’s check how it is used on the post action test inside the sample blog application:

    it('should call Meteor.call to save the post', () => {
      const Meteor = {uuid: () => 'id', call: spy()};
      const LocalState = {set: spy()};
      const FlowRouter = {go: spy()};

      actions.create({LocalState, Meteor, FlowRouter}, 't', 'c');
      const methodArgs = Meteor.call.args[0];

      expect(methodArgs.slice(0, 4)).to.deep.equal([
        'posts.create', 'id', 't', 'c'
      ]);
      expect(methodArgs[4]).to.be.a('function');
    });

This particular test will check if our action will set an error message if something goes wrong after the Meteor Method call. Just like what we did with the spy example, we are setting local objects to be used as the context for our action function.

In Mantra, the LocalState is a Meteor reactive-dict data structure (a reactive dictionary) which is mainly used to handle the client side state of the app, although it is mostly used to store temporary error messages. We are creating a LocalState object here to mimic the app context’s LocalState. We are setting its set property as a spy function, so we can later see if our action will set the appropriate error message by checking the arguments that were passed into it.

Notice that this time, we are using a stub() instead of a spy() for our local Meteor object. The reason for this is that we are no longer just observing how it is going to be called, but we are also forcing it to respond in a specific way.

We are checking our action’s behavior once the call to a remote Meteor method returns an error and if that error will be stored in the LocalState accordingly. In order to do that, we need to reproduce that behavior, or make the call() function in our local Meteor object return an error. That is something that a spy() will not be able to do since it can only observe. For this scenario, we will use the stub’s callsArgWith()* function to set our desired behavior (Line 6).

We will give callsArgWith() two arguments: 4 and the err object that we have defined in Line 5. This function will make our stub invoke the argument at index 4, passing the err as an argument to whatever function is at that position. If you are going to look at our create action above, Meteor.call() is invoked with five arguments, the last or the one in the fourth index is a callback function:

Meteor.call(‘posts.create’, id, title, content, (err) => { 

  if (err) { 

    return LocalState.set(‘SAVING_ERROR’, err.message); 

  } 

});

We have to remember that this Meteor.call() that is being invoked here is the local object that we have created and passed explicitly into our create action for testing purposes. As such, this is the stub in action, and it doesn’t know that the last argument when it is invoked is going to be a callback function, so we have to use the callsArgWith() with the err object. Inside this callback, the create action will then store the error message in the LocalState object that we have passed in. Since the set() function of that LocalState object is a spy, we can conclude our test by checking if the arguments that were passed to this spy function matches the error message that we are expecting (Line 9).

This wraps up our discussion of how Sinon’s Spy and Stubs methods are being used in Mantra Unit Testing. As a recap, a spy just observes a certain function, or can take the place of a anonymous callback function so we can observe its behavior. A stub does more than that, by allowing us to pre-program a function’s behavior. If I have provided any wrong information, please feel free to correct me in the comments. :)

*The callsArg and yields family of methods have been removed as of Sinon 1.8. They were replaced with the onCall API.

Moblin (short for Mobile Linux) is a new Linux distribution that is designed by Intel to support multiple platforms and usage models ranging from Netbooks to Mobile Internet Devices (MID), to various embedded usage models, such as the In Vehicle Infotainment systems.

Moblin 2.1 was released recently and you could check out the screenshots hereor watch the intro video here for a quick look on how the Moblin 2.1 Netbook looks like. You could check for tested netbook models here. The full release note and download link could be found here.

Looks promising, but the problem is, as with any other Linux distributions, it does not play mp3 and other proprietary codecs out of the box for legal reasons. It only plays Ogg Vorbis audio and Ogg Theora video upon installation and the Gstreamer packages needed to play mp3 and other video codecs are not available from Moblin's repository or from the Moblin Garage. So we have to compile these packages from source.

DISCLAIMER: Try this at your own risk.

Step 1: Download the souce code here. We need the following source codes:

gst-ffmpeg-0.10.9.tar.bz2

gst-plugins-bad-0.10.16.tar.bz2

gst-plugins-base-0.10.25.tar.bz2

gst-plugins-good-0.10.16.tar.bz2

gst-plugins-ugly-0.10.13.tar.bz2

gstreamer-0.10.25.tar.bz2

After installing these, extract them to a directory of your choice (ex. /Home/Downloads).

Step 2: Open the terminal and type this command to download and install necessary development tools and build packages:

yum install gcc bison flex *glib* *diff* liboil*dev*

Step 3: Compile and build the source code. Using the Terminal, use the cd command to navigate to the folder where you have extracted the downloaded source codes (ex. cd /Home/Downloads). Then type these commands in order (press Enter after each line):

cd gstreamer-0.10.25

./configure -prefix=/usr && make && make install

cd ..

cd gst-plugins-base-0.10.25

./configure -prefix=/usr && make && make install

cd ..

cd gst-plugins-good-0.10.16

./configure -prefix=/usr && make && make install

cd ..

cd gst-plugins-bad-0.10.16

./configure -prefix=/usr && make && make install

cd ..

cd gst-plugins-ugly-0.10.13

./configure -prefix=/usr && make && make install

cd ..

cd gst-ffmpeg-0.10.9

./configure -prefix=/usr && make && make install 

Then reboot and have fun with your media. I might write an automated script later to do all of these upon execution of the script, but I'm busy at the moment.

sudo yum install fusion-icon

xfce4-autostart-editor

sudo yum remove evolution-data-server libpurple

cd /usr/acer/bin

sudo ln -s /usr/bin/thunderbird AME

sudo ln -s /usr/bin/pidgin UIM

sudo mousepad /usr/share/applications/AME.desktop

wget "http://download.mozilla.org/?product=thunderbird-2.0.0.17&os=linux&lang=en-US"

sudo tar -xvf thunderbird-2.0.0.17.tar.gz --directory /opt

sudo chown user -R /opt/thunderbird

sudo mousepad /usr/share/applications/AME.desktop

Exec=/opt/thunderbird/thunderbird

sudo cp /home/user/Downloads/thunderbird.png /usr/share/pixmaps