Learn Gulp, Jq and Nightmare

In this series we will take a look at Gulp. Gulp is a task runner. It is the commander of our army of helper robots that take critical, yet totally mundane, tasks and perform them hundreds of times per day so we don't have to. When we sit down to work, we want to enter a single command, and then spend the rest of our day writing code, testing, and basically not thinking about how it is all glued together in the background.

Automation is critical to any project of significant size. We will start with the Eggly Bookmarks application, and develop proper build automation that will include **production ready** tools that you can start using in your applications today!

AngularJS is an open-source JavaScript framework, maintained by Google, that assists with running single-page applications. Its goal is to augment web-based applications with model–view–controller (MVC) capability, in an effort to make both development and testing easier.

angularjs

Angular Automation with Gulp

jq is a lightweight and flexible command-line JSON processor.

jq can help you quickly filter massive amounts of JSON data, but how can you turn it into a CSV, so you can process it quickly with a database or spreadsheet application?

This lesson shows how to select the data you want for your CSV with jq, and uses the helper utility linked below to output as CSV.

https://github.com/joelpurra/jq-hopkok/tree/master/src/tabular

Convert JSON to CSV with jq

In this lesson, we are going to learn how to create a config file to store common values so that we can reuse them multiple times in our `gulpfile.js`. As our `gulpfile.js` grows in complexity, we may find that we are introducing complex and multiple file path references. By creating a `gulp.config.js` file, we can simplify these references and make it easier to refactor and extend in the future. 

Angular Automation: Gulp Config

In this lesson, we are going to learn how to use `gulp.watch` to detect local file changes and execute additional Gulp tasks. To illustrate this concept, we are also going to hook up `gulp-jshint` to automation lint our JavaScript as we make changes to our code.

Angular Automation: Gulp Watch

Angular projects require automation. Period. 

In this series we will take a look at Gulp. Gulp is a task runner. It is the commander of our army of helper robots that take critical, yet totally mundane, tasks and perform them hundreds of times per day so we don't have to. When we sit down to work, we want to enter a single command, and then spend the rest of our day writing code, testing, and basically not thinking about how it is all glued together in the background.

Automation is critical to any project of significant size. We will start with the Eggly Bookmarks application, and develop proper build automation that will include **production ready** tools that you can start using in your applications today!

Angular Automation: Introduction to Gulp

In this lesson, we are going to learn how to copy assets from one place to another using Gulp. This is important as you start to prepare your source files to be deployed into production. We will learn how to implement `gulp.src`, `gulp.pipe` and `gulp.dest` as well as the **del** plugin as we copy our files from the source directory into a build directory.

Angular Automation: Copy Assets with Gulp

In this lesson, we are going to learn how to use the `gulp-inject` plugin to sanely and efficiently manage our file references with our HTML. Manually declaring JavaScript references will quickly get out of hand as your application grows in size and complexity. The `gulp-inject` plugin solves this problem by taking a stream of files, converting them into string references and then injecting those references into your HTML via annotations. 

Angular Automation: Gulp Inject

In this lesson, we are going to learn how to automatically serve our project via an HTTP server using `gulp-serve`. Being able to serve our project from an HTTP server allows us to bypass a myriad of development issues such as loading remote source files.

Angular Automation: Gulp Serve

In this lesson, we will learn how tasks work in Gulp. We will talk about how to define a task and what each parameter in the definition does. We will see how task dependencies work and talk about how to handle asynchronous issues using `run-sequence`.

Angular Automation: Gulp Tasks

In this lesson, we learn how to get up and running with Gulp by initializing our project with ```npm init``` and then installing the Gulp CLI globally and then installing it locally to our project. We finish the lesson off by creating our first Gulp file and running our first Gulp task.

Angular Automation: Installing Gulp Globally and Locally

Browserify lets you require('modules') in the browser by bundling up all of your dependencies.

browserify

Building upon the `watch` task that we begin this lesson with, we’ll look at how to create a local development server using the `app` directory as the web root. By using Browsersync to achieve this, we can also instruct all connected browsers to automatically reload each time our bundle is re-generated. Browsersync has a public `.stream()` method that is designed exactly for this purpose.

Gulp and Browserify - Adding Live Reload with Browsersync

Nightmare is a high-level browser automation library.

The goal is to expose just a few simple methods, and have an API that feels synchronous for each block of scripting, rather than deeply nested callbacks. It's designed for automating tasks across sites that don't have APIs.

nightmare

Many pages you'll want to grab data from will require you to log in first. This video demonstrates how to log in and grab whatever data off the page you will need.

Scraping Data from Sites with Login Forms with Nightmare

Gulp is a toolkit for automating painful or time-consuming tasks in your development workflow, so you can stop messing around and build something.

gulp

What is already a nice workflow using Gulp, Browserify, Watchify & Browsersync to enable auto rebuilds & live browser reloading can be further improved by adding Babelify. A simple wrapper around the popular transpiler Babel, Babelify can ‘transform’ source files as they pass through our stream to enable certain Javascript features not yet available in the Browser or Node. We end the lesson by looking at how to generate & extract sourcemaps using `exorcist`.

Gulp and Browserify - Adding Babel & Source Maps

jq is a lightweight and flexible command-line JSON processor. It is extremely useful for processing data from APIs to better understand the data your code will be consuming, before you write the code. In this lesson I introduce you to jq and show you the basic features it provides using the Star Wars API (http://swapi.co) as our data source. Topics covered include filtering, piping, object construction and data manipulation.

Using jq to parse the Star Wars API

Many websites have more than just simple static content. Dynamic content which is rendered by JavaScript requires browser to be able to scrape data. This video demonstrates how to use Nightmare (which is a wrapper around Electron) to launch a url and scrape dynamic data.

Scraping Dynamic JavaScript Websites with Nightmare

Let's look at the basics of setting up Gulp and Browserify. Creating a Gulp task from scratch, we'll cover how to utilise the Browserify API to generate a single `bundle.js` file, catch and log any errors that occur in the compilation process, transform the stream into a format that can be consumed by `gulp.dest` and finish by writing the bundle to a `dist` directory.

Gulp and Browserify - Initial Setup

A brief introduction to using GulpJS in your JavaScript application and creating your first Gulp task.

Installing and Writing a Default Task with Gulp

Beginning with a single Gulp task capable of generating a one-time Browserify bundle, we enable file-watching and incremental rebuilds by incorporating Watchify. We add a second task `watch` & refactor our initial code to enable the bulk of it to be re-used. We’ll look at exactly how to ‘wire’ these tools together and how the relationship between Watchify & Browserify can manifest into a seriously efficient workflow.

Gulp and Browserify - Hooking up Watchify

We will look at bundling your JavaScript files using Gulp, Gulp-Concat, and Gulp-Uglify

Bundling Your JavaScript Files with Gulp

Bash, unfortunately, doesn’t ship with a command that can work with JSON natively. In this lesson, we’ll learn how to read and do basic queries on JSON with `jq`, an installable command line tool that makes working with JSON in bash really easy. We'll pipe the JSON output of the Github API to jq to extract values. 

Note: jq has to be installed. On macOS, the easiest way to do this is to run `brew install jq`.  To view install instructions for all platforms, see https://stedolan.github.io/jq/download/. 

Read and Use JSON in Bash with jq

In this lesson, we'll apply our `jq` skills and write a script that gets a list of dependencies from package.json and greps for usages of every package. If a package is unused, it's marked for removal. 

Topics

Instructors

Content Type