DataPipe

Contents

• Overview
• Example
• Construction
• Methods
• Type Coercion

Overview

A DataPipe offers a way of piping and transforming items from one DataSet into another DataSet. It can be used to coerce data types, change structure or even generate brand new items based on some data specific to your problem.

Example

The following example shows how to use a DataPipe to divide app specific and Vis specific data to prevent name collisions (like having app specific and Vis specific labels).

// TypeScript interfaces to illustrate the structure of the data:
//
// interface appDSItem {
//   id: number | string;
//
//   appLabel: string;
//   appPosition: number;
//
//   visLabel: string;
//   visColor: string;
//   visX: number;
//   visY: number;
// }
//
// interface VisDSItem {
//   id: number | string;
//   label: string;
//   color: string;
//   x: number;
//   y: number;
// }

const appDS = new DataSet([
  /* some app data */
]);
const visDS = new DataSet();

const pipe = createNewDataPipeFrom(appDS)
  // All items can be arbitrarily transformed.
  .map((item) => ({
    id: item.id,
    label: item.visLabel,
    color: item.visColor,
    x: item.visX,
    y: item.visY
  }))
  // This builds and returns the pipe from appDS to visDS.
  .to(visDS);

pipe.all().start();
// All items were transformed and piped into visDS and all later changes
// will be transformed and piped as well.

Construction

A DataPipe can be constructed as:

const pipe = createNewDataPipeFrom(sourceDataSet)
  .filter(item => item.enabled === true)
  .map(item => ({
    /* some new item */
  }))
  .flatMap(item => [
    /* zero or more new items */
  ])
  .to(targetDataSet);

where:

• sourceDataSet is a DataSet or DataView that will supply the items into the pipe line.
• targetDataSet is a DataSet that will receive the processed items from the pipe line.
• .filter's argument is a function that takes an item from the pipe as it's argument and returns true if the item should be processed further down the pipe line or false if it should be skipped.
• .map's argument is a function that takes an item from the pipe as it's argument and returns a new item that will be processed further down the pipe line.
• .flatMap's argument is a function that takes an item from the pipe as it's argument and returns an array of zero or more items that will be processed further down the pipe line.

Note that the items are passed through the pipe in the order in which the factory methods (.filter, .map and .flatMap) were called. Also note that it's possible to use the same method multiple times (for example .filter, .map and .filter again) or only use some of them (for example .filter and .map but no .flatMap).

Methods

DataPipe contains the following methods.

Method	Return Type	Description
all()	this	Sends all the data from the source DataSet or DataView down the pipe line.
start()	this	Starts observing the changes in the source DataSet or DataView and sending them down the pipe line as they happen.
stop()	this	Stops observing changes in the source DataSet or DataView.

Type Coercion

DataPipe can be used as a replacement for the now deprecated type coercion of DataSets (configured by the type constructor parameter).

// --[BEGIN]-- The original implementation of type coercion.

// !!!! Make sure to import Moment.js as this code depends on it.
const convert = (() => {
  // parse ASP.Net Date pattern,
  // for example '/Date(1198908717056)/' or '/Date(1198908717056-0700)/'
  // code from http://momentjs.com/
  const ASPDateRegex = /^\/?Date\((-?\d+)/i;

  /**
   * Test whether given object is a number
   *
   * @param value - Input value of unknown type.
   *
   * @returns True if number, false otherwise.
   */
  function isNumber(value) {
    return value instanceof Number || typeof value === "number";
  }

  /**
   * Test whether given object is a string
   *
   * @param value - Input value of unknown type.
   *
   * @returns True if string, false otherwise.
   */
  function isString(value) {
    return value instanceof String || typeof value === "string";
  }

  /**
   * Get the type of an object, for example exports.getType([]) returns 'Array'
   *
   * @param object - Input value of unknown type.
   *
   * @returns Detected type.
   */
  function getType(object) {
    const type = typeof object;

    if (type === "object") {
      if (object === null) {
        return "null";
      }
      if (object instanceof Boolean) {
        return "Boolean";
      }
      if (object instanceof Number) {
        return "Number";
      }
      if (object instanceof String) {
        return "String";
      }
      if (Array.isArray(object)) {
        return "Array";
      }
      if (object instanceof Date) {
        return "Date";
      }

      return "Object";
    }
    if (type === "number") {
      return "Number";
    }
    if (type === "boolean") {
      return "Boolean";
    }
    if (type === "string") {
      return "String";
    }
    if (type === undefined) {
      return "undefined";
    }

    return type;
  }

  /**
   * Convert an object into another type
   *
   * @param object - Value of unknown type.
   * @param type - Name of the desired type.
   *
   * @returns Object in the desired type.
   * @throws Error
   */
  return function convert(object, type) {
    let match;

    if (object === undefined) {
      return undefined;
    }
    if (object === null) {
      return null;
    }

    if (!type) {
      return object;
    }
    if (!(typeof type === "string") && !(type instanceof String)) {
      throw new Error("Type must be a string");
    }

    //noinspection FallthroughInSwitchStatementJS
    switch (type) {
      case "boolean":
      case "Boolean":
        return Boolean(object);

      case "number":
      case "Number":
        if (isString(object) && !isNaN(Date.parse(object))) {
          return moment(object).valueOf();
        } else {
          // @TODO: I don't think that Number and String constructors are a good idea.
          // This could also fail if the object doesn't have valueOf method or if it's redefined.
          // For example: Object.create(null) or { valueOf: 7 }.
          return Number(object.valueOf());
        }
      case "string":
      case "String":
        return String(object);

      case "Date":
        if (isNumber(object)) {
          return new Date(object);
        }
        if (object instanceof Date) {
          return new Date(object.valueOf());
        } else if (moment.isMoment(object)) {
          return new Date(object.valueOf());
        }
        if (isString(object)) {
          match = ASPDateRegex.exec(object);
          if (match) {
            // object is an ASP date
            return new Date(Number(match[1])); // parse number
          } else {
            return moment(new Date(object)).toDate(); // parse string
          }
        } else {
          throw new Error(
            "Cannot convert object of type " + getType(object) + " to type Date"
          );
        }

      case "Moment":
        if (isNumber(object)) {
          return moment(object);
        }
        if (object instanceof Date) {
          return moment(object.valueOf());
        } else if (moment.isMoment(object)) {
          return moment(object);
        }
        if (isString(object)) {
          match = ASPDateRegex.exec(object);
          if (match) {
            // object is an ASP date
            return moment(Number(match[1])); // parse number
          } else {
            return moment(object); // parse string
          }
        } else {
          throw new Error(
            "Cannot convert object of type " + getType(object) + " to type Date"
          );
        }

      case "ISODate":
        if (isNumber(object)) {
          return new Date(object);
        } else if (object instanceof Date) {
          return object.toISOString();
        } else if (moment.isMoment(object)) {
          return object.toDate().toISOString();
        } else if (isString(object)) {
          match = ASPDateRegex.exec(object);
          if (match) {
            // object is an ASP date
            return new Date(Number(match[1])).toISOString(); // parse number
          } else {
            return moment(object).format(); // ISO 8601
          }
        } else {
          throw new Error(
            "Cannot convert object of type " +
              getType(object) +
              " to type ISODate"
          );
        }

      case "ASPDate":
        if (isNumber(object)) {
          return "/Date(" + object + ")/";
        } else if (object instanceof Date || moment.isMoment(object)) {
          return "/Date(" + object.valueOf() + ")/";
        } else if (isString(object)) {
          match = ASPDateRegex.exec(object);
          let value;
          if (match) {
            // object is an ASP date
            value = new Date(Number(match[1])).valueOf(); // parse number
          } else {
            value = new Date(object).valueOf(); // parse string
          }
          return "/Date(" + value + ")/";
        } else {
          throw new Error(
            "Cannot convert object of type " +
              getType(object) +
              " to type ASPDate"
          );
        }

      default:
        throw new Error(`Unknown type ${type}`);
    }
  };
})();

// --[END]-- The original implementation of type coercion.

const rawDS = new DataSet([
  /* raw data with arbitrary types */
  { id: 7, label: 4, date: "2017-09-04" },
  { id: false, label: 4, date: "2017-10-04" },
  { id: "test", label: true, date: "2017-11-04" }
]);
const coercedDS = new DataSet(/* the data with coerced types will be piped here */);

const types = {
  id: "string",
  label: "string",
  date: "Date"
};

const pipe = createNewDataPipeFrom(rawDS)
  .map(item =>
    Object.keys(item).reduce((acc, key) => {
      acc[key] = convert(item[key], types[key]);
      return acc;
    }, {})
  )
  .to(coercedDS);

pipe.all().start();
// All items were transformed and piped into visDS and all later changes
// will be transformed and piped as well.