<img height="1" width="1" src="https://www.facebook.com/tr?id=2464304033899319&amp;ev=PageView &amp;noscript=1">

Graphs Are Everywhere - Easily Integrate Them With Ardoq

30 Mar 2020

by Tobias Laundal

After working with Ardoq for a while, you will start to see graphs where you previously only saw lists and flat data. To get the best value, you need to have all of your data in the platform.

Ardoq comes with many out-of-the-box integrations to get you started. However, we understand that everyone has unique needs - So we built a software development kit that makes it super easy to write integrations for any graph data you can find.

Ardoq’s Javascript Software Development Kit

Ardoq’s javascript software development kit, ardoq-sdk-js, does all the heavy lifting of writing an integration. The only thing it needs is a simple representation of how the graph should look in Ardoq and a field for consolidating existing data. The SDK takes care of everything, from creating the needed component and reference types to calculating the changes, and only updating what needs updating.

Internally at Ardoq, we have used the SDK to push our codebase, reactive streams, and analytics infrastructure to Ardoq. Because the SDK only updates what has changed, we can annotate the data by applying human labels and connecting the data to other parts of the business. This allows us to connect what was previously separated in different silos.

Tarantino’s Favorites

To showcase just how easy it is to write an integration using ardoq-sdk-js, we will integrate with The Movie Database and look at the cast of Quentin Tarantino's movies. The question we will answer is: Which actor has appeared in most of Tarantino’s films?

We have included the code for the integration here in this post. There is, of course, a tiny bit of set up behind the scenes, such as getting authentication tokens for Ardoq and TMDb and creating the workspaces in Ardoq, but once that is complete, we can dive into the code that does the work.

An integration like this needs to do three things

  1. Fetch the necessary data
  2. Find a graph representation of the data
  3. Push the data to Ardoq

Our software development kit takes care of point 3 completely and makes point 2 super simple.

The only thing we need to know in order to fetch the data from TMDb is that Quentin Tarantino has person ID 138 in their database. We can get all the movies he has worked on with a simple request to TMDb’s person/:person/movie_credits endpoint. As you will see from the code, we think of our data as a graph from the get-go, by representing Tarantino himself as a component.

 const components: Record<stringSimpleComponent<{}>> = {};

 const references: SimpleReference<{ job: string }>[] = [];

 const director: SimpleComponent<{}> = {

   name: "Quentin Tarantino",

   customId: "138",

   workspace: "people",

   type: "Director"

 };

 components[director.customId] = director;

 

 const response = await fetch(

   `${TMDB_URL}person/${director.customId}/movie_credits?api_key=${TMDB_KEY}`

 );

 const { crew } = await response.json();

 

Next, we simply loop over the films Tarantino has been crew in, filter for the ones he’s directed, and create a component and reference for each one. We then get the full cast of the movie, and create a component and a reference for each of the actors.

 for (const work of crew) {

   if (work.job !== "Director"continue;

   const customMovieId = "m" + work.id;

   components[customMovieId] = {

     name: work.original_title,

     customId: customMovieId,

     workspace: "movies",

     type: "Movie"

   };

   references.push({

     customId: work.credit_id,

     source: director.customId,

     target: customMovieId,

     type: "Directed"

   });

 

   const response = await fetch(

     `${TMDB_URL}movie/${work.id}/credits?api_key=${TMDB_KEY}`

   );

   const { cast } = await response.json();

   for (const actor of cast) {

     const customActorId = "a" + actor.id;

     components[customActorId] = {

       name: actor.name,

       customId: customActorId,

       workspace: "people",

       type: "Actor"

     };

     references.push({

       customId: `${actor.credit_id}`,

       source: customActorId,

       type: "Plays in",

       target: customMovieId

     });

   }

 }

 

This is where we get to the magic. We have built up a static representation of the graph, as we want it to look in Ardoq. The sync function from ardoq-sdk-js will then take this representation and update the graph in Ardoq to match it. Existing components will not be recreated, and changed components will be updated in-place rather than being replaced.

 sync(

   { url: "https://home.ardoq.com/api/", authToken: "<snip>", org: "home" },

   { movies: "ddfb94b8fcdee541071a283c", people: "61fb1ebef6889fb2e373570c" },

   { components: Object.values(components), references }

 );

 

And this is all we need! After running the script, which will only take a few seconds, we can look at the data in Ardoq.

Blog_Post_GraphsAreEverywhereEasilyIntegrateThemWithArdoq_Image_1

Looking at this all at once makes it hard to gather any insights. When we add a calculated field for the number of movies each actor plays in, and filter for actors that appear in more than 3 movies, we get a much more readable image.

Blog_Post_GraphsAreEverywhereEasilyIntegrateThemWithArdoq_Image_2

And would you look at that, Tarantino himself stars in ten of the movies he has directed, while Samuel L. Jackson is only in six of them!

To take a closer look at the data, you can browse through this live presentation of the data in Ardoq:

Code as a Graph

There is, of course, not much business value in knowing that Tarantino stars in most of his movies, but the concept of integrations and how to apply our SDK is the same. One of the integrations we use at Ardoq is for our codebase. Through multiple hackathons, we’ve written an integration that reads our source code and parses how our different source files depend on each other and external libraries.

We use this graph to look at the sources for subsystems of our application, or for analyzing the dependency paths between different systems. This allows us to identify risks by tracking security issues from a vulnerable library right up to the affected functionality in the app.

We can also use the graph to identify technical debt in the code base by looking for big modules with a lot of dependencies and dependents. The screenshot below shows some of the source files for Ardoq, coloured by whether they have deprecated dependencies and if they are TypeScript or JavaScript. They’re also ordered by their file size and how many other modules depend on them.

Blog_Post_GraphsAreEverywhereEasilyIntegrateThemWithArdoq_Image_3

The most valuable part of this integration is that it runs as a part of our continuous integration process. The graph always reflects the code that is actually running in production. Working with the source modules as a graph is not only useful for analyzing dependency paths and technical debt but also allows us to track the progress over time of some of the initiatives in our code base.

Over the past year, we’ve been adopting TypeScript in our code base. From the source code graph, we have created a dashboard that tracks our progress of rewriting files. It shows us how much of our code that is currently TypeScript and how much we have left to migrate. This could be extended to any migration case you might have, for example, moving to the cloud or migrating off an expensive vendor.

Human Interpretation

The high-level view we get from looking at the source files is certainly useful, but sometimes we need to interpret and translate the data before we push it to Ardoq. 

Ardoq’s front end is built on functional reactive programming with streams, which means we need to watch out for glitches. To save ourselves from drawing parts of our stream architecture on paper every time we need to reason about how data flows, we’ve made tooling that can extract the graph of our streams and push it into Ardoq. This saves us the time and uncertainty of drawing by hand.

The tooling is split between a custom framework on top of rxjs and a simple integration that identifies the streams we define with this framework and pushes them to Ardoq. This graph complements the source code files graph, and both are part of the bigger graph of Ardoq’s source code. The stream graph is a much more suitable level of abstraction for many issues and use cases we have when working as developers. In general, it is important to create integrations on the correct level of abstraction.

In the following visualization, we can clearly see how dashboardContainer$ contains glitches through asset$ and workspacesPermissions$, by the “circles” the references form between the components. This let’s us quickly see that we should examine the dashboard for glitches in the rendering and possibly refactor the streams if it leads to a poor UX.

Blog_Post_GraphsAreEverywhereEasilyIntegrateThemWithArdoq_Image_4

Previously we would sit at developer meetings and draw up these diagrams by hand to get an understanding of the issue in front of us, but the confidence in those drawings was never particularly high. Knowing that this diagram is updated automatically every time we push a code change gives us confidence in analyzing our code and in finding good solutions.


The software development kit is available on NPM as “ardoq-sdk-js”, and the examples above hopefully serve as some inspiration for what is possible with Ardoq and this SDK. The code examples should also point you in the right direction for getting started. As you can see, you can document almost anything with Ardoq.

If you're not already an Ardoq user, you can try it yourself with our 7-day trial

Try Ardoq

Tobias Laundal

  

Subscribe to our newsletter to get the latest news, views and opinions straight to your inbox.