The Data Model (Intermediate)

6. Print
To be able to see what's going on, I'm going to introduce another very handy module: Print found in the Debugging Category.

Add a Print to your net. It has no outputs! So how do we get anything useful out of it? Easy: its answers always appear in the Message Window. This is another example of a "side-effect" module (like Image), that always runs when you Execute the program regardless of whether you connect anything to its inputs. It shows a "required" input, but if you leave it unconnected, you'll just see a message "Null Object". (Go ahead: prove it to yourself.) If you've lost the Message Window, you can get it by choosing Windows: Open Message Window.

Print, in its default setting, is only mildly useful. Open its CDB and note that the second input is "o". This is shorthand for "high-level Object description" and means that only the Object type itself will be reported.

Let's try it on some Objects to see what we get. Connect the output of Import to Print's first input and Execute. Study the Message Window. You should see it report that you have a Field with 4 components. The Field also has an Attribute called "name". The "name" Attribute's value is "temperature". (This Attribute was added to the data file by the person who created the data. They should have added another Attribute about the units of measure!)

Unhook Print from Import and connect it to AutoColor's left output. Execute again and read the Message Window. Another Field, but this one has 7 components. For fun, try the right-hand output of AutoColor.

Note that you would normally leave the entire net hooked up: Print is just an additional "parallel" wire coming off a module output. If you don't leave the net hooked up, you'll keep getting pesky Errors from downstream modules with required inputs, like Image. Print will work, but we don't need the Errors interrupting us at this point.

 

However, when you start debugging your own nets with Print, breaking a wire and inserting a Print is sometimes a handy way to force the net to Print the information but then terminate further downstream processing. This is like setting debugging stop points in other debuggers you may have used. Of course, in DX we aren't "debugging", we're "gaining greater understanding".

Before you go on, rehook Print to the output of Import. We'll examine a more detailed view, then advance down through the program by progressively moving Print down.

 

Think of Print as a "wiretap" allowing you to peek inside the network wires to see what is flowing.

 

Optional: View this Technical Aside if you want more detail on this subject; the information is not required to understand the upcoming material.

Now a bit about managing the Message Window display. By default, the Message Window is rather small, but you can stretch the window to be as tall and wide as you like. I tend to make it as tall as the screen, but maybe 1/3 to 1/2 the width. Modify yours so you can see the network and the Messages at the same time.

The Message Window gets filled up with junk every time you execute, especially with Print pumping stuff into it. So, whenever you want to clear it, choose Message Window: File: Clear. (Do it now.)

 

When you are all finished with Print, remember to either disconnect it or Delete it from your net. Print will continue to dump output into the Message Window every time you Execute and this can make the net run more slowly, especially if the Object being Printed is large. If you've closed the Message Window, you may forget this is going on and wonder why things are so slow!

Return to the Print CDB and pull down the ellipsis button menu for the "options" input. You see "o", "rd", and "rD". There are other options but these are the most commonly used. We talked about "o" as a high-level description of an Object. "rd" is short for "recursive small (or abbreviated) dump". "rD" is "recursive BIG DUMP". "rd" is my favorite, so choose that.

Execute Once and inspect the Message Window. Whooee! Fill 'er up!

Note that you probably now have a scrollbar in the Message Window, so scroll it back to the start of the current "dump". Again, we see Field with 4 components.

The next line reads "Component number 0, name 'data'". Then the description of this Component begins: Generic Array, 2800 items, float, real, scalar. After this, you see an abbreviated dump of the contents of the data array. "rd" only shows the first and last 25 items in long arrays which is often enough to tell you what you need to know, such as, do the data values look reasonable?

At the end of the "data" array listing, notice the Attribute "dep", String "positions". This is one of those required Attributes I mentioned a long time ago: this is the declaration required by the Field to tell it how the data is mapped to the spatial system. In this case, the data is "dep" or dependent on positions.

 

How many positions does this Field have to have? Don't peek in the Message Window.

Answer

 

Optional: View this Technical Aside if you want more detail on this subject; the information is not required to understand the upcoming material.

In this Field, the next Component description is for "connections". Since this is a regular positions, regular connections type of grid, you see that it is trivial to describe the mesh of cubic elements that bind the regular positions together. The "Mesh Array" is another internal DX Object type, fully described in the Native File Format documentation. Here we see how DX took the data file description of the grid and rewrote it in its internal syntax. Three Path Arrays are combined to make the Mesh Array. The elements are "cubes": the "element type" Attribute is required for the "connections" Component to make sense in the Field. The "ref" Attribute declares that "connections" are References to the "positions". This alludes to the discussion of how "connection" elements are references to the implicit position indices. The peculiar Attribute "dep" "connections" is best forgotten about. I can't explain why that's there. I'm sure there's a good reason lost in the sands of time.

The next Component description is for the "positions". We can see the origin and delta vectors for each of the 3 axes and the counts (called repetitions here). By multiplying these three directed axes together, DX constructs a regular arrangement of 3D points in space (remember they are not connected until we apply the "connections" to the points). Again, there's an apparently redundant declaration that the "positions" is dependent on the "positions". Ignore it like I do.

This particular data set is very odd. Look very carefully at the "positions" delta vectors. For whatever reason, the data stream was written in such a way that X varies slowest, Z second fastest, and Y fastest of all. How do I know that? Note that the second delta ([0,0,4153.85]) has the changing value last, and claims 14 repetitions. The third delta listed ([0,2214.29,0]) has the changing value second with 8 repetitions. I'm not sure who or what program wrote this file, but because the deltas are described this way, the grid once loaded into DX will become a 25x8x14 grid in which the Y dimension has 8 repetitions, each 2214.29 apart, and Z has 14 spaced at 4153.85 units each. Very strange, but you should note that DX can deal with this as long as you define the "majority" correctly. This "majority" is neither "row" nor "column", but some bizarre version without a name.

Finally, there's a peculiar Component I've never mentioned: "box". This is the bounding box of the spatial coordinates described by the "positions" and is manufactured inside DX on the fly. Several modules, like Image, need to know this information, so DX creates it and attaches it to the Field so that every module that needs it does not have to recalculate it. However, if a module modifies the spatial positions in some way, that module will also modify the "box" Component.

 

You can see the "box" by attaching a Realization: ShowBox module to the output of a module destined for Image; then hook the ShowBox output up to Image. The result is a wireframe rectangular box (or rectangular prism) that completely encloses the Object you've sent into ShowBox. Try this on your own. Add a Collect and you can have both the Object realization and the ShowBox at the same time in the Image. Now try this: attach the ShowBox to Import's output (the whole 3D data set) and Collect it with the output of AutoColor (which is only the 2D plane). This shows you where the plane is within the 3D volume. Use Rotate in Image to inspect this. When you are finished, select ShowBox and Collect by clicking on one, then shift-clicking on the other and choose Edit: Delete to remove them from your net. Reconnect AutoColor to Image.

 

This image shows the solution to the Exercise: try not to peek until you've done the Exercise yourself, or gotten hopelessly stuck.

At the end of our printout of the Imported data file, we see again the Field Attribute "name" whose value is "temperature".

So that's what a 3D array of data looks like to DX. Nothing here should be too hard to understand since you know all about Components and Fields and how they are associated.

Let's now apply Print to the output of Compute so we can check that Compute actually does something.

I still have Compute next, right after Import. Attach Print to Compute's output. Bring up the Message Window. Normally, I would advise you to Clear the Message Window before you Execute, but if you want to leave the previous printout there for comparison, that's OK. The only warning is that it is sometimes a bit difficult to find where one Print ends and the next begins. You will always see a Begin Execution line at the start of each Execution, but these lines are difficult to find if the Message Window is full of multiple Prints.

 

Always scroll to the bottom of the Message Window, then scroll slowly back up until you find the most recent Begin Execution. Or use Clear often.

Notice that the Execute menu also appears in the Message Window. You can use this menu to execute the net without having to bring the VPE back up, then bring the Message Window to the top again. Big timesaver when debugging!

This time, we see a Field with 4 Components (what did you expect?). Examine the object and you'll see it has the same "connections", "positions", and "box" as before. The "data" has the same structure, item count, type, etc., but if you look carefully, you'll see that all the data values are different. (If you did Clear the Message Window and you don't have a photographic memory, hook Print to Import and Execute once more. This time the Import printout will appear below the Compute printout in the Message Window.)

So Compute did do something after all. I wasn't just handing you a line! Feeling frisky? Go ahead and try some other expressions within Compute. Since you have only one input at this point, your choices are somewhat limited. You can perform any kind of math expression using sums, products, squares, nested parentheses, constant values, and copies of the input "data" (currently called "F" within your Compute module). Refer to the Compute Context-Sensitive Help for inspiration. Return the expression to the F to C conversion when you are done.

 

Instead of mucking up your perfectly good F to C expression, why not get another Compute module onto the net to use as your playground? Power user trick: middle-mouse drag sideways on the existing Compute module on the net. You will get a cloned copy of the Compute including the expression and named inputs. If you have a 2-button mouse, hold both buttons down then drag on a module to clone. If this does not work, click on Compute once, then use the Edit: Copy command followed by Edit: Paste, then click where you want the clone to drop.

Before I show you more of Compute's power, let's finish our inspection tour of this net's Objects.

Move the Print down and attach it to MapToPlane's output. Execute and locate the most recent Begin Execution (or if you Cleared the Message Window, scroll to the top).

This Field has 6 Components and the Print is rather long (even using "rd").

 

Before you read my comments, go down through the Print out and see how much you can understand from what you already know. Make particular note of the "counts" of items in each Component, the "dependency" relationships, and the general syntax of how DX internally describes Objects and Components.

OK, here's my description of this Object. First, have in mind what MapToPlane module does. We gave it a 3D regular positions, regular connections mesh of data values. It gives us back a 2D plane surface with data values mapped onto the surface. It derives these values by interpolating data from the 8 corners of each cube element through which the 2D plane surface passes, then maps the interpolated data values onto the positions of the 2D plane (which is a mesh itself, though not a regular connections mesh).

Before we look at the formal object description of the mesh, let's look at the mesh itself. Add Realization: ShowConnections to the net. Hook the output of MapToPlane to ShowConnections, then ShowConnections can either go to Image or to AutoColor-Image (your choice).

 

Showbox image

 

Execute the following arrangements:
  1. Import-Compute-MapToPlane-ShowConnections-Image
  2. Import-Compute-MapToPlane-ShowConnections-AutoColor-Image
  3. Import-Compute-ShowConnections-Image

What are the differences?

Answer

Wire your net using arrangement 2 in the Exercise you just completed. Add a Print module connected to MapToPlane's output in addition to the ShowConnections module that is attached to MapToPlane. Look at the ShowConnections mesh image. You see that it is made up of triangular elements ("triangle connections"). These triangles were not created in ShowConnections, but in MapToPlane. ShowConnections only shows the connections (get it?) of the input Field. (To be more precise, ShowConnections shows line boundaries of connection elements, regardless of their type.)

 

The first thing I do when I import unfamiliar data is to pop in a ShowConnections to check that the input mesh looks "reasonable". So my first net is often Import-ShowConnections-Image since ShowConnections provides a color for free. I may then discard the ShowConnections and begin working on other visual realizations, or I may follow the next Pro Tip …

 

Use ShowConnections, followed by a Color set to "black", and Collect to group the original colormapped mesh and the black ShowConnections Object. This gives you both a colorized surface and nice black lines on top that show the underlying mesh. Revealing the mesh is often a good visualization technique since many viewers may want to know how fine a resolution your sampling space has.

 

ShowConnections image

Now that we've viewed the Object with this "diagnostic" method, the Printed Object description may make more sense to you. Look first at the "data" Component. There are only 200 items. This number is how many positions and therefore data values MapToPlane created internally when it made a plane that fit through the center of the 25x8x14 volume in X-Y (in other words, it's almost like it took a slice at a fixed value of Z, yielding a 25x8x1 (or 25x8) piece.

 

Optional: View this Technical Aside if you want more detail on this subject; the information is not required to understand the upcoming material.

Though there are fewer data values, the Component still has the same type (float) and structure (scalar).

The next component is "connections". We see there are 336, integer, real 3-vectors. Right away, we know these must be triangular since that is the only 3-vector "connection" type in DX. You see the list of 3-vectors and of course remember our discussion of position indices. As before, "rd" has trimmed this long list to just the first and last 25 items for a quick scan. At the end of the abbreviated list you will find the actual Attribute that declares the element type of these connections to be "triangles" which of course "ref" the "positions" index values.

Now we see the "positions" list written out as individual 3-vector float values. How many? Of course, 200 (25x8). We knew that because the "data" was "position dependent". Notice that there is no explicit position index (0..n-1) anywhere, but of course, DX can calculate this internally when it needs it for the "connections" elements.

 

Optional: View this Technical Aside if you want more detail on this subject; the information is not required to understand the upcoming material.

After the "positions", we find our friend "box". Then there are two more Components that weren't there before. "colors" are added by default by MapToPlane. ShowConnections does not override the already set colors. Notice how compactly the "colors" can be described since they are a constant value of RGB = [0.5, 0.7, 1.0] (light blue), using the DX Constant Array Object type. There are 200 colors: why? Because colors derived from "position dependent data" are also "position dependent". (Very important to remember: nothing in DX is "data dependent"! Data is a dependent variable itself.)

The last component is called "normals". We mentioned this in our discussion of how DX connects points to make polygonal surfaces. A normal is a direction vector that is perpendicular to the face or to the tangent of a curve that passes through a vertex (the first is called a "face normal", the second a "normal", or a "point normal"). You can see that all these 200 "position dependent" normals face the same way: right at your face ([0,0,1]), which means that the polygon corners they are attached to are also facing you.

Last, but not lost in the shuffle, is the user-added Attribute "name" which is still "temperature". Think about that: although DX has completely reorganized the input data object into a very different object, it has inherited the original Attribute assigned by the user. Thus, we could extract this attribute and display it in the image as a reminder of the source of the data.

Hey, good idea! Let's do that.

While you are rewiring, you can disconnect Print, but leave it in the net. Add Structuring: Attribute, Structuring: Collect and Annotation: Caption. Wire them according to this illustration:

 

Attribute caption

Execute and observe the image. It turns out that the default "attribute" is preset to "name". If you had a data file with a Field Attribute "Experiment date" whose value was "2000/04/20", you would have to open the Attribute module's CDB and edit the second input to "Experiment date". Then the Caption would receive and display the string "2000/04/20".

 

Optional: View this Technical Aside if you want more detail on this subject; the information is not required to understand the upcoming material.

 

I find preprocessing data in Excel a useful operation. In Excel, you can convert dates and times to decimal numbers. These can then be used "logically" in DX if you need a "time" axis; for example, you might want to visualize data plotted in an X-Y-Time space. If you also pass in the original date or time string ("2000/04/20", etc.) in a separate Component of type string, DX can do a Lookup and place these strings along your time axis. To sum up, import a "time" Component of type float and a "timelabel" Component of type string (the names are arbitrary). Use the Lookup module to reassociate the strings with their numeric counterparts. Since each Component is a "data-like" Component, each is dependent on "positions", so Lookup can simply look down the "positions" indices in the "time" and "timelabel" Components and find the matching records.