How to Improve Spotfire Load Times

Facebook Tweet LinkedIn

Long load times are a problem all Spotfire users struggle with eventually. The first step to reducing load time involves understanding which tables are taking the longest to load. Spotfire provides this information in the Help menu, but it’s not in a form that is easy to work with. This week, I need to help one of our project teams optimize their Spotfire project, so I built an Excel template to parse this data. Now I know what to go after in order to improve Spotfire load times.

Blog or Video

Diagnostic Information

This is the Diagnostic Information tab in the Help menu, Support Diagnostics and Logging. This table displays the size of each table in the analysis and it’s load time. This information can be copied and pasted for further analysis. Unfortunately, it’s not in a very useable form. But, it’s simple to parse the data using either Excel or Spotfire formulas. See below for detailed steps and a sample file.

Approximate Table Size vs View Size

The screenshots below are from different DXPs, and you will notice that the diagnostic information says “Approximate table size” and “Approximate view size”. In the background, the Spotfire data engine creates data views for each visualization. The approximate view size indicates memory held by that visualization. Fortunately, these views are small, so they don’t take a long time to load. But, it does mean that the more visualization in a project, the longer it will take to load.

Downloadable Excel Template

You might be wondering why I did this in Excel rather than Spotfire. The diagnostic data comes in two formats. I don’t need all the calculations for each format. Thus, to create a Spotfire project with only 1 table would require adding a lot of IF logic or splitting it into two tables and duplicating calculations. Not to mention the fact, that I would have to go into and out of the add calculated column dialogs to see the result. Lastly, I don’t need to have this task in Spotfire, and I don’t need to make it perfect. Also, WordPress won’t let me up load a DXP. They block the file format.

Help-Decoder-v2 Download

Detailed Steps

Go to the Help menu
Go to Support Diagnostics and Logging
Find the Diagnostic Information tab
Click the Copy Info button
Paste into Excel and delete unnecessary content as shown below. Keep the records that start with “Approximate table size” and “Load time and size”. When pasting into the template, the “Approximate table size records go into the first “chunk” of calculations. You may need to delete or carry down formulas.

Warning

I had a lot of trouble getting Excel to sort the columns correctly. I ran multiple tests and didn’t get consistent results on whether the use of the VALUE function, adding a comma/decimals/formatting, or removing a comma/decimal/formatting allowed it sort.

The End Result

Ultimately, the 2 questions I am trying to ask and answer are:

Which tables are the slowest to load?
Which processes slow down the load?

I can answer those questions with the information in the calculated columns.

Table size is correlated to the load time but not perfectly. Transformations and other processes also contribute to load time. Note, KB conversions done in decimals, not binary.

Information

Notice that Spotfire breaks data loading steps like loading source data (SourceColumnProducer), transformations (DataTransformationColumnProducer), and adding columns (AddColumnsColumProducer). This can further help you understand what’s taking so long!

Improvements

In order to improve the load time on this particular project I’ll take 3 actions.

Because Geomodel Data is an information link that is loaded once a day, I’ll cache the information link from the Information Designer.
I’ll setup a Scheduled Update to make the web player version load faster.
I’ll setup an Automation Service job to make the desktop version load faster.

Now, you have the tools needed to improve Spotfire load times!

Spotfire Version

Content created with Spotfire 10.3.