The Guardian’s Miso Project, might one day provide us with a truly open visualisation alternative, so worth keeping an eye on it.

If you’ve not seen Google Charts in action do check out their “playground” and also, this Building Interactive Dashboards video, demonstrating some of the newer, and even easier to use, controls.

Utilising Google Charts API could be an alternative method of publishing PowerPivot generated datasets when the option to use SharePoint is not available (or perhaps not affordable).

One way to do this would be to publish “tabular reports” to a Google Docs account using a “steam-powered server” approach. This actually could be a very powerful method of disseminating PowerPivot generated reports, particularly if mobile devices are the target (most Google Charts are now HTML5 enabled); and I’ll come back to this in a future post.

But, Google Charts can consume data from any server, and can do so very easily if that server implements its Data Source Protocol. So, as a POC I’ve added such a server protocol to my InProcess-oData server example. The new end-point is /tq and like the /range endpoint expects to be followed by a valid range pointing at a table. See the example index.html file (no need for a server  just open in browser, having first started the InProcess-oData server on port 8081).

The protocol is not fully implemented (only supports JSONP, and only supports one request-at-a-time from any client), but it gives a flavour of what’s possible.  To use this you’ll need to download the latest version of HAMMER.

Ah, but what if your heart is set on all the SharePoint PowerPivot goodness (and the sight of all that JavaScript doesn’t appeal), but IT refuses to upgrade your ancient SharePoint farm  (or maybe even refuses to let SharePoint in any shape or form anywhere near their servers). Is all lost? Not at all, check this out, a PowerPivot hosted service (not just any old service, @powerpivotpro‘s service), now offering 30-day free trials.

To download the latest version of HAMMER, go to this page on my website.

Follow the HAMMER tag on  this blog for information on commands and examples (best start with the oldest and work forward …)

It uses the “SAP.Functions” COM object as exposed by the SAP GUI Client, fetching the data via RFC_READ_TABLE; an automated SE16 in effect.

The credentials required are the same as those you would use to log into your desktop client, and whatever internal tables you can see via SE16, those same tables will be fetchable via RFC_READ_TABLE.

This automated fetching of data is ideal when some self-service reporting is a requirement (you know, standard DW extracts offer most of what you need, but there’s always something missing ).

I figured this would be a good candidate as a HAMMER command. Not just to take advantage of HAMMER’s natural table handling but also its multi-threading capability. Being able to spawn one or more background threads (or delegate to HAMMER.exe command line process(es) ) would be very handy for SAP datasmiths.

Problem is, the code below works, and I’ve converted it to VB.NET, made it more generic and added it as a HAMMER command; but I can’t test it, as I no longer have access to a SAP R3 Instance!

The command SAPREADTABLE takes three parameters:

• a CSV list of SAP logon credentials: System,Client,User,Password,Language
• a CSV list of table information, 1st argument the table name, the rest field names e.g. KNA1,KUNNR,NAME1,NAME2,LAND1
• a filter statement (like a SQL where) e.g. LAND1 in (‘DE’,'NL’)

Example:

“Test SYS,600,tom,pwd,EN”,”KNA1,KUNNR,NAME1″,”LAND1 = ‘DE’”

UPDATE: April 29, 2012

Could somebody with access to SAP R3 test this out for me?  Done, tested (found a small bug, now fixed) and working (thanks to a kind person who allowed me access to a test server, you know who are, thanks again).

Fetch the modified latest version below (fixed bug that produced an extra blank column and extra blank row in result table, my typical “1 off” bug when converting from VBA to VB.NET, obviously I’ll never learn )

If you get a scary “ABEND – SYSTEM FAILURE” error, don’t panic you haven’t broken the company ERP system, it’s usually due to a malformed filter statement e.g. LAND1=’NL’ (no spaces) rather than LAND1 = ‘NL’.

To download the latest version of the code, go to this page on my website.

Follow the HAMMER tag on  this blog for information on commands and examples (best start with the oldest and work forward …)

Need a pure VBA version, here it is :

SAP RFC_READ_TABLE VBA Example:

Option Explicit
Option Base 0

Public Function RFC_READ_TABLE(tableName, columnNames, filter)

Dim R3 As Object, MyFunc As Object, App As Object

' Define the objects to hold IMPORT parameters
Dim QUERY_TABLE As Object
Dim DELIMITER   As Object
Dim NO_DATA     As Object
Dim ROWSKIPS    As Object
Dim ROWCOUNT    As Object
' Where clause
Dim OPTIONS As Object
' Fill with fields to return.  After function call will hold
' detailed information about the columns of data (start position
' of each field, length, etc.
Dim FIELDS  As Object
' Holds the data returned by the function
Dim DATA    As Object
' Use to write out results
Dim ROW As Object

Dim Result As Boolean
Dim i As Long, j As Long, iRow As Long
Dim iColumn As Long, iStart As Long, iStartRow As Long, iField As Long, iLength As Long
Dim outArray, vArray, vField
Dim iLine As Long
Dim noOfElements As Long

'**********************************************
'Create Server object and Setup the connection
'use same credentials as SAP GUI login
On Error GoTo abend:
  Set R3 = CreateObject("SAP.Functions")
  R3.Connection.SYSTEM = ""
  R3.Connection.Client = ""
  R3.Connection.User = ""
  R3.Connection.Password = ""
  R3.Connection.Language = "EN"

  If R3.Connection.logon(0, True) <> True Then
   RFC_READ_TABLE = "ERROR - logon to SAP Failed"
   Exit Function
  End If
'**********************************************

'*****************************************************
'Call RFC function RFC_READ_TABLE
'*****************************************************

  Set MyFunc = R3.Add("RFC_READ_TABLE")
   Set QUERY_TABLE = MyFunc.exports("QUERY_TABLE")
   Set DELIMITER = MyFunc.exports("DELIMITER")
   Set NO_DATA = MyFunc.exports("NO_DATA")
   Set ROWSKIPS = MyFunc.exports("ROWSKIPS")
   Set ROWCOUNT = MyFunc.exports("ROWCOUNT")

   Set OPTIONS = MyFunc.Tables("OPTIONS")
   Set FIELDS = MyFunc.Tables("FIELDS")

   QUERY_TABLE.Value = tableName
   DELIMITER.Value = ""
   NO_DATA = ""
   ROWSKIPS = "0"
   ROWCOUNT = "0"
   OPTIONS.Rows.Add
   OPTIONS.Value(1, "TEXT") = filter ' where filter

    vArray = Split(columnNames, ",") ' columns
    j = 1
    For Each vField In vArray
        If vField <> "" Then
            FIELDS.Rows.Add
            FIELDS.Value(j, "FIELDNAME") = vField
            j = j + 1
        End If
    Next

   Result = MyFunc.CALL

   If Result = True Then
     Set DATA = MyFunc.Tables("DATA")
     Set FIELDS = MyFunc.Tables("FIELDS")
     Set OPTIONS = MyFunc.Tables("OPTIONS")
     R3.Connection.LOGOFF
   Else
     R3.Connection.LOGOFF
     MsgBox MyFunc.EXCEPTION
     Exit Function
   End If

  noOfElements = FIELDS.ROWCOUNT
  iRow = 0
  iColumn = 0
  ReDim outArray(0 To DATA.ROWCOUNT, 0 To noOfElements - 1)
  For Each ROW In FIELDS.Rows
    outArray(iRow, iColumn) = ROW("FIELDNAME")
    iColumn = iColumn + 1
  Next

'Display Contents of the table
'**************************************
iRow = 1
iColumn = 1

For iLine = 1 To DATA.ROWCOUNT

       For iColumn = 1 To FIELDS.ROWCOUNT
         iStart = FIELDS(iColumn, "OFFSET") + 1
    '       If this is the last column, calculate the length differently than the other columns
         If iColumn = FIELDS.ROWCOUNT Then
            iLength = Len(DATA(iLine, "WA")) - iStart + 1
         Else
             iLength = FIELDS(iColumn + 1, "OFFSET") - FIELDS(iColumn, "OFFSET")
        End If
    '       If the fields at the end of the record are blank, then explicitly set the value
        If iStart > Len(DATA(iLine, "WA")) Then
             outArray(iRow, iColumn - 1) = Null
        Else
            outArray(iRow, iColumn - 1) = Mid(DATA(iLine, "WA"), iStart, iLength)
        End If

       Next

       iRow = iRow + 1
Next

RFC_READ_TABLE = outArray
Exit Function

abend:

RFC_READ_TABLE = Err.Description

End Function

Public Sub Paste_sheet1()
Dim lArray
Dim lAdjust As Long

lArray = RFC_READ_TABLE("KNA1", "KUNNR,NAME1,NAME2", "LAND1 = 'DE'")
If TypeName(lArray) = "String" Then
    MsgBox "Problem calling RFC is it here " & CStr(lArray)
Else

    ' adjust if zero based array
        If LBound(lArray, 1) = 0 Then lAdjust = 1 Else lAdjust = 0

    [Sheet1!A1].Resize(UBound(lArray, 1) + lAdjust, UBound(lArray, 2) + lAdjust) = lArray

End If

End Sub

But why, what purpose do these mini servers serve?

They’re obviously not intended as beyond-the-firewall public servers, they wouldn’t scale or be secure enough for such a task. Providing in-house feeds to other web enabled clients would be a more sutable task. For example, providing a feed from a “hub” workbook containing a PowerPivot model to other “spoke” workbooks (PowerPivot enabled or not) – a poor man’s alternative to doing the same via a SharePoint farm, if you like.

But it’s another seemingly unrelated technology that’s really sparked my interest in perfecting the fractional horsepower server: HTML5.

Generally when people think of HTML5 (if indeed they think of it at all), it’s mobile platforms that come to mind. (As it’s primarily Apple and Google, through their shared WebKit browser core, that have driven the development and adoption of HTML5). So what has this to do with Excel and the boring, but oh so profitable world, of corporate IT? Well, next time you’re in any spot where the global mobile workforce gathers, airport waiting lounges, hotel lobbies, etc. look at the technology kit that they’re using.

Only a few years back, the vast majority would have had a Windows laptop, if indeed they had any “data processing” device. Now, many, if not all, will either be using a smart-phone or a tablet device (iPhone or iPad but also increasingly Android powered phones/pads). All of these workers are still likely to have a laptop in a carry case or back in the hotel room, and certainly will have a laptop/desktop or their workplace desks. But on the move mobile is where it’s going.

So how do front-line datasmiths respond to this? Currently many of us build reporting solutions and really-simple-systems using Excel as the delivery agent, moving all or part of this to a mobile delivery agent will inevitably become increasingly attractive and/or demanded.

MS is already responding to this, e.g. PowerPivot and standard Excel spreadsheets are capable of being rendered via SharePoint’s Excel Services. But what if you don’t have access to a SharePoint farm, or you need a more robust UI, such as could currently be built using a VBA/.NET add-in? This is where HTML5 and fractional horsepower servers come in.

For me, there are two aspects of HTML5 that I think will make developing and deploying such “systems” possible and relatively easy:

• In built graphics and animation support within HTML5 supporting browsers, this makes it easier to display and build business charts and dashboards (see Google’s chart API playground to see what’s possible).
• Local off-line storage.

It’s HTML5′s local storage, that’s makes a fraction horsepower server scenario possible. In traditional web apps, it’s assumed that:

• 1st the client is always connected to a server,
• and that the server provides both the layout (html, javascript, css) and all the data (REST APIs etc.) that the web app consumes.

Now with HTML5 apps, the client doesn’t need to be always connected to its main server or to its data server(s). It can go offline, or it can stay connected to its main server (perhaps a public-internet-facing S3 hosted domain), and every now and then make contact with one or more data servers (which can be safely positioned behind the firm’s firewall).

An example:

A firm’s Sales Reps come into the office every Friday for wash-up meetings, to record sale completions and to get their journey-plans for the following week.

Each rep has a desktop computer, where they interface with the firm’s various systems. One such set of “systems” are PowerPivot based models that report on the year’s forecasts and actual sales.  Part of the process of preparing for a sales visit is creating a set of sales reports for each customer to be visited, last year sales, this year’s targets, and so on, sourced from the various PowerPivot models. Although the production of the reports is largely automated via Excel macros, currently the resulting sheets have to be printed.

There’s been talk of company supplied laptops for years and the budget for them has now at long last materialised. The reps however, have expressed a preference for using iPads when customer-facing, mainly because the sales conversation often require not only presenting the prepared sales reports and charts but also flicking through many of the 100 odd product manuals. Being able to hand around an iPad with high quality glossy images (and videos) of this year’s new products, plus a sales projection chart for the same products, is, they contend, a winner.

A simple mobile sales reporting app is therefore developed (using the JoApp framework and Google’s Chart API and this pure JavaScript columnar database) to cater for the type of sales reports the reps require. The existing Excel automation code is enhanced with a HAMMER server. The reps new iPads’ web-apps are configured to automatically download prepared and ad-hoc reports when they log-in to the office network.

This has worked so well, the reps now want the ability to feed back sales target changes, that they also wish to record on their iPads via another really-simple-system, to their personal Sales Plan workbooks.

Is this as simple as using a “pure” spreadsheets solution, no, but it’s nearly as simple as building a VBA/.NET powered Excel application to do the same. The problem with many Excel “applications” is that they often push Excel beyond its “comfort zone”. The benefit of a hybrid solution like above, is that Excel gets used where it’s really useful and powerful (reports and models, data gathering and dispersal) while at the same time taking advantage of the freedom and cost-benefits (and fun!) of the emerging mobile web.

Oh Data!

To test out the new threading facilities in HAMMER I picked on that perennial question that I get asked “Can your PowerPivot refresh code, refresh linked tables?” to provide me with a suitable task.

As regards the original code, the answer is no; linked tables cannot be refreshed via the XMLA method I use. You could of course, simply automate the export of the Excel tables to, say, a CSV format and then refresh from the disk. Biggest problem with that approach is PowerPivot’s insistence on using absolute file addresses, which makes sharing the resulting workbook more cumbersome (requires that the saving folder on each machine be the same). Another solution is to use some variation on SendKeys to automate the “button push”, can be done, but pretty it’s not.

So what’s an automate’r to do? Use PowerPivot’s oData import facility, that’s what.

In a previous example I had demonstrated using oData to provide a feed from a master-workbook to client work-books, so I already had code in Python to generate a simple oData feed (it’s essentially an ATOM feed; yeah, the same format used by many blog engines (usually alongside RSS2.0), and offered as a output format by many web services, Twitter and Google Docs, for example).

Rather than using HTTPlistener (which is very powerful, but requires Admin privileges) this time I used HAMMER’s internal simple HHTP/1.0 socket based server (which is multi-threaded and uses callbacks to communicate with either Python or Javascript GET/POST handling functions). Using a PYTHONTHREAD command enabled me to spin of the oData-generating-code as an in-process server running on an independent thread; leaving the main Excel thread to fetch the necessary ranges from the workbook and then issue a PowerPivot refresh command (both of theses operations require being run single-threaded and in the main thread).

In PowerPivot itself, the range tables are fetched using a “127.0.0.1:8081/range/” endpoint.

So, for example, to fetch a table from a range whose top-left-hand-corner is Sheet2!A1, use the URL “127.0.0.1:8081/range/Sheet2!A1″.

For an Excel 2007/2010 table named InvoiceHeaders use “127.0.0.1:8081/range/InvoiceHeaders[%35All]” (the %35 is URL encoded “#”).

While the workbook is in “manual mode”, the in-process server will fetch on-demand any valid tabular range (but be careful not to “lock” the workbook, by, for example, editing a cell, while this is happening).

When in “automated refresh mode”, the VBA code will lock out any on-demand refreshes (it’s best in any case, to close PowerPivot’s window before doing any XMLA refreshes); then use the list of ranges provided to fetch the tables required, and initiate an auto-refresh.

Everything should go smoothly as long as the list matches exactly (case sensitive) the URLs assigned to tables with PowerPivot.

To try this out, download the latest version of HAMMER, go to the distribution folder, open the InProcess_oData workbook.

In the Control sheet, click the Enable HAMMER button, followed by the Start oData Server button. If you think port 8081 is likely to be already in use on your PC, go to the PythonCode sheet and change to a free port.

Then go to the Data sheet, change the tables, and refresh PowerPivot using the “arrow” button provided.

Make sure to click the Stop Server button, before exiting Excel, otherwise the port-listening thread will keep Excel alive in the background.

Of course, auto refreshing same-workbook “linked” tables is but one use this could be put to. Other possible uses would be taking advantage of Excel powers that the PowerPivot import facility still lack, such as XML Maps. Or, using 3rd party libraries or bespoke code to access “non-standard” (i.e. the majority that don’t use ATOM!) web services or data sources (such as XBRL).

Have fun…

Dark satanic mill by Mark Nelson

I use the term “micro ETL” a lot when writing about tools such as HAMMER, but what do I mean by the term?

The ETL bit is easy to explain:

ETL, as all you data-warehousing and business intelligence folks will know, is the Extracting, Transformation and Loading of data from source systems into a reporting/data warehousing system. The techniques of ETL are not unique to the DW/BI worlds but are used anywhere transfers of data are needed between one computer system and another, for example, master data take-on for new systems or transactional interfaces between front and back-office systems – this is often referred to as DI (data integration) but is essentially the same problem domain.

So what’s the “micro” bit about?

You might assume that the micro adjective implies small or indeed tiny datasets, and in many cases you would be correct. Most final-mile data analysis, like politics, is local. Most business decisions along with their implementation and monitoring require ‘localised’ data. That data will be pre-filtered and summarised to some degree, but a fair degree of data shaping will still happen close to the decision makers. Excel is often the tool of choice when data gets to this stage.

HAMMER is optimised for this world, it sees the world how Excel sees it, but also adds the power of SQL and scripting languages to pick up where Excel stops. But enabling better Excel based data shaping is not HAMMER’s only function. It can operate outside of Excel (HAMMER.exe) and it can be used to craft task-specific ETL tools (HAMMER Inside). In both cases I continue to use Excel as my IDE, teasing out a problem before fixing it in code or in an external HAMMER call; and I can also use Excel as the UI for the end products.

In such scenarios, micro applies not so much to the datasets (which can be anything from tiny to very large) but to the concept of deploying simple micro “fractional horsepower” data engines to solve complex ETL, DI or RSS (Really SImple Systems) requirements.

HAMMER is built to take advantage of the distributed grid of powerful data crunchers (be that PCs, laptops, in-house cheap servers or just-in-time pay-as-you-go cloud-based CPUs) that every business, big or small, can now call on.

This revolution in distributed power is similar to what happened with the deployment of fractional horsepower AC-powered electrical motors in the last century. No longer was manufacturing restricted to “dark satanic mills” which had to be built close to natural power sources (water and later coal seams); and had to conform to the multi-story classic mill design to harness that captured power through belts, pulleys and shafts. With the expansion of the AC power grids (and the parallel expansion of internal combustion engine carrying roadways) the factory began to take on its modern single-story (or single story with mezzanine) distributed profile than can be seen everywhere from China to Cork. A similar landscape change is happening in IT.

HAMMER can take advantage of the “distributed engines” easily enough but the workflow, the actual control and distribution of tasks, data and decisions requires the ad hoc implementation  of either steam-powered or classic centralised server processes. I badly needed a more pre-built modular approach, micro Workflow to complement  micro ETL (and micro BI via PowerPivot ?), if you like. Last week I had started to think seriously about how/what to do about this (JSDB powered grids were featuring high on the list) when this appeared.

Perfect timing, Amazon’s SWF (Simple Workflow service) is exactly what I need!

SWF allows for the control and distributed deployment of stateless data processors. HAMMER was designed primarily as a stateless data processor (with state being persisted either in Excel or on disk as simple CSV/JSON flat files). Its default use of in-memory, rather than disk-based, SQLite assumes both abundant CPU and RAM (like is the case with your average 64bit laptop) and the existence of an external state-machine (which Excel and now SWF provide).

I’ve spent any spare time I had this week doing a deep dive into SWF and figuring out how HAMMER can take full advantage of this technology, not just for classic ETL, but for distributed decision control processes and RSS solutions. The result, in Dublin slang, is that I’m both “delira and excira” (delighted and excited). This is, to use that term again, yet another AWS game- changer.

But, when “going outside” is not desirable or feasible, another solution is required, namely threads. There are those who regard threads as evil and I too would have to admit I generally would not use them without good reason. But sometimes they’re needed, usually to keep a controlling UI (in this case Excel) responsive and capable of monitoring and managing background long-running tasks.

I’ve already added threading capability to HAMMER via the hammerThreadEnabled function and via “internal HAMMER threads” (using NET4.0′s latest parallel programming additions).

Both of these “lock” the Excel main thread until complete, with hammerThreadEnabled being also dependent on Excel’s dependency graph to determine if threading will happen or not. So I figured I needed to bite the bullet and add a new set of threading commands to better handle long-running tasks, be they tasks that eventually return data or a status to the main Excel thread (such as a long running save to ( or fetch from) a high latency web service end point), or those that run continuously (such as a Excel as a web service). The main new commands are:

• PYTHONTHREAD – like PYTHON but spawns a thread to run to Python script. (NET4.0 runtime required)
• JAVASCRIPTTHREAD – as above, but using JAVASCRIPT plus no need for NET 4.0.

Both commands “take ownership” of the HAMMER session’s SQLite database  and return the new thread’s ID (in the form PYTHREAD1,PYTHREAD2, JSTHHREAD3 etc), and are likely to be the last command in a HAMMER function call’s sequence.

The thread ID can then be used by the following helper commands to monitor, re-join and clean-up such threads:

• SQLFROMTHREAD – like SQL but takes two arguments, 1st is the thread id, and the 2nd is the SQL to apply against that thread’s SQLite database. e.g. = … “JSTHREAD1″,”Select * from messages”,”SQLFROMTHREAD” … In general, only Select statements would be issued against this database, but SQLite is thread-safe so data updates are possible but with obvious potential for SQLITE_BUSY locks and time outs. (Likewise, the thread itself can communicate back to main thread using an APPDB connection, as APPDB is always attached uniquely to the main Excel thread).
• THREADRETURN – will report on whatever return values (be that an error, or on success, a string or a table value) once the thread completes. Again, it uses the thread ID as its argument. If the thread is still running, the command returns “Waiting”.
• DELETETHREAD – will attempt to terminate an already running thread. It will then, or if already terminated, clean-up and close any resources associated with the thread, mainly its database, but also its Python and/or JavaScript engine instances.

It’s also possible to issue a HAMMER call that will re-join a completed thread’s database, this will happen if a HAMMER function call’s 1st argument is a valid thread-name e.g. =HAMMER(“JSTHREAD2″,”Select * from results”,”SQL”). If the thread is still running, the HAMMER function will exit with a return value of “Waiting”. With this, it’s possible to not only access the thread’s database but also its Python and/or JavaScript engine context. Multiple such HAMMER calls can be made until a DELETETHREAD command is issued or the Excel session is terminated.

As SQLite is thread-safe, SQL can be used as a simple and robust method of inter-thread communication. If the pattern of threads only updating their “own” database while the main Excel thread updates only “APPDB” is followed, very little locking or timeout issues should arise (ha, famous last words – okay, not as famous as Scott’s). There’s also nothing to stop more adventurous inter-thread single database usage if the need or impulse arises (“Do ya feel lucky Punk, well do ya!”). And don’t forget, normal disk-based databases can also be used alongside, or instead of, the more usual in-memory ones.

This use of SQLite falls neatly into my view of SQLite not so much as a database but as means of mapping and managing  shared memory using SQL as an  API.

It’s often said that SQLite should be viewed not as an alternative to Oracle/MySQL/MSAccess/whateverRDBMS but as an alternative to a file-open statement. But for me, I primarily use SQLite as an alternative to rolling-my-own in-memory shared data structures  (be that across threads or application stacks).

Two other new commands added with this version are:

• VERSION – will return the version of the internal HAMMER (XLite) engine is use.
• LISTCOMMANDS – returns the list of commands available.

There’s also been a small breaking change to the JAVASCRIPT command. Scripts will now only return a table if the return value is a JavaScript Array of Arrays (List of Lists) object or a JSON string representing a LOL. All other objects will be converted to a JSON string representation or scalar string value.

A few extra JavaScript focused helper methods have also been added to the passed-in “db” object:

• db.jsCreateTable(tableName, JSONaLOLorLOOobject) where the 2nd argument can be a “JSON table” in either List of Lists or List of Objects format. A table will be created using either the first “line” of the LOL or the “names” of the objects fields in the LOO to provide the column names (no types applied, as no need most of the time).
• db.jsInsertRows(tableName,JSONaLOLorLOOobject) – as above but will insert/append the pass’d “rows” to the specified table.
• db.jsonSQLQuery(SQLSelectStatement,typeOfJSON) will return a table in List of Lists format if typeOfJSON = “LOL” or as a List of Objects if “LOO”.

There’s also a new JavaScript command: loadAssembly(dotNetAssemblyName) – which will load a .NET assembly allowing the use of any .NET library within JAVASCRIPT.

Both the JAVASCRIPT and threading commands are at a very early stage, only minimal testing so far, so use with care.

To download the latest version of HAMMER, use this link.

There’s three main reason’s why I’ve added JavaScript as a partner scripting language to HAMMER’s existing Python (and VBA) integration.

• JavaScript is everywhere, a lot more people know it and use it than have ever used, or likely to use Python, (or VBA). JavaScript is now the undisputed “glue language”.
• I needed a scripting language that would work in all versions of HAMMER (the current IronPython implementation requires NET 4.0). For that reason I decided to use the NET2.0 based JINT rather than IronJS as my choice of JavaScript engine. JINT, is simpler, and probably slower that IronJS, but being C# based (IronJS is being developed in F#) I can more easily dig as deep as I need into JINT’s codebase without exhausting my technical skills.
• Increasingly I’m coming across JSON as data transfer protocol and although I have inbuilt JSON processing commands in HAMMER, JavaScript and JSON are natural partners.

The main use-case I see for JavaScript scripts will be the manipulation of data, which is obviously equally possible with Python, but I see IronPython’s main use as a scripting language for .NET facilities (IronPython’s integration with .NET is complete, the JINT engine’s integration is much less advanced). And as I pointed out above, removing the NET4.0 requirement (for both Excel and the HAMMER.exe command line version) is very useful when building HAMMER Inside bespoke applications.

So my scripting language choices are:

• VBA, when manipulating the Excel Object Model
• IronPython when accessing .NET internals, or for complex coding requirements when not  using C# or VB.NET.
• SQL , for tabular set manipulations, and for the fact “that SQL  does exactly what it says on the tin“.
• JavaScript for everyday data-focused IF-THEN-ELSE’ing or JSON’ing.

Calling JavaScript from HAMMER is similar to Python. So …

=HAMMER(“myVal=’cat’; return myVal;”,”JAVASCRIPT”) will return “cat”.

Like PYTHON, a object “db” is passed into the script, this allows access to the SQLite instance via several functions:

• db.SQL(arg) will take a string arg holding a SQL command and execute it.
• db.SQL(arg,callback(arg)) as above but expects a SELECT statement and will return a JSON encoded object string via the callback’s arg for each row returned.
• myJSONTable  = db.jsSQLQuery(arg) expects a SQL SELECT statement and will return the table as a single JSON encoded string as a List of Objects.
• myJSONReturn = db.JSONhammer(arg) will make a call as an internal HAMMER command where arg is a JSON encoded string of a List of commands. Will return a JSON encoded table or a string value.
• argValue = db.getArgNoArg(argNo) will get the value of a HAMMER argument, where argNo=1 is the last argument before the JavaScript script argument, argNo=2 is the argument prior to that, and so on.
• tableName  =  db.getArgNoTable(argNo) like above but returns a table name if the argument position holds a table rather than a string value.

A predefined JSON object is also passed in, offering the usual JSON.parse(arg)  and JSON.stringify(obj) methods.

There’s also an alert(arg) command which will display a Window’s message box and a println(arg) command which will append to a _JSSTDOUT specified log file if one has been specified.

Use the “return” command to return either a table (a JavaScript object containing either a List of Lists or a List of Objects) or a string value.

I’ve not fully tested this, or settled completely on the interface between HAMMER and its new JAVASCRIPT engine, but I’ll do so over the next few weeks and will then include a proper examples of its use.  So use with caution.

Here’s a list of the HAMMER commands implemented so far …

Download  the latest version of HAMMER from here …

The more I come to know PowerPivot, the more I believe that modelled data warehouses‘ days are numbered. I didn’t say data warehouses per se, rather those that attempt to centrally model end user reporting structures (usually as star-schemas).

There will continue to be a need for centrally controlled data warehouses (or at least simplified data views (and/or copies) of operational datasets, either provided by system vendors of by in-house IT) to bridge the raw-to-actionable data gap. But I suspect the emphasis will change from providing finished goods to providing semi-processed raw materials.

So, will the star-schema become redundant? No, as it’s still a valid method of modelling a reporting requirement in order to make many queries simpler to phrase (this obviously applies to SQL , but also to DAX queries). But, those who build them will be doing so closer to the problem at hand, and specific to that problem (I’ve discussed this before in http://blog.gobansaor.com/2011/01/08/slowly-changing-dimensions-time-to-stop-worrying/).

For many reports the barely modified operational data model will be all that’s required (for example, DAX doesn’t require “fact” header/detail tables to be flattened to detail level, as would be the case with a classic star).

“Good Enough” models will become the norm; classic “Everything You Ever Wanted to Know” centralised models a luxury for most (especially as such models tend to “age” very quickly).

If you’re about to invest or re-furbish your data warehouse or your reporting data sub-systems, don’t do so without first taking a serious look at PowerPivot. This is a game-changer, not just for full-stack Microsoft BI shops, but for any business that finds that their reporting datasets invariably end-up in Excel.

If you need any help evaluating PowerPivot or modelling your reporting needs in PowerPivot,  I’m for hire.

Update:

Just in case you think I’m an dimensional-model un-believer or likely to abandon my star-schema roots read this….

Star Schemas: to explore strange new conformed dimensions, to seek out new measures, to boldly go where no Excel spreadsheet has gone before.

Follow @gobansaor

I’ve also tested the code against the just released SQLServer 2010 RC0 version of PowerPivot and it appears to work.

I’ve updated the equivalent PPREFRESH code in HAMMER and this is now part of the tool’s latest release (V1.3.4(Beta)). My previous post, Excel as a Book of Record, previewed the most important new commands available in this release. Alongside those, I’ve also added the following:

• ISTABLE, if the previous COMMAND’s result  or the previous argument is not a table, this will abend the command sequence.
• ISARG, as above, but this time checks for an argument (a HAMMER parameter may either be a table, a command or an argument- aka, an ARG).
• ISOK, previous argument must be the string value “OK”.
• TABLESARETHESAME, will fail if the last two tables are not identical. Intended mainly for automated regression testing.
• ARGSARETHESAME, as above. but this time for ARGS.
• _GUID, will return a globally unique identifier.

Download the latest version of HAMMER from here …

When such systems are primarily reporting in emphasis, the justification for using Excel is usually straight forward and compelling (and getting even more compelling with the appearance on the scene of Excel 2010′s  PowerPivot.) But, alarm bells sound across the world of professional IT when Excel is proposed as a “book of record” i.e. when it’s to be used to store and maintain a business critical dataset. And, with some considerable justification, the nightmare that is linked (or even worse, shared) workbooks is very real indeed. But yet, businesses continue to do so, and do so quite successfully.

I myself record my business as a series of Excel documents (Invoices, Timesheets, Expenses) in a set of folders (Financial Years subdivided into major customers).  Essentially a document-oriented database.

In the past I simply then used a VBA powered workbook to open the required “documents” and extracted the data necessary for whatever report I required (VAT, year-end etc.).  To better automate (i.e. less bespoke VBA) this task I’ve have made changes to HAMMER to help with this and with similar tasks for clients.

The following list of commands will be added to the next release of HAMMER. (In the meantime these new functions can be previewed here …)

LISTOFFILES

This command takes a single argument, the folder to search, and will return a table of files in that folder and in any sub-folders. The result can then be used to select a list of files for further processing.

Example:

=HAMMER(“C:\a\rss”,”LISTOFFILES”)

_XLTOJSONDOC

This command takes a list of workbooks, opens each one, checks for a list of named ranges and generates a JSON document. The command is intended to be called from within a VBA macro (as opening and closing workbook breaks the “no side effects” rule of UDFs). Most “_” commands such as “_MD5″ etc. are likewise intended for “programming use”, but any command beginning with “_XL” must be restricted to macro (i.e. non-UDF) use.

See the example workbook FetchInventory for an example of this function in action. The function takes one argument (the name of the document to load) and expects a table where the last column is the full name of the workbook to open. Any columns in the source table will be copied to the new “JSON document” table with an additional column called “Document” which will hold a JSON document representing the key-name pairs and table(s) extracted from the workbook.

On opening a workbook, it is searched for a named range with the name of document concatenated with “_fields” (e.g. INVENTORY_fields). The value of this range is expected to be a CSV list of fields and tables to load. A single Excel “document” could contain multiple logical documents (each specified by its own “_fields” list) .

See the PartsInventory_bin4 for an example of a multi-document workbook (INVENTORY and EXAMPLE). The EXAMPLE document in this workbook also demonstrates the various types of tables handled.

Example:

lJSONObjects = oHammer.HAMMER(“C:\a\rss\StockTake1″,”LISTOFFILES”,”Select name,fullname from table2 limit 1″,”SQL”,”INVENTORY”,”_XLTOJSONDOC”)

lReturn = oHammer.HAMMERToRange(lJSONObjects,”Sheet2!A27″)

… will output

JSONDOCVIEW

This command is where the previous commands are leading to, i.e. extracting some real information value from your documents. It converts JSON documents into Excel friendly tables. It is, in essence, a Map function as in MapReduce. In a previous example I used a Python Map and a SQL Reduce, here, both Map and Reduce are via SQL (the command uses a series of SQL commands to perform its task).

Before I describe the function let me explain why I use an intermediate JSON format. I could just extract the data directly from each document and either store directly in Excel or create tables in SQLite of Access to hold this data. And in fact, that’s what I would have done in the past (seeExcel as a document-oriented NoSQL database). Now , however, I tend to favour using a free-format (i.e. no need for a fixed database schema) structure like a JSON document, so as the source documents evolve over time (which tends to happen not just during design stages but as the system matures) this will not break older documents.

So, for example, original Invoice workbooks might not have a backing time-sheet while newer Invoices do. As long as new and old documents share a core sub-set of data fields they can continue to be analysed together.

The command takes 5 arguments and a driving table (a record so far for HAMMER commands, most have a max of two arguments). The driving table’s last column is assumed to contain the JSON document to process, columns prior to this (if any) will be output unchanged for each resulting row.

The first argument specifies the name of the “inner” table to fetch (if any). Most real life documents consist of header details (the “outer”document) and one or more tables (“inner” details). Invoices, time-sheets,  stock-takes, all tend to follow this pattern. This command will effectively join each document’s outer details to a single inner table (if more than 1 inner table, a call for each one is required).

The second (field list in SQL format) and third (SQL where predicate format) arguments specify what inner fields to extract (if blank, then all) and what restrictions to impose (if any). So “InvNo, Date”,”InvNo > 12″ would only fetch documents where the InvNo > 12 and only include the InvNo and Date fields.

The fourth and fifth arguments do the same for the outer table (i.e. Header data).

If any of the columns specified  (inner or outer) can not be found, or if the predicates (inner or outer) result in no selection, no error is returned, the document simply returns no rows. Likewise if an inner table is specified and no such table exists, then no rows are returned for that document – in other words this is not an outer join, which is not usually a problem as in most cases a “header” without detail lines is meaningless. If an outer join is required, then extract the headers (outers) and details (inner tables(s)) separately and join using SQL.

Example:

=HAMMER(“Select Name,FullName,Document from invoice_docs”,”SQL”,”table_2″, “[PART NUMBER],QTY”, “QTY >30″, “Bin_Number”, “Bin_Number > 1″, “JSONDOCVIEW”)

would result in:

For more complex JSON objects use the JSON command to incrementally parse the text or use the VBA JSON module within microETL.  [UPDATE: Feb 2012 - ... or use the JAVASCRIPT command]. But for most situations (especially if you control the expected format) JSONDOCVIEW should handle it.

As JSON is fast becoming the preferred transport format for web and mobile applications having the ability to parse and produce JSON form within Excel is very useful. It is possible, for example, to use a simple web technology such as http://robla.net/jsonwidget/ to craft another type of Really Simple System. This time with the collection happening on the web (most likely using AWS S3 pre-signed forms, so no HTML server required – keep it simple) but with the control and reporting remaining within Excel (a variation on my Steam Powered Server idea).

For an example of a really simple system  download this.

Latest version of HAMMER including the above commands now released …