The more I come to know PowerPivot, the more I believe that modelled data warehouses‘ days are numbered. I didn’t say data warehouses per se, rather those that attempt to centrally model end user reporting structures (usually as star-schemas).

There will continue to be a need for centrally controlled data warehouses (or at least simplified data views (and/or copies) of operational datasets, either provided by system vendors of by in-house IT) to bridge the raw-to-actionable data gap. But I suspect the emphasis will change from providing finished goods to providing semi-processed raw materials.

So, will the star-schema become redundant? No, as it still a valid method of modelling a reporting requirement in order to make many queries simpler to phrase (this obviously applies to SQL , but also to DAX queries). But, those who build them will be doing so closer to the problem at hand, and specific to that problem (I’ve discussed this before in http://blog.gobansaor.com/2011/01/08/slowly-changing-dimensions-time-to-stop-worrying/).

For many reports the barely modified operational data model will be all that’s required (for example, DAX doesn’t require “fact” header/detail tables to be flattened to detail level, as would be the case with a classic star).

“Good Enough” models will become the norm; classic “Everything You Ever Wanted to Know” centralised models a luxury for most (especially as such models tend to “age” very quickly).

If you’re about to invest or re-furbish your data warehouse or your reporting data sub-systems, don’t do so without first taking a serious look at PowerPivot. This is a game-changer, not just for full-stack Microsoft BI shops, but for any business that finds that their reporting datasets invariably end-up in Excel.

If you need any help evaluating PowerPivot or modelling your reporting needs in PowerPivot,  I’m for hire.

Update:

Just in case you think I’m an dimensional-model un-believer or likely to abandon my star-schema roots read this….

Star Schemas: to explore strange new conformed dimensions, to seek out new measures, to boldly go where no Excel spreadsheet has gone before.

Follow @gobansaor

I’ve also tested the code against the just released SQLServer 2010 RC0 version of PowerPivot and it appears to work.

I’ve updated the equivalent PPREFRESH code in HAMMER and this is now part of the tool’s latest release (V1.3.4(Beta)). My previous post, Excel as a Book of Record, previewed the most important new commands available in this release. Alongside those, I’ve also added the following:

• ISTABLE, if the previous COMMAND’s result  or the previous argument is not a table, this will abend the command sequence.
• ISARG, as above, but this time checks for an argument (a HAMMER parameter may either be a table, a command or an argument- aka, an ARG).
• ISOK, previous argument must be the string value “OK”.
• TABLESARETHESAME, will fail if the last two tables are not identical. Intended mainly for automated regression testing.
• ARGSARETHESAME, as above. but this time for ARGS.
• _GUID, will return a globally unique identifier.

Download the latest version of HAMMER from here …

When such systems are primarily reporting in emphasis, the justification for using Excel is usually straight forward and compelling (and getting even more compelling with the appearance on the scene of Excel 2010′s  PowerPivot.) But, alarm bells sound across the world of professional IT when Excel is proposed as a “book of record” i.e. when it’s to be used to store and maintain a business critical dataset. And, with some considerable justification, the nightmare that is linked (or even worse, shared) workbooks is very real indeed. But yet, businesses continue to do so, and do so quite successfully.

I myself record my business as a series of Excel documents (Invoices, Timesheets, Expenses) in a set of folders (Financial Years subdivided into major customers).  Essentially a document-oriented database.

In the past I simply then used a VBA powered workbook to open the required “documents” and extracted the data necessary for whatever report I required (VAT, year-end etc.).  To better automate (i.e. less bespoke VBA) this task I’ve have made changes to HAMMER to help with this and with similar tasks for clients.

The following list of commands will be added to the next release of HAMMER. (In the meantime these new functions can be previewed here …)

LISTOFFILES

This command takes a single argument, the folder to search, and will return a table of files in that folder and in any sub-folders. The result can then be used to select a list of files for further processing.

Example:

=HAMMER(“C:\a\rss”,”LISTOFFILES”)

_XLTOJSONDOC

This command takes a list of workbooks, opens each one, checks for a list of named ranges and generates a JSON document. The command is intended to be called from within a VBA macro (as opening and closing workbook breaks the “no side effects” rule of UDFs). Most “_” commands such as “_MD5″ etc. are likewise intended for “programming use”, but any command beginning with “_XL” must be restricted to macro (i.e. non-UDF) use.

See the example workbook FetchInventory for an example of this function in action. The function takes one argument (the name of the document to load) and expects a table where the last column is the full name of the workbook to open. Any columns in the source table will be copied to the new “JSON document” table with an additional column called “Document” which will hold a JSON document representing the key-name pairs and table(s) extracted from the workbook.

On opening a workbook, it is searched for a named range with the name of document concatenated with “_fields” (e.g. INVENTORY_fields). The value of this range is expected to be a CSV list of fields and tables to load. A single Excel “document” could contain multiple logical documents (each specified by its own “_fields” list) .

See the PartsInventory_bin4 for an example of a multi-document workbook (INVENTORY and EXAMPLE). The EXAMPLE document in this workbook also demonstrates the various types of tables handled.

Example:

lJSONObjects = oHammer.HAMMER(“C:\a\rss\StockTake1″,”LISTOFFILES”,”Select name,fullname from table2 limit 1″,”SQL”,”INVENTORY”,”_XLTOJSONDOC”)

lReturn = oHammer.HAMMERToRange(lJSONObjects,”Sheet2!A27″)

… will output

JSONDOCVIEW

This command is where the previous commands are leading to, i.e. extracting some real information value from your documents. It converts JSON documents into Excel friendly tables. It is, in essence, a Map function as in MapReduce. In a previous example I used a Python Map and a SQL Reduce, here, both Map and Reduce are via SQL (the command uses a series of SQL commands to perform its task).

Before I describe the function let me explain why I use an intermediate JSON format. I could just extract the data directly from each document and either store directly in Excel or create tables in SQLite of Access to hold this data. And in fact, that’s what I would have done in the past (seeExcel as a document-oriented NoSQL database). Now , however, I tend to favour using a free-format (i.e. no need for a fixed database schema) structure like a JSON document, so as the source documents evolve over time (which tends to happen not just during design stages but as the system matures) this will not break older documents.

So, for example, original Invoice workbooks might not have a backing time-sheet while newer Invoices do. As long as new and old documents share a core sub-set of data fields they can continue to be analysed together.

The command takes 5 arguments and a driving table (a record so far for HAMMER commands, most have a max of two arguments). The driving table’s last column is assumed to contain the JSON document to process, columns prior to this (if any) will be output unchanged for each resulting row.

The first argument specifies the name of the “inner” table to fetch (if any). Most real life documents consist of header details (the “outer”document) and one or more tables (“inner” details). Invoices, time-sheets,  stock-takes, all tend to follow this pattern. This command will effectively join each document’s outer details to a single inner table (if more than 1 inner table, a call for each one is required).

The second (field list in SQL format) and third (SQL where predicate format) arguments specify what inner fields to extract (if blank, then all) and what restrictions to impose (if any). So “InvNo, Date”,”InvNo > 12″ would only fetch documents where the InvNo > 12 and only include the InvNo and Date fields.

The fourth and fifth arguments do the same for the outer table (i.e. Header data).

If any of the columns specified  (inner or outer) can not be found, or if the predicates (inner or outer) result in no selection, no error is returned, the document simply returns no rows. Likewise if an inner table is specified and no such table exists, then no rows are returned for that document – in other words this is not an outer join, which is not usually a problem as in most cases a “header” without detail lines is meaningless. If an outer join is required, then extract the headers (outers) and details (inner tables(s)) separately and join using SQL.

Example:

=HAMMER(“Select Name,FullName,Document from invoice_docs”,”SQL”,”table_2″, “[PART NUMBER],QTY”, “QTY >30″, “Bin_Number”, “Bin_Number > 1″, “JSONDOCVIEW”)

would result in:

For more complex JSON objects use the JSON command to incrementally parse the text or use the VBA JSON module within microETL. But for most situations (especially if you control the expected format) JSONDOCVIEW should handle it.

As JSON is fast becoming the preferred transport format for web and mobile applications having the ability to parse and produce JSON form within Excel is very useful. It is possible, for example, to use a simple web technology such as http://robla.net/jsonwidget/ to craft another type of Really Simple System. This time with the collection happening on the web (most likely using AWS S3 pre-signed forms, so no HTML server required – keep it simple) but with the control and reporting remaining within Excel (a variation on my Steam Powered Server idea).

For an example of a really simple system  download this.

Latest version of HAMMER including the above commands now released …

I call this “HAMMER Alongside”, to differentiate it from HAMMER Inside where I use HAMMER internals to craft stand alone XLLs. With this COM Server method, the standard HAMMER add-in (installed or just-in-time registered) is needed alongside either a VBA add-in or a VBA enabled workbook. The extra “moving parts” are easily justified by the extra flexibility that this method allows (particularly to those with a reasonable grasp of VBA, but perhaps lacking any familiarity with the .NET world).

By learning a small amount of IronPython (here’s a good starting place) it’s relatively easy to hook up VBA to any .NET library through HAMMER, without having to invest time and money in learning heavy duty development environments such as Visual Studio. Using “Internal Threads, it’s also possible to take advantage of .NETs multi-threading capability from VBA. (Note: both these options require .NET4).

So with VBA becoming a first-class language with regards to HAMMER, why offer Python as an alternative scripting choice? Well, one reason is to offer a means to access .NET’s power as per the last paragraph. But, the primary reason, is related to the ability of HAMMER transformations to be “detached” from Excel and run via HAMMER’s command line tools. The .NET 4 version of the command line tool supports Python, so it is possible to initially build out a micro-ETL transformation within Excel (using Python to perform the business logic alongside SQL) and then transfer that logic to the command-line tools with a minimum of modification.

This could be useful , for example, when a PowerPivot model moves to the server, its related HAMMER-powered micro-ETL processing could do likewise (most enterprise ETL tools support call-outs to command-line processes).

The COM server can be accessed from VBA only via late-binding like so:

Set comServ = CreateObject(“hammerCOMServerV1″)

The server’s methods are:

• hammer(…) – works the same way as the UDF version.
• hammerVersion() – returns HAMMER version, again the same as UDF version.
• hammerVersionOK(version) – e.g.  isOK= comServ.hammerVersionOK(120) will return TRUE if the current version is >= V1.2.0
• arrayResize(anArray) enables the creation of “toFit” UDFs.
• arrayToSheet(anArray) likewise for “toSheet” UDFs.
• arrayToRange(anArray,pasteToWhereString) enables “toRange” UDFs.
• hammer_ppRefresh_inline(optional table,optional timeout) – refreshes a PowerPivot model, again the same as its UDF equivalent.

Download the latest version of HAMMER from here …

The functionality introduced by Excel Tables is a huge improvement and if you haven’t checked it out I recommend you do; it and 2010′s PowerPivot are two of the most important enhancements to Excel since the PivotTable.

Nevertheless, if you are to write UDFs to handle tabular data you’re still talking arrays (although you can reference a table within a UDF, returning tabular data still requires an array). Array formulas also continue to a very powerful skill for those who wish to master Excel and again if you’ve not done so, do check them out. But for those of you who have an aversion to  CTRL+SHIFT+ENTER, HAMMER (and HAMMER Inside UDFs) offers a number of array helper functions:

• HAMMERtoFit – will resize the selected destination range to fit the returned array (result remains an array). Has the disadvantage that the underlying function call will be called twice if the array area requires resizing.
• HAMMERtoSheet – will output the array to a new sheet as a non-array table. This (and a properly sized array call to HAMMER) is the fastest method of returning variable sized datasets to Excel.
• HAMMERtoRange – will paste the array to the range address specified (as a non-array table), will also run an optional VBA macro before/after the paste.

The first two functions have been available for some time. The toRange helper is new to this release (V1.1.1). Before I describe the new function in detail, it’s useful to understand how HAMMER (and  any C based XLL ) sees and serves-back array data. An array range passed to such a function will be converted into a two dimensional multi-type array. The two types of data that will be passed over are: Doubles  (all numerics and dates), and Strings (everything else). For those of you familiar with the Excel Object Model, that’s the equivalent of the VALUE2 property of the Range object. The apparent date conversion to a Double (e.g. 01-JAN-2001 passed as 36892) is, in fact, not a conversion, as dates in Excel are always simply numbers with date formatting applied.

Likewise,when data is returned to Excel, the data comes back in the same “raw” format, requiring date or currency formatting to be applied to the relevant cells.

The HAMMER function and the two helper functions toSheet and toFit operate to these conventions. The third helper function HAMMERtoRange (along with the XLRANGE set of commands) can work around this restriction (at a certain performance cost).

HAMMERtoRange:

The function is called in the exact same way as the parent HAMMER function but with an additional argument. This argument is a comma separated list of 1,2 or 3 elements.

If only one element supplied it’s assumed to be the address to paste the table to; if two elements, first is the address, 2nd is the name of a VBA macro to run after the data is pasted; if 3 elements, the 1st is a macro to run prior to the paste , 2nd is the address, and 3rd is  the post-paste macro.  Examples:

• =HAMMERtoRange(…,”Sheet2!a4″)
• =HAMMERtoRange(…,”MyNamedRange”)
• =HAMMERtoRange(…,”Sheet2!a4,myCleanupMacro”)
• =HAMMERtoRange(…,”,Sheet2!a4,”)
• =HAMMERtoRange(…,”mySetupMacro,Sheet2!a4,myCleanupMacro”)
• =HAMMERtoRange(…,”onlyASetupMacro,Sheet2!a4,”)

The address is specified as a “string” i.e. doesn’t reference a range object (otherwise the result would be an infinite loop). If you do mange to initiate a loop, exit it using the Esc key, if you don’t, the loop will eventually end in one of two ways, either with a nice exception or a, not so nice, failure of Excel. The toFit helper will never loop as it is using Excel native array functionality and Excel will protect against this; the toSheet has the potential to loop but less likely than the toRange.

You may find the HAMMERtoRange function much slower that the other helpers (due to its dependence on COM Automation rather than the C API) but only for large datasets. Making use of the pre/post macros doesn’t require much VBA skills as the outputs of “record macro” sessions are ideal for this type of processing.

The address examples above,such as “Sheet1!A1″, are the “top left-hand corner” of the resulting table. The function will clear any data within the “.CurrentRegion” of that cell, but will first check if an Excel Table already occupies that region, and if so, reconstitute a table of the same name after the new dataset is pasted.

If a cell is sourced from a SQLite date column (actually a date “cell” as SQLite, like Excel, uses manifest typing) the data will be formatted as a date (this is unlike the behaviour of toFit and toSheet). This will not normally be obvious if the source data is loaded into HAMMER via a range (see discussion above) as the data will have been delivered to SQLite as a Double. To get around this either:

• Update the affected column within HAMMER using the SQLiteDate function via SQL e.g. “Update table1 set datejoined = SQLiteDate(datejoined)”
• or, use the pre/post macros to format the date columns as dates
• or, load the dataset using these newly added COMMANDS: XLRANGE or EXCELRANGEASTEXT.

XLRANGE: This command expects an address in “string” format (i.e. not a referenced range) from which it will load a table using Excel Range’s Value property. Using Value rather than Value2 means the internal function logic will know if a cell is a date or not and store that data in SQLite date format. The address can either be a “top left-hand corner” type single-cell address or a full range; if a single-cell address, then the cell’s “.CurrentRegion”  is taken to be the extent of the table to be loaded.

Being able to directly access the PowerPivot model opens up all sorts of interesting options, one such is a web service.

In the past I’ve written about exposing Excel as a simple DIY web service, see here for a JavaScript example and here for a Python version. Both were capable of serving PowerPivot data but they both had to trigger events in Excel to do so.

With this newly discovered ability to query the model directly, I figured a third attempt at a PowerPivot Web Service was overdue.  This time I’m using the new kid on the block, HAMMER, and in particular using its in-built IronPython interpreter.

I could have built a simple server using VB.NET or C# and added it as a HAMMER command, but the nature of a web service makes it hard to frame as a parametrised command. It was for this type of requirement that I added the ability to script using Python, i.e. situations where a generic command would always come up  short, for example:

• Should the service allow only local connections?
• Should it use authentication, and if so, what type?
• Should it allow free-format commands or offer only packaged commands?
• Should it use HTTPListener (which is in-built and very powerful, but requires Admin privileges) or use a more basic socket based utility that can run without Admin privileges.
• Should it block or handle asynchronous requests. (For services that access the Excel Object Model, it has to be blocking, but perhaps if the service simply accessed the AS engine, it night support multi-threaded queries? Must try it sometime.)

For this example I decided to use HTTPListener (in blocking mode and without authentication) and to expose services that accept free-format MDX,DMV or (if Denali) DAX Table Queries.

See the PPWebService workbook example in the latest (V0.5.0) version of HAMMER. To activate, make sure latest version of HAMMER is installed (or use activate button on Hammer sheet) and click the Expose PowerPivot Model as Web Service button (again on Hammer sheet). This will start a server on port 8070.

Dax table queries are a new feature of DAX (part of the Denali PowerPivot upgrade) that in my opinion takes the DAX language away from its (very useful) pivot table birth-place and makes it a truly powerful alternative to SQL/MDX for analytical reporting.

It was Chris Webb’s excellent  series of articles on DAX that first wetted my appetite for this feature and then hinted that it could be accessed from Excel and ultimately automated if need be.

An so it can, the VBA code that I used to query DMVs and send XMLA process commands to refresh the PowerPivot model can also be used for MDX and DAX tabular queries.

And it gets better; you don’t need to use code to do this, there’s a very simple trick which will expose this functionality in plain Excel.

This trick only applies to Denali as it takes advantage of one of Denali’s most welcome new features, show detail (aka drill-thru). (An alternative method which can be used in PowerPivot V1 is detailed here  - if using V1, only DMV and MDX queries would be possible).

First thing to do is (in a PowerPivot Denali powered workbook) make a pivot table (doesn’t matter of what), such as this …

… then right-click and show-detail on any measure cell, which should result in a new sheet with a Excel table showing the drill-through, like so …

… if you then right-click, pick Tables and then Edit Query, you’ll see the MDX query associated with the table …

… you’re now free to edit this query, replacing it with whatever takes your fancy, here’s an MDX tabular query …

… you can also issue DMV queries such as “Select * from [Model].[$InvoiceHeaders]” or its equivalent DAX table query …

Note: both the DMV and DAX ‘dump a dimension’ are unlikely to work for very large tables (such as a multi-million row facts) unless you have loads of free memory and the ability to use it (i.e. 64 bit), without triggering a “could not allocate memory error”.

DAX is not limited to DMV type queries but can be used to express very complex OLAP queries, Being able to define new  temporary just-in-time Measures (and Variables) for use within a particular queries will help not just to produce the result required, but will make DAX easier to master for newcomers.

The current version of HAMMER only allows DMV and MDX queries as it restricts the “ADO” command  to “SELECT” statements; the next release will remedy that.

I’m at present working on an example of using HAMMER to build a DMV/MDX/DAX Web Service such as I did with MicroETL in Python-Powered PowerPivot. This time using IronPython and the .NET’s HttpListener and simply passing the queries straight-through to PowerPivot without any need for Excel Pivot Table involvement.  So stay tuned … Update: (here it is)

Although the resulting tool is proving to be very useful in its standalone formats (as a UDF formula, VBA tool via Application.Run and as a command-line EXE) its use as  a powerful utility library within VB.NET or C# add-ins is still what justifies the cost of its development.

Over the last few weeks I’ve used HAMMER in the development of several ETL-focused add-ins with great success. That success is not just to be measured in speed and accuracy of development but in the extra features I can now deliver for little or no cost (caching of API or database calls for example). I term them my “HAMMER Inside” add-ins.

The final destination is to allow me to offer fixed-price (or at least transparent price) development of  ETL-focused add-ins. Not just the classic database or flat file backends, but WEB APIs and calculation engines (such as pricing /discounting tables) etc.).

Such add-ins would be “opinionated software”, in that they’d have a limited look’n'feel, with a concentration on the hewing and hauling of data, not on fine-detail Excel automation or advanced/pretty UI features.

In the meantime, HAMMER has been updated with a series of new COMMANDs:

• JSON – Load a JSON document into a table.
• TOJSON – Convert a table to a JSON document. If a single row table, will generate a JSON “dictionary” object with the column headers as the Keys, and the column data as the Values. If a multi-row table will generate a JSON List of Lists.
• TOJSONLOL  - force a JSON List of Lists (when a single row table is really a table not a dictionary).
• TOJSONLOO – force tables to be represented as a List of (Dictionary) Objects rather than a List of Lists.
• TEXTTOFILE – output an argument as an UTF8 encoded text files e.g. HAMMER(myTable,”TOJSON”,”C:\files\myTable.json”,”TEXTTOFILE”)
• TABLETOSCALAR – converts a table to a scalar value by picking the first column of the first row.
• SQLRAW – Like SQL, but doesn’t perform any token substitutions. The range of tokens that can be substituted by SQL has also been increased. The existing “:n” tokens and “from tablen”, where n is the argument number, continue to be supported; but new formats of :ARGn ,:<ARGn ,:TABLEn, and :<TABLEn have been added.  As before, n will be the argument number for :ARG and :TABLE but in the case of :<ARG and :<TABLE, n will represent the previous nth argument or table (think “<’ pointing backwards).
• URLGET and URLPOST – fetch or post data to a URL.
• There’s also a series of “_” functions, mainly intended for use within VBA or .NET but could be useful as formula calls too, I guess. These are: _URLENCODE (encode a URL), _HAMCSHA1 (calculate a HMACSHA1 hash),_MD5 (MD5 hash),_MD5FILE (same but for a file),_GETCONFIG (reads addin’s .config file’s HAMMER JSON section) and _GETCONFIGNOERROR (same but doesn’t complain if no .CONFIG file or no JSON section found).

A live specimen - not a datasmith but a related data consuming species: The PacMan

If you’ve been following my ramblings posts for any length of time you’ll have come across the term “civilian datasmith” quite a lot. For example, in my last two posts on DAX I’ve been looking at whether they’re likely to use PowerPivot’s DAX.

Professional datasmiths, like myself, will have little choice but to learn and adopt DAX if we intend to continue to service either the high-end Excel management information market or the general MS BI tools market, but will our civilian peers follow suit? I think many will, particularly those who are already comfortable with high-end “excel formula programming” or those who already have extensive SQL and/or data modelling experience. But who are these non-IT datasmiths?

Well, they’re very unlikely to have data anywhere in their job titles and often share their job titles with dataphobic colleagues. But you’ll know them, they’re usually the “go-to-guys/gals” for all sort of data analysis/integration/discovery tasks and they tend to favour making decisions based on hard facts rather than gut instinct. They’ve learned to make use of whatever data and whatever tools are at hand, while others complain about tools/systems not doing what they want or IT not reading their minds, datasmiths just get on with making things happen.

An example of a real live civilian datasmith in action is Richard Baxter of SEOGadget. I know Richard from Excel development work he’s commissioned from me, including this Google Adwords Excel Add-in that appears to have been all the rage at this year’s MozCon conference.

I was at first somewhat surprised that an SEO professional was  commissioning Excel add-ins, as the typical SEO tools tend to be web-based (usually by default I think,rather than by rational choice), but I soon realised I’d found yet another datasmith enabled profession (to add to the more typical accountants, engineers, sales & marketing managers etc.).

Richard’s a classic datasmith.  To see him in action  check out the slides of his well received MozCon presentation. Note how he sources data from multiple sources but uses Excel to merge and shape the data (one other tool I’d suggest he should check out would be my HAMMER, ideal when multiple tables need to be queried,merged, cross-joined etc.).

So if you’re an SEO pro (or indeed any data dependent professional) struggling with too much data, or worrying that decisions are been made based on too little data, check out the following:

1. Microsoft Excel for SEO http://www.distilled.net/excel-for-seo/ not just for SEO types, an excellent run down on the Excel skills any would-be datasmith must have.
2. See Richard’s Adwords Add-in for Excel, a good example of the sort of thing Excel can do through the power of custom-coded add-ins. (UPDATE: Here’s a video of the tool in action)
3. Then check out HAMMER, my swiss-army-knife-like  datasmithing tool for those whose working lives revolve around  data tables (primarily an Excel add-in for use as an array formula, but can also be called from VBA and via the command-line (aka DOS box)) .

When I started in this business in the late 1970s, the database world I encountered was a noSQL one. Database structures were in the main laid out by DBAs using either hierarchical (e.g. MUMPS) or network topologies (the various CODASYL offerings). Programmers would then navigate these predetermined access paths using a cacophony of APIs and query languages. It worked quite well, and if fact many developers found the move to a SQL world difficult, as they regarded the pure relational model to be sub-optimal when developing applications (and many still do!).

But there were two main problems with such databases:

• Although the pre-defined access paths worked well for development, they were often next to useless for downstream reporting, which usually meant that data had to be dumped to files for “reshaping” and de-normalisation, with resulting flattened data often being consumed by early end-user reporting tools such as Datatrieve. (Sound familiar?)
• Likewise, although the original data design tended to be ideal for the original application design, downstream changes in requirements were not easy to incorporate. DBAs were busy, expensive and usually very grumpy, which meant that many enhancements, as with their reporting cousins, had to make do with using external files.

The SQL era was a huge improvement. Nevertheless, many developers found the impedance mismatch between this relational SQL world and their own  ”networked”  object-oriented world to be a continuing problem, leading to the constant search for the perfect ORM and eventually the re-emergence of noSQL database engines.

PowerPivot’s VertiPaq engine is one of these new noSQL databases. It resembles those pre-SQL databases in that its access and relationships paths are largely determined not by its DAX query language but by whoever builds the database in the 1st place. The big difference is of course, PowerPivot’s primary (and only) use-case is reporting while most  databases have in the past targeted system datastorage as their primary purpose in life, with reporting as a by-the-way. The other difference is that the creators of PowerPivot “databases” are less likely to be DBAs and more likely to be end-users or at least those in IT, like data/business analysts, who are close to the business i.e. more common on the ground, less expensive and hopefully less grumpy. Indeed many civilian DAX practitioners will have the luxury of being able to populate their reporting models with suitable datasets that have already been constructed by existing data warehousing systems.

It’s this separation of filtering- aggregation-transformation from topology that I think will help endear DAX to a less technical audience that has been the case with SQL. MDX also had this concept, but the underlying “more natural” tabular nature (business users live by lists!) that DAX builds on, will I think,be easier for many to engage with.

I used “I think” at lot in the previous paragraph as it’s very difficult for somebody like myself (a professional programmer and veteran of many a database mind-share war) to know how non-technical folks will actually view something like DAX. Although my career has been one largely based on business-user interaction, I’m still a techie at heart.

So what do you civilian datasmiths think? DAX, yep I’ll learn it, or nah, that’s what those geeks down in the basement are for!