SQL Driver for Big Data JSON Query large volumes of configurable JSON data. The Big Data JSON driver covers 7 tables and 273 columns. ## Big Data JSON Clients Invantive UniversalSQL is available on many user interfaces ("clients" in traditional server-client paradigma). All Invantive UniversalSQL statements can be exchanged with a close to 100% compatibility across all clients and operating systems (Windows, MacOS, Linux, iOS, Android). The clients include Microsoft Excel, Microsoft Power BI, Microsoft Power Query, Microsoft Word and Microsoft Outlook. Web-based clients include Invantive Cloud, Invantive Bridge Online as OData proxy, Invantive App Online for interactive apps, Online SQL Editor for query execution and Invantive Data Access Point as extended proxy. The [[https://cloud.invantive.eu/bigdatajson|Big Data JSON Power BI connector]] is based on the Invantive UniversalSQL driver for Big Data JSON, completed by a high-performance OData connector which works straight on Power BI without any add-on. The OData protocol is always version 4, independent whether the backing platform uses OData, SOAP or another protocol. For technical users there are command-line editions of Invantive Data Hub running on iOS, Android, Windows, MacOS and Linux. Invantive Data Hub is also often used for enterprise server applications such as ETL. High-volume replication of data taken from Big Data JSON into traditional databases such as SQL Server (on-premises and Azure), MySQL, PostgreSQL and Oracle is possible using [[https://cloud.invantive.com/data-replicator/|Invantive Data Replicator]]. Invantive Data Replicator automatically creates and maintains Big Data JSON datawarehouses, possibly in combination with data from over 75 other (cloud) platforms. Invantive Data Replicator supports data volumes up to over 1 TB and over 5.000 companies. The on-premise edition of Invantive Bridge offers an Big Data JSON ADO.net provider. Finally, online web apps can be build for Big Data JSON using App Online of [[https://cloud.invantive.eu/bigdatajson|Invantive Cloud]]. ## Specifications The SQL driver for Big Data JSON supports partitioning: data from multiple companies are all listed together in one table. The value of the column partition indicates which company the data belongs to. An introduction into the concepts of Invantive UniversalSQL such as databases, data containers and partitioning can be found in the [[Grammar|Invantive UniversalSQL grammar]]. The configuration can be changed using various attributes from the database definition, on log on and during use. A full list of configuration options is listed in the [[Attributes|driver attributes]]. The catalog name is used to compose the full qualified name of an object like a table or view. The schema name is used to compose the full qualified name of an object like a table or view. On Big Data JSON the comparison of two texts is case sensitive by default. The driver has the following properties: - Driver code for use in settings.xml: `BigDataJson` - Recommended alias: `bdn` - More technical documentation as provided by the supplier of Big Data JSON on the native connection used can be found at [[https://cloud.invantive.eu/product-manuals|Technical Documentation]]. - Technical support (URL): https://forums.invantive.eu - Function documentation (URL): https://cloud.invantive.eu/data-models - Function support (URL): https://forums.invantive.eu What the driver reads The driver reads a directory of JSON files as a set of tables. Every file holds one record per line (NDJSON) or one record as a whole file, plain or compressed with gzip or Zstandard; a file of a million records is read record by record and never held whole. Which tables exist and which columns they have is stated by an OpenAPI specification, not guessed from the data, so a column keeps its name and its data type however few files hold a value for it. Setting up a connection The directory is named in 'directories' and the specification in 'swagger-specification-files'. With 'recurse-directories' the directories under it are read as well, which is how a folder per day or per machine is read as one table. 'extension' says which names are data, by default '*.*json,*.*json.gz,*.*json.zst'. Partitions A first-level subdirectory which holds a data file is a partition, and the directory itself is one for the files in its root. Every table therefore starts with the columns 'partition', 'file_path', 'file_name' and 'file_row_number', which say where a row comes from. A statement confined to a partition never walks the other directories: use ws253 select count(*) from Logging select count(*) from Logging where partition in ('ws253', 'ws267') select * from Logging where file_name like '%bridge%' A filter on 'partition', 'file_path' or 'file_name' chooses the files before one of them is opened. The index A full read of a directory describes every file it read in an index beside the data, under 'index'. The index holds per file how many records it has and, per column, the smallest and the largest value, how many records hold no value, and either the values themselves or a filter over them. A later statement uses it to skip the files which cannot hold a matching record, and to answer a count, a smallest or a largest value without opening a file at all: select count(*) from Logging select min(OccurredUtc), max(OccurredUtc) from Logging select count(*) from Logging where MessageCode = 'itgenlic125' A file which the index does not describe as it is now is described first and the answer follows from the index, so a directory which keeps growing costs a statement the files which arrived, not the whole collection. What the index holds is readable: select * from System.FileIndexFiles select * from System.FileIndexColumns where file_name = 'trace-001.ndjson.zst' select * from System.FileIndexColumnValues where column_name = 'MessageCode' select * from System.FileIndexBloomFilters The index is maintained where the data is written to; with 'file-index-directory' it is kept elsewhere, for a directory which cannot be written to. 'file-index-maintain' and 'file-index-use' switch the writing and the reading of it, and 'file-index-budget-percentage' says how much of a file's contents its description may be worth, ten percent by default: a larger percentage keeps the value lists and filters which do the skipping, a smaller one keeps the bounds and little else. Files which are not as expected A line which is not valid JSON is skipped with a warning while 'ignore-erroneous-records' is on, so one damaged line does not cost the other million; switched off, the statement fails on it. 'max-erroneous-files' says how many files may be unreadable before a statement fails. A file which holds several JSON records on one line is read as several records. What is read and what is not Only the columns a statement names are built from a record and only the records which match its filters are handed to the engine, so a narrow statement over a wide record costs what it asks for. A count of rows asks for no column at all. ## Objects Per object category a list is available: - [[Tables]] - [[Procedures]]