Added wikidata data source.

wikidata/README.md

+# Wikidata
+
+Requires a `json2bracket.py` conversion script by Thomas from
+https://frosch.cosy.sbg.ac.at/thuetter/json-datasets. Currently included with
+this repository. Better, enable getting it directly from Thomas's repo.
+
+Download the latest version of an entity in JSON format (Q40 is Austria).
+Current revision as of 09.04.2020 is `1153700198`.
+```
+wget https://www.wikidata.org/wiki/Special:EntityData/Q40.json
+```
+
+Download a specific revision of an entity to file `Q40_1151291349.json`.
+```
+wget https://www.wikidata.org/wiki/Special:EntityData/Q40.json?revision=1151291349 -O Q40_1151291349.json
+```
+
+To download and convert all data files, execute
+```
+./download.sh
+```
\ No newline at end of file

wikidata/download.sh

+#!/usr/bin/env bash
+
+mkdir raw
+
+# Austria:
+wget https://www.wikidata.org/wiki/Special:EntityData/Q40.json?revision=1153700198 -O raw/Q40_1153700198.json
+wget https://www.wikidata.org/wiki/Special:EntityData/Q40.json?revision=1151291349 -O raw/Q40_1151291349.json
+
+# Albert Einstein:
+wget https://www.wikidata.org/wiki/Special:EntityData/Q937.json?revision=1152908796 -O raw/Q937_1152908796.json
+wget https://www.wikidata.org/wiki/Special:EntityData/Q937.json?revision=1152908745 -O raw/Q937_1152908745.json
+
+# European Union
+wget https://www.wikidata.org/wiki/Special:EntityData/Q458.json?revision=1151107735 -O raw/Q458_1151107735.json
+wget https://www.wikidata.org/wiki/Special:EntityData/Q458.json?revision=1145385373 -O raw/Q458_1145385373.json
+
+# United States
+wget https://www.wikidata.org/wiki/Special:EntityData/Q30.json?revision=1156772506 -O raw/Q30_1156772506.json
+wget https://www.wikidata.org/wiki/Special:EntityData/Q30.json?revision=1156772394 -O raw/Q30_1156772394.json
+
+mkdir bracket
+
+for file in raw/*.json
+do
+  python3 json2bracket.py --filename $file > bracket/$(basename -- $file .json).bracket
+done
\ No newline at end of file

wikidata/json2bracket.py

+#!/usr/bin/env python3
+
+# The MIT License (MIT)
+# Copyright (c) 2020 Thomas Hütter.
+# 
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+# 
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+# 
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+"""json2bracket.py: Transforms a given JSON document into bracket notation."""
+
+import sys
+from argparse import ArgumentParser
+import json
+# JSON         | Python
+# -------------+--------
+# object       | dict
+# array        | list
+# string       | str
+# number(int)  | int
+# number(real) | float
+# true         | True
+# false        | False
+# null         | None
+
+def json2bracket(x):
+    if isinstance(x, dict): # OBJECT
+        print('{\{\}', end='')
+        for key, val in x.items():
+            print('{"' + key + '"', end='')
+            json2bracket(val)
+            print('}', end='')
+        print('}', end='')
+    elif isinstance(x, list): # ARRAY
+        print('{[]', end='')
+        cnt = 1
+        for val in x:
+            print('{' + str(cnt), end='')
+            json2bracket(val)
+            cnt += 1
+            print('}', end='')
+        print('}', end='')
+    else: # VALUE
+        if isinstance(x, str):
+            print('{"' + x + '"}', end='')
+        elif isinstance(x, int):
+            print('{' + str(x) + '}', end='')
+        elif isinstance(x, bool):
+            print('{' + str(x) + '}', end='')
+        else: # NULL
+            print('{' + 'null' + '}', end='')
+
+    return
+
+def main():
+    # Read command line arguments.
+    parser = ArgumentParser(description='Input parameters for JSON to bracket notation converter')
+    parser.add_argument('--filename', type=str, default="",
+                        help='Filename/-path where the collection is stored.')
+    parser.add_argument('--collection', default=False, action='store_true',
+                        help='Parse a collection of JSON documents surrounded by an array.')
+    parser.add_argument('--print', default=False, action='store_true',
+                        help='Print the header with dataset info.')
+    args = parser.parse_args()
+
+    with open(args.filename) as json_file:
+        data = json.load(json_file)
+        records = len(data)
+
+        # Print header with dataset statistics.
+        if args.print:
+            print("BRACKET NOTATION:")
+            print("#record: " + str(records))
+            print()
+
+        # Based on input parameters either parse (1) a collection 
+        # nested in an array or (2) a single document.
+        if args.collection:
+            for d in data:
+                json2bracket(d)
+                print()
+        else:
+            json2bracket(data)
+            print()
+
+    print()
+
+    return
+
+if __name__ == '__main__':
+    main()