DCAT-AP dump to LP-DAV (NKOD)
type: Pipeline
tag: NKOD
version: 2
tag: dcat-ap
profile: https://etl.csu.opendata.cz/resources/pipelines/1546611885800/2

Chunked RDF to wrapped JSON-LD
type: Component
configuration graph: https://etl.csu.opendata.cz/resources/pipelines/1546611885800/66
template: http://etl.linkedpipes.com/resources/components/t-rdfToWrappedJsonLdChunked/0.0.0
x: 1400
y: 760
description: Distributions

CouchDB
type: Component
configuration graph: https://etl.csu.opendata.cz/resources/pipelines/1546611885800/graph/52
template: http://etl.linkedpipes.com/resources/components/l-couchDb/0.0.0
x: 1800
y: 760

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/12
type: Connection
source binding: OutputRdf
source component: Chunk splitter
target binding: InputRdf
target component: Chunked RDF to wrapped JSON-LD

Chunk splitter
type: Component
configuration graph: https://etl.csu.opendata.cz/resources/pipelines/1546611885800/69
template: http://etl.linkedpipes.com/resources/components/t-chunkSplitter/0.0.0
x: 1180
y: 760

SPARQL construct to file list
type: Component
configuration graph: https://etl.csu.opendata.cz/resources/pipelines/1546611885800/graph/76
template: http://etl.linkedpipes.com/resources/components/t-sparqlConstructToFileList/0.0.0
x: 540
y: 740
description: Separate datasets to graphs

Files to RDF Chunked
type: Component
configuration graph: https://etl.csu.opendata.cz/resources/pipelines/1546611885800/graph/56
template: http://etl.linkedpipes.com/resources/components/t-filesToRdfChunked/0.0.0
x: 880
y: 760

SPARQL construct
type: Component
configuration graph: https://etl.csu.opendata.cz/resources/pipelines/1546611885800/graph/54
template: http://etl.linkedpipes.com/resources/components/t-sparqlConstruct/0.0.0
x: 1520
y: 600
description: Prepare for SOLR

SPARQL construct
type: Component
configuration graph: https://etl.csu.opendata.cz/resources/pipelines/1546611885800/graph/74
template: http://etl.linkedpipes.com/resources/components/t-sparqlConstruct/0.0.0
x: 260
y: 900
description: Generate graph separation queries

Files to RDF single graph
type: Component
configuration graph: https://etl.csu.opendata.cz/resources/pipelines/1546611885800/graph/58
template: http://etl.linkedpipes.com/resources/components/t-filesToRdfGraph/0.0.0
x: 120
y: 740

HTTP get
type: Component
configuration graph: https://etl.csu.opendata.cz/resources/pipelines/1546611885800/graph/64
template: http://etl.linkedpipes.com/resources/components/e-httpGetFile/0.0.0
x: 120
y: 640
description: DCAT-AP dump

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/20
type: Connection
source binding: OutputRdf
source component: Files to RDF Chunked
target binding: InputRdf
target component: Chunk splitter

CouchDB
type: Component
configuration graph: https://etl.csu.opendata.cz/resources/pipelines/1546611885800/graph/78
template: http://etl.linkedpipes.com/resources/components/l-couchDb/0.0.0
x: 1800
y: 860

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/23
type: Connection
source binding: OutputFiles
source component: SPARQL construct to file list
target binding: InputFiles
target component: Files to RDF Chunked

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/24
type: Connection
source binding: OutputRdf
source component: Files to RDF Chunked
target binding: InputRdf
target component: SPARQL update chunked

SPARQL update chunked
type: Component
configuration graph: https://etl.csu.opendata.cz/resources/pipelines/1546611885800/68
template: http://etl.linkedpipes.com/resources/components/t-sparqlUpdateChunked/0.0.0
x: 880
y: 860
description: Delete distributions

SPARQL construct
type: Component
configuration graph: https://etl.csu.opendata.cz/resources/pipelines/1546611885800/graph/84
template: http://etl.linkedpipes.com/resources/components/t-sparqlConstruct/0.0.0
x: 1320
y: 960

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/27
type: Connection
source binding: OutputRdf
source component: SPARQL construct
target binding: Tasks
target component: SPARQL construct to file list

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/28
type: Connection
source binding: FilesOutput
source component: HTTP get
target binding: InputFiles
target component: Files to RDF single graph

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/29
type: Connection
source binding: OutputRdf
source component: SPARQL construct
target binding: InputRdf
target component: Mustache

Mustache
type: Component
configuration graph: https://etl.csu.opendata.cz/resources/pipelines/1546611885800/graph/60
template: http://etl.linkedpipes.com/resources/components/t-mustache/0.0.0
x: 2020
y: 600
description: Build index

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/32
type: Connection
source binding: OutputFiles
source component: Chunked RDF to wrapped JSON-LD
target binding: FilesInput
target component: CouchDB

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/33
type: Connection
source binding: OutputRdf
source component: Files to RDF single graph
target binding: InputRdf
target component: SPARQL construct

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/34
type: Connection
source binding: OutputRdf
source component: Files to RDF single graph
target binding: InputRdf
target component: SPARQL construct to file list

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/35
type: Connection
source binding: OutputRdf
source component: Files to RDF single graph
target binding: InputRdf
target component: SPARQL construct

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/36
type: Connection
source binding: OutputRdf
source component: Files to RDF single graph
target binding: InputRdf
target component: SPARQL construct

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/37
type: Connection
source binding: OutputRdf
source component: SPARQL update chunked
target binding: InputRdf
target component: Chunk splitter

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/38
type: Connection
source binding: OutputFiles
source component: Mustache
target binding: FilesInput
target component: Load to Solr

Files to RDF single graph
type: Component
configuration graph: https://etl.csu.opendata.cz/resources/pipelines/1546611885800/graph/80
template: http://etl.linkedpipes.com/resources/components/t-filesToRdfGraph/0.0.0
x: 1780
y: 400

CouchDB
type: Component
configuration graph: https://etl.csu.opendata.cz/resources/pipelines/1546611885800/graph/82
template: http://etl.linkedpipes.com/resources/components/l-couchDb/0.0.0
x: 1800
y: 960

HTTP get
type: Component
configuration graph: https://etl.csu.opendata.cz/resources/pipelines/1546611885800/graph/71
template: http://etl.linkedpipes.com/resources/components/e-httpGetFile/0.0.0
x: 1780
y: 320
description: File Type EU MDR NAL

SPARQL construct
type: Component
configuration graph: https://etl.csu.opendata.cz/resources/pipelines/1546611885800/72
template: http://etl.linkedpipes.com/resources/components/t-sparqlConstruct/0.0.0
x: 1780
y: 500
description: Filter English labels

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/42
type: Connection
source binding: OutputRdf
source component: SPARQL construct
target binding: InputRdf
target component: Mustache

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/43
type: Connection
source binding: OutputRdf
source component: Files to RDF single graph
target binding: InputRdf
target component: SPARQL construct

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/44
type: Connection
source binding: FilesOutput
source component: HTTP get
target binding: InputFiles
target component: Files to RDF single graph

Mustache
type: Component
configuration graph: https://etl.csu.opendata.cz/resources/pipelines/1546611885800/65
template: http://etl.linkedpipes.com/resources/components/t-mustache/0.0.0
x: 1600
y: 960

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/46
type: Connection
source binding: OutputFiles
source component: Mustache
target binding: FilesInput
target component: CouchDB

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/47
type: Connection
source binding: OutputRdf
source component: SPARQL construct
target binding: InputRdf
target component: Mustache

Load to Solr
type: Component
configuration graph: https://etl.csu.opendata.cz/resources/pipelines/1546611885800/graph/62
template: http://etl.linkedpipes.com/resources/components/l-solr/0.0.0
x: 2000
y: 760

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/6
type: Connection
source binding: OutputRdf
source component: Chunk splitter
target binding: InputRdf
target component: Chunked RDF to wrapped JSON-LD

Chunk splitter
type: Component
configuration graph: https://etl.csu.opendata.cz/resources/pipelines/1546611885800/graph/50
template: http://etl.linkedpipes.com/resources/components/t-chunkSplitter/0.0.0
x: 1160
y: 860

Chunked RDF to wrapped JSON-LD
type: Component
configuration graph: https://etl.csu.opendata.cz/resources/pipelines/1546611885800/67
template: http://etl.linkedpipes.com/resources/components/t-rdfToWrappedJsonLdChunked/0.0.0
x: 1400
y: 860
description: Datasets

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/9
type: Connection
source binding: OutputFiles
source component: Chunked RDF to wrapped JSON-LD
target binding: FilesInput
target component: CouchDB

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/65
type: Configuration
add first flag: true
add first flag control: None
class: http://localhost/temp/Root
class control: None
template: {{! PREFIX mustache: <http://plugins.linkedpipes.com/ontology/t-mustache#> }} { "_id" : "keywords_by_publishers", "json": [ {{#http://localhost/temp/entry}} {{^mustache:first}},{{/mustache:first}} {"value" : "{{http://www.w3.org/ns/dcat#keyword}}", "count" : "{{http://www.w3.org/1999/02/22-rdf-syntax-ns#value}}" } {{/http://localhost/temp/entry}} ] }
template control: None

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/66
type: Configuration
id resource type: Distribution
id resource type control: None
template: { "_id":"{{ID}}", "jsonld":{{JSONLD}} }
template control: None

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/67
type: Configuration
id resource type: Dataset
id resource type control: None
template: { "_id":"{{ID}}", "jsonld":{{JSONLD}} }
template control: None

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/68
type: Configuration
query: DELETE { ?distribution ?p ?o } WHERE { ?distribution a <http://www.w3.org/ns/dcat#Distribution> ; ?p ?o . }
query control: None

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/72
type: Configuration
query: construct { ?s <http://www.w3.org/2004/02/skos/core#prefLabel> ?label . } where { ?s <http://www.w3.org/2004/02/skos/core#inScheme> <http://publications.europa.eu/resource/authority/file-type> ; <http://www.w3.org/2004/02/skos/core#prefLabel> ?label . FILTER(LANG(?label) = 'en') }
query control: None

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/49
type: Configuration
incoming level depth: 1
incoming level depth control: None
type: http://www.w3.org/ns/dcat#Dataset
type control: None

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/51
type: Configuration
batch size: 8
batch size control: None
clear before loading: true
clear before loading control: None
database: distributions
database control: None
password control: None
url: http://127.0.0.1:5984
url control: None
use authentication: false
use authentication control: None
user name control: None

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/53
type: Configuration
query: PREFIX dcat: <http://www.w3.org/ns/dcat#> CONSTRUCT { <http://localhost/resources/root> a <https://data.gov.cz/slovník/nkod/Root> ; <http://plugins.linkedpipes.com/ontology/t-mustache#fileName> "nkod_dump_for_solr.json" ; <https://data.gov.cz/slovník/nkod/dataset> ?dataset . ?dataset <http://purl.org/dc/terms/modified> ?modifiedStr ; <http://purl.org/dc/terms/accrualPeriodicity> ?accrualPeriodicityStr ; <http://purl.org/dc/terms/description> ?escaped_description ; <http://purl.org/dc/terms/issued> ?issuedStr ; <http://purl.org/dc/terms/publisher> ?publisherStr ; <https://data.gov.cz/slovník/nkod/publisherName> ?escaped_publisherName ; <http://purl.org/dc/terms/title> ?escaped_title ; <https://data.gov.cz/slovník/nkod/keywords> ?keywordEntity ; <http://localhost/ontology/themes> ?theme ; <http://purl.org/dc/terms/format> ?format ; <http://purl.org/dc/terms/license> ?license ; <https://data.gov.cz/slovník/nkod/temporalStart> ?temporalStartStr ; <https://data.gov.cz/slovník/nkod/temporalEnd> ?temporalEndStr ; <http://purl.org/dc/terms/spatial> ?spatialStr . ?keywordEntity <http://localhost/value> ?keywordStr . ?format <http://localhost/value> ?formatStr ; <http://localhost/name> ?formatName . ?license <http://localhost/value> ?licenseStr . } WHERE { ?dataset a <http://www.w3.org/ns/dcat#Dataset> ; <http://purl.org/dc/terms/description> ?description ; <http://purl.org/dc/terms/title> ?title . OPTIONAL { ?dataset <http://www.w3.org/ns/dcat#distribution> ?distribution . ?distribution a <http://www.w3.org/ns/dcat#Distribution> . } OPTIONAL { ?dataset <http://purl.org/dc/terms/modified> ?modified . } OPTIONAL { ?dataset <http://purl.org/dc/terms/accrualPeriodicity> ?accrualPeriodicity . } OPTIONAL { ?dataset <http://purl.org/dc/terms/issued> ?issued . } OPTIONAL { ?dataset <http://purl.org/dc/terms/publisher> ?publisher . ?publisher <http://xmlns.com/foaf/0.1/name> ?publisherName . } OPTIONAL { ?dataset <http://www.w3.org/ns/dcat#keyword> ?keyword . } OPTIONAL { ?dataset <h…
query control: None

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/55
type: Configuration
commit size: 1
commit size control: None
mime type control: None
soft fail: false
soft fail control: None

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/57
type: Configuration
commit size: 50000
commit size control: None
mime type control: None
soft fail: false
soft fail control: None

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/59
type: Configuration
add first flag: true
add first flag control: None
class: https://data.gov.cz/slovník/nkod/Root
class control: None
template: {{! PREFIX mustache: <http://plugins.linkedpipes.com/ontology/t-mustache#> }} [ {{#https://data.gov.cz/slovník/nkod/dataset}} {{^mustache:first}},{{/mustache:first}} { "iri": "{{{@id}}}", "modified" : "{{http://purl.org/dc/terms/modified}}", "accrualPeriodicity" : "{{http://purl.org/dc/terms/accrualPeriodicity}}", "description" : "{{{http://purl.org/dc/terms/description}}}", "issued" : "{{http://purl.org/dc/terms/issued}}", "publisherName" : "{{{https://data.gov.cz/slovník/nkod/publisherName}}}", "title" : "{{{http://purl.org/dc/terms/title}}}", "keyword" : [{{#https://data.gov.cz/slovník/nkod/keywords}}{{^mustache:first}},{{/mustache:first}}"{{{http://localhost/value}}}"{{/https://data.gov.cz/slovník/nkod/keywords}} ], "theme" : [{{#http://localhost/ontology/themes}}{{^mustache:first}},{{/mustache:first}}"{{{@id}}}"{{/http://localhost/ontology/themes}} ], "formatName" : [{{#http://purl.org/dc/terms/format}}{{^mustache:first}},{{/mustache:first}}"{{{http://www.w3.org/2004/02/skos/core#prefLabel}}}"{{/http://purl.org/dc/terms/format}} ], "license" : [{{#http://purl.org/dc/terms/license}}{{^mustache:first}},{{/mustache:first}}"{{{http://localhost/value}}}"{{/http://purl.org/dc/terms/license}} ], "temporal-start" : "{{https://data.gov.cz/slovník/nkod/temporalStart}}", "temporal-end" : "{{https://data.gov.cz/slovník/nkod/temporalEnd}}", "spatial" : "{{{http://purl.org/dc/terms/spatial}}}" } {{/https://data.gov.cz/slovník/nkod/dataset}} ]
template control: None

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/61
type: Configuration
core: dcat-ap-viewer
core control: None
replace: true
replace control: None
server: http://localhost:8983/solr
server control: None

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/63
type: Configuration
encode url: false
encode url control: None
file name: test.trig
file name control: None
file uri: https://raw.githubusercontent.com/linkedpipes/dcat-ap-viewer/develop/lp-etl/test.trig
file uri control: None
hard redirect: false
hard redirect control: None
user agent control: None

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/70
type: Configuration
encode url: false
encode url control: None
file name: filetypes-skos.rdf
file name control: None
file uri: http://publications.europa.eu/resource/distribution/file-type/rdf/skos_core/filetypes-skos.rdf
file uri control: None
hard redirect: true
hard redirect control: None
user agent control: None

configuration
type: Configuration
query: PREFIX srtfl: <http://plugins.linkedpipes.com/ontology/t-sparqlConstructToFileList#> PREFIX dcat: <http://www.w3.org/ns/dcat#> CONSTRUCT { ?dataset a srtfl:Task ; srtfl:fileName ?filename ; srtfl:fileFormat "text/turtle" ; srtfl:hasTaskQuery ?tqIRI . ?tqIRI a srtfl:TaskQuery ; srtfl:query ?query . } WHERE { ?dataset a dcat:Dataset . BIND(UUID() as ?tqIRI) BIND(CONCAT(STR(?tqIRI), ".ttl") as ?filename) BIND(CONCAT(""" PREFIX dcat: <http://www.w3.org/ns/dcat#> PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#> PREFIX skos: <http://www.w3.org/2004/02/skos/core#> PREFIX dcterms: <http://purl.org/dc/terms/> PREFIX foaf: <http://xmlns.com/foaf/0.1/> PREFIX schema: <http://schema.org/> PREFIX vcard: <http://www.w3.org/2006/vcard/ns#> PREFIX void: <http://rdfs.org/ns/void#> PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> PREFIX adms: <http://www.w3.org/ns/adms#> PREFIX spdx: <http://spdx.org/rdf/terms#> PREFIX org: <http://www.w3.org/ns/org#> PREFIX tú: <https://data.gov.cz/slovník/nkod/typ-úložiště-datové-sady/> PREFIX tz: <https://data.gov.cz/slovník/nkod/typ-datové-sady-dle-zdroje/> PREFIX nkod: <https://data.gov.cz/slovník/nkod/> CONSTRUCT { ?catalog dcat:dataset <""",STR(?dataset),"""> ; dcat:record ?record. <""",STR(?dataset),"""> a dcat:Dataset, ?typZáznamu ; dcterms:title ?title ; dcterms:description ?description ; dcterms:issued ?issued ; dcat:distribution ?distribution ; dcterms:modified ?modified ; dcterms:spatial ?spatial ; dcterms:temporal ?temporal ; dcterms:language ?language ; dcterms:publisher ?publisher ; dcat:keyword ?keyword ; dcat:landingPage ?landingPage ; dcterms:accrualPeriodicity ?accrualPeriodicity ; nkod:accrualPeriodicity ?LaccrualPeriodicity ; nkod:ruian_code ?Lruian_code ; nkod:ruian_type ?Lruian_type ; nkod:lkod ?lkod ; dcterms:identifier ?identifier ; dcat:theme ?theme ; foaf:page ?page ; dcterms:type ?type ; dcat:contactPoint ?cp . ?theme skos:inScheme ?scheme . ?temporal a dcterms:…
query control: None

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/75
type: Configuration
deduplication: false
deduplication control: None

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/77
type: Configuration
batch size: 8
batch size control: None
clear before loading: true
clear before loading control: None
database: datasets
database control: None
password control: None
url: http://127.0.0.1:5984
url control: None
use authentication: false
use authentication control: None
user name control: None

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/79
type: Configuration
commit size: 50000
commit size control: None
mime type control: None
soft fail: false
soft fail control: None

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/81
type: Configuration
batch size: 8
batch size control: None
clear before loading: true
clear before loading control: None
database: static
database control: None
password control: None
url: http://127.0.0.1:5984
url control: None
use authentication: false
use authentication control: None
user name control: None

https://etl.csu.opendata.cz/resources/pipelines/1546611885800/83
type: Configuration
query: PREFIX dcterms: <http://purl.org/dc/terms/> PREFIX dcat: <http://www.w3.org/ns/dcat#> PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> PREFIX local: <http://localhost/temp/> CONSTRUCT { <http://localhost/resource/root> a local:Root; <http://plugins.linkedpipes.com/ontology/t-mustache#fileName> "publisher_keywords.json" ; local:entry ?entryID. ?entryID a local:Entry ; rdf:value ?počet_poskytovatelů ; dcat:keyword ?kwstring. } WHERE { { SELECT ?keyword (count (distinct ?poskytovatel ) as ?počet_poskytovatelů) WHERE { ?datova_sada a dcat:Dataset ; dcterms:publisher ?poskytovatel ; dcat:keyword ?keyword . } GROUP BY ?keyword HAVING (?počet_poskytovatelů > 1) } BIND(IRI(CONCAT("http://localhost/temp/", LANG(?keyword), "/", ENCODE_FOR_URI(STR(?keyword)))) as ?entryID) BIND(STR(?keyword) as ?kwstring) }
query control: None