PySpark
E702182
PySpark is the Python API for Apache Spark, enabling large-scale data processing, analysis, and machine learning using Python.
All labels observed (7)
| Label | Occurrences |
|---|---|
| PySpark canonical | 2 |
| pyspark.accumulators | 1 |
| pyspark.files | 1 |
| pyspark.ml | 1 |
| pyspark.mllib | 1 |
| pyspark.profiler | 1 |
| pyspark.sql.functions | 1 |
How this entity was disambiguated
This entity first appeared as the object of triple T7984784 — resolving that mention is where its identity was fixed. The disambiguator weighed these candidate entities and picked the highlighted one (or “None”, minting a new entity). This is how homonymy is resolved: the same surface form can point to different entities.
NED1
Entity disambiguation (via context triple)
gpt-5-mini-2025-08-07
Target entity: PySpark Context triple: [Apache Spark, supportsLanguageAPI, PySpark]
-
A.
Apache Spark
Apache Spark is an open-source, distributed data processing engine designed for large-scale data analytics, machine learning, and stream processing.
-
B.
Spark
"Spark" is a virtuosic jazz fusion composition by Japanese pianist Hiromi Uehara, showcasing her signature blend of technical brilliance and energetic, genre-blurring style.
-
C.
Apache Pig
Apache Pig is a high-level platform for creating MapReduce programs used to analyze large data sets in the Hadoop ecosystem.
-
D.
Python (via Snowpark)
Python (via Snowpark) is Snowflake’s integration of the Python language for building and running data pipelines, machine learning, and other data applications directly within the Snowflake data platform.
-
E.
Apache Flink
Apache Flink is an open-source distributed stream-processing framework designed for high-throughput, low-latency data processing and real-time analytics on large-scale data.
- F. None of above. chosen
- G. Unsure - the case is ambiguous/there is not enough information to decide.
NED2
Entity disambiguation (via description)
gpt-5-mini-2025-08-07
Target entity: PySpark Target entity description: PySpark is the Python API for Apache Spark, enabling large-scale data processing, analysis, and machine learning using Python.
-
A.
Apache Spark
Apache Spark is an open-source, distributed data processing engine designed for large-scale data analytics, machine learning, and stream processing.
-
B.
Spark
"Spark" is a virtuosic jazz fusion composition by Japanese pianist Hiromi Uehara, showcasing her signature blend of technical brilliance and energetic, genre-blurring style.
-
C.
Apache Pig
Apache Pig is a high-level platform for creating MapReduce programs used to analyze large data sets in the Hadoop ecosystem.
-
D.
Python (via Snowpark)
Python (via Snowpark) is Snowflake’s integration of the Python language for building and running data pipelines, machine learning, and other data applications directly within the Snowflake data platform.
-
E.
Apache Flink
Apache Flink is an open-source distributed stream-processing framework designed for high-throughput, low-latency data processing and real-time analytics on large-scale data.
- F. None of above. chosen
Statements (65)
| Predicate | Object |
|---|---|
| instanceOf |
Apache Spark component
ⓘ
Python API ⓘ big data framework component ⓘ software library ⓘ |
| canBeUsedWith |
AWS EMR
ⓘ
linked to:
Amazon EMR
Azure Synapse Analytics ⓘ Databricks ⓘ Google Dataproc ⓘ
linked to:
Google Cloud Dataproc
|
| compatibleWith |
Hadoop ecosystem
ⓘ
linked to:
Hadoop
|
| developer | Apache Software Foundation ⓘ |
| documentation | https://spark.apache.org/docs/latest/api/python/ ⓘ |
| hasComponent |
pyspark.RDD
ⓘ
pyspark.accumulators ⓘ
linked to:
PySpark
pyspark.broadcast ⓘ pyspark.conf ⓘ pyspark.context ⓘ pyspark.files ⓘ
linked to:
PySpark
pyspark.ml ⓘ
linked to:
PySpark
pyspark.mllib ⓘ
linked to:
PySpark
pyspark.profiler ⓘ
linked to:
PySpark
pyspark.resource ⓘ pyspark.serializers ⓘ pyspark.sql ⓘ pyspark.sql.DataFrame ⓘ pyspark.sql.SparkSession ⓘ pyspark.sql.Window ⓘ pyspark.sql.functions ⓘ
linked to:
PySpark
pyspark.sql.types ⓘ pyspark.storagelevel ⓘ pyspark.streaming ⓘ pyspark.taskcontext ⓘ |
| interoperatesWith |
Jupyter Notebook
ⓘ
NumPy ⓘ pandas ⓘ scikit-learn ⓘ |
| license | Apache License 2.0 ⓘ |
| partOf | Apache Spark ⓘ |
| programmingLanguage | Python ⓘ |
| repository | https://github.com/apache/spark ⓘ |
| runsOn |
Apache Spark cluster
ⓘ
linked to:
Spark
Kubernetes ⓘ YARN ⓘ standalone Spark cluster ⓘ |
| supports |
ETL workloads
ⓘ
SQL queries ⓘ batch processing ⓘ distributed computing ⓘ graph processing ⓘ large-scale data processing ⓘ machine learning ⓘ stream processing ⓘ |
| supportsLanguageFeature |
DataFrame API
ⓘ
RDD API ⓘ SQL API ⓘ pandas user-defined functions ⓘ structured streaming ⓘ user-defined functions ⓘ |
| typicalUseCase |
data analytics
ⓘ
data engineering ⓘ data warehousing ⓘ machine learning pipelines ⓘ |
| uses |
Spark Core
ⓘ
linked to:
Arduino core for ESP8266
Spark MLlib ⓘ Spark SQL engine ⓘ Spark Streaming ⓘ |
How these facts were elicited
The pipeline generated the facts above by prompting gpt-5.1 with this entity's name + description and the instruction below.
Instruction
You are a knowledge base construction expert. Given a subject entity and a description of it, return factual statements that you know for the subject as a JSON list of dictionaries(triples), where keys must be "subject", "predicate" and "object". The number of facts may be very high, between 25 to 50 or more, for very popular subjects. For less popular subjects, the number of facts can be very low, like 5 or 10. # Requirements - If you don't know the subject at all, return an empty list. - If the subject is not a named entity, return an empty list. - Include at least one triple where predicate is "instanceOf". - Do not get too wordy. - Separate several objects into multiple triples with one object.
Input
Subject: PySpark Description of subject: PySpark is the Python API for Apache Spark, enabling large-scale data processing, analysis, and machine learning using Python.
Referenced by (8)
Full triples — surface form annotated when it differs from this entity's canonical label.
linked to: PySpark
linked to: PySpark
linked to: PySpark
linked to: PySpark
linked to: PySpark
linked to: PySpark