Visitar URL original

Home · holdenk/spark-testing-base Wiki · GitHub

Repository navigation

Home

Jump to bottom Edit New page

Mahmoud Hanafy edited this page Mar 3, 2020 · 7 revisions

Welcome to the spark-testing-base wiki!

Project Main Modules:

SharedSparkContext: Provides SparkContext to be used in testing.
RDDComparisons: Compares two RDDs.
DataFrameSuiteBase: Checks for dataframe equality.
DatasetSuiteBase: Checks for dataset equality.
StreamingSuiteBase: Applies given operation on given input stream and compare with expected output.
StreamingActionBase: Tests actions that don't have output on input streams.
RDDGenerator: Generates arbitrary RDDs.
DataFrameGenerator: Generates arbitrary DataFrames.
DatasetGenerator: Generates arbitrary Datasets.
HDFSCluster: Simulate working with HDFS in tests.