pyspark读取hive数据实例

使用pyspark读取hive中的数据，测试代码：

vi test.py

#!-*- coding:utf-8 -*-  from pyspark import SparkConf, SparkContext from pyspark.sql import HiveContext  conf = (SparkConf().setMaster("yarn").setAppName("My app").set("spark.executor.memory", "1g")) sc = SparkContext(conf = conf) sqlContext = HiveContext(sc)  my_dataframe = sqlContext.sql("Select * from database.table limit 10") my_dataframe.show()  sc.stop()

报错：
python pyspark_hive.py
Traceback (most recent call last):
  File "pyspark_hive.py", line 2, in <module>
    from pyspark.sql import HiveContext,SparkSession
ModuleNotFoundError: No module named 'pyspark'

环境变量设置：
vi ~/.profile
 

export SPARK_HOME=/usr/lib/spark-current export PYTHONPATH=$SPARK_HOME/python/:$SPARK_HOME/python/lib/py4j-0.10.7-src.zip:$PYTHONPATH

参考：
https://www.cnblogs.com/tong775131501/p/7582258.html