Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
doku:bigdata [2023/05/10 13:11] – katrin | doku:bigdata [2023/05/11 08:52] (current) – katrin | ||
---|---|---|---|
Line 93: | Line 93: | ||
# you can choose one of the following python versions | # you can choose one of the following python versions | ||
# with the current hadoop/ | # with the current hadoop/ | ||
+ | # note: the current spark version does NOT work with python 3.11 | ||
module load python/ | module load python/ | ||
# module load python/ | # module load python/ | ||
Line 145: | Line 146: | ||
module purge | module purge | ||
- | module load openjdk/ | + | module load openjdk |
module load hadoop | module load hadoop | ||
+ | |||
+ | export PDSH_RCMD_TYPE=ssh | ||
prolog_create_key.sh | prolog_create_key.sh | ||
Line 176: | Line 179: | ||
To use Big Data on VSC use those modules: | To use Big Data on VSC use those modules: | ||
- | * python/3.8.0-gcc-9.1.0-wkjbtaa | + | |
- | * openjdk/ | + | |
- | * hadoop | + | * python/3.9.13-gcc-12.2.0-ctxezzj |
- | * spark | + | * python/ |
+ | * openjdk/ | ||
+ | * hadoop | ||
+ | * spark (spark/ | ||
* r | * r | ||
+ | |||
Depending on the application, | Depending on the application, |