Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
Last revisionBoth sides next revision
doku:bigdata [2023/05/10 13:11] – katrindoku:bigdata [2023/05/11 08:52] – [Available modules] katrin
Line 93: Line 93:
 # you can choose one of the following python versions # you can choose one of the following python versions
 # with the current hadoop/spark installation on VSC-4 # with the current hadoop/spark installation on VSC-4
 +# note: the current hadoop version does NOT work with python 3.11
 module load python/3.10.7-gcc-12.2.0-5a2kkeu module load python/3.10.7-gcc-12.2.0-5a2kkeu
 # module load python/3.9.13-gcc-12.2.0-ctxezzj # module load python/3.9.13-gcc-12.2.0-ctxezzj
Line 145: Line 146:
  
 module purge module purge
-module load openjdk/11.0.2-gcc-9.1.0-ayy5f5t+module load openjdk
 module load hadoop module load hadoop
 +
 +export PDSH_RCMD_TYPE=ssh
  
 prolog_create_key.sh prolog_create_key.sh
Line 176: Line 179:
 To use Big Data on VSC use those modules: To use Big Data on VSC use those modules:
  
-  * python/3.8.0-gcc-9.1.0-wkjbtaa +  * python (note: the current spark version does NOT work with python 3.11) 
-  * openjdk/11.0.2-gcc-9.1.0-ayy5f5t +    * python/3.10.7-gcc-12.2.0-5a2kkeu 
-  * hadoop +    * python/3.9.13-gcc-12.2.0-ctxezzj 
-  * spark+    * python/3.8.12-gcc-12.2.0-tr7w5qy 
 +  * openjdk/11.0.15_10-gcc-11.3.0-ih4nksq 
 +  * hadoop (hadoop/3.3.2-gcc-11.3.0-ypaxzwt) 
 +  * spark (spark/3.1.1-gcc-11.3.0-47oqksq)
   * r   * r
 +
  
 Depending on the application, choose the right combination: Depending on the application, choose the right combination:
  • doku/bigdata.txt
  • Last modified: 2023/05/11 08:52
  • by katrin