SDS Project Configuration 1 - Setup

Now that I have LaTeX and computations working in Emacs Org Mode (see LaTeX with Emacs and Setting Up Computational Notebook in Emacs), it is time to create an entire Statistics and Data Science Project covering a variety of topics. This note shall document the setup process.

Prerequisites

1. Basic Emacs Configurations. Note Git (Magit), Org Mode, Org Babel, Vterm, and the programming languages are properly configured.
2. Emacs + LaTeX Configuration.
3. Emacs + (Python + R + Julia).

Project Structure

Create the essential folders and files inside the appropriate directory.

mkdir -p Chapters Data/{Raw,Processed} Figures Python R Julia
touch README.org

The tree structure of the project should look like this.

SDS
├── Chapters
├── Data
│   ├── Processed
│   └── Raw
├── Figures
├── Julia
├── Python
├── R
└── README.org

Edit README.org with a description of the project.

Create Virtual Environments

Python

1. From SDS, create a Python virtual environment. Generally, Git should not be track this.

python3 -m venv Python/.venv

2. Activate it. This should start showing (.venv) in the prompt.

source Python/.venv/bin/activate

3. If necessary, while the environment is activates, upgrade pip.

python -m pip install --upgrade pip

4. Install necessary packages. This will also install various dependencies.

python -m pip install numpy pandas matplotlib scipy seaborn

5. Freeze requirements.

python -m pip freeze > Python/requirements.txt

This file should be tracked by Git. All the installed packages should be listed here. Run this command from time to time to keep the list of installed packages updated.

R

1. Install renv.

install.packages("renv")

2. In .../SDS/R open R and create an R environment.

renv::init()

3. Restart R and then install the necessary packages. This can take a while.

install.packages(c("tidyverse", "janitor","skimr"))

4. Take a snapshot of the environment.

renv::snapshot()

This is saved in the lockfile, renv.lock. Again, update it as needed. This environment can be restored on a different machine with the renv::restore() command.

Julia

1. From .../SDS/Julia activate Julia.
2. Press ] to open the package manager. The prompt should say pkg>.
3. Run the following to activate a project.

activate .

4. Install packages.

(Julia) pkg> add DataFrames CSV Distributions StatsBase StatsPlots

5. Project.toml and Manifest.toml are automatically created in this process.

Setting Up Org Babel to Use Proper Environments

The org-babel-python-command and the corresponding variables for the other languages need to be set appropriately so that Org Babel knows which environment to use. For this, it is helpful to add certain function in the emacs config files.

In my-functions.el add these functions for making use of the Python environment.

(defun my/system-python ()
  (interactive)
  (setq org-babel-python-command "auto")
  (message "Org Babel: system Python environment"))

(defun my/sds-python ()
  (interactive)
  (setq org-babel-python-command
      "/home/___/Documents/Projects/SDS/Python/.venv/bin/python")
  (message "Org Babel: SDS Python environment"))

Now my/sds-python can be used to configure Org Babel to use the SDS Python virtual environment. (Note: this is different from ‘activating’ the SDS Python virtual environment using the command source Python/.venv/bin/activate.) The my/system-python function reverts to the default.

R has a library system that manages virtual environments differently. By running R from within .../SDS/R, the appropriate renv would be activated. But we need it to work with org files anywhere in the SDS directory. Hence, the function looks a little different.

(defun my/sds-r ()
  (interactive)
  (setq org-babel-R-command
    "cd /home/___/Documents/Projects/SDS/R && R --slave --no-save")
  (message "Org Babel: SDS R environment"))

Now, whenever an org file in the SDS folder needs to access this R environment, this function can be run. There is no need for a corresponding reverse function because of how environments in R works.

For Julia, add the following function. Note: The function below is deprecated due to the use of Julia-Vterm. See SDS Project Configuration 2 for the updated function.

(defun my/sds-julia ()
  (interactive)
  (setq org-babel-julia-command
    "julia --project=/home/___/Documents/Projects/SDS/Julia")
  (message "Org Babel: SDS Julia environment"))

Again, Julia does not need to escape this environment with a reverse function. Just opening it without --project= works.

Python is the only language out of the three for which the project environment has a separate interpreter that needs to be explicitly selected. Below is a table summarizing how to enter and exit the project environments.

Testing

Be sure to test all Org Babel is able to use the appropriate environments and all the programs are running as desired using an org file like this. To get the code blocks to execute, use M-x+my/sds-python,my/sds-r, or my/sds-julia as needed.

* Python

#+begin_src python :results output
import sys
import numpy
import pandas
import scipy
import seaborn

print(sys.executable)
print("numpy:", numpy.__version__)
print("pandas:", pandas.__version__)
print("scipy:", scipy.__version__)
print("seaborn:", seaborn.__version__)
#+end_src

#+RESULTS:
: /home/___/Documents/Projects/SDS/Python/.venv/bin/python
: numpy: 2.5.2
: pandas: 3.0.5
: scipy: 1.18.1
: seaborn: 0.13.2

* R
#+begin_src R :results output
library(tidyverse)
library(janitor)
library(skimr)

x <- rnorm(1000)

mean(x)
#+end_src

#+RESULTS:
: [1] -0.01990596

#+begin_src R
library(renv)

print(renv::project())
print(R.version.string)

x <- rnorm(1000)
mean(x)
#+end_src

#+RESULTS:
: 0.0194806010562875

#+begin_src R :results output
cat("renv project:", renv::project(), "\n")
cat("working directory:", getwd(), "\n")
cat("R executable:", R.home("bin"), "\n")
#+end_src

#+RESULTS:
: renv project: /home/___/Documents/Projects/SDS/R 
: working directory: /home/___/Documents/Projects/SDS/R 
: R executable: /usr/lib64/R/bin 

#+begin_src R :results value
library(ggplot2)
packageVersion("ggplot2")
#+end_src

#+RESULTS:
: 4.0.3

#+begin_src R :results value
library(dplyr)

x <- rnorm(1000)
mean(x)
#+end_src

#+RESULTS:
: 0.0354994074352827

* Julia

#+begin_src julia :results output
println("active project: ", Base.active_project())
#+end_src

#+RESULTS:
: active project: /home/___/Documents/Projects/SDS/Julia/Project.toml

#+begin_src julia :results value
using DataFrames
using CSV
using Distributions

df = DataFrame(x = randn(1000))
first(df, 5)
#+end_src

#+RESULTS:
|  0.2723598619652979 |
| 0.20312551810424087 |
|  0.1639445504167204 |
|  0.9493135192287597 |
|   1.395687213107897 |

Version Control

Initialize Git at the root (SDS). I am not going over the Magit commands here. These will be used frequently, and also my Magit keybindings are likely to be different from the defaults.

The .gitignore file should specify the following.

Python/.venv/
R/renv/library/
R/renv/staging/

At this point, the following files should be tracked by Git/Magit.

.gitignore
Chapters/test.org
Julia/Manifest.toml
Julia/Project.toml
Python/requirements.txt
R/.Rprofile
R/renv.lock
R/renv/.gitignore
R/renv/activate.R
R/renv/settings.json
README.org

Commit these files to the repository appropriately.