Starter Guide: Allen and the LHCb Stack — Installation, Development, Usage

4 minute read

This blog post combines various notes and technical instructions related to Allen, the LHCb stack, CERN computing resources, authenticating with SSH and Kerberos, configuring Git, and more.

Building Allen: Standalone

LXPLUS

Install ssh key for GitLab.

Clone the repo:

git clone ssh://git@gitlab.cern.ch:7999/lhcb/Allen.git

Build with CVMFS and CPU target:

source /cvmfs/lhcb.cern.ch/lib/LbEnv
mkdir build
cd build
cmake -DSTANDALONE=ON -DCMAKE_TOOLCHAIN_FILE=/cvmfs/lhcb.cern.ch/lib/lhcb/lcg-toolchains/LCG_101/x86_64-centos7-clang12-opt.cmake ..
make

Install Python dependencies:

pip install wrapt pydotplus pydot sympy cachetools

Put input MDF file into Allen/input/ from /scratch/allen_data/mdf_input/.

Run Allen:

./toolchain/wrapper ./Allen --sequence hlt1_pp_default --mdf ../input/upgrade_mc_minbias_scifi_v5_retinacluster_000_v1_newLHCbID.mdf -n 500 -m 500 -r 10 -t 16

LXPLUS GPU

Clone the repo:

git clone ssh://git@gitlab.cern.ch:7999/lhcb/Allen.git

Build with CVMFS and GPU target:

source /cvmfs/lhcb.cern.ch/lib/LbEnv
mkdir build
cd build
cmake -DSTANDALONE=ON -DCMAKE_TOOLCHAIN_FILE=/cvmfs/lhcb.cern.ch/lib/lhcb/lcg-toolchains/LCG_103/x86_64_v3-el9-gcc12+cuda12_1-opt.cmake ..
make -j32

MacOS M1

Install Homebrew:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Clone the repo:

git clone ssh://git@gitlab.cern.ch:7999/lhcb/Allen.git

Install dependencies:

brew install llvm cpp-gsl catch2 zeromq nlohmann-json python3 boost
pip3 install --user wrapt cachetools pydot sympy

Clone UMESIMD:

git clone https://github.com/edanor/umesimd.git /Users/<username>/umesimd
export UMESIMD_ROOT_DIR=/Users/<username>/

Symlink libclang:

ln -s /Library/Developer/CommandLineTools/usr/lib/libclang.dylib /usr/local/lib/libclang.dylib

Install CMake and ROOT:
```
brew install cmake pkgconfig root
```

Build Allen:

mkdir build
cd build
cmake -DSTANDALONE=ON ..
make

Put input MDF file into Allen/input/ from /scratch/allen_data/mdf_input/.

Run Allen:

./Allen --sequence hlt1_pp_default --mdf ../input/upgrade_mc_minbias_scifi_v5_retinacluster_000_v1_newLHCbID.mdf -n 500 -m 500 -r 10 -t 16

Building Allen: Gaudi / LHCb Stack

Setup the environment:

cd /afs/cern.ch/work/<username>/
curl https://gitlab.cern.ch/rmatev/lb-stack-setup/raw/master/setup.py | python3 - stack
cd stack
$EDITOR utils/config.json
BINARY_TAG=x86_64_v2-centos7-gcc11-opt
make Allen

Running Allen

Event loop steered by Allen:

./Allen/build.x86_64_v2-centos7-gcc11-opt/run python Allen/Dumpers/BinaryDumpers/options/allen.py --mdf Allen/input/minbias/mdf/MiniBrunel_2018_MinBias_FTv4_DIGI_retinacluster_v1.mdf

Event loop steered by Moore:

./Moore/run gaudirun.py Moore/Hlt/Moore/tests/options/default_input_and_conds_hlt1_retinacluster.py Moore/Hlt/Hlt1Conf/options/allen_hlt1_pp_default.py

Installing in CVMFS

All packages available in LCG are here: /cvmfs/sft.cern.ch/lcg/releases.
Look in the directory if you can find the package, if not, you have to install it yourself, e.g. /cvmfs/sft.cern.ch/lcg/releases/gtest.
Look in the package directory for a version and a platform/binary tag that works for you, e.g. /cvmfs/sft.cern.ch/lcg/releases/gtest/1.11.0-21e8c/x86_64-centos9-gcc11-opt.
Check if the package is included in the lcg-toolchains file that is being used. The file that handles that is in the repository lcg-toolchains in fragments/packages, look for the one that matches the platform and LCG version you’re using. In this case LCG_103cuda-x86_64-centos9-gcc11-opt.cmake.
Add the package if needed, e.g. by adding a line to the file: _add_lcg_entry("${LCG_releases_base}/gtest/1.11.0-21e8c/x86_64-centos9-gcc11-opt").

Adding a New Device Algorithm in Allen

Add a subdirectory in device called new_algo, with src, include and a CMakeLists.txt.
Modify device/CMakeLists.txt to take the new folder into account.
```
add_subdirectory(new_algo)
```

In device/new_algo/CMakeLists.txt, use allen_add_device_library() to add the new library, and give the include path with target_include_directories().

file(GLOB new_algo_sources "src/*cu")

allen_add_device_library(New_Algo STATIC
  ${new_algo_sources}
)

target_include_directories(New_Algo PUBLIC $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>)

The include AlgorithmTypes.cuh is including other files, e.g. BackendCommon.h, so use target_link_libraries with the libraries below, to use all these other libraries in Allen. In the same file add:
```
target_link_libraries(New_Algo PRIVATE Backend HostEventModel EventModel Utils)
```

To instantiate an algorithm, INSTANTIATE_ALGORITHM(), you have to link the new library New_Algo to the Stream library: modify stream/CMakeLists.txt.

target_link_libraries(Stream
  PRIVATE
  ...
  New_Algo
  ...
)

Why? Because otherwise you get the linker error:

libAllenLib.so: undefined reference to `Allen::TypeErasedAlgorithm 
Allen::instantiate_algorithm<new_algo::new_algo_t>(std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > const&)'

Define a global kernel, in order to be able to execute the algorithm on the device. In the .cuh file:
```
__global__ 
void new_algo(Parameters);
```

Add the implementation of the kernel in the .cu file:

__global__ 
void new_algo::new_algo(new_algo::Parameters parameters)
{
...
}

Define the methods of the algorithm, keeping in mind that an algorithm must define two methods: set_arguments_size and operator(). Also, in order to invoke host and global functions, wrapper methods host_function and global_function should be used.
Implement the algorithm.
Add the algorithm to the sequence.

Run:

./toolchain/wrapper ./Allen --sequence new_algo --mdf /scratch/allen_data/mdf_input/upgrade-magdown-sim10-up08-30000000-digi_01_retinacluster_v1_newLHCbID.mdf -n 500 -m 500 -r 100 -t 16

Allen Commands

Measuring Physics Performance

./toolchain/wrapper ./Allen --sequence hlt1_pp_validation --mdf ../input/Upgrade_BsPhiPhi_MD_FTv4_DIGI_retinacluster_v1_newLHCbID.mdf -n 500 -m 500 -r 1000 -t 16

Measuring Computational Performance

./toolchain/wrapper ./Allen --sequence hlt1_pp_default --mdf ../input/Upgrade_BsPhiPhi_MD_FTv4_DIGI_retinacluster_v1_newLHCbID.mdf -n 500 -m 500 -r 1000 -t 16

LHCb Stack Instructions

Setup on LHCb Online

Clone and configure:

curl https://gitlab.cern.ch/rmatev/lb-stack-setup/raw/master/setup.py | python3 - stack
cd stack
utils/config.py binaryTag x86_64_v3-el9-gcc13+detdesc-opt+g
make Moore BUILDFLAGS='-j 10'

Generate MDF Files

Initialize the lhcb proxy:

source /cvmfs/lhcb.cern.ch/lib/LbEnv-stable
lhcb-proxy-init

Dump an MDF file:

./Moore/run gaudirun.py Moore/Hlt/Moore/tests/options/default_input_and_conds_hlt1.py Moore/Hlt/RecoConf/options/mdf_for_standalone_Allen.py

CERN Systems Access

LXPLUS

Connect:
```
ssh -XY <username>@lxplus.cern.ch
```

Connect with port forwarding:

ssh -XY -L 1235:localhost:1235 <username>@lxplus.cern.ch

Launch Jupyter notebook:

jupyter notebook --port=1235 --no-browser

LXPLUS GPU

Direct connection:
```
ssh <username>@lxplus-gpu.cern.ch
```

If problems occur, connect first to lxplus and then to gpu:

ssh <username>@lxplus.cern.ch
ssh <username>@lxplus-gpu.cern.ch

CERN LHCb Online GPU

Connect through LXPLUS:
```
ssh -tt lbgw ssh n4050101
```

Chain ssh directly:

ssh -tt <username>@lxplus.cern.ch ssh -tt lbgw ssh n4050101

File Storage at CERN

File storage at CERN is normally organized as follows:

User directory: /afs/cern.ch/user/<username>/
Work directory: /afs/cern.ch/work/<username>/
Private EOS: /eos/user/<username>/
Public EOS: /eos/lhcb/user/<username>/

VSCode Troubleshooting

If you cannot connect with VSCode, log in manually via terminal first, then retry from VSCode.
Connect via terminal and delete the VSCode server folder
```
rm -rf ~/.vscode-server
```
and then retry connecting.
Sometimes can cause problems: Untick Remote.SSH: Use Flock in the remote VSCode server settings.

Obtaining a CERN Grid Certificate

Finally, if you need help obtaining a grid certificate see my earlier post: Obtaining and Renewing a CERN Grid Certificate in LHCb.

SSH Configuration File

For help configuring your ssh configuration file see this file.

Authenticating with Kerberos

For full instructions see this website.

Instructions on MacOS

Install kerberos (MacOS normally has kerberos already installed).
Copy the CERN krb5.conf file (found on the website above) to your /etc/krb5.conf.
Modify your ssh config file (~/.ssh/config), under the lxplus host add:
```
GSSAPIAuthentication yes
GSSAPIDelegateCredentials yes
```
Get kerberos tickets using /usr/bin/kinit:
```
/usr/bin/kinit <username>@CERN.CH
```
Check which kinit is being used:
```
which kinit
```

Git/GitLab Configuration

SSH authentication to GitLab:

Generate key:

ssh-keygen -t ed25519 -C "<cern_email>@cern.ch"

Copy its contents:

tr -d '\n' < ~/.ssh/id_ed25519.pub | pbcopy

Add to GitLab.
Verify key:
```
ssh git@gitlab.cern.ch -T -p 7999
```

Recommended Git configuration:

git config --global user.name "<name>"
git config --global user.email "<cern_email>@cern.ch"
git config --global color.ui "auto"

git config --global core.autocrlf input

git config --global credential.helper 'cache --timeout=28800'

Done! ✅

Share on

X Facebook LinkedIn Bluesky

Starter Guide: Allen and the LHCb Stack — Installation, Development, Usage

Building Allen: Standalone

LXPLUS

LXPLUS GPU

MacOS M1

Building Allen: Gaudi / LHCb Stack

Running Allen

Installing in CVMFS

Adding a New Device Algorithm in Allen

Allen Commands

Measuring Physics Performance

Measuring Computational Performance

LHCb Stack Instructions

Setup on LHCb Online

Generate MDF Files

CERN Systems Access

LXPLUS

LXPLUS GPU

CERN LHCb Online GPU

File Storage at CERN

VSCode Troubleshooting

Obtaining a CERN Grid Certificate

SSH Configuration File

Authenticating with Kerberos

Instructions on MacOS

Git/GitLab Configuration

Done! ✅

Share on

Comments

You May Also Enjoy

The 2025 Breakthrough Prize in Fundamental Physics Awarded to the LHCb Collaboration

Notes for New Developers on LHCb’s Allen Codebase

Work on the LIP6 GPU Cluster: A Quick Guide

Obtaining and Renewing a CERN Grid Certificate in LHCb