[Docs]

Knowledge-Debugger is an open-source interactive tool for inspection and knowledge editing in transformer-based language models. This repository includes the code and links for data files required for running Knowledge-Debugger on GPT2-XL. This tool is easily adaptable to other models, new knowledge editing methods and metrics. These modifications only require a few changes.

KnowledgeDebugger is able to run Models that build upon the Transformer Architecture. Tested with GPT2-XL and Llama2/Llama3-style Models.

Contributions are welcome!

⚙️ Requirements

The tool runs in a React and python environment with Flask and Streamlit installed. In addition, the exploration view uses an Elasticsearch index. To set up the environment, please follow the steps below:

1. Clone this repository:

   git clone https://github.com/ebnz/lm-debugger
   cd lm-debugger

2. Create a Python 3.10 environment and install dependencies

   conda env create --file=environment.yaml

3. Install Yarn and NVM, and set up the React environment:

   cd ui
   nvm install
   yarn install
   cd ..

4. Install Elasticsearch and make sure that the service is up.

🔎 Running Knowledge-Debugger

Creating a Configuration File

Knowledge-Debugger executes one model at a time, based on a given configuration file. This file's config fields can be seen in the Docs. Exemplary Files are provided in config_files/

Creating an Elasticsearch Index

The keyword search functionality in the exploration view is powered by an Elasticsearch index that stores the projections of feed-forward parameter vectors from the entire network. To write this index into the server files directory (specified in Config), run:

python es_index/create_offline_files.py \
--config_path CONFIG_PATH

To upload the index from the server files directory to an ElasticSearch Database, run:

python es_index/index_value_projections_docs.py \
--config_path CONFIG_PATH

Executing Knowledge-Debugger

To run Knowledge-Debugger:

bash start.sh CONFIG_PATH

Adding an EasyEdit Knowledge-Editing Method

Simply create an EasyEdit-Hparams-File and store it in your EasyEdit Hparams-Path (default: config_files/ee_hparams).

In case you are interested in running only one of the two views of Knowledge-Debugger, this can be done as follows:

1. To run the Flask server (needed for the prediction view):

   python3 -m flask_server.app --config_path CONFIG_PATH

2. To run the prediction view:

   python ui/src/convert2runConfig.py --config_path CONFIG_PATH
   cd ui
   yarn start

3. To run the exploration view:

   streamlit run streamlit/exploration.py -- --config_path CONFIG_PATH

Origins of Knowledge-Debugger

Knowledge-Debugger is based on LM-Debugger, originally developed by Geva et al.

Docs to be extended soon!