NL2SQL360: A Multi-angle Evaluation Framework

NL2SQL Overview

**NL2SQL360** is a multi-angle evaluation framework. It can assess different NL2SQL methods against established benchmarks or tailor their evaluations based on specific criteria (e.g., varying data domains or SQL characteristics).

Abstract

Translating users' natural language questions into SQL queries (i.e., NL2SQL) significantly lowers the barriers to accessing relational databases. The emergence of Large Language Models has introduced a novel paradigm in NL2SQL tasks, enhancing capabilities dramatically. However, this raises a critical question: *Are we fully prepared to deploy NL2SQL models in production?* To address the posed questions, we present a multi-angle NL2SQL evaluation framework, NL2SQL360, to facilitate the design and test of new NL2SQL methods for researchers. Through NL2SQL360, we conduct a detailed comparison of leading NL2SQL methods across a range of application scenarios, such as different data domains and SQL characteristics, offering valuable insights for selecting the most appropriate NL2SQL methods for specific needs. Moreover, we explore the NL2SQL design space, leveraging NL2SQL360 to automate the identification of an optimal NL2SQL solution tailored to user-specific needs. Specifically, NL2SQL360 identifies an effective NL2SQL method, SuperSQL, distinguished under the Spdier dataset using the execution accuracy metric. Remarkably, SuperSQL achieves competitive performance with execution accuracy of 87% and 62.66% on the Spider and BIRD test sets, respectively.

NL2SQL360 Framework

infrastructure

Overview of **NL2SQL360**, which is featured by: - Easy-to-use Evaluation: Command Line Usage / Python Code Usage. - Integrated Metrics: Execution Accuracy / Exact-Match Accuracy / Valid Efficiency Score / Question Variance Testing. - Multi-angle Performance: Fine-grained performance (JOIN, Sub-query, etc.) / Scenario-based (Business Intelligence, etc.)

NL2SQL360-AAS: NL2SQL Automated Architecture Search Algorithm

executable environment

- A modular NL2SQL design space. - An algorithm to automatically search NL2SQL solutions for specific scenarios. - Our case study of SuperSQL shows competitive results in Spider and BIRD datasets.

Leaderboard

We experiment with state-of-the-art LLM-based and PLM-based NL2SQL methods in Spider Dataset. We show the overall results into different subsets, including JOIN, Subquery, QVT, and Competition Domain. **We are actively updating the benchmark with new methods. Pull requests welcomed!** 👏

All
JOIN
Subquery
QVT
Competition Domain

Rank	Model	Details	Score
1 Aug 29, 2024	SFT CodeS-15B	LLM-based Finetuning-based	84.9
2 Aug 29, 2024	RESDSQL-3B + NatSQL	PLM-based Finetuning-based	84.1
3 Aug 29, 2024	DAILSQL + Self-Consistency	LLM-based Prompting-based	83.6
4 Aug 29, 2024	DAILSQL	LLM-based Prompting-based	83.1
5 Aug 29, 2024	DINSQL	LLM-based Prompting-based	82.8
6 Aug 29, 2024	C3SQL	LLM-based Prompting-based	82.0

BibTeX

@misc{li2024dawn,
        title={The Dawn of Natural Language to SQL: Are We Fully Ready?}, 
        author={Boyan Li and Yuyu Luo and Chengliang Chai and Guoliang Li and Nan Tang},
        year={2024},
        eprint={2406.01265},
        archivePrefix={arXiv},
        primaryClass={id='cs.DB' full_name='Databases' is_active=True alt_name=None in_archive='cs' is_general=False description='Covers database management, datamining, and data processing. Roughly includes material in ACM Subject Classes E.2, E.5, H.0, H.2, and J.1.'}
  }