About SNAILS

What is SNAILS?

SNAILS is a set of NL-to-SQL-related artifacts designed to explore the effects of schema naming on LLM-based NL-to-SQL performance. It includes a set of real-world databases and associated natural language (NL) : SQL gold query pairs, human-labeled data containing naturalness classifications the tables and columns in the database collection, a model and method for ML-based naturalness classification, and prompting strategies for improving schema identifier naturalness.

Access SNAILS Artifacts

SNAILS Artifacts on GitHub

SNAILS Word Naturalness Classifier on HuggingFace

SNAILS Technical Report

What SNAILS Isn’t:

SNAILS is not a fully-featured benchmark similar to Spider or BIRD that maintains a leaderboard, and provides a training, and dev dataset while also retaining a hidden test set. Instead SNAILS artifacts are a starting point for further research into schema-specific effects on NL-to-SQL. It can also be used to support the development of new NL-to-SQL methods. We encourage other researchers and developers to use our artifacts for their research objectives, and invite collaboration and feedback from the community.

Contents:

• Installing SNAILS
• SNAILS artifacts
  • SNAILS databases
  • SNAILS Identifier Naturalness data
  • Naturalness Classifiers
  • Naturalness-Modified Identifiers
  • Naturalness Modifier
  • SNAILS NL Question - Gold SQL Query dataset
• Natural Views
• Reproducing SNAILS Experiments
  • Manual SQL Correctness Validation Tool
  • SNAILS Reproducibility Notebooks
• src
  • end_to_end_data_prep_and_prediction
  • natural_view_builder
  • nl_to_sql_inference_and_prompt_generation
  • query_manual_evaluation
  • query_profiler
  • schema_graph
  • schema_linking_eval
  • schemarenamer
  • semantic_compare
  • word_list_matching
  • util