On this tutorial, we exhibit a totally purposeful and modular information evaluation pipeline utilizing the Lilac library, with out counting on sign processing. It combines Lilac’s dataset administration capabilities with Python’s purposeful programming paradigm to create a clear, extensible workflow. From organising a venture and producing life like pattern information to extracting insights and exporting filtered outputs, the tutorial emphasizes reusable, testable code constructions. Core purposeful utilities, reminiscent of pipe, map_over, and filter_by, are used to construct a declarative circulation, whereas Pandas facilitates detailed information transformations and high quality evaluation.
!pip set up lilac[all] pandas numpyTo get began, we set up the required libraries utilizing the command !pip set up lilac[all] pandas numpy. This ensures now we have the complete Lilac suite alongside Pandas and NumPy for clean information dealing with and evaluation. We must always run this in our pocket book earlier than continuing.
import json
import uuid
import pandas as pd
from pathlib import Path
from typing import Checklist, Dict, Any, Tuple, Non-obligatory
from functools import scale back, partial
import lilac as llWe import all of the important libraries. These embrace json and uuid for dealing with information and producing distinctive venture names, pandas for working with information in tabular kind, and Path from pathlib for managing directories. We additionally introduce sort hints for improved operate readability and functools for purposeful composition patterns. Lastly, we import the core Lilac library as ll to handle our datasets.
def pipe(*capabilities):
"""Compose capabilities left to proper (pipe operator)"""
return lambda x: scale back(lambda acc, f: f(acc), capabilities, x)
def map_over(func, iterable):
"""Purposeful map wrapper"""
return record(map(func, iterable))
def filter_by(predicate, iterable):
"""Purposeful filter wrapper"""
return record(filter(predicate, iterable))
def create_sample_data() -> Checklist[Dict[str, Any]]:
"""Generate life like pattern information for evaluation"""
return [
{"id": 1, "text": "What is machine learning?", "category": "tech", "score": 0.9, "tokens": 5},
{"id": 2, "text": "Machine learning is AI subset", "category": "tech", "score": 0.8, "tokens": 6},
{"id": 3, "text": "Contact support for help", "category": "support", "score": 0.7, "tokens": 4},
{"id": 4, "text": "What is machine learning?", "category": "tech", "score": 0.9, "tokens": 5},
{"id": 5, "text": "Deep learning neural networks", "category": "tech", "score": 0.85, "tokens": 4},
{"id": 6, "text": "How to optimize models?", "category": "tech", "score": 0.75, "tokens": 5},
{"id": 7, "text": "Performance tuning guide", "category": "guide", "score": 0.6, "tokens": 3},
{"id": 8, "text": "Advanced optimization techniques", "category": "tech", "score": 0.95, "tokens": 3},
{"id": 9, "text": "Gradient descent algorithm", "category": "tech", "score": 0.88, "tokens": 3},
{"id": 10, "text": "Model evaluation metrics", "category": "tech", "score": 0.82, "tokens": 3},
]On this part, we outline reusable purposeful utilities. The pipe operate helps us chain transformations clearly, whereas map_over and filter_by permit us to rework or filter iterable information functionally. Then, we create a pattern dataset that mimics real-world data, that includes fields reminiscent of textual content, class, rating, and tokens, which we are going to later use to exhibit Lilac’s information curation capabilities.
def setup_lilac_project(project_name: str) -> str:
"""Initialize Lilac venture listing"""
project_dir = f"./{project_name}-{uuid.uuid4().hex[:6]}"
Path(project_dir).mkdir(exist_ok=True)
ll.set_project_dir(project_dir)
return project_dir
def create_dataset_from_data(title: str, information: Checklist[Dict]) -> ll.Dataset:
"""Create Lilac dataset from information"""
data_file = f"{title}.jsonl"
with open(data_file, 'w') as f:
for merchandise in information:
f.write(json.dumps(merchandise) + 'n')
config = ll.DatasetConfig(
namespace="tutorial",
title=title,
supply=ll.sources.JSONSource(filepaths=[data_file])
)
return ll.create_dataset(config)With the setup_lilac_project operate, we initialize a novel working listing for our Lilac venture and register it utilizing Lilac’s API. Utilizing create_dataset_from_data, we convert our uncooked record of dictionaries right into a .jsonl file and create a Lilac dataset by defining its configuration. This prepares the information for clear and structured evaluation.
def extract_dataframe(dataset: ll.Dataset, fields: Checklist[str]) -> pd.DataFrame:
"""Extract information as pandas DataFrame"""
return dataset.to_pandas(fields)
def apply_functional_filters(df: pd.DataFrame) -> Dict[str, pd.DataFrame]:
"""Apply numerous filters and return a number of filtered variations"""
filters = {
'high_score': lambda df: df[df['score'] >= 0.8],
'tech_category': lambda df: df[df['category'] == 'tech'],
'min_tokens': lambda df: df[df['tokens'] >= 4],
'no_duplicates': lambda df: df.drop_duplicates(subset=['text'], preserve='first'),
'combined_quality': lambda df: df[(df['score'] >= 0.8) & (df['tokens'] >= 3) & (df['category'] == 'tech')]
}
return {title: filter_func(df.copy()) for title, filter_func in filters.gadgets()}We extract the dataset right into a Pandas DataFrame utilizing extract_dataframe, which permits us to work with chosen fields in a well-recognized format. Then, utilizing apply_functional_filters, we outline and apply a set of logical filters, reminiscent of high-score choice, category-based filtering, token rely constraints, duplicate removing, and composite high quality circumstances, to generate a number of filtered views of the information.
def analyze_data_quality(df: pd.DataFrame) -> Dict[str, Any]:
"""Analyze information high quality metrics"""
return {
'total_records': len(df),
'unique_texts': df['text'].nunique(),
'duplicate_rate': 1 - (df['text'].nunique() / len(df)),
'avg_score': df['score'].imply(),
'category_distribution': df['category'].value_counts().to_dict(),
'score_distribution': {
'excessive': len(df[df['score'] >= 0.8]),
'medium': len(df[(df['score'] >= 0.6) & (df['score'] Dict[str, callable]:
"""Create numerous information transformation capabilities"""
return {
'normalize_scores': lambda df: df.assign(norm_score=df['score'] / df['score'].max()),
'add_length_category': lambda df: df.assign(
length_cat=pd.lower(df['tokens'], bins=[0, 3, 5, float('inf')], labels=['short', 'medium', 'long'])
),
'add_quality_tier': lambda df: df.assign(
quality_tier=pd.lower(df['score'], bins=[0, 0.6, 0.8, 1.0], labels=['low', 'medium', 'high'])
),
'add_category_rank': lambda df: df.assign(
category_rank=df.groupby('class')['score'].rank(ascending=False)
)
}To guage the dataset high quality, we use analyze_data_quality, which helps us measure key metrics like whole and distinctive data, duplicate charges, class breakdowns, and rating/token distributions. This offers us a transparent image of the dataset’s readiness and reliability. We additionally outline transformation capabilities utilizing create_data_transformations, enabling enhancements reminiscent of rating normalization, token-length categorization, high quality tier project, and intra-category rating.
def apply_transformations(df: pd.DataFrame, transform_names: Checklist[str]) -> pd.DataFrame:
"""Apply chosen transformations"""
transformations = create_data_transformations()
selected_transforms = [transformations[name] for title in transform_names if title in transformations]
return pipe(*selected_transforms)(df.copy()) if selected_transforms else df
def export_filtered_data(filtered_datasets: Dict[str, pd.DataFrame], output_dir: str) -> None:
"""Export filtered datasets to information"""
Path(output_dir).mkdir(exist_ok=True)
for title, df in filtered_datasets.gadgets():
output_file = Path(output_dir) / f"{title}_filtered.jsonl"
with open(output_file, 'w') as f:
for _, row in df.iterrows():
f.write(json.dumps(row.to_dict()) + 'n')
print(f"Exported {len(df)} data to {output_file}")Then, by apply_transformations, we selectively apply the wanted transformations in a purposeful chain, guaranteeing our information is enriched and structured. As soon as filtered, we use export_filtered_data to jot down every dataset variant right into a separate .jsonl file. This allows us to retailer subsets, reminiscent of high-quality entries or non-duplicate data, in an organized format for downstream use.
def main_analysis_pipeline():
"""Important evaluation pipeline demonstrating purposeful method"""
print("🚀 Establishing Lilac venture...")
project_dir = setup_lilac_project("advanced_tutorial")
print("📊 Creating pattern dataset...")
sample_data = create_sample_data()
dataset = create_dataset_from_data("sample_data", sample_data)
print("📋 Extracting information...")
df = extract_dataframe(dataset, ['id', 'text', 'category', 'score', 'tokens'])
print("🔍 Analyzing information high quality...")
quality_report = analyze_data_quality(df)
print(f"Unique information: {quality_report['total_records']} data")
print(f"Duplicates: {quality_report['duplicate_rate']:.1%}")
print(f"Common rating: {quality_report['avg_score']:.2f}")
print("🔄 Making use of transformations...")
transformed_df = apply_transformations(df, ['normalize_scores', 'add_length_category', 'add_quality_tier'])
print("🎯 Making use of filters...")
filtered_datasets = apply_functional_filters(transformed_df)
print("n📈 Filter Outcomes:")
for title, filtered_df in filtered_datasets.gadgets():
print(f" {title}: {len(filtered_df)} data")
print("💾 Exporting filtered datasets...")
export_filtered_data(filtered_datasets, f"{project_dir}/exports")
print("n🏆 Prime High quality Information:")
best_quality = filtered_datasets['combined_quality'].head(3)
for _, row in best_quality.iterrows():
print(f" • {row['text']} (rating: {row['score']}, class: {row['category']})")
return {
'original_data': df,
'transformed_data': transformed_df,
'filtered_data': filtered_datasets,
'quality_report': quality_report
}
if __name__ == "__main__":
outcomes = main_analysis_pipeline()
print("n✅ Evaluation full! Test the exports folder for filtered datasets.")Lastly, within the main_analysis_pipeline, we execute the complete workflow, from setup to information export, showcasing how Lilac, mixed with purposeful programming, permits us to construct modular, scalable, and expressive pipelines. We even print out the top-quality entries as a fast snapshot. This operate represents our full information curation loop, powered by Lilac.
In conclusion, customers can have gained a hands-on understanding of making a reproducible information pipeline that leverages Lilac’s dataset abstractions and purposeful programming patterns for scalable, clear evaluation. The pipeline covers all vital phases, together with dataset creation, transformation, filtering, high quality evaluation, and export, providing flexibility for each experimentation and deployment. It additionally demonstrates the right way to embed significant metadata reminiscent of normalized scores, high quality tiers, and size classes, which will be instrumental in downstream duties like modeling or human evaluate.
Try the Codes. All credit score for this analysis goes to the researchers of this venture. Additionally, be at liberty to observe us on Twitter and don’t overlook to affix our 100k+ ML SubReddit and Subscribe to our E-newsletter.
Nikhil is an intern advisor at Marktechpost. He’s pursuing an built-in twin diploma in Supplies on the Indian Institute of Expertise, Kharagpur. Nikhil is an AI/ML fanatic who’s all the time researching functions in fields like biomaterials and biomedical science. With a powerful background in Materials Science, he’s exploring new developments and creating alternatives to contribute.
