On-device GenAI APIs as a part of ML Equipment assist you to simply construct with Gemini Nano

Posted by Caren Chang – Developer Relations Engineer, Chengji Yan – Software program Engineer, Taj Darra – Product Supervisor

We’re excited to announce a set of on-device GenAI APIs, as a part of ML Equipment, that will help you combine Gemini Nano in your Android apps.

To begin, we’re releasing 4 new APIs:

• Summarization: to summarize articles and conversations
• Proofreading: to shine brief textual content
• Rewriting: to reword textual content in numerous types
• Picture Description: to offer brief description for photos

Key advantages of GenAI APIs

GenAI APIs are excessive degree APIs that enable for simple integration, much like present ML Equipment APIs. This implies you’ll be able to count on high quality outcomes out of the field with out further effort for immediate engineering or high-quality tuning for particular use instances.

GenAI APIs run on-device and thus present the next advantages:

• Enter, inference, and output information is processed regionally
• Performance stays the identical with out dependable web connection
• No extra price incurred for every API name

To forestall misuse, we additionally added security safety in numerous layers, together with base mannequin coaching, safety-aware LoRA fine-tuning, enter and output classifiers and security evaluations.

How GenAI APIs are constructed

There are 4 principal parts that make up every of the GenAI APIs.

1. Gemini Nano is the bottom mannequin, as the inspiration shared by all APIs.
2. Small API-specific LoRA adapter fashions are educated and deployed on prime of the bottom mannequin to additional enhance the standard for every API.
3. Optimized inference parameters (e.g. immediate, temperature, topK, batch dimension) are tuned for every API to information the mannequin in returning the perfect outcomes.
4. An analysis pipeline ensures high quality in numerous datasets and attributes. This pipeline consists of: LLM raters, statistical metrics and human raters.

Collectively, these parts make up the high-level GenAI APIs that simplify the trouble wanted to combine Gemini Nano in your Android app.

Evaluating high quality of GenAI APIs

For every API, we formulate a benchmark rating based mostly on the analysis pipeline talked about above. This rating is predicated on attributes particular to a process. For instance, when evaluating the summarization process, one of many attributes we take a look at is “grounding” (ie: factual consistency of generated abstract with supply content material).

To offer out-of-box high quality for GenAI APIs, we utilized characteristic particular fine-tuning on prime of the Gemini Nano base mannequin. This resulted in a rise for the benchmark rating of every API as proven beneath:

As well as, this can be a fast reference of how the APIs carry out on a Pixel 9 Professional:

Pattern utilization

That is an instance of implementing the GenAI Summarization API to get a one-bullet abstract of an article:

val articleToSummarize = "We're excited to announce a set of on-device generative AI APIs..."

// Outline process with desired enter and output format
val summarizerOptions = SummarizerOptions.builder(context)
    .setInputType(InputType.ARTICLE)
    .setOutputType(OutputType.ONE_BULLET)
    .setLanguage(Language.ENGLISH)
    .construct()
val summarizer = Summarization.getClient(summarizerOptions)

droop enjoyable prepareAndStartSummarization(context: Context) {
    // Verify characteristic availability. Standing will probably be one of many following: 
    // UNAVAILABLE, DOWNLOADABLE, DOWNLOADING, AVAILABLE
    val featureStatus = summarizer.checkFeatureStatus().await()

    if (featureStatus == FeatureStatus.DOWNLOADABLE) {
        // Obtain characteristic if needed.
        // If downloadFeature will not be known as, the primary inference request will 
        // additionally set off the characteristic to be downloaded if it isn't already
        // downloaded.
        summarizer.downloadFeature(object : DownloadCallback {
            override enjoyable onDownloadStarted(bytesToDownload: Lengthy) { }

            override enjoyable onDownloadFailed(e: GenAiException) { }

            override enjoyable onDownloadProgress(totalBytesDownloaded: Lengthy) {}

            override enjoyable onDownloadCompleted() {
                startSummarizationRequest(articleToSummarize, summarizer)
            }
        })    
    } else if (featureStatus == FeatureStatus.DOWNLOADING) {
        // Inference request will robotically run as soon as characteristic is      
        // downloaded.
        // If Gemini Nano is already downloaded on the gadget, the   
        // feature-specific LoRA adapter mannequin will probably be downloaded very  
        // rapidly. Nonetheless, if Gemini Nano will not be already downloaded, 
        // the obtain course of might take longer.
        startSummarizationRequest(articleToSummarize, summarizer)
    } else if (featureStatus == FeatureStatus.AVAILABLE) {
        startSummarizationRequest(articleToSummarize, summarizer)
    } 
}

enjoyable startSummarizationRequest(textual content: String, summarizer: Summarizer) {
    // Create process request  
    val summarizationRequest = SummarizationRequest.builder(textual content).construct()

    // Begin summarization request with streaming response
    summarizer.runInference(summarizationRequest) { newText -> 
        // Present new textual content in UI
    }

    // It's also possible to get a non-streaming response from the request
    // val summarizationResult = summarizer.runInference(summarizationRequest)
    // val abstract = summarizationResult.get().abstract
}

// Make sure you launch the useful resource when not wanted
// For instance, on viewModel.onCleared() or exercise.onDestroy()
summarizer.shut()

For extra examples of implementing the GenAI APIs, take a look at the official documentation and samples on GitHub:

Use instances

Right here is a few steering on the best way to finest use the present GenAI APIs:

For Summarization, contemplate:

• Dialog messages or transcripts that contain 2 or extra customers
• Articles or paperwork lower than 4000 tokens (or about 3000 English phrases). Utilizing the primary few paragraphs for summarization is often adequate to seize a very powerful info.

For Proofreading and Rewriting APIs, contemplate using them throughout the content material creation course of for brief content material beneath 256 tokens to assist with duties resembling:

• Refining messages in a selected tone, resembling extra formal or extra informal
• Sharpening private notes for simpler consumption later

For the Picture Description API, contemplate it for:

• Producing titles of photos
• Producing metadata for picture search
• Using descriptions of photos in use instances the place the pictures themselves can’t be displayed, resembling inside an inventory of chat messages
• Producing various textual content to assist visually impaired customers higher perceive content material as a complete

GenAI API in manufacturing

Envision is an app that verbalizes the visible world to assist people who find themselves blind or have low imaginative and prescient lead extra impartial lives. A standard use case within the app is for customers to take an image to have a doc learn out loud. Using the GenAI Summarization API, Envision is now in a position to get a concise abstract of a captured doc. This considerably enhances the consumer expertise by permitting them to rapidly grasp the details of paperwork and decide if a extra detailed studying is desired, saving them effort and time.

Supported units

GenAI APIs can be found on Android units utilizing optimized MediaTek Dimensity, Qualcomm Snapdragon, and Google Tensor platforms by AICore. For a complete record of units that assist GenAI APIs, consult with our official documentation.

Study extra

Begin implementing GenAI APIs in your Android apps in the present day with steering from our official documentation and samples on GitHub: AI Catalog GenAI API Samples with Compose, ML Equipment GenAI APIs Quickstart.